When I was into grey I eventually, painfully realized there's a huge range of greys and trying to mix and match them can be very difficult, so I know exactly what you mean when you say you hold it up to other grey stuff and suddenly, it's brown.
Have you looked at Maria Killam's blog on decorating with colour? She's all about recognizing undertones.
As an aside, I realized today out in the sun that my one oatmeal garment has a blush undertone. Which is probably why it suits me well and other "oatmeal" items haven't so much.