I've never thought to put words to this before, but I'm confident I know a neutral when I see it. In art and design, we're talking no hue at all -- greyscale only, from black to white and everything in between -- but in fashion, which is more practical, a neutral is something with little hue, as Diana says. It's not just a desaturated colour, because black and navy are both highly saturated and they're definitely neutrals.
Basically a neutral is not on the colour wheel. If you imagine mixing paint, it's what you'd get if you used only black and white, or mostly black or white along with all the primaries mixed together to get some shade of brown. Beige is included as a brown (with a lot of white pigment added) so it counts as a neutral. I'd also include navy, which is a dirty (i.e. brownish) blue with lots of black mixed in.
If you point to a neutral that isn't black, white or grey, and a child can't immediately come up with a word to describe it, it's probably a neutral. So that excludes cobalt (which any child can easily identify as "blue"), teal ("greenish-blue") and burgundy (probably "purple" or "red"). Kids don't have words for things like beige and stone and ivory, though.
A desaturated olive can be a neutral, but the greener it gets, the more likely that it's just "green", and not neutral.
Angie thinks red is a neutral because it goes with pretty much everything. I think she's wrong. Well, she's correct about the second part -- it does go with quite a range of other colours -- but it's not a neutral by my standards. It's RED. It's in the rainbow. Rainbow colours are not neutral.