My neutrals are black, gray (from light to charcoal), cream/white, and navy (in that order). I have a handful of camel items but I keep that to a minimum. I draw the line at brown -- because I find that adding in shades of brown complicates factors and I'll wind up with a second wardrobe of warm shades and items that don't mix and match so well with other items. Camels/cognacs are okay in small doses; it's chocolate browns, beiges, khakis that get me into trouble.
When I first joined YLF, I had a "double personality" closet like this -- I look best by far in cool rather than warm tones, but I'm just neutral enough, I can get away with warm tones too, it's just not ideal and requires different color hair and makeup. That said I do adore army green: accents my eyes. I consider army green a neutral, but I am using it these days as more of an accent color, as I'm using cognac/camel as an accent color.
I don't think my wardrobe could operate without black, gray and white, but navy is optional as I find it plays the same basic role as black. Except when worn with black, then I view it as a color not a neutral, if that makes sense?
I do need color in my wardrobe but I run about 75% neutrals and 25% color. I would get sad if my wardrobe was just neutrals, yet I cannot operate or get dressed without a heavy dose of them.