I think we've had a renaissance of neutrals in fashion in recent years. It used to be much less common to wear an outfit of only neutrals, and different neutrals together - would have been considered frumpy or boring. Monochromatic, or neutral plus bright, were much more common.
I think it's Janice at The Vivienne Files that often talks about a light neutral, a dark neutral, and a color or two. Her color schemes are some of the best I've seen, in blogs or in fashion books, as far as where you should start if you're looking to build a mix and match wardrobe or need to figure out where your holes are.
I personally have black, gray, navy, denim - and off white, if that counts. I used to have a lot more black - if I wasn't wearing denim I was wearing black - but have backed off it a lot because it felt boring and not so flattering. It's so useful, though, that I doubt I'd ever get rid of all of my black. I've never really loved brown and it doesn't really suit my coloring; the only items I have in brown are shoes, no clothes. I also have several pieces in olive, which depending on the shade can really act like a neutral.
(I think we're due for a rebirth of brown, actually, in fashion; it's been years since its been the trendy color. Recently navies and grays have been the neutrals in stores, seems like.)