Definitely black- several times over like Janet! I will wear columns of black, as long as my bags and shoes are different (burgundy or cognac). I can easily wear black for any season or refinement level!
Dark brown and burgundy I have in most types of pieces, though I don't like flat brown tops on me, just bronze.
Cognac is more of a supporting neutral instead of a main one. I just wear it in shoes, socks, bags, and prints. Outerwear would be fine too, or things with sheen (to contrast my skin).
I'm quite well represented with olive, but since I wear it with black, it's a color to me, not a neutral. My rule is, if I wouldn't wear a color with a "proper" neutral- like black or brown- it's a color.