I go back and forth on this. I totally get the idea of "body neutrals" and do tend to gravitate towards clothing colours that refer to my personal coloring in some way. So blues are easy (eyes), dark brown is pretty easy (hair), rust/cinnamon is easy (freckles and sun highlights), and olive and ochre are represented in my eye flecks. But I still treat all of those as colours. If I put on a cinnamon top and denim bottoms that is a *colorful* outfit to my eye. It's practically red and blue! Cinnamon top with black jeans -- that's a colour plus an achromatic. Achromatics are the closest to neutrals for me in the sense of goes-with-almost-everything. If I'm only packing one pair of jeans, you can bet it will be black and not blue.
So I do treat my browns like colours. Rust serves as my version of orange, ochre as my version of yellow. The cognac thing is interesting; I've always found most cognacs to be too bright and orange for me, especially next to blue denim. Like Sarah says, it's a complementary colour combination, and feels very bright and high impact. I'm always looking for lighter mid-browns that are softer and more muted than cognac, but not as pale as beige or tan. Walnut or toasted coconut...
When I did try to treat brown as a neutral, I ended up purging it all, because I was expecting it to behave like black and it just wouldn't!