For myself, I'm firmly on Team "use my naturally occurring colours in my wardrobe." But it wasn't always thus...or at least not where my hair is concerned. My eyes are blue and I have always worn a lot of blue. I've also always loved the berry tones, which appear in my lips. I don't wear a lot of pink but magenta, fuchsia, orchid, and blue reds are usually in my closet.
For most of my life my hair was a sort of dirty blonde, but I highlighted it, so it wasn't entirely natural. Meanwhile, yellow is one of my worst colours. I also look awful in beige. So I did not wear those colours when I was blonde and I also tried to reduce the brassy tones in my blonde hair because whenever things went too yellow in the hair, my complexion looked horrid. When I dyed my hair red, I didn't wear more red; instead I wore more chocolate brown and purple and green (all of which I really like but they didn't suit me as well, as a blonde).
Going grey changed things.I like grey or silver or white for bookending my hair. I do have a light grey bag, and like it a lot.Like Jenni, I've found that with silver hair, black and white patterns work better than ever. I also find that white in a pattern has become so good that I've basically stopped buying patterns that don't include a bit of white or silver or light grey. And I wear charcoal and light greys happily.
Having said that—I find that unless I also add a colour and/or some contrast or shine or pattern, I feel less than fab in grey, esp. light grey. It flatters me, but also has the potential to make me disappear, much as you say wearing your hair colour does. I prefer outerwear in my other key neutrals (navy or white) or in red. So, for me, my hair colour's a lot better as an accent than as a whole outfit.