As a blonde, I wore rose gold, yellow gold, and silver and sometimes mixed metals. It wouldn't have been too noticeable, though, because I don't wear a lot of jewellery and most of my bags at that time would have had minimal hardware.
When my hair went silver, I transitioned to all silver hardware and jewellery. Except for one jacket that has gold buttons (and they drive me crazy, so.....). Gold still looks okay on me, I think, But I like the way silver emphasizes the shine in my hair and makes an outfit cohesive. My complexion is cool to neutral. Eyes cool toned, dark blue.
I think your glasses are the issue -- once you fix that problem, you will be able to wear both metals well. Given your preference for matching, you may still prefer outfits that feature one metal or the other, but you won't be as limited in your choice of items at least.