I am the same way: very close to neutral, but gold looks slightly better on my skintone. I have some metal sensitivities, and because of that, I gravitate toward sterling silver rather than gold (sometimes gold causes a problem, and I'm not sure why). Otherwise I would probably buy more gold than silver. I think if I were in your shoes, I would see what looks best with your outfits and then buy accordingly.
My goal is to have a fine jewelry complement with a few pieces in each color metal for a few categories: hoop earrings, dainty necklaces, and stud earrings. I often buy pendants separately from chains and just swap out the pendant.
I have found recently that if I make sure that my silver pieces are very shiny (that is, cleaned up with a jewelry cleaning cloth), they look better on my skintone than if not.