I've got a couple pairs of loafers, including a plain black pair, and I like them fine - I think the slightly lower vamp, and the fact that I tend to wear them without socks or with no-show socks, makes the effect different than with oxfords - you get some bare skin showing even with full-length pants and that makes them look not so heavy. Oxfords are different, but as Diana said, it's not really about heaviness exactly - I wear black boots happily enough.
Sally, I also used to do the 90s thing and even wear oxfords with dresses all the time, but I have been wearing less black in recent years. I still have some black shoes - how can one avoid it, really - but maybe that's why this pair feels so out of place, because I'm not necessarily anchoring the black shoes with anything else black in my outfit, and my hair is a light enough brown that it doesn't come across as bookending.
The brogue detailing on these is lovely, but not really obvious from any distance, since they're so dark. And the leather is not shiny, which I appreciate from a easy-care perspective, but also makes them very visually heavy. I'm leaning toward returning, but I'll wait a couple days to see if I have a change of heart.