In my own closet (which has a lot of loafers and oxfords both), they do fit a slightly different niche. Now that you are making me pay attention; mostly I try stuff on and go "Naah".
If I had thought about it before, I would have pegged the oxfords as "masculine" and the loafers as "feminine" (or maybe neutral). I had orthopedic saddle shoes that I hated so much, I accidentally left them in Montreal after Expo 67. I have only fairly recently come around to wearing oxfords. I like the look with a dress or skirt (sometimes); it feels like a bit of juxtaposition. (I also think of it as a 30's or 40s look which I like to channel sometimes.) Wearing oxfords with pants feels more "masculine", in that I make an effort to put something more traditionally "feminine" in the outfit, even if it is just to wear my long hair loose rather than tied back.
Loafers on the other hand... They look OK to me with a shirtdress, or a pleated skirt, or something else with a preppy or "librarian" vibe, but I'm less likely to wear them with an A-line dress than I am oxfords. They are my winter default with trousers for work though. Largely because stopping to tie on oxfords when I change out of my boots in the morning, and again if I go out at lunch, is a pain.
It's entirely possible that once warm weather comes and I start wearing shoes outdoors again, I will remember all kinds of dress/skirt outfits that I like with loafers... I am fickle.
I am one of those who has a "thing" about matching vamp height and pants length. Mostly, I like to see a discrete amount of skin (or in winter sock!) between the hem and the top of the vamp. So with full length skinnies I like ballet flats, with the top of the foot visible, but with ankle baring pants I like a higher vamp shoe. It's the same principle I guess as the wearing of booties or high-ankle-strap shoes with true crops or midi skirts.
