Are loafers and oxfords interchangeable?

I like both shoes below very much but am wondering if they both fill the same niche.

Can I wear loafers with a dress? (I think I can wear oxfords with a dress.) Or are they a pants-only shoe?

On a budget and don't want to overbuy but tempted by both and not sure how they're going to work in my wardrobe. Could use your help articulating the difference.