It's fine Sarah.
I carry my wallet the same way, and for the same reasons as you - I like to be hands free.
But I think you have overlooked another reason - it also think it looks much nicer to have (especially) a nice wallet neatly tucked into your waistband (or belt) with just the top showing than to have a bulge in one of your pockets; or worse having the heavy wallet (or a purse strap) pulling your nice dress out of shape.
Absolutely the same for guys - As you said, many guys carry their wallets this way, but again, it looks waaay nicer than having their trousers pulled out of shape, or a giant "tumor" on their ass!
I have a collection of wide belts to wear with dresses solely for the purpose of having somewhere to put my stuff (phone in bra, wallet and keys under belt - nothing to carry!