Of course the styling of the specific cross-body bag matters, but just a few associations off the top of my head, in case they help you pin down the problem and show you a direction you can go to resolve it...
It doesn't have the feminine connotations of a handbag, either, to add to what Raisin's already pointed out.
It can have more youthful connotations too --- like a schoolboy's satchel.
Traditional it is not, for either sex. Most crossbody bags don't suggest 'classic', though detailing will matter too.
Some of us just more 'stabilized' by having something to hold! Hands-free may be all kinds of practical, but can make some people feel at a loose end...
Could the crossing strap be at odds with the outfit you're wearing? Maybe it needs to match your style persona better, or maybe your style persona does not like to have its clothes 'crossed off'...
There is a suggestion of 'active' lifestyle about a crossbody, even more than a sling or shoulder bag (a backpack too, but it has additional 'carefree' connotations as well as contrary 'baggage' connotations). Certainly the handbag or clutch is a far more 'leisured' or sedate creature. If you are someone who habitually moves in a careful, deliberate way, if you are someone who *hates* rushing, or if you carry a bunch of stuff and move at a relatively erratic pace that makes said bag bounce around your hip, your very posture and gait may militate against the 'racy' style of a crossbody.
Any of that resonating with you?