Well, for a lot of people, gender isn't necessarily binary, either. People aren't always born in bodies that correlate with their gender identity, in the way that most people still think about it. There are men who menstruate, and women who need jock straps. Sounds like you've just begun your journey of learning about gender; maybe start a thread in the off-topic section, for suggestions about where to deepen your understanding.
I'm a hetero cis-woman, yet I wear a lot of men's clothing, because it fits, is better quality than the corresponding "women's" items, and because the price is usually a better value. But my body type works with that; most women have more curves, and proportions that don't work with labeled-men's clothing. Being able to "just buy menswear" and look presentable wearing it is a privilege I don't take for granted.
There's also the question of size ranges it's viable to manufacture in any given line. As an extreme example: Manolo Blahnik is not going to make high heels in a size and width available to Mr. Maximal, just like they're not going to make shoes intended to fit orthotics. So the men who want to wear certain "women's" items will need to have bodies that conform to a narrow range of sizes and shapes...just like women do.
Tl;dr: clothes are sold as gendered because it's efficient and economical.
