Ahahaaa I would do this all the time if I were bigger. I love the fabrics, textures and above all colours they use for menswear, but I'm a 5'1" pixie, so men's garments are always too big. Except belts. Most of my belts are men's or unisex.
I have had success shopping in the boys' department on occasion. One time I bought an argyle sweater vest from Hanna Andersson, and it's great. I wanted the v-neck and looser fit of a men's vest. I already had a woman's shell vest, which is a finer gauge knit and more form-fitting -- also great, but a totally different look.
Things like trousers don't really work, of course, because I'm shaped like a woman, not a 12-year-old boy.
Oh, I thought of something else. My one scarf (which I wear in the winter for warmth) is from Harry Rosen and it's grey and black plaid. I love it. Women's wool scarves are usually too "pretty" for my taste.
And lastly, I wear converse sneakers, but I don't know if this counts, because they are unisex. They started out as men's fashion, though.