For a man-- I would say not. However, If I am not mistaken, some of the ladies at work know a bit about your interests in fashion (women's fashion) if not for any other reason but because you often notice little details about their outfits and make compliments. A woman that was interested enough in fashion to do that would most likely be a woman that plays with her own wardrobe, so the ladies at work might expect you to be more into your wardrobe. Also, women in general have a tendency to love to play dress up with male friends ect, or at least think about how they would dress someone differently, especially if the guy wears a lot of very plain very casual clothes and that doesn't necessarily mesh 100% with his personality. I know for one that I've commented to a guy friend that I liked it when he wore something different than his usual, or said that I liked him in button downs or in a certain color.
My SO has like 8 pairs of the same pants, and 4 pairs of the same boots in different colors. He is hard to fit and those are his favorites . . . so no, not unusual for your average guy.