In a perfect world, we would all wear what we wanted without being subjected to negative judgements and critical comments.
Work environments, though, are tricky. If your wardrobe choices are subject to workplace gossip and leering glances, your superiors will view your appearance as a negative distraction. If your performance is outstanding, the negative distraction may be tolerated. If your performance is average, the negative distraction may become a deciding factor that hinders your chance for a promotion. If your performance is questionable, your inability to "fit in" might be a reason for dismissal.
If you are willing to accept the possibility your wardrobe choices could have a negative effect on your employment, then continue to wear what you like. Alternatively, if your job is important to you and you'd like to be considered for promotions, you may want to consider adapting to the norms of your workplace. Getting along with colleagues, clients, and bosses is generally easier if your appearance is working for you instead of against you.