Hmm. Interesting question.
Sometimes there is discomfort/pushback if what seems appropriate is dressier than what others are wearing -- sort of a "Oh, you think you're better than us" thing.
If what seems appropriate is more casual than what others are wearing, it might be read as disrespectful.
Which direction do you generally find yourself in?