Because I mostly wear jeans (albeit attempted in a smart way), I don't get this very often. However, sometimes if I pull out a dress or skirt, I get a comment like this. Even when I make pains to casual-ize the look.
I don't take the "where are you going you are so dressed up" comment lightly. I take it that, in some way, I have failed. That I have either thrown on too much stuff, or I am over-dressed for the occasion, which I dislike. I haven't gotten one of these since the summer, happily. (When I last wore a dress for a casual occasion.)
So even though these comments can be hurtful, I have taken them as learning occasions.