There's definitely a level of clothing-related inappropriateness that is noticeable and negatively distracting--think extremely stained or torn/hole-y clothing, or over-exposed skin for the context, to give two common examples. And I think this is true for most reasonably observant people, whether or not they are into clothes/fashion.
As for clothes in general, I notice, and based on comments I get, people around me notice, too. And this is NOT at all limited to women who are interested in fashion, or people who know me. I get comments from the boys in my lab, both positive and negative, and I get comments from strangers, too (all positive, and this time mostly from females) --so it's clear that people *do* notice, in all sorts of contexts. I would say that the "meeting tons of people" at parties context is objectively different, since it's sort of person overload.
And yes, face is the most important/most memorable thing. But I find I notice people's clothing, the way they dress/carry themselves, etc. a LOT more now than I did in high school, where it seemed like *all* I noticed was how pretty a person's face was.
I only find *extreme* overdressing equivalent to *extreme* underdressing--in general, I prefer moderate overdressing to moderate underdressing, because it is more visually interesting. From the point of view of people watching, I find American street life distinctly more boring than Indian or Spanish street life, since the women in those countries dress with more flair than most American women do.