OK, one more comment for now.
I think that the extent to which people care about fashion may be on some natural continuum. So some people are going to care no matter what. Others aren't going to care, no matter what. But the people in the middle may develop an eye for fashion to varying degrees based on their influences growing up. Having a particularly fashionable older relative, or living in an artsy area, or developing an associated interest in art, culture, photography, etc., may cause a person to develop a greater interest in fashion than they otherwise would. A person with some interest may likewise never develop it if they have other interests or needs that come first.
It also helps greatly if you have someone knowledgable to help you develop such an interest. I remember my aunt telling me that a particular dress or top didn't fit me because the shoulder seams were hanging off my shoulders, when they should have been sitting right on top. Another aunt refused to buy a particular item of clothing that I wanted because the pattern on the seams didn't match up. She considered this to be a sign of low quality. She was also a very good seamstress herself. These are things my mother never mentioned to me. In addition to these little lessons in quality, the first aunt mentioned above had tastes in clothing that were a bit more fashionable and cosmopolitan than anything I saw on people in my home town. If not for her, I may have still ventured outside the fashion comfort zone of those around me, but maybe not as soon or as much.
Also...
I do think it is possible for people with very little income to dress well, and even fashionably. But it takes more time and effort when you don't have the cash. I have a greater respect and appreciation for those people who manage to thrift, upscale, and repurpose a terrific wardrobe for practically nothing, than I do for those who can buy a terrific wardrobe. Both of these take effort, of course, because you can't just buy a sense of style. You have to develop it. But doing it on a tight budget takes *more* effort. So I'm inclined to give a pass to those who may look "sloppy" by my standards, but who are too busy with jobs or kids or sick relatives or whatever to give what they look like much thought.
And...
I wonder if people who dress very casually, but who are label conscious (as someone previously mentioned) are using the labels as some kind of shortcut to fashionable or appropriate dress. Maybe they believe that wearing the "right" brand makes up for not giving any thought to whether or not what they are wearing looks good or is appropriate for the situation. Maybe they have that inclination to care about fashion, but it has been completely and totally undeveloped? Or maybe they really don't care, but the labels matter because they are status conscious. Maybe it's the influence of fast fashion, to an extent. Back when clothing cost a lot more to manufacture, people had fewer clothes and they tended to keep them longer, wear them more often, and take better care of them. So quality mattered more, because you wanted the few clothes that you had to last a long time. And even if you made your own clothing, the quality of the fabric and the notions used, as well as the fit, still mattered. Now that clothing can be made so cheaply in developing countries, the branding and image associated with a clothing label matters more than it used to. I can't even say it's style over substance, because there is often so little style...
My rambling thoughts have now reached a dead end.