I think it's mainly a subset of the teens to 30-something set who don't have kids and other responsibilities. A subset, because even in that age group, not all are interested in fashion.
I think there's another SIZEABLE tribe of women (and men, too) in their 40s, 50s, and beyond -- if that weren't true, YLF would not exist, nor would all the over 40 bloggers exist.
It's just that this demographic is spread out, not all in the same location. So the internet has been a boon for that.
Thinking about my close friends in town:
1. Super into fashion, but has her own quirky individual style not really trend driven, though she is trend aware. Super arty.
2. Not at ALL interested in fashion.
3. Interested in gear fashion only. For the purpose of doing stuff in said gear.
I have other friends here in town who enjoy clothes but I would not call them into trends. Many of them dress in a sort of leftover hippie boho/ academic/ artsy way.
A friend in Toronto who is very into fashion. A friend in Vancouver who is really into Eco fashion but more because she's into Eco than into clothes, per se -- though she supports great Canadian designers.
I'm the only one who has made it a kind of hobby, I think. Others have different hobbies, e.g. gardening.