I think you'll find you're in good company here, Mel. A lot of us are trying to reduce our reliance on gear as casual wear, and many others (including Angie) who view gear as a sort of "necessary evil" so to speak.
Here's me take: lot of folks in my cohort (young moms) spend all of their downtime in gear. Lululemon tights, Athleta, etc. As far as I can tell, it's because they view two categories of clothes as essential: the stuff they wear to work (if they work), and the stuff they wear to work out. Casual clothes are pretty far down the list of necessities, especially if everyone else around you is wearing their gym leggings as casual clothes. And if you're trying to plan a wardrobe budget around a changing body (which a lot of young moms are, as they're working off baby weight), then it may make more sense to splurge on gear for a while and scrimp on things like jeans, when you don't even know what size you'll end up in.
Also, different types and brands of fitness clothes communicate all kinds things about social status/values/etc, just like any other type of clothes. I think younger people/millennials definitely view their gear as "real" clothes, and are often willing to spend more on a piece of technical apparel than they are on something like jeans. So, I honestly don't think athleisure/gear-as-casual-wear is going away any time soon.
Personally, I've put a lot of work into rebuilding my casual wardrobe and *not* relying on fitness gear for daily wear, but I did wear my yoga pants almost daily for close to a year after my son was born. I'm also really lucky in that my workplaces are very casual (and one is an outdoor school, so my "work clothes" sometimes have to be gear!), so I don't need a separate work capsule at this point in my life. If I needed office wear, I expect I would have very little left over in my budget for "casual" clothes, because I absolutely need outdoor gear for the weekends, which I mostly spend hiking, camping, etc. Casual clothes would end up being somewhat low-wear for me in that scenario, and more difficult to justify.