I think it's really choose-your-own-adventure, with a hefty nod to the climate you're living in. I wear cashmere when I have the perfect alignment of cold day + a specific 'occasion', like going out to dinner, or visit friends, or see a performance. Otherwise, I nearly always grab cotton, or merino wool on very cold days.
I've had a number of cashmere sweaters over the years, but the two that have lasted are Lands End, and J Crew**. I purchased both secondhand, and they have really held up so far. In addition, I have a couple of sweaters that are merino wool/silk/other textiles, blended with cashmere, which makes it a little less delicate. Banana Republic and Talbots, again, purchased secondhand, in impeccable condition.
Because I shop secondhand, I don't know how any of these brands are doing now, if you were to shop new. I would examine online reviews carefully.
**This is the only J Crew cashmere sweater I have encountered in the past decade, that didn't develop small holes or really bad pilling, within 4-5 wears and a couple of careful handwashes.