I've never counted my sweaters. My first reaction was "I'd own 80 if I could find that many that FIT" because one of my big frustrations is finding quality sweaters in petite sizes. That's improved over recent years so I shouldn't kvetch too much! I easily have 40 sweaters/sweater dresses. Over that, I cannot say.
For storage, I am very very limited. I use two rolling wire mesh kitchen carts that fit nicely underneath my hanging tops in my closet. Each has three drawers. I can pack a lot of sweaters in each drawer. I swap out seasonally, so 1/3rd of my collection will be tucked away in storage at any given time. I do not roll, but that is a great idea and I'm going to try that.
Advantage to wire mesh carts, in addition to being able to roll them in and out to see what I have and to tidy up etc, is that the wire allows air to circulate. I find this is important for sweaters, which you probably don't wash every wear.
When it comes to thinning the sweater horde, each time I rotate, I cull out those that are pilled/overly worn/threadbare; those that are past their fashion prime (I'm looking at you, ruffled sweater); those that are not classic/neutral colors but appealed of-the-moment, yet did not flatter all that much.
Assuming in good shape, I'd hold on tight to any skirt-length sweaters, or body con sweaters. Right now oversized shapes are in style, but fashions change like the wind. Glad to see you held on firm with the sweater sets. Tuck away any out of fashion items for now, if you have storage. Because I have noticed a very definite downward trend in terms of sweater quality over the years. So a high quality wool or cashmere sweater is totally worth holding on to, even if you are just not feeling it right now.