Do you feel the same way about cardigans? If not, if they feel fine, it may be you are after a high comfort level.
I find sweaters very comfortable, jackets less so, and I did notice a substantial increase in comfort when I first forayed into knit blazers. I feel I can forget they're on. But 2 of mine are unlined and ruched sleeved so give a less "high end" look, and another is lined but styled more casually. So I've started keeping an eye out for more upscale knit blazers. Also I think they have the potential to have a more "delicate" shoulder fit, which is actually good for me,as my shoulders are small but kind of squared off, so a structured but not too sharp look is enough.
With structured woven, lined blazers, Ifind it hard to get the right armhole fit. Cut too high and it looks good, well-fitted, but is not comfortable and hard to wear over anything with sleeves. Cut lower and looks too big through chest, and boxy. So that's kind of a HEWI.
Being very cold-natured may help--I always need something. Not enough to forego decent comfort, but comfort requires warmth.