Great question, and such interesting responses. Anna, I can really see how in your non-work life, comfort would have less priority. You are looking for fun, and trying things out. And with a work life where comfort and practicality completely rule and aesthetics are out of the picture, no wonder!!
Comfort is highly important to me, but the degree of comfort and kind of comfort I seek depends on the situation.
When I'm working at home, I like clothes that feel like workout gear (i.e. clothes that move with me and, in the cold winter months, hug me and keep me warm). But at the same time, I don't want to wear actual gear. I do go very casual, but aim for stylish casual. So that's one kind of comfortable. Soft, some stretch or give, non-binding, cozy.
But when I'm out and about, I actually find some degree of structure very comfortable. Too much softness feels...unfinished. That's one reason I feel more like me with a good jacket.
I do need shoes that I can walk in. But evidently I tolerate more discomfort than some people. For example, Sarah mentioned that she was ruling out a pair of shoes because they left red marks on her feet. I wouldn't do that, as long as they weren't creating permanent blisters. I'd figure, oh, a couple of red marks, big deal. And I'd see if I could soften them up. (I mean, assuming I loved them.) Or if they were giving me a corn, I'd stick a corn protector thingie on my toe. I guess it's a question of degree -- I really hate shoes that pinch badly. And can't stand heels I cannot walk fast in.
I agree with what K said, that lots of times people blame certain types of clothes for being uncomfortable when they simply aren't wearing the right size or cut.
And I also think that lots of "regular" clothes are every bit as comfortable as pajamas. My outfit today is no exception. Tube skirt (stretchy and soft), peplum (made of a soft knit, not at all binding), and kimono with wedge shoes. I am walking all over town like this. And curling up on a bed on my computer, right now.