Well, if you are a celebrity, I guess the big brimmed hat + big sunglasses affords a little more anonymity. But really I think it's exactly what you said - that hats are a pain to pack. I was thrilled to find out that the red one I bought in England is packable!
This reminds me though - I need to acquire a hat box for my fancy pillbox hat. I carried it home from the UK in my big Longchamp tote (and carried my other purse separately) inside one of those clear plastic boxes, but I really need a hard sided box for it. The hat boxes they had for sale at the hat shop were all pretty enormous, and I didn't need one quite so large as that.
ETA: I'd imagine you would HAVE to take the hat off while you were sitting in the airplane seats, otherwise you'd have to sit with your head tilted forward the whole time? Or maybe this is because I'm short and my head never clears the top of airplane seats...