I prefer it in the XS.
Suz, let's face it: you look stunning in a sheath dress. You need more of them. This wouldn't replace your black one, but would serve as an exploratory field. And I'm sure you would end up bringing its CPW down quite a bit. The argument that you already have a navy skirt only serves the purpose to say: go for it, because 1) this is not a skirt 2) you already have jewellery and accessories to match blue.
Now about the folding and travelling issue: Do you absolutely need to travel with this particular one? Does it have to fit that criteria at all costs? and if you did, rolling the item with a plastic sheet (i.e. like what you get from the cleaners) greatly minimize wrinkles.
Then on the casual issue: are you crazy? No, it is not too casual. It all depends what you style it with and especially the footwear. A lower vamp would make it less casual. Have you tried it with heeled knee high boots?
