I wear skirts at home, casually, in WARM weather, too. It has to be warm, though, or my legs get too chilly. I dress them down with a tee or a simple top and (if wearing footwear, to go out) sneakers or sporty sandals. I find them much cooler and more comfortable than shorts for some activities.
I agree with you, Janet, that shorts are generally even more casual, but they can also be dressed up a bit with footwear/ tops/ toppers, and "our" sarong shorts, I treat more like a skirt, and will wear out into town.
Like you, I see it as a kind of continuum of casualness, if that makes sense.
To qualify as "casual" (to me) a skirt has to be easy fitting (either flowy and longer or stretch and longer -- I don't want to feel exposed or constrained.) It also has to be easy care. I do have to iron my Liberty skirt once in a while, but not that often, because the print disguises wrinkles and the cotton is so lovely that they mostly fall out anyway.