I'm another work-from-home person who lives in a casual university city with yucky weather for a large part of the year, and an active person. Oh, and my natural preference is for more polished and smart looks. Necessity and practicality dictate a more casual style than I might like, but I've become happier in my choices since joining YLF.
I'm nodding along with others about the importance of irregular outfit juxtaposition, fit, footwear, grooming/ hair, makeup and owning one's own preferences.
I also give thought to what I am actually doing. There is casual as in GEAR casual (sports, workouts, hiking, etc). or CHORE casual (as in gardening, major housework etc.) -- for these you need dedicated WORK clothes, i.e. clothes that let you get the job done. I have a gear capsule and one or two reserved items that I can wear if I'm doing heavy housework. I don't wear these outside those settings.
Then there is clothing and even here, what I'll wear depends entirely on what I'll be doing. Pub with friends demands something different from visit to the dentist. Attending a reading is different from going to the market. I don't mean that I wouldn't use the same essentials (i.e. jeans) but I might change up the top, footwear, topper, jewellery for different types of event. This helps me find a balance between being true to myself and respectful/ mindful of the environment.
I also agree that dresses and skirts can be casual So if you enjoy them, wear them!
Typically, people make note of what you are wearing if you seem "dressed up" for a brief period of time only. Once they get used to it, they stop.