Ah. Okay - I get it now. It's the truly casual wear that is throwing you.
You know, I used to loathe summer fashion. Now I like it more than winter fashion! Part of that is the terrible winters we've had the past few years. Part is my lifestyle. No fun booties or jackets for me in winter. And in a slushy period like the one I am living through, not even any wool coats. I'm on foot in waterproof snow boots, in my puffer coat, and I have to choose uber-practical in order to survive. This is what I wear every. single. day. For months on end.
Meanwhile, in summer, I can wear a skirt or dress, fun and different footwear, sleeveless tops, jeans on a cool evening, jackets in AC, shorts for casual, so many options!! I have variety!!!! Which I love.
Having said that, during the dog days, I do sometimes miss the third piece even in my dressier outfits -- though shoes, a cuff, great earrings help. TexStyle has been an inspiration there.
For at home super casual (my typical wear) I pretty much have a uniform and I don't really expect it to be terribly fashionable. Just fresh for each season. I have a couple of pairs of shorts that I don't hate (despite the fact that my legs are not great). These are the essentials. For me, white shorts have been a fantastic addition; for you, looking for your light neutral might be key.
Meanwhile, I mostly try to focus on tops and footwear. A really flattering and current top makes a big difference, along with the low heeled sandal of the season.
Is it runway-ready? Of course not. But I think on most days I look -- at minimum -- "dressed" vs. schlump. In fact, I think I probably look more fashion conscious on most summer uber-casual days than winter casual days.
Another trick for me, since like Aida, I often have days where I want to move from one activity to another very different one, is to carry a tote bag or fashion backpack. I do quick changes in bathrooms all the time. I keep a core (maybe my shorts, or maybe the top) and sub out. I'll switch a casual tee to a silky blouse (if wearing shorts) or a pair of shorts for a tube skirt. I might or might not change footwear and jewellery.
My figure flattery issues are a bit different than yours; I fear not my tummy (so much) as my cellulite riddled legs. However, faced with your figure flattery priorities I would probably take a cue from Aida and Debs. Floaty tops with slim shorts or even a stretchy mini (since your legs are fab)? Or a midi tube skirt if going out. If you can find in linen or some other super cool material, pants similar to the ones I had from J. Crew a few seasons ago (and still wear) might work, as well. I am also considering culottes in the right fabrications.