You've already had great advice here. I like the specific suggestions that GreenJeans made (about starting with footwear) and then Angie's suggestions as well.
I concur with the advice to look for casual dresses and also a skirt. I almost lived in my tube skirts for years -- they could be dressed down or up. I'm now in need of replacements/ upgrades to the style as well as the skirts themselves.
Tops -- look for wovens if you can. They will tick more of your boxes and feel dressier. Do you want sleeves or are sleeveless okay? My summer options below. The short sleeve gingham one is a bit transparent (i.e I wouldn't wear to an office) but it is the coolest thing ever and with a nude bra is fine for casual wear.
Pants -- linen pants are a great summer option. The ones in Finds are old and no longer available, but they've been great.
I do wear some knits in summer also, either as evening toppers, or on cooler days.
The boatneck linen tees from J. Crew are great -- I wear a cotton tank underneath. I have these in a bunch of colours -- metallic, grey, navy, white, cobalt.