I live in a neighbourhood with a lot of religious families who dress as you describe - I often see women wearing a thin crew neck 3/4 sleeve t-shirt (in a dressy fabric, not cotton) under a sleeveless dress - in a coordinating or contrasting colour. The top needs to be fairly tight to work, but it looks very pretty, and turns a less modest dress into a modest one. It is pretty easy to find sleeveless dresses that are long enough so this might be an option. Then you can wear the dress afterwards without the undershirt. Otherwise a shrug would work as well on top,

Sounds like you're situated, but if the dress doesn't work, try Talbots. They have quite a few pretty fit and flare midis, but they would need toppers.