The main exercise I do is running, so I have dedicated clothes for that because I can't think of anything I'd want to wear running that would work for anything else.
I generally just get cheap stuff from Target or Old Navy because that works just fine for my purpose (which isn't exactly training for a marathon or anything). The only more expensive thing I have is this for really hot days because I got really sick of choosing between short-shorts that rode up and capris that felt like way too much fabric. Oh, and my shoes, because that's something I didn't want to skimp on and regret it later.
I worry about flattery only to the point that I don't want to wear anything that I'd feel really self-conscious or ugly in. For example, you may notice that the bottom I linked to strategically has a skirt on top to keep it from just being really short bicycle shorts. And I don't wear tops in super-bright fluorescent shades because I absolutely hate them on me, but I don't care if I'm wearing colors I just look kind of meh in.