Do you really think women in short skirts and heels is more attractive to men than other types of outfits? Or is the head turner factor, as you put it? I'm also a little uncomfortable with limiting it to opposite sex attraction, but that is example you used.
It may be generally accepted, but I have not found it to be fact. From my experience, men aren't really attracted to the outfit, as they are the selfconfidence a particular woman pocesses. And, the same for women being attracted to a particular man.
Anyways, I like a white button down, jeans, and loafers or jeans, tee, sportscoat, and loafers. I like a more casual, but put together look on a man.