Patty, I always found it quite hilarious when this has happened to me. I have been a curvy hourglass since age 13 (D cup or better, big behind, big feminine lips) and it still was and is a frequent occurrence. It didn't help that I was as tall or taller than many of the men and boys in my community and often ended up wearing men's outerwear, ect because I couldn't find women's that was long enough to keep me warm. My brother's own friends used to mistake me for him and I have had co-workers of my fathers run me down. I think I finally figured out what it was that gave people the wrong impression though---- it was my mannerisms. When I was around people that I didn't know/was uncomfortable with I would unconsciously copy my father's mannerisms. I have his face, so mimicking his expressions wasn't hard and the body language was easy too.
Now wearing only women's clothing, with long hair down my back, and finally shorter than at least some men, when I get uncomfortable around people I wasn't expecting to meet or don't trust, I still get sir . . . .and frequently the other common connotation about masculine women.
Perhaps it isn't your body or face or clothing that is throwing people off . . . maybe it is your mannerisms?