My mum, as she has aged, has worn baggier and baggier clothing. She always wears things too big. I doubt I will change her at this point, but it is clear that she believes that unless she has a "perfect" body or unless she loses "X" pounds, she doesn't deserve to wear clothing that flatters.
And it is difficult for me (as I go into my 40's) to not feel similarly. But then when I wear something that fits well, even my own mum will say things like, "Wow, have you lost weight? You look great!" I haven't lost weight, but the difference in fit can work magic. And then what Shiny said really resonates with me. I always felt that I couldn't wear something or other, even when I was many pounds thinner and many years younger. I've finally decided that while I always want to be in good taste, I'm not getting any younger and I refuse to hide in shapeless sacks of clothing.
There is a mentality among many women that after one has children, one should disappear as a woman. There ought to be no effort to look good because that would be vain and imply that you somehow cared less about your children. So a woman should cut her hair, wear no makeup, and give in to wearing baggy jeans and teacher sweaters. This is a very predominant attitude in some areas (including where I live), even if it isn't as directly expressed as this. I believe this is probably where your friends were coming from, even if they couldn't or wouldn't articulate it that way. A woman "of a certain age" focusing on looking good is somehow inappropriate.
Like others have said, you always look appropriate in your WIWs. Please don't give in to the mentality of your friends.