Update: So I wore the red dress! I did bring a dark denim jacket and I'm glad I did (thanks Suz for the idea). On the way out, my male neighbour said, "We can certainly see you coming!" I didn't say anything, I just smiled.
When I arrived at the restaurant, two ladies were there already. One of the ladies commented that she didn't recognize me as I usually wear baseball caps and my hair tied back and we have our tennis clothes. We laughed at that because it's true!
Our attire was quite varied.. One lady golfs (I don't think she had golfed), but she was wearing a tech zip polo in a pretty coral, blue, and white pattern with matching coral capri pants. The second lady wore a v-neck, blue dress with white flowers with a pretty blue polished rock pendant. The third lady wore cute cropped chambray blue pants and a plain white t-shirt. The lady in the pants did comment that she felt underdressed and that she should have dressed up more. I felt a bit overdressed, but I was glad I wore the dress, as Angie says, "If not now, when?" Very true.