My 2-cents: I worry about looking seasonally confused or too teen when wearing shorts in cold weather. Nonetheless, I am often in wool/leather or poly-blend "formal" shorts, long sleeves/sweaters and tall boots. I like how the look fits me, I feel good when wearing such combo, and, to me, it's warmer and more comfortable to wear a wool, a 80 or a 100 hosiery with shorts than just jeans.
(Using a fine hosiery under jeans is of course the warmest option for really cold winters, but, oh man, that's uncomfortable! Digression ends here.)
So I guess it boils down to your priorities. Do you believe the look as flattering to your figure? Is it more comfortable to you to wear shorts than pants or skirts? Do you hv enough fairly casual occasions in your life where shorts can be worn without causing distress? If you answer yes to at least 2 of these questions, I would say you can pull the look off (and not minding that it might read a tad seasonally confused or teen to others - and yourself included)
I agree that similar fabric weights are less "out there", but poly-blends are quite light and I nonetheless pair them with heavy knits. By keeping dark and neutral colors in the shorts, you can IMO get away wth a heavier fabric top and a lightweight short.