I wear the same coat every day, but that is only because for a long time I had no money for a second one. This year, I had the money, but I couldn't find one that fit right. Next year, I want to add another coat. It's like you say: when it's that cold out, all people really see is the coat because you don't take it off. So it needs to be a stylish coat.
I have some silky button-downs that I will wear on mild winter days, but they are made from rayon, not poly or silk. I also wear a cami underneath or some other type of long underwear. But when it gets to be really cold, I want sweaters, not button-downs. I want to feel cozy! So I guess my rule is, if it's hovering around freezing, I'm OK in the silky button-downs, but if it gets colder than that, I need sweaters.