Rachylou, for me the 3 sweaters are crucial! It's winter here now (which is a pretty mild winter), so I'm almost always wearing a layer, even indoors where heating is scarce. Realistically, people are looking at whatever I'm wearing on top the most and are thus most likely to remember whether I wore it before when they see me again, so in terms of variety from other people's perspective, I think that actually adding another layer would give me more bang for my buck than adding another of any other type of item. When I know I'm going to see people a second time (or more), I always make sure I'm wearing a different layer, even if everything else ends up being the same.
A second warm jacket would also have a HUGE pay-off in terms of variety - I'm wearing that teal fleece in every single picture of me outside on this trip! But packing another warm layer would also take up a ton of space...