Actually, yes, size matters to me --- where the wardrobe is concerned. I do currently have my stuff scattered across rooms in our three-bedroom house, and it's driving me batty! Definitely, the aim is to have them all corralled into the one room, if not the one closet.
I dislike having to go fetch shoes from one room, wellies and raincoat from another etc --- and even if that could be sorted with near-the-door storage, I truly hate having to search for scarves in a different room from tops, say.
Mostly, for me, it has to do with a certain level of laziness. I find it easier to cook if my cookbooks and oven and fridge and sink are within steps of each other, in the same room (it's oddly common in India for the fridge to take up residence in the dining or living room instead of the kitchen). I find it easier to work with phone in arm's reach of computer, paper and pen to hand, and the scanner and printer in the same room (my workplace requires traversing two rooms to get to the printer). I'd prefer the drying rack and clothes line to be on the same floor as the washing machine. I want my toiletries and cosmetics in ONE place --- either bathroom or dressing table, not distributed across both (two exceptions for lip balm and foot cream by bedside).
So also with clothes. I do HAVE the space to spread out. I just hate it enough to keep pruning until I can stuff it into the one closet space and at most an additional dresser and one shoe rack.
ETA: I should add that I didn't actually let me wardrobe *expand* across rooms. Rather, my partner and I shared a walk-in closet in an earlier apartment and then moved to a place without closets or built-in wardrobes, so everything had to be distributed into armoires and dressers and hanging racks and chests spread all over.
I find I'm too lazy to actually wear much of the stuff not in the bedroom! So yes, what Shannon and IK say about seeing it all better---and at a glance!---definitely applies.