Scarlet - yes, I think drape and fit are key.
Examples:
* close-fitting (too tight!) clothes that show or create lumps and bumps = boy, do I feel huge!
* close-fitting clothes that skim my curves = do not feel bigger
* close-fitting clothes that actually reduce my measurements (i.e. Spanx) = do not feel bigger
* loose fitting clothes that drape beautifully = do not feel bigger
* loose fitting clothes that add bulk in places or proportions I feel are not balanced = feel big (i.e. a skirt with a lot of extra fabric around the hips)
* bulky clothes (sweaters, in particular) with no drape = can feel bigger...bulky knits need to have some drape, imo
But I'm just on my first cup of coffee, so I'll stop now!