I've been to a few swap parties, and as long as you invite people you know well, I think the free-for-all approach is best. These are things that would be donated, anyway, so IMO it's even better if they go to my friends - regardless of whether or not I find a stitch for myself. When I've gone, the minimum has been around 5 pieces, to be sure everyone is contributing a fair share.
A 1-1 swap feels icky to me, because then you get into whether my piece is "worth" two of yours and oh, my blouse looks perfect on you, but you can't have it because your stuff doesn't fit me. To me, that's not fun.
As for the sizes, my experience has been that all sizes should be represented. Most of us purge stuff that *doesn't fit us* right? I may be a size X, but can bring to the swap all those size As I've been saving for 10 years to torture myself. Plus people can bring shoes, bags, scarves, and jewelry.