Hi Gryffin,
I have never tried the pants, but I did try a VB pencil skirt in size 4, which fit perfectly on the hips but was huge in the waist. For the reference, I'm 5'6" and ~120lb these days. My Theory and Vince bottom sizes are 2/4 - 2s for bootcuts , 4s for slimmer cuts, although I often have to take their waists in. The sizes you mention sound too big for your figure to me, even with a looser fit. I would start with a VB 4 or 6.