I would have been okay with the S and I outweigh you a bit! I usually wear 6 or 8 so I'm thinking the XS sounds more like you. The S was still very loose and fluid on me.
It just didn't work on me because it is very long...and I'm two inches taller than you. I just couldn't get it to look right with anything, proportions just not happy for me. But, I don't wear leggings or skinnies so that is probably the problem.
Since they are free shipping and returns if you move fast, I'd order the S and the XS. I think you will be happier with the XS but I'm just guessing.