Rae, I think the jacket looks fantastic on you. It's not overly gearish at all. I honestly can't tell a big difference between the XS and S, looks wise, so I think you need to go with whatever will allow you to move around and be most active. Being able to layer with a fleece would make this even more versatile.
Fill this wardrobe hole without guilt! For whatever it's worth, I bought a similarly styled jacket from Lole and it's quickly become a workhorse for me. It dresses up for work, but it's not too precious to wear on my parents' farm, which is very important to me.