Hey GG!
I tried on that vest at ATL, and put it back because it just didn't fit perfectly, esp. in the shoulders. I also realized that I needed something with more coverage, a narrower v-neck, and more buttons, since I'm short-waisted. This vest has a wide v-neck, almost u-neck, and only 4 buttons, versus a narrower v-neck and 6 buttons. Sounds nitpicky, but for someone built like me, it makes a difference in the look.
The vest looks better buttoned over a button-down shirt, with your skirt. Opened, it makes your outfit look unstructured and not as pulled together.
I think that a vest can work opened over a fitted tee, especially a graphic tee. I plan to wear mine buttoned regardless, just cuz I prefer a structured, sleek look.
If the vest is uncomfortable, then take it back. There are lots out there to choose from.
Hope this helps!