Vicki and Elizabeth, I wear a size 2 shirt in Brooks Brothers, 0-2 in Ann Taylor tops, 6 in Karen Millen, 2 in Tahari, XXS in Danier, XS in my Banana tops, XS in the Nordstrom Nexx silk blouse, and I think I was a 2 in the last Gap blazer I tried on (don't currently own any).
In the Lanai, I take a 6! The 4 fit and looked nice, but it did not leave much room for movement in the shoulders. And I typically find most shoulders cut too big for me.