I actually am so glad you bought it to try on! I think the bold colours you wear with it will make all the difference, and it could be a great alternative for white as a base colour in your wardrobe. I'm with Inge that it will look great with your tweed blazer - not to mention how great it looks with your signature colours of tomato red and citron. Worth a bash, I say!
Also - with grey particularly, I find - it comes down to the right shade. For me, for example - it has to be very light, not medium, so that it provides sufficient contrast against my skin. Also, it has to be a warm grey, not cool, it has to be heathered, not flat, and it can't be too muddy. If it meets all these criteria - it's a great colour on me. If not - back it goes!