Una, we're all different when it comes to pattern and colour. What seems fabulous to one woman might feel garish and eyewatering to another.
I'm personally drawn to desaturated colour palettes, and not just in clothing. I prefer it in architecture, landscapes, and art. I like shades of grey. I like the odd shot of colour, but in small doses. In clothing, that usually means a colourful accessory. In a landscape, it could be something like a red phone box on a grey London street scene. I don't consider myself frumpy at all. It's just a personal preference. I also don't think that women who wear bright colours or wild patterns necessarily look bad, either -- I get that they prefer that sort of thing, and it suits their personality.
I also don't totally buy into the "it's not your colour" thing. I think there are a few "bad" colours for certain people (for example, I know I look terrible in lemon yellow), and a few strikingly "good" ones too (certain shades of olive make my skin tone look great), but most colours are somewhere in between.
A neutral, taupe-y top might not be your best colour ever, or it might look better on you than you think.