I believe that, as in most things - frumpiness is in the eye of the beholder, and there are exceptions to every rule.
Having prefaced my remarks with this, and specifically speaking in the context of business attire ... :o) ... I think that it's rare to find a woman in a pencil skirt and flats who can carry it off. If a person is petite, in height and size, and they wear the pencil skirt at the top of the kneecap, I've seen it work. But to my eye, the balance of a figure is off if the pencil skirt in question provides a lot of visual impact in terms of length and width, while paired with visually diminutive flats.
My pet peeve with business attire is a suit with the jacket's hemline cutting straight across a woman's hips. Bad with skirts ... *worse* with pants, since the two suit pieces are disproportionate in relation to one another. It's so unfortunate that most suits on the market include jackets in that awkward length - but it *is* possible to find longer-lined jackets. I have two serious suits - one skirt suit, one pants suit - with jackets that skim the bottom of the bum. When I found each of them, I didn't even look at the price tag! I knew that these were the most flattering look, and worth the investment. Years later, I still trot them out for important meetings/events.
But in any case, as I mentioned above, everyone has their own perspective and, in the end, we wear what we know works best for us. :o)