I agree it can be bad when tossed around at other people--the reason, I think, is that it's kind of a dismissive, one-word thing that sounds more negative than an outfit "critique". But still I have some items and outfits that look that way on me. I guess I am the worse offender because I chose a version of it for my moniker, ho ho.
I think Jenn gets closest. Anything I remotely consider frumpy is unflattering in some significant way. There could be exceptions--the whole clip-on earring thing (hides in case people throw things). I don't have pierced ears so I did wear various clip-ons, especially at times I've had especially short hair, and I thought they, ah, highlighted my features, but then I got self-conscious about whether they were, maybe not frumpy, but unsophistocated and dated, or something.
Sometimes I use the word "schtumpy" for what's wrong with my outfits, because there is a close relationship between that appearance and frumpy, because it's closely tied to proportions and also and to a kind of sloppy or poor fit, too boxy, saggy. Too heavy. Like a poorly fitting, boxy blazer. Though the reverse could be true, too fitted or prim.