I have been known to have to handle this type of rude questioning (and it *is* rude, pure and simple). I may answer in one or many of the following ways:
1) Dressing well (no matter what your budget) is a matter of self-respect. (Said to the person who's being downright rude and deserves a snappy retort).
2) Dressing well can lead to a job (or a better one). Even when simply running errands. You just never know who you may run into, and how they may help you along in the future. (Said to the person who just lost her job or is in fear of losing it... and then I'll shift the convo away from clothes and on to gentle reassurance and encouragement).
3) You can buy a few crappy, inexpensive tee-shirts.... and end up replacing them each and every year, because they are shoddily made. Or you can invest in well-made clothes with quality fabrics and classic cuts, and they will last for years (if not a lifetime). Which is better for the environment and our burgeoning landfills? I vote the latter. (Said to the person who is Green, which I strive to be too. Since YLF I have a smaller wardrobe but it's much more functional and more and more it is focused on quality pieces that will last for a long time - instead of a disposable wardrobe of constantly replaced pieces that never quite work together).
4) Finances are all about choices; none of us can have it all. I deliberately chose a smaller house and one 10 year old car, so I can have room in my budget to indulge in nice clothing (and also travel). It's what gives me pleasure and puts a smile on my face. If a fancy new car or double Lattes or ... whatever.... puts a smile on your face - then go for it (if it doesn't put you into debt)! We all need a bit of mad-money in our lives. (Said to the person who is really asking about whether I am racking the clothing up on credit; I'm not. I have a clothing allowance. Just happens to be generous right now. If I lost my job, I'd *still* have a clothing allowance... it would just be a lot smaller).
5) I actually have a lot of anxiety about the future. I'm pretty anxious by nature. Psychographically speaking, dressing nice gives me something I can control. Life can't be falling apart if I look awesome. I probably inherited this from my grandma, who came from wealth but lost it all in the great depression, yet to her dying day, would throw on blue powder eyeshadow, pink blush, lipstick, pearls and a mink to hit the local grocery store. This is what makes me feel comfortable in my own skin - if you feel more emotionally comfortable dressing differently, than all the power to you; I can understand it. Dressed up to the nines or dressed in a "who cares" way - often it's not about the clothes - it's about the emotions. (Said to my friends who "get" this sort of emotional analysis stuff! I actually had this same exact convo with my BFF since 6th grade yesterday.... )