Torontogirl- I feel your pain! I have been ruthlessly editing my closet for the past year and have reached a state of contentment by doing the following:
1- I only keep things that: a) are in good shape (no holes, stains, etc), b) fit me well (no holding onto the "in case I lose weight" etc), c) fit my CURRENT lifestyle (not a fantasy life of dress balls, office work, etc), and d) work well with the rest of my closet (no items that I will wear if only I could find the right pair of shoes, etc.)
2- The items that don't make test "a" get thanked for their service (tip of the hat to Marie Kondo for this wonderful way of paying tribute to items entering retirement) and then get donated, etc,etc
3- The items that don't make test b,c,d get put away into what I call the "boutique" (thanks to YLF'er Lisa for this idea). This has proven invaluable:
By moving items about which I am unsure into the boutique, I can create a very curated closet of items that fit well and mix well, and that I will likely wear often (because they suit my lifestyle). I track wears and any item not getting worn gets a second look and is considered as a candidate for the boutique, etc.
I look at the boutique items every couple of months: by having them out of my closet (and off my mind), I find I am able to be much more objective about their suitability for me, and it is therefore easier to let them go. When I really get a cohesive, functional closet, I find I am really reluctant to add anything back about which I have any reservations.
The same goes for purchases: an item really has to be worth it for me to add it to my closet now. I keep tags on, track purchases and return policies/return dates, and am quite ready to return (or pass on if worn) any item that doesn't seem to fit in. I've had a few purchasing mistakes since starting the cleanup and tracking, but much less than before, and I try to find a lesson from my mistakes and do a mental recall of my mistakes when I do go shopping again.
Remember: once an item has been worn, it is a sunk cost. No point in spending more time, energy, or money on something that just doesn't work for you (but may work for someone else).