Others have already said it: Look to the underlying emotional influences and (try to) deal with those first and you might find your dissatisfaction with your closet diminishes. The wardrobe is a symptom, not the problem.
As others have also said, you have terrific style and you always look fantastic! If you don't feel fab, then absolutely, shift your purchasing to stay truer to your stated preferences. Find an "accountability buddy" to help you keep on track, as Gryffin suggested. Do whatever it takes to keep yourself to your plan.
You asked how others stay true to their own style and avoid the endless churn. It seems to me that it all boils down to inventing or accepting limitations of some kind.
1. Deciding on a colour palette, or a theme (as per Brooklyn), or a silhouette or "formula outfit" (as others have mentioned, e.g. Shevia) or some other form of aesthetic limit. Whatever limit works for you.
I don't really do this myself terribly consciously but, having said that...despite my love of colour, if you looked into my closet, what you'd actually see is a sea of blues with some white and grey. Why? Because I don't tire of blue and feel happy wearing my blue items for a long time. The same item in a different colour wouldn't fare as well. Even if I buy something a bit trend-driven (e.g. my now 5 year old Zara sarong shorts) chances are, if it's blue, it will stay in my closet long after the trend is over and I will still enjoy wearing it. This allows me to have my cake (a trend) and eat it, too.
2. Setting a firm budget and sticking to it. I admit I fudge this one myself all the time. I just take on extra contract work to pay for my new things. As long as I am paying for them and enjoying them, I'm okay with it. It's my hobby.
3. Setting a clear number of purchases you will make per month or year or a top end number or a 30-wears (or similar) goalpost for each new purchase. Again, I don't do this religiously myself but it seems to work for others, and I have found that questioning an item I'm thinking of buying ("Would I want to wear you 30 times?") is a good way of preventing the most egregious mistakes.
As for the guilt over shopping mistakes...I like Sterling's idea: it's like tuition. And donating while the clothes are still current allows someone else to get some joy from what torments you.
Looking forward to seeing your before and after wardrobe/closet and watching you refine your style even further into continued fabulousness!