I've been mulling this question myself recently, after my weight loss. So far I've gotten rid of everything that is 2 sizes up, and most of what's a full size up, and am only keeping a small pile of things that are a half size up - as you say, TOM/bloated sizing. Exceptions so far are two pairs of Jackson trousers from BR in a size up that are not made anymore, and are just perfectly my shape.
Some people say that keeping larger clothes around makes it easier for them to gain the weight back, so they don't. I personally don't subscribe to that philosophy and think that having a few 'in case' outfits makes me less stressed. Especially with dressier stuff which I don't wear that often - what if I suddenly need to go for a job interview or to a funeral and I don't have time to go shopping for appropriate clothes? The more casual stuff I will either wear or cull more often, because it's my daily wear.
I think it's also a-ok if you want to get rid of everything, or keep it all, if you like it and you have the space to store it without it stressing you out.