I would let it go also if you know you will not develop a little love for it. Last year I made a list of colours that I would wear and tried hard to stick to it. I felt I would have fewer orphans and way less money spent on Ooolaalaas which is what I call my crazy draw to colours that are not great for me but look wonderful on others. I try to only buy in certain colours to avoid. However, I must confess I have fallen off that wagon here and there.

What about wearing a great scarf, vest, blazer, etc., with it?

If you can't (won't) dye it, then let it go. You deserve to not have a simple mistake make you miserable every time you wear it. You miscalculated. Not really a big deal. Time to move on from that mistake though. You've got some really nice sweaters that DO make you feel fab. Wear THEM to death!

Letting it go will cause you a bit of mental anguish when you do it. You'll probably feel bad for a day or two, and then it'll be over, you'll forgive yourself and move on. You'll wear the nicer sweaters and smile inside every time you do. (And really, when you look back have you ever truly regretted donating something you don't love? I never have. It's a good thing to know about yourself.)

Keep wearing that sweater and the mental anguish could go on for (years?)....

Your choice...

I'm the opposite. I love my not quite right coloured clothes for around the house, especially if it's a colour I love but cannot wear for flattery. I am super duper cold and always wear another layer so would love the cardigan to death. Wear it over the nice top (cashmere turtle) when baking, using bleach, scrubbing tiles and don't worry about it. The difference is that I love mine and feel great in them, they do not make me feel badly. My sister gave me a yellow premiere cashmere sweater I adored for years around the house. Felt lucky to be able to wear a quality garment for doing all those icky things at home without ruining my clothes.
If it makes you feel bad get rid of it, just make sure it isn't the winter doldrum effect.

I'm totally guilty of doing this, although I'm getting a little better now that I realize it. But I totally understand the urge to wear things I don't like as much to justify having them, while trying to "preserve" the ones I like better. I should know by now that, when I wear something to work only on days where I'm sure I'll just be sitting at my desk not talking to anyone all day and heading straight home in the evening, it's a good sign something's up with it.

For your sweater, I'd probably be tempted to downgrade it from working-at-home wear to just plain loungewear because I have a hard time giving up really soft and comfy things even if I don't like them much on me. If that doesn't appeal to you, I think you'd be absolutely entitled to just get rid of it.