I think pink, black or ice blue would all look right on you with this chestnut brown. I believe pure white would be better than ivory or cream. I haven’t perceived that you lean towards autumn colors, and there isn’t a law that anyone has to combine brown with other fall colors like orange, gold and olive, or even cream. I challenge you to do an outfit lab and pair the sweater with all your different color tops and dresses. You might be surprised.
My personal single, unbreakable rule for brown has always been to NOT combine with green. I have very few brown items now, just a couple of mushroom/taupe layering tops, and I’m not sure they qualify as brown. I usually pair them with navy cords, blue or burgundy jeans, or black pants.
The nice thing about chestnut brown is the clarity. This color rarely looks murky or muddy, the biggest risk in browns.