This is stunning on you!! I don't think the contrast is a problem at all in this case. There are two reasons.
1. Your own beautiful high-contrast colouring makes higher contrast easier for you to wear in general.
2. The proportions. In my situation, it was the combination of the ensemble's proportions and MY personal proportions that created a problem. Here, your sweater is longer, revealing just an inch or so of shirttail. Plus, you may not have my short legged frame to contend with. (This makes a longer top a challenge for me to wear -- even though I am high waisted I have a long rise, and relatively shorter legs -- so longer tops truncate my leg line.)
I think you look great, in short!