I just read through the reviews and most women said they wear it with a white button-down (not very adventurous!). I love the idea of a plaid shirt to mix patterns, or with a big contrasting scarf at the neck (solid or plaid), or with a long button down poking out underneath (maybe tucking in only one half, a la JCrew). I would also try non-black bottoms -- your plaid Kut skinnies maybe, or some slouchy jeans or pants.
And keep a sock handy, in case you still run into spoilers.
PS I happen to love this sweater. I coveted it for a long time!