You need a finer gauge sweater for starters. Perhaps scrunch the sleeves. Add shoes that pop.

I find the drapey "volume on volume" look very elegant - and I like elegant - so the combination speaks to me.

Hope that helps.