I've learned to stay away from buying sweaters that itch - coarse wool and arcyrlic - no thank you! Even if I wore a long-sleeved fitted pure cotton tee underneath, I'd still feel the ickiness on my neck and wrists. I need absolute comfort in my clothes. ALL of my sweaters are cotton or a blend of cashmere/rayon/cotton. Once I had a gorgeous red sweater dress that looked cotton but was arcrylic and it just hung in my closet, never worn once.
If your sweater is itch-free, not a problem! I love a cotton sweater next to my skin!