Another vote for wool! I either feel very cold or very hot in humid weather; it tends to make either temperature feel more extreme. Wool has the property of not feeling clammy in humid weather, although cotton can. It also breathes well and makes it more unlikely for warmth to build up unpleasantly. I've been in that situation with puffers - too cold without it, but start getting overly warm with it on, then take it off and feel extra chilly with the dampness, rinse, repeat. Wool cardigans or boiled wool jackets are more comfortable in damp weather.