Horseback archery sounds so cool!!
I've actually never had a fleece vest, but have always had a down vest or two (my entire adult life anyway). A thin/ultralight puffer vest, something along the lines of Uniqlo packable down or Patagonia Nanopuff, should be comfortable as an outer layer in 40s-50s F if you have a reasonably warm layer underneath (fleece, merino sweater, etc) in addition to a baselayer. You can always shed a layer if you (or the day) start to warm up. A puffy does such a good job trapping body heat and blocking wind compared to fleeces and knits, it's nice to have on hand for an instant boost of warmth even if you don't wear it all day.
The thing I love about puffer vests is that, without the sleeve bulk of a jacket, they're easy to layer under just about any coat. And they don't look as out of place indoors as a puffer jacket! I fell in love with them back in my organic farm/produce department days, jobs where I was in and out of cold storage, but also often wrist deep in water, so jacket sleeves were a no-go. A hoodie with cuffed sleeves that would stay pushed up, plus a synthetic-fill puffer vest, was the perfect combo.
Will you be able to wear some sort of glove? When I rode horses, I definitely needed gloves for shoulder seasons; you can't just stuff your hands in your pockets to warm them up!
