I don't think you will look too young or trendy. I have a similar white jacket (and black and purple and khaki...) and I do find it's extremely versatile. The question is whether it fits your definition of Sunday casual?
Personally, whenever I put on a jacket like this, I feel I am more smart-casual than casual. On Sundays I'm most likely to opt for a cardigan or jean jacket because that to me says casual more than a structured blazer. However, as we age, structure comes in handy so I can see myself at some point reaching for a blazer even on a Sunday.
I'd say if you wear the military jacket on a Sunday, then you will wear this white jacket on a Sunday, so go for it!
Also another idea is to pair the jacket with a graphic tee, which will make it more on the casual-Sunday side.