Rachylou, I think I get what you mean. The collar, the pinstripes, and the drapey fabric all tend toward formal or business looks to me. But it's shorts (and short ones at that) and short sleeves. Casual toppers are a way to anchor it more in casual territory.

I like these pieces, together and paired with different items! The only option I'm not as keen on is the longer cardigan. Re. sizing: I definitely prefer the smaller shorts, but I would say to go with whatever size shirt feels best, accounting for some shrinkage.

I love your sweater and jeans outfit--gorgeous sweater knit and perfect with those jeans!