Ah, that blazer! Now, why did you have to remind me of that?
It's such a gorgeous jacket and would look lovely paired with the dress.
To be completely honest, I don't think it would work so well over the sweaters. Keep in mind, I have a low tolerance for "winter" and "summer" fabrics being used together. But also -- I had this jacket on. There is something about it that begs for lighter weight fabrics underneath it. It's not flimsily lightweight (it does have leather, after all!) and could work pretty well with cotton or other lightweight knits (e.g. linen) underneath. But I think wearing cashmere under it would feel and probably look a bit odd.
I also think sizing up to wear with sweaters beneath is the wrong approach for this style. It's a blazer. It wants to be worn as a lightweight over layer or as an indoor jacket. What's important to it is the cut and shape. If you size up you lose that. It won't lie properly, and you won't be able to wear it well without those layers.
It's absolutely gorgeous, and I was really tempted -- but for me, it wouldn't get worn enough. My climate is sort of the opposite of yours, but the issues might be similar. I think it would be most useful to someone who lived in a temperate place where lots of 3-season or even all-season clothes made sense.
For me, it's too warm and heavy for my true summers ( if I worked in air conditioning it would be useful -- but I don't). Meanwhile, it's too lightweight for my winters and even my falls. It's really a late spring jacket. The only months I could really see myself wearing it were June and maybe September. When temperatures are around 18C.
Now...that might be perfect for San Diego like temperatures! But I can't imagine wanting to layer it with cashmere in that kind of weather.
Even so, it is so gorgeous and at the current price is, I think, a genuine steal. It might be worth having anyway as a spring and early fall jacket.