I'm usually a lurker here, but had to de-lurk to answer this question... I have lived in San Diego for 11 years now. I live near the coast, where it is usually quite cool. Inland can often be 10-15 degrees warmer, especially during the summer.
I've made the mistake of buying too many summer clothes during most of the years I've lived here. In actuality, our summers are quite short, lasting from early to mid-July through October (and sometimes not all of October). The rest of the year is mostly cool (not usually cold but it has been the past few weeks) with some unseasonably warm days thrown in from time to time. There are days it's in the 70s in January or February, but those days are relatively rare.
I feel the cold more than a lot of people, but I find I need some sort of jacket here about 85-90% of the time. I definitely think it's a good idea to have lots of layers to peel on and off. I often wear a lightweight top with a cardigan and a coat or warmer jacket. It's pretty damp here so it feels colder than it is much of the time. I used to live in the Lake Tahoe area and in Colorado and the same temperatures here feel quite a bit colder. The same thing is true of the San Francisco Bay Area, where I'm from.
I hope this is helpful. How one dresses here depends upon which area of San Diego they live in as well as to what degree they feel the cold. Since I live by the coast and get cold easily, I need more warmer clothes than a lot of other people in San Diego. Overall, this is a beautiful place and the weather is better than most places in the world, but it's not the warm and sunny climate that most people think it is. Hope you'll like it here!