Since I only ever wear my blazer over a skirt to feel dressier or more formal, I have to agree that it seems blazers on skirts makes for a very formal look. Even though mine is brightly printed, lightly quilted cotton --- plenty casual in fabrication.
Though, I have to say I find the same thing happens to me if I add the blazer over trousers. Only jeans and flannels seem to go to casual territory.
Assessing further, I suspect all these bottom halves I've ever paired are casual to start with, except the jeans and the flannel pants --- so it may not be the blazer that's to blame! I wonder what would happen if I had a denim skirt (instead of tweed pencil or silk maxi) to pair with my blazer...
But yeah, cardis and casual jackets are 100% easier on a skirt or dress for MOTG.