I could have written your post!
Same experience--tailored sweater ( but yes, I have sized up one or sometimes 2 sizes from how the stock photo may look--say in Talbots or AT --can work with pencil skirts but can look like just too much classic with pants, especially very typical work trousers. For example, I have bought the Talbot's Audrey sweater in several strategic colors but still have to be careful as they "need" something else to jazz them up.
What helps: boots with the trousers; sometimes contrasting or unexpected color, sleek or trim; also pants that are more updated, such as are tapered and scrunched over loafer (ankle pants would work but are problematic for me in winter). Possibly interesting jewelry.
Lesson 1 I learned (stll learning): Experiment with what types of footwear seem to elevate basic pants. Depends on season. Keep updating in those directions.
Lesson 2 I learned: don't overbuy in one style or silhouette.Wear fewer, but more modern, tops This year I specifically added a few high-low, some cocoon, sweaters, fluid to slightly oversized; merino hi-low; curved or shirttail hems. These are giving an updated look and the not-straight bottom is good for pear shape. I tried to resist duplicatiing even some of those--I am assuming that some time next year or the next, I may want a different look.
Note to self has been: sweaters over pants are workhorses for me in winter, but need to be updated; pants can be updated but are inherently harder for me to use too many styles since fit and proportion is just more problematic; tops are easier to update/trend. Sweaters should not be simply fast-fashion, but also may not all be 5-year items.