Hmm, I usually do not have an issue with this, but it does happen from time to time. Because button-downs are much stiffer than knitted tops, I did need a bigger size once or twice for a sweater to layer over them. Maybe a very stretchy sweater is a solution if you want to layer it over both knits and wovens? It could be one of those things like pants lengths, that you have to commit to layering it over certain item in advance and pick the size accordingly.
Another thing I failed to mention, is that I noticed that for some brands sweaters tend to run smaller than knitted tops. I have purchased tops in XS and sweaters in S to achieve the same fit.