Wardrobe Architect has several sections on color palette and uses a color grouping she calls "nearly neutrals" that are actually colors, of course, but not too bright or arresting. Burgundy, gray blue, olive, rust, blush are some of her examples.

Of course I've used colors in this way before but I enjoyed seeing her organize her palette with these separated out. Other color organizing "color naming" might call these the secondary colors or something as opposed to bright accent colors.

Not to get hung up on color naming, but these colors or similar ones, depending on degree of warm/cool, are exactly the types of colors I like to use and that seem to satisfy some color cravings but also are not overwhelming or overly "memorable". To me they add richness and interest in the right versions and fabrications, are more flattering face-colors than most "typical" neutrals. And they can be layered tonally or combined.

So it's not just about "red is a neutral" but that I want to work in more of these "quiet colors", maybe I actually want these to be a higher percentage of total items, whilst others might do more true-neutrals of black, navy, white, cream.
I do think it's a challenge, though because it can be harder to find better quality items, which are often produced only in one or 2 neutral colors. So need to balance color goals with other aspects.