I have been mulling this over, and I think I am either taking things to an illogical extreme or being dense. A column of colour - almost any colour - can function as a great backdrop for a pattern or one other bright against it. A print with solids is fine, or even a printed top and a different print on a shoe separated by solid trousers. But in those cases, the column of colour is the neutral, or the prints are broken up by a solid in between. In those cases, colour can function as a neutral.
However, one cannot start from the point of saying that polka dots or leopard print is a neutral in the same way a solid colour is. Most people could not wear leopard trousers with a polka dot top, because just claiming that either print is a neutral doesn't make it so.
IMO, a neutral is a backdrop against which almost anything can be placed. Brights can function like a neutral with certain other colours, but definitely not all. Patterns do not function as neutrals to my eye, and pairing any and all of them together doesn't always work. In my mind, it almost always takes using a coordinating colour or solid to go with a bright or print, and that coordinating colour or solid serves as the true neutral. All of these things are conditional neutrals in my mind, meaning that I cannot throw on a polka dot top with anything (like leopard pants) the same way I can throw anything on with black pants.