Team Gryffin, unless they're pesky or redundant. Most I'm aware of are useful; I probably haven't come across some of the types and locations mentioned her.
I remove most labels, always open jacket/coat vents, and usually open pockets I'm likely to use. Used to request surgeons cuffs when I had suits made, because why have a mock opening? but it wasn't always accommodated. One tailor just made working buttonholes on a cuff that looked like it would open.
I used to remove labels inside bras, but now that I have duplicates in slightly different sizes, leave labels in for quick identification. Like rachylou, I'd rather wait for the right time to clip than to damage a garment.