While is hard to fathom how moth damage might occur in the same places on multiple garments, do consider that the evil little beasts are drawn to perspiration. So once you've ruled out friction-related sources of damage, do not rule out the insect kingdom.
Also, while moths much prefer woolens (their absolute favorite being cashmere), it's my understanding that they will eat pretty much any natural textile in a pinch. And the moth mantra is "the dirtier the fabric, the better, the darker the surroundings, the better." Keep things clean, keep the surroundings sunlit, if possible.
If moths are the culprits, it's the larvae that do the damage, not the actual flying moths. So if you're seeing moths flying about, take that as a symptom of a problem and not as the actual problem.