This happens to me too, and it's super annoying! For me it mostly happens on thinner tees and lightweight merino or cashmere knits (which have weaker fibers in general). My husband's theory is that it's caused by office desks, because they are largely made from crappy particle board with rough edges these days. When he wore long ties more regularly, they always got shredded around that area; he claims that's one reason he switched to bow ties!
I think it's actually a combo of the jeans button and the counter/desk (where the two basically rub together with the tee in between).
If the holes are small, I just darn them closed with some matching thread. It's not perfect but not too bad either.
ETA: I almost never wear belts, so for me at least I think I can rule that out.