What a great question. I have asked myself this question in the past.
I agreed with Rabbit who said she'd put the money into shoes, jeans perhaps and bag. For a solid colored light weight tee, it's more about how it fits you and the materials, to my mind. So if I wanted to go high end I'd go high end in materials and ethical production for a tee shirt -- which does have a cost impact, not brand name unless it was some very unique pattern.
But as I continued to read, I agreed with April who said if you're a tee person (I am), a beautifully-fitting tee is worth a lot. And Firecracker who said if they are good quality fabric, they fit the way you want them to, they hang straight, and they are the style you want, I'd say they are worth what you are willing to spend.
I think what happened is that I agreed with Rabbit's logic, but then went on to love what April and Firecracker said because I wanted to be convinced that an expensive tee shirt is worth the money. What I came away with is that an expensive tee shirt can be worth the price if you love it. I would suggest buying one and road testing it. If it meets your needs then buy more.