I don't wash my face very often as a rule, but I don't change out cloths every time either (I do actually use a wash cloth in the shower more often, as my dry skin seems to get irritated by the plastic puffy loofahs or even the natural loofahs).
I figure I just need to use a little extra cleanser (face wash, shower gel, soap) to wash it up and then I hang it up to dry, and we're good. Surely what's good enough for cleaning my skin is good enough to clean the cloth? I keep two to hand to alternate in case it's slow air-drying weather. They go in the wash at the end of a week or so, with the other towels and such --- only because the build-up from hard water needs to be pounded out by hot water!