Yeah, mid-value-range colors work much better for me because cat fuzz is inevitably floating in the air, no matter how much I lint-roll my clothes.
I also use anti-static spray on the worst offenders (navy oxford cloth, black felt...). It smells awful but helps a lot.