Grace, I actually really like camel, but it has to be the right shade of camel. I think camel sometimes gets mixed up with khaki or tan, and to me, camel has almost a reddish tint to it. I think it can be very pretty, but if you end up with just a tan cardigan, it could easily wash you out (it sounds like you and I have the same coloring, and I have this problem with lots of tan/khaki shades). If you aren't too keen on bright colors, what about a darker jewel tone color, like burgundy or dark purple? Something like this in mulberry or the navy color:
http://www.eddiebauer.com/cata....._ite=Tier3
Those colors would look nice with black and white and other neutrals also, but also add some interest to an all neutral outfit.