I have a hard time judging what colors are best for me too, actually. I just know that they need to be clean and very saturated. Narrowing down my neutrals was more helpful than anything, especially since I also wear both warm and cool shades. Since I stick to black, gray, taupe, cream, and white I can always use those to ground whatever oddball color pairing I have going. If I'm mixing warm/cool colors, I'll often also mix warm/cool neutrals in the same outfit to make it look more purposeful (like black with brown leather when wearing both turquoise and tomato red).
I wonder if choosing to pick soft colors mixed with either brown OR gray would solve your problem? The brown shades would likely be more warm, the gray more cool.