I like the idea of a neutral rather than a color, and I lean toward the Cole Haan. It looks so classic. The Hobo looks really nice, too. A major difference between the two is how they could be carried. The Hobo could go over your shoulder or over your arm, and the CH over the arm or cross-body.
I have one colored bag, a shade of purple that I couldn't resist, and I've carried it some, but not as much as I thought I would. I reach for my neutral bags--taupe, gray, black, brown--all the time. I do switch them out often now, though I didn't used to. (I use a purse organizer that I can just shift with all its contents from one bag to the next.) I am the same way with shoes--always favor my neutrals over any colors that I might decide to try.
If you like Fossil bags, you might check out the Vickery shoulder bag or the Sydney satchel. Both come in warm brown shades, and I could see you carrying either of them. I have the Vickery in a cream/taupe combo and carried it everywhere all summer.