I also feel like the bag needs to go with the outfit, but that may or may not mean that it ends up matching the shoes. I am not adverse to shoes and bag in the same color if it makes sense in the overall context of the outfit, but I don't go out of my way to make them match.
However, if my shoes are a neutral, like cognac, that is not apparent anywhere else in my outfit, I might feel inclined to choose a belt or bag in a similar tone or shade.