I don't have the ideal bag for each category, by a long way.
In no particular order:
-- Flat, relatively small crossbody for when I don't need much and don't have pockets, or don't want to ruin garment lines by weighing down pockets. Have two non-optimal (EF, Coach, both thrifted), not actively looking for ideal solo replacement. Like yours, seldom used but essential when needed.
-- Large catch-all tote for errands. The ideal version of this is a HEWI. Must have crossbody strap as well as short handles, and a lot of other details difficult to find in one bag.
-- Canvas bags to hold errand-specific items and go inside the catch-all errand bag.
-- When in climate with extreme hot and/or humid summers: braided paper/raffia/straw type tote bag in black or beige. Eric Javits options have worked well in the past.
-- I'm a new convert to the category of non-work-bag everyday non-errand handbag that can hang on shoulder. Have two: one robust pebbled leather, the other more interesting design-wise but was showing first signs of wear. Both have ideal (for me) pocket number, type, and placement. Was looking at platinum bags for summer wear, then found I was enjoying high-contrast black/white.
-- Clutch for semi-formal and other events: have just one, but perfect: pale, versatile 220-lb, 5'11 with pockets. Often senses when item is needed and offers it.