Oh goodness, so many options! I like lighter weight, looser cotton or linen pants when it's warm. My current pairs are both olive, different shades. Both sort of simple straight leg cuts. I always dream of a swishy, wide-legged pair of linen pants but I really like being able to roll up my summer pants (for wading, beachcombing, biking, etc), so a wide leg isn't as versatile.
For colder weather, I like corduroy and wool; both are warmer than denim and offer a big texture boost to otherwise subdued winter outfits. I also have some mid-weight cotton pants--twill, canvas, double gauze--that are effective at spanning multiple seasons. I have found that olive is a more versatile pants colour for me than almost anything else. If I could only choose two pairs of pants, I'd go with black denim and olive twill.
I do wear my gear pants quite a lot too in casual-but-mostly-outdoor contexts (park, playground, bike path, skating rink, teaching outdoor classes, etc). The black joggers, loose fitting nylon khakis, and slim-fitting grey hiking pants below are all arguably jeans substitutes for me. The grey ones are so versatile I've added a second pair in sage green, and may go pack for a third pair this spring in a darker green or brown. I would love to add a baggier, more voluminous pair of track pants too, but haven't found the right ones yet.