Definitely don't bother with the car. Fun story: Years ago I went to San Francisco on business and rented a car. It was a nightmare: Horrible traffic, plus it was $35 a night to park at my hotel. So the next morning, bright and early, I called the car rental place and said "Help! I need to dump this car!" They directed me to the famous Mark Hopkins Hotel (home of Top of the Mark, which I highly recommend), where there was a rental car place in the underground parking garage. So I dumped the rental car and went to the front of the hotel to grab a cab to my business meeting. Lo and behold, the hotel's limousine driver wasn't doing anything at the moment, so he gave me a lift to my meeting! Big fun!
In other news, I love the restaurants at Belden Place. It has a variety of small European restaurants to choose from. If you like French food I definitely recommend Cafe Bastille there. It's like dining in Paris! (My first YLF live meetup was dinner with Natalie [GoldenPig] at Cafe Bastille!)
Oh, and if you have any teens (or adults) who are into that sort of thing, check out the Cartoon Art Museum. My son loved it and so did I!
And finally, this is super dorky but years ago my son and I did a Segway tour and it was a blast!
Couple other things:
Dottie's True Blue Cafe for breakfast or brunch. You will have to stand in line but it will be worth it.
City Lights Bookstore and the Beat Museum right around the corner. Fun look into San Francisco's Beat Generation history.
Oh, and we took a walking tour of Haight Ashbury that we really enjoyed. But we're history nerds that way, as you can probably tell!