It's hardly fair for me to answer the question, but I would HATE to leave behind the camera while traveling. It is my job, after all.
However, there are plenty of times while traveling when I leave the camera at my side and stop to enjoy things as they unfold, rather than experience them through the viewfinder. I don't bother with taking pictures in places like museums -- I'm there to LOOK at the art, not document it.
I was just in Cuba for 10 days, and my most memorable interactions with the people there occurred without any photos being taken. Granted, I'm not a great "people photographer," but it was still noteworthy. I also spent a full afternoon on the beach without taking any photographs, and it was a wonderful break for my eye, mind, and soul. I don't think I did my best photography on this trip, but it was more than worthwhile to take the photographs anyway.
To each his/her own. A camera is a critical part of my travel experience, because I'm always open to the experience of creating art, and I'd hate to miss an opportunity. However, some experiences are best viewed without anything between the eyes and the scene.