I wash most things before wearing, although I have made "emergency" purchases that I wear right away. Most garments are treated with sizing and other compounds to make them look nice on the rack/in the package, and they're not good for your skin. Same goes for bed and table linens!
Something like a hand knit wool sweater, I probably would not bother washing.
Also -- with rigid denims (heavy, all-cotton, selvedge types) many of the makers recommend *not* washing or washing very minimally. The idea is to break the pair of jeans in and let them mould to your body... I have very minimal experience with this process b/c all of my jeans are secondhand.
With shoes -- I've noticed more and more of them come with a strong chemical smell -- probably they are treated to protect against moisture, mold, insects, etc. Most of the time, I send shoes right back if I can't stand the smell. But if they're perfect in every other way then I will leave them out in the shed for a few days to air out before I wear them. I don't like the idea of them off-gassing in my house.