I guess I've just about given up perfume and heavily scented lotions. I am very smell-sensitive and have massive allergies. I can't tell you how many times I've been stuck on a bus or a plane near someone wearing something that either was too heavily scented or that I was allergic to. It's not just a matter of disliking the smell. In many cases, I will become itchy and get a dreadful headache as a result of someone's scent.
I like the idea of perfume, but when I think of my own reaction to heavy scents I don't want to risk putting someone else through that. People have really widely differing levels of smell-sensitivity, many people have allergies, and smell is such a subjective thing that it's impossible to know who I might be negatively impacting with my scent. This is why I choose not to wear perfume or heavily scented lotion, but I want to make it clear that I don't expect everyone to give up their beloved perfume!
It would be very interesting to see whether any scientific studies have been conducted on smell perception. I wonder whether there are easily identifiable categories, like "scents in such and such family tend to bother +/- 25% of subjects," or "scents in this family tend to be innocuous for +/- 60% of subjects," etc. I know that something like this exists for tree pollens, but I doubt it's been done for perfumes.
That said, I get how nice it can be to have a scent that makes you feel pretty and sweet.