Liz' comment (with suggestions on how to finish the question about "trying to hard to ...) is food for thought.
In my opinion, the old Chanel rule (about removing one item before you leave the house) are among the rules we no longer need to follow. It might have made sense in her circles and time, when she and others wore a lot of accessories. Today, many of us wear just a few, perhaps too few. A modern rule (if we need any at all) perhaps should be: "Before you leave the house, take a look in the mirror, and consider to add at least one great accessory!"
However, personality is a very important ingredient in personal style. Some people manage to look gorgeous LOADED in accessories. (It wouldn't make sense to ask Iris Apfel "remove one item" ...) Others manage to look stunning practically WITHOUT accessories. Most of us are somewhere in between.
You simply have to experiment to find your own accessory style. And this can be an exciting journey. Please don't be too afraid of others' opinions. Rember to have fun with fashion.