Don’t want to sound discouraging, but having a wardrobe of “perfectly fitting” items will always be an elusive goal. Our bodies aren’t static. Age, dietary changes, medications, exercise patterns, and fashion trends will always play havoc with fit; that’s why stylists and dressers are employed by celebrities and those in the public eye. The rest of us mere mortals usually settle for a “good enough” fit using styling tweaks, forgiving designs, and judicious pinning.
The good news is that quality garments can be tweaked by a competent tailor to adjust sizing as our bodies change. If the fit of your pants bothers you, then find a good tailor who can take them in, but do so realizing another visit to let the pants out in a year or so might be required. A perfect fit in wardrobe which holds quality items means making friends with your tailor.