I love button front shirts! But like Angie suggested, I opt for a looser, relaxed fit for most shirts. As long as the shirt fits me well along the shoulder line, I don't mind a bit of looseness around the body. Anyways I tuck the shirt in for more structure and tend to wear them with a bit of 70's vibe. I leave the first two buttons undone if possible to elongate the neckline. But I think opting for a fit that is just a lil bit oversized plays a big role in my comfort with shirts. I love patterned shirts - polka, gingham, plaid & mostly wear cotton ones.
If I do buy a fitted one which is rare, I look for some amount of lycra / spandex in the material to ensure a close fit and always use a safety pin to ensure smooth fit around the bra area.
In fact button downs are a staple for me, my daily uniform is skinny pants + button down shirt + booties with a scarf / topper for my freezer like office.