You look absolutely beautiful, and that dress is perfection! Love that you actually have a sleeve!
I'm not sure just how ivory is your dress, but my DD's wedding dress was ivory--though not a deep antique ivory--and the groom and groomsmen still wore white shirts, and the photos did not look odd at all ( that's me saying, though). That could work better than trying to get ivory men's shirts--you'd almost have to hand-pick them to avoid that yellowed or beige look.
I applaud your trying to keep utility in mind for the guys, but as some others said, jackets and khakis will create a lot of motion in the groomsmen (all varying lines for where the jackets hit) so I'd still think of some kind of suit, whether it's a rented tux or navy or gray suits.
My DD also used David's Bridal and picked a purple color and knee-length styles, but each bridesmaid picked her own style from 6 or so available. Straplesss, halter, V -necks were represented, and I was really knocked out by how each young lady knew what suited her, and so each was comfortable and pretty, instead of trying to make one style work for everyone. They also went with nude-type shoe color but each used or found her own, and that worked fine, too. This was all outdoors in a rustic setting, too.
