I have the hat with short hair and glasses thing mastered. I have learned that For me the brim can be no more than 3 inches. 4 inches is pushing things and would be for practical sun blocking purposes at poolside, not for out and about. Any more than that and it gets too large. Visors are a great choice because they allow your hair to show. They make some nice straw visors, not all of them are sporty looking, and I have a nice collection. Some can look good with a dress. and everyday clothes.
Watch the shape of the hat. I find I can wear fedora's, boaters, western looking hats, and even floppy brims as long as the brim doesn't drop too much in front. You may want to experiment with some styles to see what works for you, but when the brim droops in front it makes me look bald, if it is more uplifted, you can usually see some of my hair.
Below is a hat that I purchased this year. The brim is on the larger end but it rolls upwards to shorten and ends up looking nice. I tie a scarf around the top which also makes the proportions look better on me, otherwise it looks very large for my frame.
Good luck.