It is a pretty hat, especially the blue version!
Just as with clothing size being a number on a tag no one can see, the 'gender' of accessories especially (but also some other clothing without fly closures etc) is largely invisible when worn. If the shoe --- or hat! --- fits, I'd wear it!