I have decided to place the battery on the underside of the hat brim. It makes it easy to get at and it means it is away from my head. I'm not going to do anything fancy - just stick it onto the hat with some velcro.
On the back of this velcro is double sided sticky tape (with a strong adhesive). (The pieces are slighter smaller than an inch squared.) The loop side (the piece on the right in the picture above) was stuck to the back of the battery.
A small cut was then made just below the Adafruit Gemma to allow the connector through.
The other side of the velcro was then stuck to the underside of the brim. (A few stitches with black thread were used to keep it in place.) The battery was used as a guide to see where the best place was to put this piece of velcro.
The 2 pieces together look something like this:
(The black thing you can see in the hat is the switch. As you can see I have sewn the ends under the seam of the hat. When the hat is the correct way up, the switch dangles down and is near the brim of the hat and is now easy to locate due to the battery being there.)
This is how the hat looks from the other side. As you can see the battery is not noticeable:
So that's the battery taken care off. Next is to work on DAB's excellent ping pong ball suggestion!
