After writing down all the changes I wanted, I started to play with the PCB to see if they were possible, and of course I made another version with all the changes I wanted:
And the back:
I managed to keep the back clear of parts, save for the battery and speaker, which really can't fit anywhere else. I also modified the battery charging circuit a little, to make it possible to charge the battery with the unit powered off. Hopefully it won't drain the battery.
I'm going to sit on this a little bit and if I don't have any more ideas, I will order it tomorrow.