This was the biggest step, getting it all a lot smaller and fitting in the small gap in front of the speaker.

First was to get the strip the parts off the USB sound card, the 3.5mm jacks were the easy part and the USB jack, even though the shield was not soldered in. In the end I got it loose enough to use metal fatigue (wobble the jack back and forth) the the pins broke.

With that out of the way, time to solder everything to the Pi Zero. I used this guide to solder the USB sound card directly to the Pi: https://www.grappendorf.net/tutorials/nerves-pizero-edimax.html

After a lot of de-soldering and soldering, I gave it a quick test, nothing fried, fortunately. Next I soldered the 5V straight to where the header should go.

After soldering the resistors in place and the speaker to the USB sound card, time to give it a final test before stuffing it all in the case. All worked again.

Finally I made the USB sound card the default alsa device using this guide: https://nixingaround.blogspot.com/2016/11/raspberry-pi-zero-usb-audio.html

And finally, all stuffed in, case screwed down and boot up the Pi, winning!