So I still can't get the looper working. I tried a workaround where I would start a subprocess to run SOX aplay to play the looping audio. But.....it conflicted with the already running pyo server and could not get access to the audio output at the same time. So, I am now seeing if using pulseaudio or jack server might fix the SIGSEGV I get from pyo's SfPlayer.
Why? Because I had SfPlayer working on my Arch Linux laptop and not running on the Raspberry Pi and one difference is the laptop uses Jack. I've been trying to avoid this, because it adds another layer into the audio stack and everything feels a bit fragile already.
So I am recompiling pyo with Jack support.
In the meantime, I broke everything when I installed pulseaudio. I almost lost my mind. But I found the problem, it was ALSA had been reset to use the mic input, and not line input. Who knows why?
So here's a picture of how the ALSA setup should look when you run alsamixer. (Also added this to the google slides doc.)