For the second test I went with the 74HC595 8-bit shift registers, I used 4 of them, one for each color and the last one to toggle the MOSFETs on the anode.

This allowed me to decrease the number of GPIOs used from 32 to... 3. I used the CircuitPython 74HC595 library which makes use of SPI to write to the shift registers, it made redrawing the matrix a breeze!

One problem is that I still had only 7 colors, since it's a bit hard to do PWM with shift registers, but that's something for the next test.