For the second test I went with the 74HC595 8-bit shift registers, I used 4 of them, one for each color and the last one to toggle the MOSFETs on the anode.
This allowed me to decrease the number of GPIOs used from 32 to... 3. I used the CircuitPython 74HC595 library which makes use of SPI to write to the shift registers, it made redrawing the matrix a breeze!
One problem is that I still had only 7 colors, since it's a bit hard to do PWM with shift registers, but that's something for the next test.
For the first test I just went with connecting all 32 pins of the matrix to a board (the 8 anodes were connected over a MOSFET). The board I used was my SAMD51 Breakout Board, I got CircuitPython running on it an used an array of 32 DigitailInOut objects to control the matrix. I don't have the code on me right now, will update later.
It was, of course, possible to drive the matrix this way but it used way too many GPIOs, even though it was kinda fun ;)