The matrix is controlled by an ESP32. This was my first ESP32 project and my first foray into ESP-IDF. So I cut some corners and instead of using LCD-Mode, I bitbanged it. With one core fully dedicated to the display and one dedicated to IO I can draw about 3300 monochrome frames per second. Enough for me.
Normally, you redraw the same line a few times (like PWM), to achieve color depth. By only drawing a line once, I trade in color depth for fullscreen refresh rate.
The Kinect spews out a texture with every pixel color representing the depth. This texture is fed into a HLSL compute-shader which is amazing for this use-case because its so great to parallelize.
The whole thing is then lightly RLC-compressed and send over a raw TCP stream right into the displays framebuffer.