Created the button matrix PCB, with connections for the ESP32, an Adafruit PowerBoost 500, and an LCD/SD card combo board. Ordered all the parts for a prototype.

It should come out roughly the same size as a TI-83. The prototype will be thicker, since it'll be mounted on pin headers. I plan to keep the ESP32 as a separate dev board, but soldered in place. That might be with male pins, or I might try soldering directly to the surface of the button matrix PCB. Surface mounting will be harder to solder by hand, but gives more options in placement since the througholes have to place around the buttons.

Speaking of which, the current placement means the ESP32's USB port is off center. There's just no way to do it without changing the button locations while also being able to mount the boards together.

Adafruit doesn't have the PowerBoost 500 in stock at the moment, so I ordered the 1000. The pinout looks like it isn't quite the same, so I'll have to use jumpers for the prototypes.

The other problem is that the PowerBoost board has its own USB for charging. It doesn't pass through the USB data pins directly. It could be done with a y-splitter cable, but as it is, there would be two ports. One for programming (on the ESP32 dev board) and one for charging (PowerBoost). That's not ideal, but fine for a prototype.

The second version would likely integrate the lipo pass through charging directly on the main PCB, rather than having a daughter board. Could integrate an ESP32 module, as well, but I think it would over complicate the design.

Prototype parts for three boards come out to around $60 each. Add a dollar or two for the filament costs on a 3d printed shell. Using EEVblog's rule of thumb of retailing for 2.5 * costs, that would come out to $150. There should be some significant savings to be found yet, of course.

Edit: the location of the directional buttons is also reversed so that it's more in line with a Game Boy (D-pad on the left). Maybe people will want to use it as retro-style handheld, I dunno.