I realized that I was making lots of compromises and that this board was very expensive to make due to the pi.
I also realized that unlike most consoles and handhelds, I could get away with putting the microcontroller into the ROM and having it preflashed because that would mean the game is only like $1.
I plan to use ATtiny84 "ROM"s which slot into the main shell (still an OLED).
This will bring down the overall cost and complexity of the system while also boosting the performance by not using an interpreter.
Essentially, I expect the overall system to be $5 (if I get the OLEDs from overseas) and the cartridges to be $1 each.