I'm a couple months into this already, so I've already tried and discarded a dozen ideas.  Right now, I have an STM32F746 dev board driving the Sharp LCD at speeds up to 60FPS (finally), though I'm dropping that to 30-40FPS for practical usage.  I'm making some progress getting a basic FreeRTOS structure set up (using CMSIS).

I've gotten some individual module boards fabbed for basic 3.3V LDO supply and 5V boost for the LCD, and tested out a couple of strategies for connecting the LCD via a 0.5mm FIPS socket

I'm dusting off and updating the layout for handling a blackberry trackball properly (more difficult than you'd think with a two-layer board, if you want to fit in LEDs)