A few more hours finishing the display circuit (oh boy, does the SSD1306 datasheet suck in terms of example application), a bunch of capacitors added, and I think it's ready. Oops, a small DRC fail because I changed the USB connector. No problem, we will move that trace, thank you DRC! And now ordered from OSHPark.
We will see how that goes. Now I we can bet on what comes first: the buttons, the display, or the PCB?