After connecting my microcontroller and display to my board, I tried displaying test patterns with code based on sprite_tm's. I did get a reaction from the display, but I kept getting vertical streaks while trying to write a single row to the display. So I switched to my backup screen, and it worked! So I think my first screen got fried while I was attempting to use it with my v1 PCB and its improperly soldered FPC connector.
The next step is driving it properly. Each of the driver sources (essentialscrap/Sprite's mods/NekoCal) is a bit different as to when it sets the various pins high or low, especially the OE (output enable) and CKV (vertical clock) pins. I'm doing it like NekoCal now and it seems right and I get nice black and white output. But to get grayscale like NekoCal does, I need to control the timing better. This is a problem due to my use of shift registers.
I'm using the ESP8266's hardware SPI to drive my SN74HC595 shift registers, and at first I just tried using the max speed of 80 MHz. As it turns out, they can't run at that speed. If I understand it correctly, TI's datasheet says 5MHz at 2V, 25MHz at 4.5V. My board runs at 3.3V, so I set the SPI to 10MHz and it seems to be working fine. The two 8-bit shift registers are chained to get 16 bits, 8 bits of data and 8 control bits. (If I did this again, I'd split them so that data can be written faster without having to also write the control bits). Writing 16 bits at 10MHz takes 1.6us, so that's the fastest I can toggle any pin included in the shift register. Unfortunately, the CKV pin is one of those pins, and I need to time it in hundreds or maybe tens of nanoseconds - 1.6us is too much. I intend to try to work around this by removing the shift register from its socket so I can wire the CKV pin separately.
Another issue I ran into is that a full 800x600 black-and-white bitmap takes 60KB of memory. 4-bit grayscale would be 240KB. I'd thought the ESP8266 has 128KB of RAM, but it only has 96KB, of which around 40KB (maybe a bit more) is available for general use, so no storing a full bitmap for me! For now I'm streaming the bitmap over wifi in 200x200 pixel chunks, but I've ordered some 23LC1024 SPI SRAM chips so hopefully in the future I can store the full image on the ESP itself. Other options I've considered are to store the picture in flash memory instead (My ESPs should have 4MB, which would be plenty) or to try this: hackaday.com/2016/11/30/the-esp-a-new-1kb-contender-appears , as maybe with wifi off I'll have more memory.