Inspiration material (although not directly useful): The original vt100 technical manual. I didn't know they could overlay video!
This uses quite a few tricks to get around slow memory, and bus contention between CPU and display controller, like a one-line buffer for the "in-hand" line, and a quick DMA to grab that line. It also describes the very low-level way that double-height and double-width mode worked (it halved the clock for a the line counter, or shifter, logically enough).
I think that I can get away with processing the buffered serial input in the vertical blank, like ZX81 Slow Mode would, and not worry about memory contention or similar.
There's a github repo with my VHDL code in it (and eventually the Z80 assembly too): howardjones/fpga-vt. It's my first attempt at an HDL implementation (or any significant-sized digital logic actually, I'm a software person), so please be gentle.
Wow! This is an impressive first FPGA project to tackle. Kudos for sticking to it for so long.
I see it has a Z80 core. Could it be built with a real Z80? I like the idea of a vintage terminal built with vintage parts. I designed and wrote software for a Z80-based terminal back in the 1980's, so such a project would give me a chance to re-use that knowledge.
I solved the Z80/video contention problem by noting that the Z80 only reads/writes data RAM during its T3 cycle. With fast RAM, I could multiplex the address between the Z80 and video a half the Z80's clock speed to neatly interleave the accesses.