To properly simulate the video hardware, I had to move off Angelo Papenhoff's code base and move the simulator to its own repository: https://github.com/larsbrinkhoff/terminal-simulator
I rewrote a proper simulation of the video hardware, as close as the Technical Manual allows. Now it renders one scanline at a time and will do most of the text attributes like double-width characters, double height, soft scrolling, reverse video, etc.
At first this incurred a large performance penalty but after finding just the right representation for the frame buffer I was down to a reasonable percentage of the host CPU. And it even runs in real time on a Raspberry Pi 3.
Visually, it's a step back compared to Papenhoff's sleek character rendering. There's now only a harsh monochrome display.