I have been running the simulator against a "pseudo TTY", i.e. a virtual software-only serial port. But Michael has a real RS232 serial port hat attached to his Raspberry Pi, so this is an important use case. The code was easy to update and if you pass a character device instead of a program on the command line, the simulator will now talk to a real serial port.
Michael reports success, but characters are sometimes lost, especially when using smooth scroll. Searching old Usenet postings reveal that the VT100 indeed was quite bad processing incoming text in combination with smooth scrolling. Since smooth scrolling is a rather bad idea, the best fix is to use jump scrolling.
The VT100 configuration menus aren't very user friendly. Enter the SET-UP mode (F9 on the simulator) and then type 5 to get to SET-UP B. Use the arrow keys to go to the first bit in the first "switchpack", and type 6 to toggle from 1 (smooth scroll) to 0 (jump scroll).