So the marvelous 4-port serial "thats gonna save ports" solution has a wrinkle on Linux.

ov 14 22:11:14 K8Console kernel: [  377.599525] usb 1-1.2: new high-speed USB device number 12 using dwc_otg
Nov 14 22:11:14 K8Console kernel: [  377.730456] usb 1-1.2: New USB device found, idVendor=0403, idProduct=6011, bcdDevice= 8.00
Nov 14 22:11:14 K8Console kernel: [  377.730476] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Nov 14 22:11:14 K8Console kernel: [  377.730490] usb 1-1.2: Product: Quad RS232-HS
Nov 14 22:11:14 K8Console kernel: [  377.730504] usb 1-1.2: Manufacturer: FTDI
Nov 14 22:11:14 K8Console kernel: [  377.731748] ftdi_sio 1-1.2:1.0: FTDI USB Serial Device converter detected
Nov 14 22:11:14 K8Console kernel: [  377.731933] usb 1-1.2: Detected FT4232H
Nov 14 22:11:14 K8Console kernel: [  377.732489] usb 1-1.2: FTDI USB Serial Device converter now attached to ttyUSB1
Nov 14 22:11:14 K8Console kernel: [  377.733237] ftdi_sio 1-1.2:1.1: FTDI USB Serial Device converter detected
Nov 14 22:11:14 K8Console kernel: [  377.733394] usb 1-1.2: Detected FT4232H
Nov 14 22:11:14 K8Console kernel: [  377.733876] usb 1-1.2: FTDI USB Serial Device converter now attached to ttyUSB2
Nov 14 22:11:14 K8Console kernel: [  377.734550] ftdi_sio 1-1.2:1.2: FTDI USB Serial Device converter detected
Nov 14 22:11:14 K8Console kernel: [  377.734732] usb 1-1.2: Detected FT4232H
Nov 14 22:11:14 K8Console kernel: [  377.735219] usb 1-1.2: FTDI USB Serial Device converter now attached to ttyUSB3
Nov 14 22:11:14 K8Console kernel: [  377.735883] ftdi_sio 1-1.2:1.3: FTDI USB Serial Device converter detected
Nov 14 22:11:14 K8Console kernel: [  377.736046] usb 1-1.2: Detected FT4232H
Nov 14 22:11:14 K8Console kernel: [  377.736570] usb 1-1.2: FTDI USB Serial Device converter now attached to ttyUSB4
Nov 14 22:11:14 K8Console mtp-probe: checking bus 1, device 12: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.2"
Nov 14 22:11:14 K8Console mtp-probe: bus: 1, device: 12 was not an MTP device
Nov 14 22:11:14 K8Console mtp-probe: checking bus 1, device 12: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.2"
Nov 14 22:11:14 K8Console mtp-probe: bus: 1, device: 12 was not an MTP device
Nov 14 22:11:15 K8Console kernel: [  379.356674] usb 1-1.2: USB disconnect, device number 12
Nov 14 22:11:15 K8Console kernel: [  379.357691] ftdi_sio ttyUSB1: FTDI USB Serial Device converter now disconnected from ttyUSB1
Nov 14 22:11:15 K8Console kernel: [  379.357838] ftdi_sio 1-1.2:1.0: device disconnected
Nov 14 22:11:15 K8Console kernel: [  379.358956] ftdi_sio ttyUSB2: FTDI USB Serial Device converter now disconnected from ttyUSB2
Nov 14 22:11:15 K8Console kernel: [  379.359084] ftdi_sio 1-1.2:1.1: device disconnected
Nov 14 22:11:15 K8Console kernel: [  379.367419] ftdi_sio ttyUSB3: FTDI USB Serial Device converter now disconnected from ttyUSB3
Nov 14 22:11:15 K8Console kernel: [  379.367576] ftdi_sio 1-1.2:1.2: device disconnected
Nov 14 22:11:15 K8Console kernel: [  379.368846] ftdi_sio ttyUSB4: FTDI USB Serial Device converter now disconnected from ttyUSB4
Nov 14 22:11:15 K8Console kernel: [  379.368990] ftdi_sio 1-1.2:1.3: device disconnected
Nov 14 22:12:09 K8Console kernel: [  432.915225] usb 1-1.2: new high-speed USB device number 13 using dwc_otg
Nov 14 22:12:09 K8Console kernel: [  433.046149] usb 1-1.2: New USB device found, idVendor=0403, idProduct=6011, bcdDevice= 8.00
Nov 14 22:12:09 K8Console kernel: [  433.046171] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Nov 14 22:12:09 K8Console kernel: [  433.046186] usb 1-1.2: Product: Quad RS232-HS
Nov 14 22:12:09 K8Console kernel: [  433.046200] usb 1-1.2: Manufacturer: FTDI

I have replaced the quad port adapter with 4 CP2102 on a 4-port USB hub.   Now the consoles are stable.  Every time the device reset, I lost 4 consoles.

I'm working on some ideas for the hardware as well as the software stack. 

Hardware -

1. LiPo backup for two of the core nodes.   I'm considering implementing etcd and two nodes on battery should allow some resilience.

2.  Adding intake fan seizure detection.   The fans I selected to provide direct cooling have a third wire that allows detecting a problem on the management Pi and have open-collector output that can interface directly to the GPIO pins.

3.  Stickers.  For the un-painted box.  I have nginx, Log-dna (even tho that didn't go well )  ... but must obtain others.  A Postgres sticker would probably secure its place in the stack.

Software -

1. Fluent-bit for logging.  I've already deployed the daemonset - but my elasticsearch server is down ( sad face ).   There is a presentation at KubeCon tomorrow on a Fluent-bit/PostgreSQL stack that sounds interesting.  I've recently used the Crunchy Postgres operator and I'd really like to run it in this cluster.   Like Sybok,  you can let an operator take away the database pain.  Elasticsearch is (in practice) something you endure more than use.