Our remote stations update over the air with an A/B scheme: two root-filesystem slots, OTA writes the inactive one, the box boots into it on trial, commits if it comes up healthy — and if it doesn't, a bootcountcounter trips and the bootloader rolls back to the known-good slot. For a station you can't physically reach, that unattended rollback is the whole point.

Except a recent OTA didn't roll back cleanly — the trial slot just hung at boot. On the Proxmox VNC console: a blinking cursor and nothing else. No log, no panic, no clue. It sat there dead until bootcount finally rolled it back.

Two things were wrong.

First, it was invisible. The kernel command line only had console=ttyS0 — serial. On a graphical console (VNC), the kernel prints nothing; it was actually panicking the whole time, we just couldn't see it. One-line fix: add console=tty0 in front of the serial console so printk and panics render on screen too (serial stays primary, so the headless -nographic workflow is unchanged). Suddenly the "dead box" was a visible panic — and the real bug was obvious.

Second, the real bug: OTA only rewrote the rootfs, not the kernel. The kernel lived on a separate, static partition — the EFI System Partition on x86 (GRUB), a FAT boot partition on the Raspberry Pi (u-boot) — written once at flash time and never touched by OTA. It was also unpinned, so every rebuild could float its version. So an OTA delivered a new rootfs whose /lib/modules/<version> and ext4 format no longer matched the frozen bootloader-side kernel. The trial slot literally could not boot its own userland.

The fix is structural, not a patch: put the kernel (its modules, and the DTB on the Pi) inside every rootfs, and have the bootloader load the kernel from the active slot's rootfs. Because OTA already writes the whole rootfs partition as one atomic block, the kernel now updates atomically with its modules — one artifact, one write. A kernel/module mismatch is no longer something you avoid by discipline; it's impossible by construction. Bonus: A/B trial-and-rollback now covers the kernel too (a bad kernel is just a bad slot), and the EFI/boot partition becomes fully static — only the bootloader and its config live there.

We also made boot fail fast: no menu, zero timeout, and a reboot if the bootloader ever falls through — so any boot fault (bad kernel, unreadable filesystem, hang) trips the existing bootcount → rollback path automatically, no human in the loop. Same logic on both platforms: one boot logic, two bootloader dialects.

On the AI-dev side: the architecture (brainstorm → spec → implementation) came together fast with an AI assistant in the loop, and "the kernel lives in the rootfs" is clean on paper. But the last stretch was pure boot-time reality that no design foresees — GRUB scripting quirks (you can't || your way out of things there), a dangling kernel bbappend on the wrong platform, BBMASK scoping — the stuff that only surfaces when you actually build the image and watch it boot, or not. And the whole thing was gated on a human noticing that a blinking cursor was a hidden panic. AI for the shape; a person at the console for the truth.

Net result: OTA can now update the kernel, a bad update can't strand a slot, and a remote box recovers itself — the only kind of OTA you'd trust on a station you can't walk over to.