Project Logs

Collapse

• A blinking cursor, a bricked slot, and making OTA unbrickable

  Peter Buchegger • 6 days ago • 0 comments

  Our remote stations update over the air with an A/B scheme: two root-filesystem slots, OTA writes the inactive one, the box boots into it on trial, commits if it comes up healthy — and if it doesn't, a bootcountcounter trips and the bootloader rolls back to the known-good slot. For a station you can't physically reach, that unattended rollback is the whole point.

  Except a recent OTA didn't roll back cleanly — the trial slot just hung at boot. On the Proxmox VNC console: a blinking cursor and nothing else. No log, no panic, no clue. It sat there dead until bootcount finally rolled it back.

  Two things were wrong.

  First, it was invisible. The kernel command line only had console=ttyS0 — serial. On a graphical console (VNC), the kernel prints nothing; it was actually panicking the whole time, we just couldn't see it. One-line fix: add console=tty0 in front of the serial console so printk and panics render on screen too (serial stays primary, so the headless -nographic workflow is unchanged). Suddenly the "dead box" was a visible panic — and the real bug was obvious.

  Second, the real bug: OTA only rewrote the rootfs, not the kernel. The kernel lived on a separate, static partition — the EFI System Partition on x86 (GRUB), a FAT boot partition on the Raspberry Pi (u-boot) — written once at flash time and never touched by OTA. It was also unpinned, so every rebuild could float its version. So an OTA delivered a new rootfs whose /lib/modules/<version> and ext4 format no longer matched the frozen bootloader-side kernel. The trial slot literally could not boot its own userland.

  The fix is structural, not a patch: put the kernel (its modules, and the DTB on the Pi) inside every rootfs, and have the bootloader load the kernel from the active slot's rootfs. Because OTA already writes the whole rootfs partition as one atomic block, the kernel now updates atomically with its modules — one artifact, one write. A kernel/module mismatch is no longer something you avoid by discipline; it's impossible by construction. Bonus: A/B trial-and-rollback now covers the kernel too (a bad kernel is just a bad slot), and the EFI/boot partition becomes fully static — only the bootloader and its config live there.

  We also made boot fail fast: no menu, zero timeout, and a reboot if the bootloader ever falls through — so any boot fault (bad kernel, unreadable filesystem, hang) trips the existing bootcount → rollback path automatically, no human in the loop. Same logic on both platforms: one boot logic, two bootloader dialects.

  On the AI-dev side: the architecture (brainstorm → spec → implementation) came together fast with an AI assistant in the loop, and "the kernel lives in the rootfs" is clean on paper. But the last stretch was pure boot-time reality that no design foresees — GRUB scripting quirks (you can't || your way out of things there), a dangling kernel bbappend on the wrong platform, BBMASK scoping — the stuff that only surfaces when you actually build the image and watch it boot, or not. And the whole thing was gated on a human noticing that a blinking cursor was a hidden panic. AI for the shape; a person at the console for the truth.

  Net result: OTA can now update the kernel, a bad update can't strand a slot, and a remote box recovers itself — the only kind of OTA you'd trust on a station you can't walk over to.

• A real release process for the firmware — and shipping the simulator as an artifact

  Peter Buchegger • 07/04/2026 at 18:07 • 0 comments

  Until now our Zephyr firmware repo had CI that built and tested, but produced no releases. That's fine until you need actual artifacts: the real STM32 binary to flash over DFU, and — less obvious — the native_sim binary, because the whole plan is to test the station virtually before touching hardware. So we built a proper release process from scratch.

  A few decisions worth writing down:

  One monorepo, many boards, artifacts named by board. All module firmware lives in one repo; the 2 m vs 70 cm variants are just a devicetree band property (SA818-V vs SA818-U). A small declarative release-targets.yaml lists what to build; the release job iterates it and emits board-named assets — fm-sa818-2m.bin, fm-sa818-70cm.bin, and so on. A new board later is one line in that file, not a pipeline rewrite.

  The version is 26.07.04-01. Date-based (year.month.day-sequence), one repo-wide version for all boards — the same idea our Linux image uses, but with a two-digit year. That last bit is deliberate: dropping the "20" makes the whole thing fit straight into Zephyr's native app/VERSION numeric fields — VERSION_MAJOR=26, MINOR=7, PATCH=4, TWEAK=01, all inside the 0–255 byte range. So the firmware reports its own build version at runtime through the generated app_version.h, and APP_VERSION_NUMBER stays machine-comparable. No custom version header, no string hacks. The date is the version.

  The simulator ships as a first-class artifact. Next to the real firmware, every release carries the native_sim build — the host-side x86-64 executable of the same firmware, statically linked so it runs in any target image without a glibc-version dance. That's what lets the Linux image pin a known, signed firmware-simulator and bring up a fully virtual station in CI. Everything is cosign-signed with a SHA256SUMS manifest so the consuming side can pin by hash.

  First release out: 26.07.04-01 — both bands, real + sim, signed.

  On the AI-dev side, since we build this fairly openly: the design (brainstorm → spec) and the CI plumbing came together fast and cleanly with an AI assistant in the loop — exactly the kind of pattern-heavy, well-trodden work where it shines. The honest bump: the first release run tripped over the boring real-world stuff — pushing a tag to a protected main and getting the artifacts attached correctly — and needed a follow-up fix commit. The design was right; the last 10 % was CI edge cases that only surface once you actually pull the trigger. AI for the shape, a human-in-the-loop pass for the plumbing reality.

  Next up: the Linux-image side consumes these pinned artifacts to bundle image + firmware as one tested unit.

• From STM32F302 to STM32U575 — porting the FM module and getting USB composite to enumerate

  Peter Buchegger • 07/01/2026 at 20:47 • 0 comments

  The FM transceiver module in our remote station uses an STM32U575 as a USB bridge between the Raspberry Pi CM4 and an SA818 radio. To the Pi it shows up as a composite USB device - UAC2 (audio in/out), CDC-ACM (control / AT commands), and DFU (firmware update). So from Linux the radio is just a USB sound card plus a serial port, which keeps the host side clean.

  One catch: our Zephyr board definition was still written for an STM32F302 (an earlier hardware revision). The new board is a U575 - different core (Cortex-M33), different clock tree, different USB IP. So this was a cross-family port, not a pin swap. The real work lived in the devicetree: rewriting the clock tree for the U5 (HSI48 as the USB 48 MHz source, PLL1 fed from MSI, 160 MHz core), moving to the USB OTG FS controller, switching the ADC to an async clock, and remapping pins.

  Then the honest part. CI went green - but a green build proves it compiles, not that the clocks lock or that anything enumerates. The only real test is silicon. So: stencil, solder paste, hotplate reflow on the fine-pitch (0.5 mm) U575, power it up, plug it in - and UAC2 + CDC both enumerate on the host. The historically painful "Zephyr USB composite on a U5" actually works. (DFU still untested - that's next.)

  And a classic hardware-humbling moment: the SA818 wouldn't answer over the serial link, and I burned time suspecting the firmware and the pin mapping - until I realized the SA818 needs its 12 V rail connected to wake up. Not a firmware bug. Just missing power.

  On the AI angle, since we build this fairly openly with an AI coding assistant: the devicetree/Kconfig port is exactly the kind of pattern-heavy, well-documented work where the AI moved fast and got the structure right. What it can't do is tell you the 12 V isn't plugged in, or that a green build is lying to you. That line - AI for the port, hands-on-hardware for the truth - is pretty much where this project keeps landing.

  Next up: connect 12 V and verify SA818 control, test DFU, then the actual point of all this - audio and PTT over the web.

• First status log: four modules verified at v1.0, FM transceiver off to fab

  Peter Buchegger • 06/11/2026 at 20:22 • 0 comments

  Quick first log to set the baseline for where the hardware stands.

  Four of the five modules are verified and now have clean v1.0 releases:

  • Bus board — verified, v1.0
  • Power board — verified, v1.0
  • CM4 carrier — verified, v1.0
  • Device tester — verified, v1.0

  The fifth, the FM transceiver, is the one still in motion — and the one with the best story. v0.2 (STM32F302) surfaced a few issues on the bench that warranted a respin:

  • USB D+ pull-up tied straight to VCC. The external D+ pull-up was first forgotten, then hardwired to VCC — so D+ sits permanently high and the device can’t signal a disconnect. That kills DFU updates, which require a forced re-enumeration by the host: the pull-up has to be controllable (via GPIO, or the MCU’s internal software-controlled pull-up) to trigger a soft-disconnect. Hardwired, it can’t.
  • USB composite device trouble under Zephyr.
  • A few GPIOs misrouted to the FM module.

  v1.0 moves to an STM32U575 and addresses these. It just went out to JLCPCB for fabrication, with the missing parts on a Mouser order; once it’s back and populated, it goes through the same bench validation as the other four.

  On the tooling side: versioning and releases across the module repos are now handled by a release-automation script — part of the AI-assisted workflow that runs most of this project’s software and infra. The hardware itself is still hand-designed in KiCad.

  Next: FM-TRX back from fab → assemble → populate → validate. I’ll log how the respin holds up.

View all 4 project logs