## Why build this at all

Most "AI agent on your phone" work assumes the agent lives inside the device:
installed as an app, granted accessibility permissions, or driven through ADB.
That assumption is convenient and it constrains everything. A huge number of real
apps expose no API and no clean accessibility tree, so an agent that depends on
either simply cannot act there.

I wanted the opposite constraint: an agent that needs nothing from the phone at
all. No install, no root, no developer mode, no vendor cooperation. If a device
can output video and accept USB input, it should be drivable.

That leads to a fairly old-fashioned answer. Watch the screen. Move the mouse.
Type the keys. Do exactly what a human does, at the same interface a human uses.

## The two-channel approach

Perception and action run on completely separate paths:

**Perception, HDMI to CSI.** The board captures the target's display through a
TC358743 bridge. To the phone this is just an external monitor. It has no idea
anything is reading it.

**Action, USB HID.** The board presents itself as a composite USB gadget with
keyboard and pointer. When the agent decides to tap or type, it writes to
`/dev/hidg0` and `/dev/hidg1`. The phone receives what looks like ordinary human
input.

Neither channel requires the OS to cooperate, and neither requires anything
installed on the target.

## Hardware

- **Luckfox Pico Zero**, Rockchip RV1106, running Buildroot Linux
- **TC358743** HDMI-to-CSI bridge for display capture
- **USB-C hub** for the connection to the target device
- Composite USB gadget: HID keyboard, HID pointer, USB ECM networking
- Powered entirely from the phone's own USB-C port, no separate supply

Current form factor is a pocket-sized development board with the hub and cabling
visible. The direction is credit-card sized and magnetically attached to the back
of a phone, but that is a future revision, not what exists today.

## Firmware and runtime

The agent runtime is Go. Four concerns run concurrently and none is allowed to
block the others:

- **Frame capture**, pulls the display feed and hands screenshots to the agent
- **Audio**, full-duplex with hardware VAD so voice works without a wake word
- **Agent loop**, screenshot to model to decision to concrete input events
- **HID output**, writes those events to the gadget device nodes

Each is its own goroutine coordinating over channels. Goroutine-per-subsystem has
held up well for something juggling capture, audio, inference and I/O in near
real time.

Lower-level hardware services (`frame_service` for HDMI capture, `audio_service`
for record and playback) are C++. Python is tooling and benchmarks only, not in
the runtime path.

Rough language split: Go 59.7%, C++ 16.5%, Python 14.3%, shell and C for the rest.

## No backend, on purpose

There is no Aiden server. Model, speech-to-text and text-to-speech endpoints are
whatever you configure: a hosted API, a local model, your own deployment. Nothing
is collected and nothing leaves the device except to the endpoint you chose.

This is not a privacy slogan, it is the only design that survives scrutiny. The
input to this system is a live video feed of someone's phone screen. The moment
you route that through a server you control, you have built both a target and a
liability. Keeping the loop on-device and self-hostable is what makes the whole
premise acceptable.

## What works

Voice command in, multi-step task out, on apps that expose nothing. The demo I
keep showing is deliberately mundane: say what you want, and it opens the app,
navigates, and acts, with no integration behind it.

Because it drives the UI rather than an API, it does not care whether the app
"supports" anything.

## What does not work yet

The honest part.

**Action verification.** The agent infers success by re-reading the screen. That
breaks the moment a loading state lies: a spinner that resolves to an error, an
optimistic UI that shows success before the network confirms, a screen that looks
finished but is not. Reliable...