Can AI Agents Use Apps That Have No API?

Short answer: yes, by working through the same interface a human uses, but that trades a structured integration problem for a state-interpretation problem. Worth breaking down for anyone building device-control or automation projects.

The setup

An API gives you defined actions, structured data, predictable errors. When it's missing, incomplete, or doesn't cover the task, an agent (or a hardware device like Aiden, which drives phones and computers over USB HID) has to work with what's actually rendered: a page's DOM, an accessibility tree, a desktop window's controls, or in the least structured case, raw screen pixels read via OCR.

Six methods, most-to-least structured:

• Browser automation (DOM + browser protocol, e.g. WebDriver, Playwright), strongest for stable web apps
• Accessibility-tree interaction (roles/labels/values), strong when the app implements accessibility properly
• Keyboard/pointer/touch input, needed for cross-app and real-device tasks
• RPA (rules + selectors + OCR + files), good for narrow repeatable legacy workflows
• Screen + OCR, the fallback, works almost anywhere, least deterministic
• Hybrid API+UI, approved API for some steps, UI for the gaps the API doesn't cover

Rule of thumb: use the most structured method the task actually permits. Don't default to screen+OCR just because it reaches everything, reach for it because nothing more structured is available.

The loop that actually matters

Observe state → pick one bounded action → perform it → verify the result → pause/recover if the evidence doesn't check out. A UI can change layout, load slowly, or throw up an unexpected modal without warning, none of that looks like a clean API error, it just looks like the interface not matching what you expected. The system has to notice that gap itself.

Two hard lines regardless of method

• CAPTCHAs/MFA prompts get a pause and human handoff, never an attempt to defeat them
• Anything consequential (sending a message, submitting a form, moving money) needs a human confirmation step before it executes

What you actually need to instrument

Screen-state capture per step, action traces, replayable sessions, explicit post-action verification (not an assumption of success), and failure classification that distinguishes "session expired" from "layout changed" from "genuinely blocked."

We build this into Aiden at the hardware level, HDMI capture in, USB HID out, verification and human-confirmation gates built into the action loop itself. Full writeup, with the standards references (WebDriver, WAI-ARIA, Microsoft UI Automation, Android UI Automator, Apple XCTest) and worked comparison table: https://aidenai.io/blog/can-ai-agents-use-apps-that-have-no-api

Repo: https://github.com/AidenAI-IO/aiden-firmware

Anyone here doing screen-level automation on constrained/legacy hardware, curious what verification approach you landed on.

Debugging: Why Command+V Stops Working with External Keyboards on iOS

Filed under debugging logs, since this turned out not to be our bug at all, an iOS routing behavior most people would never think to look for.

Symptom

Building Aiden (drives phones over USB HID, keyboard+mouse emulation, no app on the target device), we hit this: plain letters typed fine every time. Modifier-key shortcuts, Cmd+V, Cmd+A, Shift+X, intermittently didn't register. Our write to /dev/hidg0 returned success on every attempt. Nothing happened on screen.

First real finding

iOS logs distinguish plain keystrokes from modifier-based "key commands" and log the KeyboardFocus target each command resolves to.

Working:

cmd-v -> <keyboardFocus; pid: 9085; token: MobileNotes>

Broken, identical keypress:

cmd-v -> <keyboardFocus; pid: 3738; token: com.apple.springboard>

iOS recognized Cmd+V correctly both times. It just delivered the command to SpringBoard (the home screen process) instead of the actual foreground app. Not an HID-layer failure, a routing failure after the keypress was already understood correctly.

Ruled out (didn't fix it): Full Keyboard Access toggle, Boot Keyboard protocol declaration, LED output handling, keyboard descriptor/handshake sequence, keystroke timing, stale HID file descriptors after re-enumeration.

The A/B that isolated it

With AssistiveTouch left on throughout:

Critical pair: identical keyboard, identical settings, identical test method, keyboard-only vs. keyboard+mouse. Adding the mouse brought the failure back immediately. Removing it: 22/22 clean.

Conclusion: with AssistiveTouch enabled, if iOS detects Pointer/Mouse capability on the connected device at the same time as a keyboard, it can misroute modifier-key commands to SpringBoard. Reproduced on hardware with zero relation to our stack (a standard gaming keyboard, a Bluetooth keyboard), so this rules out an HID-compatibility explanation specific to us. It's a routing bug inside iOS itself, and without source access we can't say why Pointer/Mouse capability affects KeyboardFocus resolution, only that it reliably does.

Also tried substituting a Touchscreen/Digitizer HID class for Mouse/Pointer, keeping pointer: 0. Six Cmd+V attempts, all landed correctly. Supports the same conclusion, but any keyboard interface present still makes iOS hide the software keyboard, and a one-time Eject only recovers briefly, so not viable for our actual product.

Our workaround (not a fix)

Two USB HID profiles with different Product IDs/serials, so iOS doesn't reuse stale device state:

• Normal: keyboard + pointer
• Right before a modifier-key action: re-enumerate keyboard-only, send the keypress, restore pointer after

Switching scope covers a whole agent action, not a single keypress, to avoid the software keyboard popping/retracting repeatedly mid-action.

We're just not letting iOS see the trigger combination during the narrow window it matters.

Full writeup with the complete experiment table: https://aidenai.io/blog/debugging-an-ios-bug-why-command-v-stops-working-with-external-keyboards

Repo: https://github.com/AidenAI-IO/aiden-firmware

If you're driving iOS with an external keyboard + pointer combo and have seen a shortcut silently fail, this might be why.

Aiden Live Demo

Aiden Live Demo: one voice command, zero taps.

Aiden, a physical mobile AI agent, opens Spotify, finds the song, and starts playing. Hands-free, no app integration.
What's happening under the hood: Aiden plugs into the phone over USB, reads the screen via HDMI capture, and drives the UI through USB HID input (keyboard/pointer), the same way a person would tap and type. No API, no app install on the phone, no root. The agent loop runs on-device.

It's a small but honest example of the shift from AI that answers to AI that acts operating the apps you already use, on the phone you already have.

Open source (dev board): https://github.com/AidenAI-IO/aiden-firmware