Short answer: yes, by working through the same interface a human uses, but that trades a structured integration problem for a state-interpretation problem. Worth breaking down for anyone building device-control or automation projects.

The setup

An API gives you defined actions, structured data, predictable errors. When it's missing, incomplete, or doesn't cover the task, an agent (or a hardware device like Aiden, which drives phones and computers over USB HID) has to work with what's actually rendered: a page's DOM, an accessibility tree, a desktop window's controls, or in the least structured case, raw screen pixels read via OCR.

Six methods, most-to-least structured:

• Browser automation (DOM + browser protocol, e.g. WebDriver, Playwright), strongest for stable web apps
• Accessibility-tree interaction (roles/labels/values), strong when the app implements accessibility properly
• Keyboard/pointer/touch input, needed for cross-app and real-device tasks
• RPA (rules + selectors + OCR + files), good for narrow repeatable legacy workflows
• Screen + OCR, the fallback, works almost anywhere, least deterministic
• Hybrid API+UI, approved API for some steps, UI for the gaps the API doesn't cover

Rule of thumb: use the most structured method the task actually permits. Don't default to screen+OCR just because it reaches everything, reach for it because nothing more structured is available.

The loop that actually matters

Observe state → pick one bounded action → perform it → verify the result → pause/recover if the evidence doesn't check out. A UI can change layout, load slowly, or throw up an unexpected modal without warning, none of that looks like a clean API error, it just looks like the interface not matching what you expected. The system has to notice that gap itself.

Two hard lines regardless of method

• CAPTCHAs/MFA prompts get a pause and human handoff, never an attempt to defeat them
• Anything consequential (sending a message, submitting a form, moving money) needs a human confirmation step before it executes

What you actually need to instrument

Screen-state capture per step, action traces, replayable sessions, explicit post-action verification (not an assumption of success), and failure classification that distinguishes "session expired" from "layout changed" from "genuinely blocked."

We build this into Aiden at the hardware level, HDMI capture in, USB HID out, verification and human-confirmation gates built into the action loop itself. Full writeup, with the standards references (WebDriver, WAI-ARIA, Microsoft UI Automation, Android UI Automator, Apple XCTest) and worked comparison table: https://aidenai.io/blog/can-ai-agents-use-apps-that-have-no-api

Repo: https://github.com/AidenAI-IO/aiden-firmware

Anyone here doing screen-level automation on constrained/legacy hardware, curious what verification approach you landed on.