ComputerUseAgent / AutoCua

AutoCua 
 Automatic Computer Use Agent. Mac, Windows, Linux, real phone, simulator, any browser. A multi-agent harness engine. 
 [code] 
| | Windows | Linux | macOS |
|:--|:--:|:--:|:--:|
| **Computer use** — any desktop app, with real mouse and keyboard input | ✓ | ✓ | ✓ |
| **Browser skills** — Chrome, Firefox, Edge, Brave and Opera | ✓ | ✓ | ✓ |
| **Shell use** — a coding agent in a real terminal | ✓ | ✓ | ✓ |
| **Web use** — your own Chrome, driven through the AutoCua extension | ✓ | ✓ | ✓ |
| **Mobile use** — your iPhone, iPad or an iOS Simulator | — | — | ✓ |
 Windows 10 and 11 · Linux with GNOME · macOS on Apple Silicon or Intel 
**Full capabilities:** **agent_capabilities.md**
---
## Computer use
Any desktop app, driven with real mouse and keyboard input. Agent mode: computer use.
https://github.com/user-attachments/assets/a767c117-abad-46bf-8a8b-6ecf49be6288
## Web agent: fast mode
6.1 seconds for a flight search. Agent mode: web.
https://github.com/user-attachments/assets/8cc797d5-5556-40a9-bbfc-5d6aa8880fb8
## Web agent: quality mode
Under a minute for three tasks in one run: an iPhone 18 Pro into the Apple bag, a case into the
Amazon basket, and a Google Sheet of Meta's five-year financials. Agent mode: web.
https://github.com/user-attachments/assets/82bf6163-55c9-4cc4-95e7-9c6466f2a25d
## iOS agent: iPhone use
A paired iPhone, driven over the USB cable. Agent mode: mobile use, iOS.
https://github.com/user-attachments/assets/3c22eb45-d939-4742-9053-22ecc75b416f
---
## Pick a surface
One string decides what the agent drives. Everything else stays the same.
 [code] 
| `mode` | What it drives | Where it runs |
|---|---|---|
| `"computer use"` | The desktop. Clicks, typing, hotkeys, apps, AppleScript | macOS, Windows or Linux, auto detected |
| `"shell use"` | A coding agent, straight into a real terminal | macOS, Windows or Linux |
| `"web use"` | Your own Chrome through the AutoCua extension; headless runs on a debugging port | Any host with Chrome, installed for you when it is missing |
| `"mobile use, ios"` | An iOS Simulator by default, or your paired iPhone or iPad | macOS host |
| `"mobile use, android"` | Not available yet | |
On Linux, computer use needs a GNOME desktop (tested on Wayland), and computer use and shell use
both need a few distro packages that pip cannot install. See Linux, from pip.
---
## Why it holds up on real software
**No pixel guessing.** The model never sees a coordinate. It picks a numbered element from a
view that fuses the operating system's accessibility tree with OCR, and a number it was never
shown is rejected before anything moves.
**No debugging port on your desktop.** Input goes in at the operating system's own input
layer. Mouse events posted at `kCGHIDEventTap` on macOS, pywinauto's real input path on
Windows, and a kernel driver when Windows UAC blocks everything else. An ordinary
application sees the same events it would get from your hand on the mouse.
**No debugging port on your browser either