lycorp-jp / sim-use

# sim-use
Give AI agents the ability to observe and act on iOS Simulator and Android emulator / device screens.
**Observe** — turn any screen into a token-efficient outline an LLM can reason about:
 [code] 
**Act** — tap any element by its alias, no coordinates needed:
 [code] 
Plan, code, **verify**, ship — teach this CLI to your agent and close the last gap in the agentic mobile development loop. Let agents verify what they built so you can focus on what matters.
`sim-use` is a cross-platform CLI that drives Apple's Accessibility APIs, the iOS Simulator HID pipeline, and Android's AccessibilityService through a single command surface. It emits a compact, agent-friendly screen description (`ui`) and an alias-cached tap shortcut (`tap @N`) so an LLM loop can observe → act in a few hundred milliseconds per round trip.
- The observe → act loop
- Why sim-use
- Install
- Platforms
- Commands
- Physical iOS devices
- Architecture
- Viewer
- Contributing
- Licence
## The observe → act loop
Every interaction follows the same cycle — observe, act, verify:
 [code] 
Multiple selector styles for different needs:
| Selector | Example | Best for |
|---|---|---|
| `@N` alias | `tap @9` | Speed — cached from last `ui` |
| `# ` | `tap #settingsButton` | Stability — survives layout changes |
| `--label` | `tap --label "General"` | Scripted flows with `--wait-timeout` |
| `-x -y` / `--point` | `tap --point 100,200` | Last resort — no AX data |
Selectors and explicit coordinates work in any orientation — on iOS,
sim-use self-calibrates the current rotation on each command, so `-x/-y` /
`--point` take the numbers `ui` prints. `--coordinate-space native` gives
them in the device's native portrait points instead.
## Why sim-use
- **Token-efficient.** The outline representation is ~16x more compact than a raw JSON accessibility tree. An LLM can read and reason about an entire screen in a few hundred tokens.
- **Nothing hidden.** sim-use walks the full accessibility tree including WebViews, system overlays, and embedded content — no elements are silently skipped. When the frontmost app exposes an empty tree because a remote process owns the visible UI (a system document picker, for example), `ui` automatically retries with cross-process discovery and flags the recovered, flat hierarchy via the `advisory` envelope key.
- **AI-native.** Designed from day one for agent loops, not human testers. Alias-cached taps (`@N`), structured `--json` envelopes with actionable `hint` fields on errors, and a bundled agent skill (`sim-use init --client claude`) that teaches your AI client the full command surface.
- **Fast.** A per-device background daemon amortises init cost across calls. After the first command, each observe-act round trip completes in ~300 ms.
- **Cross-platform.** One command surface drives both iOS Simulator and Android emulator/device. Same verbs, same flags, same `--json` shape — write one agent loop that works on both.
## Install
### Homebrew (recommended)
 [cod