Tongyi-MAI / OpenGCUA
git clone https://github.com/Tongyi-MAI/OpenGCUA.git
v0.1.2 · 1d ago
An open-source stack for generalist computer-use agents, combining code execution, GUI interaction, and cross-device control.
—
stars, 24h
+334
stars, 7 days · #173
+334
stars, 30 days · #799
334
★ total · +334% this week
24
forks · 1 open issues
Star history starts 2026-10-20; the chart appears after the second daily snapshot.
LanguageHTML
LicenseApache-2.0
Created2026-10-15 (5d ago)
Last pushtoday
Watchers2
OwnerOrganization
Links
- GitHub: github.com/Tongyi-MAI/OpenGCUA
- Homepage: https://tongyi-mai.github.io/OpenGCUA/
- Releases · Issues · Hacker News mentions
README excerpt
### GPT-6 made computer use the frontier. We make it open. News · Results · Demos · **Quick start** · SDK · Open stack · How it works · Contributing · Contact **Computer use has reached a turning point.** GPT-6 Astra's demos offered an exciting vision of what computer-use agents could make possible, and the focus is expanding from GUI-only interaction to both visual and programmatic interfaces. The question has changed, from how well a model operates a GUI to how well a system completes work on a real computer by whatever means works. **We call such a system a generalist computer-use agent (GCUA).** It uses a computer in more ways than you do. It reads the accessibility tree and calls app APIs, runs shell commands, and operates the GUI when it needs to. With **code execution**, it writes GUI operations as code and combines them with all of these in one program, with loops and conditions. It can also share the computer with you, working in background windows without interrupting what you are doing. **Yet frontier systems remain largely closed.** We therefore introduce **Open Generalist CUA (OpenGCUA)**, an ongoing project to develop and open the full stack for GCUAs with the community, including model, system, trajectories, efficiency optimization, and evaluation. **What we open in this version:** - 🧠 **OSWorld integration example** with reference screenshot-action and code-action agents for integrating the OpenGCUA design. - 📜 **Evaluation trajectories** of the 8 frontier models, including GPT-6 Astra, Claude Opus 5, and Gemini 3.8 Flash. - 💻 **macOS real-device harness** (preliminary), a menu-bar app that runs the agent on your own Mac. - 🔌 **Cross-Device SDK** (preliminary), one programming interface for computers, browsers, and phones. 🗺️ See the detailed plan for our open stack. ## 🔥 News - **[2026-10]** 🚀 The OpenGCUA research preview is out, with the tech report, the OSWorld integration example, the macOS real-device harness, the Cross-Device SDK, and evaluation trajectories of 8 frontier models. ## 📊 Results We test the design on two fronts: OSWorld-v2 with eight frontier models, and real Macs with our real-device harness. - **Every model improves, including GPT-6 Astra, Claude Opus 5, and Gemini 3.8 Flash.** Over a GUI-only baseline with batched actions, the mean partial score rises from 45.3 to 64.9 (+19.6 points), and steps per task fall by 53% on average. GPT-6 Astra goes from 102 to 34 steps per task. - **Weaker models gain the most.** GLM-5.3-Flash gains 35.2 points and DeepSeek-V4.1-Flash 29.8. The gap between the strongest and the weakest model shrinks from 39.1 to 18.0 points. - **Code execution adds value beyond shell access.** With the same GUI and shell access, writing operations as code adds 6.1 points and cuts steps by 14.8% over a GUI+CLI agent (mean of three models). With our real-device harness held fixed, on four macOS tasks Qwen3.8-Max used 19.4 times the reasoning tokens of GPT-6 Astra and still completed fewer