zqc3117 / Spotter

Spotter: Let the Embodied Model Lead, and the VLM Reflect for It 
 If our project helps you, please give us a star ⭐ on GitHub to support us. 
| 📄 Paper | 🏠 Project page | 🤖 Model checkpoints | 🌐 Simulator | 🚀 Get started |
|---|---|---|---|---|
| arXiv | zqc3117.github.io/Spotter | pi0.5 · Cosmos Policy (RoboCasa) · pi0.5 (RoboTwin 2.0) | RoboCasa · RoboTwin 2.0 | Installation · Quick start |
**Spotter** adds a lightweight intervention layer between a frozen embodied model (System 1) and a VLM (System 2).
Instead of having the VLM plan every step, the embodied model leads and keeps executing, while a local screener
watches in parallel and the VLM steps in only to confirm and repair an error. The VLM's job becomes much simpler,
so the whole framework works with a **locally served, open-source Qwen and no privileged information**.
## 🎬 Demos
 Real robot (Franka Research 3), stacking the blue cup onto the pink cup. Shown at 2×, with the judge's thinking time shortened; the on-screen clock is real time. Watch the full video on the project page . 
## 🔥 Highlights
- **A lightweight intervention layer.** Like a person who keeps a little attention on a task and focuses only when
 something looks wrong, Spotter lets execution flow continuously and brings in the VLM only on a confirmed error.
- **A simpler job for the VLM.** The VLM no longer drives the robot. It only has to notice a failure, repair it with
 a short primitive plan, and check the fix. A local Qwen with no access to the task's success signal is enough.
- **Continuous execution, low overhead.** The embodied model never waits for the VLM on the routine path:
 a successful episode takes only **13–16 s** longer than the policy alone.
## 💡 Why Spotter
Existing VLM-led robot agents are serial: the VLM plans a segment, the embodied model executes it, and the VLM
decides what comes next. This forces a dilemma:
- **Short segments** mean frequent VLM calls and high latency, even on easy tasks.
- **Long segments** mean a deviation during execution cannot be caught in time, and later actions keep going wrong.
Spotter removes this trade-off by supervising in parallel rather than in series.
| | VLM-led agents | Spotter |
|---|---|---|
| Who drives execution | VLM plans every step | Embodied model leads |
| When the VLM is called | Before every step | Only on a confirmed error |
| Embodied model waits for the VLM | Every step | Only during a repair |
| Needs privileged information | Usually (e.g. the success signal) | No |
| Works with a local Qwen | Only with privileged information | Yes |
### 🎯 Works with a weaker model, without privileged information
With the same local Qwen and no privileged information on RoboCasa (1,200 episodes), a representative VLM-led agent
falls **below the policy alone** (Cosmos Policy 67.9 → 65.2, π0.5 64.3 → 60.5), while Spotter still improves it
(→ 71.7 and 68.4).
### ⚡ Almost no overhead when nothing goes wrong
With Qwen, the judge is called **less than once** per successf