Motor-Mind / MotorMind-Code

# MotorMind
**Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation**
Bingxuan Li\*, Siqi Song\*, Yizhuo Wu\*, Jiarui Yao, Tong Zhang, Huan Zhang
University of Illinois Urbana-Champaign
Paper · Project page 
MotorMind bridges VLM reasoning and robot control through **mid-level actions**, **continual
refinement** and **asynchronous scheduling**. It manipulates zero-shot, with no task-specific
training, both in simulation (Franka in LIBERO-PRO) and on a real xArm6. It uses no VLA as a tool,
no action experts, no coding agent, no motion planner or IK module, no SAM, and no skill API.
## How it works
One frozen VLM takes five roles, and a deterministic **controller** turns its proposals into
motion:
| Role | Does | Code |
|---|---|---|
| Planner | splits the task into subgoals, and mends the plan after a failure | `src/planner/planner.py` |
| Executor | proposes short sequences of mid-level actions for the current subgoal | `src/executor/` |
| Monitor | checks the scene while a motion runs, and can cancel pending commands at an action boundary | `src/planner/monitor.py` |
| Verifier | judges whether a subgoal actually happened | `src/planner/verify.py` |
| Memory | keeps a short note of what has happened, updated in the background | `src/memory/` |
- **Mid-level actions.** The VLM proposes parameterized moves, rotations and gripper actions (see
 the command language). The controller validates and executes them.
- **Continual refinement.** Fresh observations and measured feedback inform the next proposal.
 The outcome decides whether to advance, retry or replan.
- **Concurrent feedback.** The monitor and the memory run alongside execution, not between steps.
The same harness drives the simulated Franka and the real xArm6; only the adapter changes.
## Layout
| Path | What |
|---|---|
| `src/schema/` | the command language: five action types, and the direction words (`directions.yaml`) |
| `src/controller/` | command → `TcpDelta`s (`convert.py`), `plan()` / `run()` (`runner.py`), the `RobotAdapter` boundary (`types.py`) |
| `src/executor/` | the per-subgoal loop: propose actions, supervise them, locate objects from camera boxes and depth |
| `src/planner/` | mission planning, the scene monitor, and the verifier that judges each subgoal |
| `src/memory/` | the running memory note handed from one subgoal to the next |
| `src/prompts/` | prompts, as data |
| `vlms/` | the model client and the role → server table |
| `adapter/libero/` | the LIBERO backend (OSC_POSE servo loop) and the LIBERO-PRO suites |
| `adapter/xarm6/` | the xArm6 backend, its HTTP bridge client, and `mock_bridge.py` (the bridge with no robot) |
| `robot/` | the xArm6 model and limits (`robot.yaml`), per-rig measured profiles (`rigs/`), and `bridge_conformance.py` |
| `web/` | the operator page: manual commands, single subgoals and whole missions over live cameras, with STOP |
| `evals/` | mission runner, resumable sweeps, reports, and the subgoal suites |
| `tools