LARK-AI-Lab / SpecAgent

# SpecAgent
**Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents**
Zilin Wang, Yinhong Liu, Chao Chen, Qun Liu, Zhijiang Guo
A small **Weak** model drafts the agent's trajectory and a large **Strong** model reviews it.
The Strong model stops writing every step: it reads a drafted segment in one prefill pass,
keeps the longest valid prefix, and repairs only the first rejected step. Reads run while the
Weak model drafts, so every step it proposes is grounded in real observations; the first
state-changing action is held until the Strong model has reviewed it.
## How it works
| | Who | What happens |
| --- | --- | --- |
| **1. Draft** | Weak | Drafts up to K steps. Read-only tool calls execute immediately. The first state-changing action (or the final answer, where it is the graded deliverable) is held, not executed. A chunk of K reads commits without review. |
| **2. Verify** | Strong, thinking off | Reviews the chunk once and returns `{reason, accepted_steps, replacement_action}`: the longest semantically valid prefix, why the next step is wrong, and optionally a replacement call. |
| **3. Repair** | Strong, thinking on | A valid replacement executes directly. Otherwise the Strong model takes one repair turn from the committed history and the reason. The repair commits without a second review; the rejected suffix is discarded. |
## Results
Qwen3.5-4B as the Weak model and Qwen3.5-397B-A17B as the Strong model, both served on H20 GPUs.
Resources are the whole system's (both models), relative to Strong-only.
| Benchmark | Weak-only | Strong-only | SpecAgent | HBM-read | Wall time |
| --- | ---: | ---: | ---: | ---: | ---: |
| τ²-Bench Airline | 80.0 | 86.0 | **86.0** | 0.44× | 0.41× |
| τ²-Bench Retail | 80.4 | 88.6 | **87.7** | 0.55× | 0.54× |
| AutomationBench | 6.8 | 15.2 | **10.5** | 0.81× | 0.57× |
| VitaBench (Delivery) | 34.0 | 48.0 | **44.5** | 0.62× | 0.69× |
Pass@1 in %. On MCP-Atlas, with the Weak model on a local RTX 5090 and three cloud Strong models,
SpecAgent closes 53–75% of the Weak–Strong gap while the Strong model decodes 36–50% of its
Strong-only tokens. Estimated FLOPs rise (by 30–87%): the saving is memory traffic and time, not
arithmetic. Baselines (AgentCollab, Speculative Actions) and the full cost tables are in the paper.
## Quick start
 [code] 
Point any agent that speaks OpenAI chat completions at `http://127.0.0.1:18081/v1`. Each client
session gets its own gateway state, and `--telemetry` writes one line per request with the
decision, the verdict, and every model call with its token usage. Python 3.10 or newer; model
serving is out of scope (the paper serves the Weak model with SGLang).
## Benchmarks
Every runner defaults to the paper's setting; each directory has a README with setup and run
commands.
| Benchmark | Directory | Held for review |
| --- | --- | --- |
| τ²-Bench Airline, Retail | `benchmarks/tau2` | writes, transfer to a human |
| AutomationBench | `benchmarks/automationbench` | wr