YuyaoGe / doc2harness

Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents 
 🌐 Website •
 📄 Paper •
 ⚡ Quick start •
 🧪 Scenarios •
 🏆 Leaderboard •
 📝 Citation 
---
Doc2Harness measures how well a model can **build a working coding agent** —
the harness itself — rather than how well it writes a patch inside one.
- 🏗️ **Stage 1 — Build.** A build model, working through
 OpenCode in an isolated container,
 receives a development document and an interface-only skeleton and has to
 implement a complete agent harness.
- 🔬 **Stage 2 — Evaluate.** The harness is frozen, moved to a separate
 container, and driven end to end by a scripted **mock model**. Each scenario
 reads four observable channels — requests sent to the model, files on disk,
 exit code, command output — and **positive controls** rule out vacuous passes.
> [!NOTE]
> 📦 **This is a partial open-source release.** The full benchmark in the paper
> has **328 test cases**; this repository releases a subset of **94 scenarios**
> (53 scored test cases plus the 41 positive controls that gate them). The
> remaining test cases are held out. Numbers on the leaderboard below come from
> the full suite and cannot be reproduced from this subset alone.
## 🏆 Leaderboard
Pass rate of **19 frontier models** on the full 328-case suite, strongest on the
left. Each model built **three independent harnesses**; a bar is their mean.
The best model reaches **53.05%** and the weakest **27.44%** — even the
strongest still fails nearly half the cases. See the
project page for per-category breakdowns
and case studies.
## ⚡ Quick start
**Requirements:** Python 3.11+ and Docker. No third-party Python packages —
both stages are standard library only.
### 1️⃣ Score an existing harness (Stage 2 only)
No model key needed: scenarios drive the product against a mock gateway.
 [code] 
- Narrow the selection with `--family headless` or `--scenario-id `.
- Output: one JSON receipt per scenario plus `catalog_receipt.json`. The
 score is `pass / (pass + fail)` over applicable scenarios; a scenario that
 cannot be observed on a product is `INCONCLUSIVE`, not a failure.
- The full catalog takes about 25–90 seconds at concurrency 12.
### 2️⃣ Build a new harness (Stage 1), then score it
Needs the builder image and a local LiteLLM
gateway holding **your** API key — setup in
`src/stage1/README.md`.
 [code] 
### ✅ Sanity check
 [code] 
Loading the catalog also verifies its digests, so an edited or inconsistent
catalog fails here instead of silently scoring a different set.
## 🧪 Scenario catalog
| Family | Total | Scored | Controls | What it observes |
|---|--:|--:|--:|---|
| 📄 `file` | 17 | 9 | 8 | Read / edit / patch tools, including hostile encodings |
| 🔒 `permission` | 16 | 8 | 8 | Whether permission rules really constrain spawned processes |
| 💾 `state` | 14 | 7 | 7 | Slash commands and session state |
| 🤖 `headless` | 13 | 6 | 7 | Non-interactive runs and contract entry points |
| 🔌 `direct` | 11 | 6 | 5 | Wire-level model protocol a