TianyuCodings / JevHarness

# JevHarness
**English | 简体中文**
**Let an LLM write a task-specific harness for Jev. Run it, inspect its decisions, and optionally improve it using rewards and complete execution traces.**
> **New paper: System1-Arena: Which Decisions Can Be Left to System 1?** — The most comprehensive benchmark for Jev-like models: nearly 100K questions, including classification and agentic tasks. Leaderboard · Code · Dataset
A harness turns task observations into useful features, constructs Jev questions and criteria, and combines the structured answers into actions. The authoring LLM can change the code, questions, graph, and memory. Once the harness is fixed, execution uses that code and its Jev calls; it does not need the authoring LLM on every decision.
**Reason deeply during development. Freeze the strategy. Let Jev make fast, fuzzy decisions.**
A strong LLM brings broad intelligence and deep reasoning, but generating that reasoning for every action adds latency and cost. Jev provides fast, lightweight judgment with more limited capacity for open-ended reasoning. JevHarness combines these strengths: let the LLM write a task-specific harness, then optionally improve it using rewards and complete execution traces.
The harness preserves the LLM's reasoning strategy in explicit code, features, state, instructions, criteria, and control flow. Code computes useful facts; Jev makes **fuzzy, context-sensitive decisions** from those facts. The authoring LLM can revise the harness and its memory during development. Once the selected harness is frozen, its code and Jev calls execute without the authoring LLM on every decision.
**Pokémon result: after 5 reflection rounds, the selected harness improved Eval win rate from 25% (3/12) to 75% (9/12).** The search retained the round-3 candidate as its best harness; Eval was used for selection.
Open the interactive demo · How it works · Latency · Install and use the skill
**Initial vs. evolved:** the two harnesses play the same evaluation scenario. The initial harness loses; the selected harness wins. **Edited highlights: turn-2 decisions, then each battle’s ending.** Watch the MP4 or open the full battle archive.
## Install in Claude Code
Run these commands inside Claude Code:
 [code] 
See skill usage, Codex installation, and manual options.
## Start with the example
The Pokémon example includes an actual archived run, the selected harness, a browsable evolution tree, complete decision records, and replay highlights synchronized with their Jev answers. Browsing the archive makes no model requests and starts no games.
 [code] 
Open localhost:8768. The viewer needs Node.js and a modern browser; battle animations download assets from the official Pokémon Showdown renderer. The archived battle log stays in the browser and is not uploaded to a replay server. See website setup for deployment and archive verification.
The reported improvement is an example result on the selection Eval set, not an independent estimate of performance