Danielxu0208 / rsi_eval

# RSI Eval
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI)
methods across agent benchmarks. Bring your own improvement loop, choose your
tasks, and evaluate each candidate through one interface.
`candidate -> evaluate() -> task feedback -> your next candidate`
Quick Start · Installation ·
Usage Examples · Custom RSI Integration ·
Custom Benchmark Integration
## Why rsi-eval?
An RSI method needs to run candidates, collect feedback, and decide what to
change next. RSI Eval supplies the evaluation boundary so that your method can
use existing benchmarks without implementing their execution and result
handling from scratch.
- **Keep control of your method.** You choose candidate generation, memory,
 selection, stopping rules, and evolve/test tasks. Call `evaluate()` from your
 own loop; no optimizer base class is required.
- **Reuse benchmark protocols.** Select pinned tasks and run their graders.
 Raw rewards, task budgets, and benchmark-specific metrics retain their meaning.
- **Get usable feedback.** Read per-task scores, execution status, token usage,
 verifier output, and trajectory paths through a common result format.
- **Record and resume experiments.** Each evaluation snapshots the candidate,
 records task and source identities, and preserves results for explicit recovery.
The catalog contains **24 evaluation configurations and two labeled legacy
subsets**. It covers coding, reasoning, engineering, and domain workflows:
| Area | Bundled benchmark families | Guides |
| --- | --- | --- |
| Terminal and software engineering | Terminal-Bench 2.0/2.1, SWE-bench Verified, Multilingual and Pro, Aider Polyglot | Additional coding benchmarks |
| Code generation | LiveCodeBench | LiveCodeBench |
| Reasoning and mathematics | MMLU, GPQA, ARC-AGI, MGSM | Reasoning, MGSM |
| Reading comprehension | DROP | DROP |
| MCE tasks | FiNER, USPTO, Symptom2Disease, LawBench, AEGIS | MCE |
| Engineering | EngDesign-Open, Frontier-Eng | EngDesign, Frontier-Eng |
| Domain workflows | JobBench, Harvey LAB, APEX | JobBench, Harvey LAB, APEX |
Use `rsi-eval list` for exact versions and task counts, or the
configuration guide for prerequisites and method
objective requirements. Task selections belong to your experiment; the catalog
assigns no permanent evolve or test role.
Optional integrations provide ready-to-run **Meta-Harness, RRSI, AHE, and
HarnessX** workflows. The core SDK has no third-party runtime dependencies;
execution backends and method dependencies are installed when needed.
## Quick Start
Preview a one-task evaluation with Python 3.12 or newer:
 [code] 
The example selects Terminal-Bench 2.0's `extract-elf` task and the bundled Codex
candidate. `preview` resolves the task references, candidate hash, and run
settings without Docker, Harbor, a login, or model calls.
For an actual evaluation, complete the execution setup below, then follow
Run an evaluation.
## Installation
### Core and execution backend
The Quick Start insta