SakanaAI / mass

MASS 
 Recursive Self-Improvement through Multi-Agent Self-Supervision 
 📜 Paper  · 
 🧠 Overview  · 
 📊 Results  · 
 🚀 Quick start  · 
 🔁 Pipeline  · 
 🧪 Tasks & evaluation  · 
 📚 Documentation 
## 🧠 Overview
**MASS alternates between improving workflows and learning from them.**
RHI searches for multi-agent workflows while the model weights stay fixed.
Post-training uses trajectories from the selected workflows to update those
weights. The next cycle starts from the updated model.
 [code] 
We use the paper's notation $\mathcal{L}^{(k)}$ for the model after $k$ cycles.
The configuration files and output directories use `L0`, `L1`, and `L2` for
the same generations. Each cycle uses the current model as executor, optimizer,
and in-loop evaluator. The base model is Qwen3.6-27B, and each generation uses
qwen-code 0.20.0 as its coding-agent runtime.
This is a research implementation for running and extending the MASS pipeline.
The experiment support and
implementation notes describe the available experiments
and differences from the reported runs.
## 📊 Benchmark results
Results for the base model $\mathcal{L}^{(0)}$ and the models after one and two
MASS cycles. Task and trial counts are per model. ScienceAgentBench and delivery
rates are shown as percentages; other scores retain their reported scales.
| Benchmark | Tasks × trials | $\mathcal{L}^{(0)}$ | $\mathcal{L}^{(1)}$ | $\mathcal{L}^{(2)}$ |
|---|:---:|---:|---:|---:|
| ScienceAgentBench (%) | 102 × 3 | 28.8 ± 1.5 | 30.4 ± 2.0 | 31.7 ± 2.0 |
| MLR-Bench | 107 × 3 | 1.71 ± 0.08 | 1.77 ± 0.09 | 2.25 ± 0.06 |
| MLR-Bench papers delivered | 107 × 3 | 57% | 59% | 76% |
| AstaBench (E2E-Bench-Hard) | 40 × 3 | 0.053 ± 0.002 | 0.055 ± 0.004 | 0.067 ± 0.017 |
| AstaBench (E2E-Bench-Hard) reports delivered | 40 × 3 | 50% | 58% | 76% |
| DSBench | 74 × 3 | 0.473 ± 0.029 | 0.465 ± 0.011 | 0.482 ± 0.025 |
| Terminal-Bench 2.0 | 89 × 3 | 0.375 ± 0.022 | 0.353 ± 0.021 | 0.352 ± 0.017 |
| SWE-bench Verified | 500 × 3 | 0.626 ± 0.015 | 0.618 ± 0.019 | 0.611 ± 0.006 |
Terminal-Bench 2.0 (official time limits) and SWE-bench Verified report mean
pass rate ± sample standard deviation across three trials. Their task counts
are the numbers attempted per trial; scores use tasks with results for all
three models within each trial: 86/88/87 for Terminal-Bench and 499/499/496
for SWE-bench. Delivery rates are reported separately from the MLR-Bench
and AstaBench scores. The
AstaBench breakdown
includes trial scores, report counts, and scores among delivered reports. See the
benchmark resources for
official papers and code or data.
The packaged MLR-Bench configuration covers an earlier 12-task subset. The
107-task configuration and evaluation adapters for AstaBench (E2E-Bench-Hard),
DSBench, Terminal-Bench 2.0, and SWE-bench Verified are not included. See
experiment support for the available runners.
## ✨ What's included
- **Workflow search:** RHI, trajectory collection, and within-task ranking.
- **Post-training:** conversa