aim-uofa / Self-Distillation-AgentRL

# More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
**Self-Distillation-AgentRL** is the official codebase for our study of self-distillation in agentic reinforcement learning.
## Overview
Do more self-distillation methods lead to better agents? We compare ten representative approaches against GRPO across four benchmarks and three model sizes, using shared environments and evaluation procedures while retaining method-specific training settings. Beyond final task performance, we examine training progress, stability, and measured computational cost.
**Study overview.** From fragmented evidence to a common benchmark for assessing the breadth of gains, training stability, and computational cost.
Our evaluation yields three main findings:
- **Gains remain setting-dependent.** No method consistently outperforms GRPO across settings.
- **Strong final performance does not imply stable training.** Final scores can conceal instability along the training trajectory.
- **More computation does not consistently yield better agents.** Additional training cost does not reliably translate into improved performance.
This repository provides a common framework for training, checkpoint conversion, and evaluation, supporting direct comparisons and further research on reliable, cost-effective agent training.
## Benchmark Coverage
The study spans **12 model–task settings** and **132 training runs**, with repeated evaluations over **eight rollout seeds** for pretrained and trained models.
### Supported Models
- ✅ Qwen3.5-2B (`MODEL_SIZE=2B`)
- ✅ Qwen3.5-4B (`MODEL_SIZE=4B`)
- ✅ Qwen3.5-9B (`MODEL_SIZE=9B`)
### Supported Benchmarks
- ✅ ALFWorld (`TASK=alfworld`)
- ✅ ScienceWorld (`TASK=scienceworld`)
- ✅ Search-QA (`TASK=search`)
- ✅ WebShop (`TASK=webshop`)
### Supported Methods
- ✅ GRPO — baseline (`METHOD=grpo`)
- ✅ GRPO+OPSD (`METHOD=grpo_opsd`)
- ✅ SDAR (`METHOD=sdar`)
- ✅ Skill-SD (`METHOD=skillsd`)
- ✅ RLSD (`METHOD=rlsd`)
- ✅ OCSD (`METHOD=ocsd`)
- ✅ OVCSD (`METHOD=ovcsd`)
- ✅ AgentOPSD (`METHOD=agentopsd`)
- ✅ ADRS (`METHOD=adrs`)
- ✅ ICSD (`METHOD=icsd`)
- ✅ PCSD (`METHOD=pcsd`)
The launchers support training-time validation, standalone evaluation, rollout recording, benchmark mean/standard deviation, and pass@k.
## Getting Started
All entry points are in launchers/. The workflow below covers environment setup, data preparation, training, checkpoint conversion, and evaluation.
### 1. Environment Setup
Use Linux x86_64 with Conda, NVIDIA GPUs, and a compatible CUDA toolchain/C++ compiler. Run all commands in Bash from the repository root.
 [code] 
The installer creates the environment, installs the local packages and pinned dependencies, and applies the bundled decord/TextWorld compatibility wheels automatically. Dependency specifications are in requirements.txt, requirements-kernels.txt, and constraints.txt.
Choose an unused `CONDA_ENV`; use `REUSE_ENV=1` only to resume an interrupted installation. If Conda is n