zfkarl / RSIPO

# RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution
RSIPO is a self-evolving agentic RL framework in which agentic skills and the policy co-evolve during training. It
builds on the hybrid of RLVR and on-policy self-distillation (OPSD) in SEED,
where the policy summarizes its own trajectories into skills and a skill-enhanced copy of the policy acts as the
teacher.
## 📰 News
- **[2026.10]** Our paper is available on arXiv [📄 arXiv].
- **[2026.10]** We released the code and the RSIPO checkpoints of Qwen2.5-7B-Instruct, Qwen3-1.7B and
 Qwen2.5-3B-Instruct on ALFWorld and search-based QA [🤗 Models].
## 🏗️ Overview
 Figure 1. Overview of the RSIPO framework. Replay gains measured by the execution check reward
on-policy skill generation and calibrate the student's trust in the skill-enhanced teacher. 
RSIPO adds an **execution check**: for every training task, the policy rewrites a skill from one of its rollouts, the
skill-enhanced teacher replays the task from the same initial state, and the replays are compared with control
rollouts of the student. The measured replay gains are used twice:
1. **On-policy skill evolution.** The gain, centered by the mean gain of the task type, rewards the skill the policy
 wrote, so skill generation is optimized on-policy together with task execution.
2. **Trust-calibrated self-distillation.** The gain, shrunk towards its task-type mean with empirical Bayes, sets the
 weight of the OPSD loss of the task, so the student follows the teacher only as far as the teacher helps.
No skill is used at test time.
## 📊 Main Results
RSIPO is compared with 13 baselines on ALFWorld and seven search-based QA datasets with three base models. It achieves
the best average success rate on ALFWorld with all three base models and the best average exact match on search-based
QA with Qwen2.5-7B-Instruct and Qwen3-1.7B; with Qwen2.5-3B-Instruct it ranks second on search-based QA, 0.4 points
below SEED.
| Method | Qwen2.5-7B ALFWorld | Qwen2.5-7B Search | Qwen3-1.7B ALFWorld | Qwen3-1.7B Search | Qwen2.5-3B ALFWorld | Qwen2.5-3B Search |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Vanilla | 12.5 | 33.9 | 12.5 | 24.8 | 21.9 | 31.7 |
| Skill-Prompt\* | 23.4 | 36.4 | 9.4 | 24.3 | 28.9 | 23.9 |
| OPSD | 32.8 | 6.2 | 14.1 | 5.8 | 28.1 | 0.0 |
| GRPO | 81.2 | 42.0 | 46.1 | 40.8 | 75.0 | 36.4 |
| Skill-GRPO | 69.5 | 40.3 | 21.1 | 40.4 | 60.2 | 34.1 |
| Skill-GRPO\* | 88.3 | 47.5 | 28.1 | 40.7 | 80.5 | 36.1 |
| GRPO+OPSD | 80.4 | 47.0 | 32.0 | 42.2 | 81.2 | 44.6 |
| Skill-SD | 85.1 | 47.8 | 52.3 | 40.8 | 73.4 | 44.1 |
| RLSD | 82.0 | 49.0 | 42.2 | 40.6 | 79.7 | 43.8 |
| SDAR | 85.9 | 49.0 | 53.9 | 41.9 | 84.4 | 43.4 |
| StepOPSD | 88.4 | 48.2 | 56.3 | 41.4 | 73.4 | 43.7 |
| OPID | 90.0 | 49.2 | 58.9 | 40.4 | 84.3 | 45.0 |
| SEED | 96.1 | 48.6 | 92.0 | 42.2 | 91.8 | **45.7** |
| **RSIPO** | **98.6** | **50.8** | **95.0** | **42.6** | **95.7** | 45.3 |
ALFWorld: success rate (%).