ifm-ai / RL360

# RL360
Reinforcement Learning codebase used in the post-training pipeline for the K2 Horizon series.
It trains language models on tasks that involve writing code, using tools, and working in a sandbox. The example below uses our K2 Horizon 7B model. To see how to switch models, see the Qwen example.
## K2 Horizon 7B: overfitting 32 coding tasks
As a working small-scale demo, we trained the K2 Horizon 7B model to overfit 32 coding tasks, sampling eight attempts per task. Mean reward rose from **0.373 to 0.901** over 99 rollout iterations. See the public W&B workspace for the run and charts, and recipes/README.md for the settings used.
Left to right: mean reward across all attempts, step time in seconds, and training step. Step time includes training and waiting.
These are training-set results. Note that Reward includes a format penalty, so it differs from the task success rate. The job hit its 72-hour Slurm limit after 99 of the planned 100 iterations, which is why W&B shows it as `crashed`.
## Repository
The bundled components (miles, megatron, sglang etc) are checked in as ordinary directories.
| Path | Purpose |
| --- | --- |
| `recipes/` | Configurable Slurm recipe for overfitting on 32 mixed coding tasks |
| `docker/` | Public container build and reference dependency versions |
| `src/agent360/harbor/miles/` | Agent execution, Harbor service, reward logging, calibration, and runtime checks |
| `src/agent360/search/` | Retrieval index construction and serving |
| `src/miles360/reward/` | Task-specific reward functions and grading utilities |
| `src/miles360/token_mean_reducer.py` | Token-level loss reduction |
| `src/merge360/` | Weight merging of RL expert checkpoints (model soup, task arithmetic, TIES, RAM, ULP-gate, Gram-soup, ISO) |
| `components/miles/` | Rollout orchestration and RL training entry points |
| `components/megatron-lm/` | Distributed model training |
| `components/sglang/` | Inference runtime and kernels |
| `components/smg/` | Model gateway and routing |
| `components/harbor/` | Sandboxes, agents, task execution, and verification |
## Architecture
Miles coordinates rollouts and training on Ray. It sends task prompts to RL360 and Harbor agent workers, which run tools and verifiers through a sandboxing service. Agents make LLM calls through session servers that record tokens and log-probabilities and return trajectories to Miles. SMG routes those requests to SGLang inference engines. Megatron-LM trains on batches from Miles and synchronizes the updated policy weights to SGLang.
Expert checkpoints trained this way can be combined into one model with `merge360`.
## Run the coding recipe
You will need a Linux x86_64 cluster with NVIDIA H100/H200 GPUs, Slurm, Docker Engine, the NVIDIA Container Toolkit, and shared storage. The container provides Python 3.12 and the training dependencies. The default allocation is 33 nodes with eight GPUs each: one service node, eight training nodes, and 24 rollout nodes. The service node does n