MirroS-Lab / AgentGarten

AgentGarten 
 Code Worlds for Evolving Agents 
 Code determines how the world changes, and the renderer learns how those changes should look. 
## Overview
Agents learn through interaction, and what they can learn is bounded by the
environment they practice in. That environment must be **faithful**, with
state, rules, and dynamics as consistent as those of the real world, and
**realistic**, with observations that follow the real world's visual
distribution.
**AgentGarten** builds real-time interactive environments that are both.
Simulators and game engines maintain persistent state and execute
program-defined rules, while a shared **neural renderer** generates the
agent's observations from the depth and surface normals each world exports.
Agents that practice in these environments improve from their own experience:
after each round they distill what happened into playbooks that later agents
inherit and build upon.
This is the official repository of AgentGarten. The neural renderer is
available now, with its training recipes and streaming inference; the code
worlds and the agent practice loop will be released here as well. The renderer
adapts Cosmos3-Nano to geometry
conditions in three stages:
- **Bidirectional** rectified-flow training of whole clips from a first frame.
- **Autoregressive** block-causal training, with teacher forcing or diffusion
 forcing, sampled with a KV cache.
- **Adversarial Forcing**, which distills the autoregressive model into a
 few-step student. The replay of each rollout is exact, so losses on later
 blocks update how the student encodes its history, and a real-data
 discriminator with exact R1/R2 regularization improves visual quality.
## News
- [2026/10/08] AgentGarten technical report and MirroS blog released.
- [2026/10/08] Training and inference code of the neural renderer released.
## Get started
### Installation
Python 3.11+ and PyTorch 2.9+ on a CUDA host.
 [code] 
### Weights
Training starts from Cosmos3-Nano
and encodes video with the
Wan 2.2 VAE:
 [code] 
Point `WM_COSMOS3_NANO` and `WM_WAN22_VAE` in `.env` at them.
### Inference
`Cosmos3Stream` runs the renderer block by block with a bounded KV cache.
Each block is denoised in a few steps and then committed to the cache, so the
conditions of the next block can depend on what was just generated:
 [code] 
`prepare_serving` serves the transformer with hand-written Triton kernels,
cuBLAS matrix products, and CUDA graphs, without `torch.compile`. Kernel JIT
and graph capture happen on first use, so warm the input shapes before
serving.
To render the validation clips of a checkpoint without training, add
`train.max_iterations=0 train.validate_at_start=true
trainer.callbacks.samples.every_n=1` to its training command. Comparison
videos (depth, normal, ground truth, sample) are written under `samples/`.
To decode RGB faster, attach the
TAEHV `taew2_2_super.pth` decoder with
one more override; encoding stays on the Wan encoder:
 [code] 
### Training
**Data.** A JSON