AetherLabsAI / Video2World

# Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
**Jinzhou Tang** 1,2* , **Zijun Zhang** 1* , Jing Yang 1 , Yuchen Yan 1 , Kun Zhou 1† ,
Lingjun Mao 1 , Ruobing Han 1 , Jinglin Cao 1 , Wenpeng Xu 1 , Lukun He 1 ,
Minghao Fu 1,2 , Fan Feng 1 , Biwei Huang 1,2 
 1 Aether AI    2 University of California, San Diego 
 * Equal contribution    † Corresponding author and project lead
Building interactive simulators from real-world observations is a promising way to scale embodied data, but current
pipelines still rely heavily on manual environment construction and calibration. **Video2World** asks whether frontier
foundation models and coding agents can automate this process end to end. We formulate *autonomous video-to-simulation*
as a software engineering task: an agent observes an embodied video, constructs the corresponding simulated
environment and robot behavior, and iteratively refines the result through execution feedback.
Video2World comprises **222 reconstruction instances** in **39 task families**, built from **189 video clips** of robot
executions, egocentric human recordings and third-person human demonstrations. Reconstructed worlds are measured along
**geometric fidelity**, **dynamic fidelity** and **functional correctness**, capturing spatial perception, physical
reasoning and executable interaction.
## Overview
Each instance pairs a source video with a target configuration (simulator, robot embodiment, control interface). The
coding agent works in an isolated sandbox with the video, simulator APIs, documentation, assets and general-purpose
coding tools; it receives no scene model, object trajectory, robot state or task specification. The submitted scene and
robot behavior are executed by the evaluator from a fresh initialization, and every change of object state must arise
from simulator dynamics. The rollout is compared with hidden, object-centric annotations:
- **Geometry**: interaction-weighted scene Chamfer distance, object shape Chamfer distance and size error;
- **Dynamics**: translation APE, rotation APE and translation RPE of the object motion produced by the submitted robot
 behavior, against the demonstrated trajectory;
- **Functionality**: task success and stage-wise task progress, judged on object-centric task requirements.
**V2WScore** (0–100) combines build, functionality, geometry and dynamics per instance, macro-averaged over task
families. Every instance also has a **human-assisted reference (HAR)** built with source annotations, tracking, scene
modeling and manual refinement, scored with exactly the same code.
## Installation
Scoring needs only the base package. Evaluation uses three simulator environments, pinned to the versions the
reference results were produced with:
 [code] 
`v2w setup` stores these locations in `v2w.local.json`; `--graphics-libs DIR` adds a library directory for headless
Isaac rendering, and Isaac Sim needs GPUs with a working Vulkan device. ManiSkill