mose3-tracker / MoSE3

MoSE3: Learning World-Space SE(3) at Every Pixel 
 NeurIPS 2026 (Spotlight) 
Jiahuan Cheng\*, Zhiyi Li\*, Tian Xia\*, Ruojin Cai, Yilun Du, Qianqian Wang
\* Equal contribution
## Updates
- **[Oct 2026]** Art-Kubric generation code and dataset released.
- **[Oct 2026]** Inference code and model weights released.
## Overview
**MoSE3** is a feed-forward model that predicts world-space **SE(3) motion for every pixel** of a
monocular RGB video, generalizing across **rigid, articulated and deformable objects**.
## Quick Start
### Installation
 [code] 
No custom CUDA extension is needed. `pip install -e .` makes `mose3` importable from anywhere.
### Model weights
Download the weights from Hugging Face into `ckpt/`:
 [code] 
### Run
 [code] 
- Inputs: `--video`, `--image_dir` (frames ordered by the numbers in their file names), or `--json`
 with a list of clips (same keys as the flags; see `examples/demo.json`).
- SE(3) fit: `--num_ref_pts` (default 12000) is the number of query-frame pixels each fit pools over.
 For locally deforming surfaces such as cloth, add `--se3_dist_weight` to also weight that pool by
 pixel distance (`--se3_sigma_px`, default 8 px). The `demo.json` clips use it.
- `python inference.py --help` lists every option. The keys of `predictions.npz` are documented in
 `run_clip` in `inference.py`.
### In Python
 [code] 
## Visualization
Both tools read only `predictions.npz` and lay a grid of query points over the query frame
(`--stride`, default 48 px).
`visualize.py` writes, each video as `input | visualization`:
- `query_points.png`: the grid on the query frame
- `tracks.mp4`: the predicted 3D tracks with a fading trail (hollow dot = occluded)
- `se3.mp4`: each point's SE(3) motion as a moving x/y/z frame, placed at the query-frame point moved
 by its SE(3); at the query frame blue z is the outward surface normal, and frames of occluded points
 are dashed
- `rigidity.mp4`: rigidity embedding, PCA → RGB
`visualize_viser.py` shows the dense tracks as a moving 3D point cloud, coloured by RGB or by the
rigidity embedding, with the same SE(3) frames and the trails of the tracks. Point size, axis length
and width, and trail lengths are sliders in the GUI.
`--help` lists the options of both tools.
## Checklist
- [x] Inference code and model weights
- [x] Art-Kubric dataset repository and generation pipeline
- [ ] Evaluation code
- [ ] Training code
## Acknowledgements
MoSE3 uses π³ as its frozen geometry branch, and this repository
includes π³'s model code and weights. It also contains code from
DINOv2, CroCo,
DUSt3R and marepo.
The model is trained on Art-Kubric, Kubric, PointOdyssey, Dynamic Replica, SynthVerse and Syn4D. The example
videos are from Pexels and the DyCheck
iPhone dataset. We thank the authors of all of these.
## Citation
 [code] 
## License
The code is released under the BSD 3-Clause license, except for a few third-party files that keep
their own licenses (some non-commercial), listed in LICENSE. The model weights include