Maelic / RelateAnything

# RelateAnything
**Real-time open-vocabulary relation prediction from any inputs.**
Paper ·
Project page ·
Models ·
Training data ·
Documentation
Give RelateAnything **an image, object regions, and the relations you want to
look for**. It returns scored `(subject, relation, object)` triplets, such as
`person → riding → horse`.
- **Use your own regions.** Boxes can come from a detector, a segmenter, or a
 person. Masks are optional.
- **Choose the relation vocabulary.** Supply phrases at inference time without
 retraining. A small text encoder embeds them once; no language model runs
 per frame.
- **Get one graph or two.** Return a single ranked list, or separate spatial
 and semantic graphs from the same forward pass.
https://github.com/user-attachments/assets/76544f31-53ac-407b-9445-157e83946938
 The same relation model across five scenes, with tracked regions and relations. Clip credits · Reproduce the video 
**Try the browser demo** —
no installation; inference runs on your device using WebGPU or WASM.
Quickstart · Models · Deployment ·
Performance · Results · How it works
## Quickstart
Use **Python 3.12+**. Install a CUDA-enabled PyTorch build for NVIDIA GPU
inference, or use `device="cpu"` in the example below.
 [code] 
Run this from the repository root. A detector finds the objects in the
included sample photo; RelateAnything predicts the relations between them.
 [code] 
The relation model reads the image and the boxes; the detector's labels are
never an input. Boxes can come from any detector, a segmenter (pass `masks=` as
well) or a person: an `[N, 4]` array in original-image pixels is all it needs.
Released checkpoints include the backbone configuration and text encoder;
**no gated DINOv3 login is needed for inference**. The `detector` extra
installs ultralytics, which is
AGPL-3.0 and downloads the YOLO26 weights on first use.
API guide: inputs, vocabularies, masks, and scores ·
Installation and offline use
## Models
Start with **`relsgg-vits16plus`**, the recommended balance of quality and speed.
For more accuracy, use **`relsgg-convnext-base`**: within 0.3 OVS of
ConvNeXt-L at half its parameters. All seven checkpoints
share the training recipe and differ in their vision backbone; the ConvNeXt
towers also train their backbone at a lower learning rate.
| Checkpoint | Backbone | Parameters |
|---|---|---:|
| `relsgg-vits16` | DINOv3 ViT-S/16 | 46.1 M |
| **`relsgg-vits16plus`** | **DINOv3 ViT-S/16+** | **53.2 M** |
| `relsgg-vitb16` | DINOv3 ViT-B/16 | 113.8 M |
| `relsgg-convnext-tiny` | DINOv3 ConvNeXt-T | 58.6 M |
| `relsgg-convnext-small` | DINOv3 ConvNeXt-S | 80.2 M |
| `relsgg-convnext-base` | DINOv3 ConvNeXt-B | 123.5 M |
| `relsgg-convnext-large` | DINOv3 ConvNeXt-L | 245.7 M |
 Accuracy (OVS-F1, four benchmark axes) against latency of the relation model on a laptop RTX 3080 with TensorRT. The shaded line is the Pareto frontier. 
Compare model quality and speed.
Each model card includes evaluation results and calibration details.