PolyU-VCLab / OmniCompass

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding 
 English | 中文 
Xuyang Liu 1,2 ,
Shihao Wang 1 ,
Xinyu Wei 1,2 ,
Minghan Li 3 ,
Zeqing Wang 1,2 ,
Zuhao Yang 4 ,
Zichen Wen 5 ,
Lei Zhang 1,2 
 1 The Hong Kong Polytechnic University,
 2 OPPO Research Institute
 3 Colorado School of Mines,
 4 Nanyang Technological University
 5 Shanghai Jiao Tong University
OmniCompass is a holistic benchmark for omni-modal video grounding and understanding, comprising **820 videos** and built with approximately **1,800 human annotation hours**. It covers **2 target specifications** (Text and Text+Image), **2 complementary evaluation tracks** (OmniCompass-VL and OmniCompass-AV), **3 core tasks** (temporal grounding, spatial grounding, and question answering), and **3 video duration categories** (short, medium, and long). We evaluate **50 MLLMs** on both task performance and cross-specification consistency.
## 📝 Content
- News
- Overview
- Evaluation Pipeline
- Experiments Results
- Citation
- Contact
## 🔥 News
* **`2026.10.17`** ✅✅ We have integrated OmniCompass into LMMs-Eval! You can now evaluate your models on our benchmark by following the instructions here.
* **`2026.10.06`** 🤗🤗 We release OmniCompass, a holistic benchmark for omni-modal video grounding and understanding.
## ✨ Overview
> **TLDR:** OmniCompass spans four dimensions: target specification, video evidence,
> task coverage, and video duration. Paired Text/Text+Image specifications are
> evaluated on the same targets. OmniCompass-VL contains 820 videos and 2,460
> instances, while the 240-video OmniCompass-AV subset contains 720 instances
> with native audio. Both tracks cover temporal grounding, spatial grounding,
> and question answering.
OmniCompass-VL evaluates general VLMs, while OmniCompass-AV evaluates
OmniLLMs with native audio-video understanding capabilities.
## 📐 Evaluation Pipeline
As for Coding Agent users, we recommend first installing the bundled
omnicompass-eval skill
for evaluation with tools such as Codex or Claude Code.
Without installation, ask your agent to read
`skills/omnicompass-eval/SKILL.md` in this checkout and follow it.
**Self-hosted VLM example:**
> Use the omnicompass-eval skill to evaluate Qwen3.5-9B on OmniCompass-VL
> for TG, SG, and QA, using 32 frames and both Text and Text+Image settings.
> Run the model locally. First check the environment and data, list the
> configuration and request count, and wait for my confirmation before starting.
**API model example:** replace the placeholders with your provider's values.
> Use the omnicompass-eval skill to evaluate ` ` through the API at
> ` ` on OmniCompass-AV for TG, SG, and QA, using 64 frames and both
> Text and Text+Image settings. Read the key from `OMNICOMPASS_API_KEY`.
> First check the data and API's image/audio support, list the configuration
> and request count, and wait for my confirmation before any paid requests.
Provide the data path when asked. You can change the model,