SeerRay-Lab / Xiaomi-Lens-0

Xiaomi-Lens-0 
 Pixel-level grounding and visual reasoning in a compact 4B multimodal model. 
 News · What it does · Method · Showcases · Results · Quick Start · Citation 
 English | 简体中文 
Xiaomi-Lens-0 is a 4B multimodal large language model that learns from pixel-level supervision through a shared visual backbone. Starting from Qwen3.5-4B, it decodes each generated ` ` token into a mask with a lightweight decoder, so the same model that answers in text can point at the regions it refers to, in a final answer or in the middle of its reasoning. It improves referring comprehension over Qwen3.5-4B by 5.6 points on average across eight RefCOCO-family splits and improves scores on all eight fine-grained perception benchmarks.
https://github.com/user-attachments/assets/d7ed080d-f815-4eb5-80b0-85273974f687
*98-second overview. Also available on the project page (full-resolution MP4).*
*Recorded outputs across the six capability domains: Referring Expression Comprehension, Referring Expression Segmentation, Multi-Instance Grounding, Reasoning Segmentation, Embodied Referring, and Fine-Grained Perception.*
## 📰 News
- The Xiaomi-Lens-0 online demo is now live on Hugging Face Spaces.
- The Xiaomi-Lens-0 model weights and the LensBench benchmark dataset are now publicly available on Hugging Face.
- The Xiaomi-Lens-0 Technical Report is now publicly available on arXiv.
## ✨ What it does
1. **Referring Expression Comprehension**: finds the object a phrase describes and returns its bounding box. First on seven of eight RefCOCO, RefCOCO+, and RefCOCOg splits, 5.6 points above its Qwen3.5-4B backbone on average.
2. **Referring Expression Segmentation**: returns a pixel mask for the referred object. 81.4% mean cIoU across the eight splits, leading the compared models on every split.
3. **Multi-Instance Grounding**: lists every instance of a category, each with its own box and mask. 63.9 mean F1 on COCO 2017, highest among the compared models and 0.9 points above GroundingPI.
4. **Reasoning Segmentation**: resolves functional and implicit requests after a short reasoning trace. Best on all four ReasonSeg scores.
5. **Embodied Referring**: picks the object a person asks to be handed in cluttered scenes. Best on all four RoboRefIt scores.
6. **Fine-Grained Perception**: thinks with visual latents, grounding each region it refers to while counting or answering high-resolution questions. Improves over its Qwen3.5-4B backbone on all eight perception benchmarks, including +6.8 points on CountQA.
We also release **LensBench**, a segmentation benchmark with 4,800 instruction-level samples from 1,200 paired evaluation units, covering General, Aerial, Driving, and Panoramic images. Each domain contains 300 units, each with four instructions crossing Single/Multiple targets with Explicit/Implicit wording.
## 🧠 Method
Each generated ` ` token has a contextual state, or visual latent, that represents a target region given the image, instruction, and preceding response. A ligh