xiaomi-research / xiaomi_spatialembodied

**[ 中文版 ]**
**[ Model Weights ]**
**[ Paper ]**
 Xiaomi-SpatialEmbodied 
 A Xiaomi Autodrive and Robotics Team research project, developed with academic collaborators. 
Official implementation of **Xiaomi-SpatialEmbodied**, the released codebase for the paper **XEmbodied: A Foundation Model with Enhanced Geometric and Physical Cues for Large-Scale Embodied Environments** (arXiv 2604.18484).
The model is designed for cloud-side autonomous-driving and robotics VQA in data closed-loop systems. It improves domain semantics and 3D understanding while preserving general VLM capabilities. The implementation follows the paper's two complementary enhancement paths: a geometric adapter for intrinsic spatial representations and an embodied adapter for compact physical cues.
**Key Contributions:**
- Present Xiaomi-SpatialEmbodied, a cloud-side embodied closed-loop VQA generalist fusing intrinsic geometric representations with physical cue interaction.
- Propose 3DA and EIEA-powered implicit physical cue alignment for adaptive geometric priors injection and efficient physical-augmented reasoning.
- Develop a progressive domain curriculum for robust adaptation with reduced forgetting, validated on 18 benchmarks to demonstrate consistent embodied understanding gains.
## Model Introduction
Xiaomi-SpatialEmbodied uses Qwen3-VL-30B-A3B-Instruct as its multimodal language backbone and adds trainable spatial and embodied interfaces. Images are processed by the native Qwen vision tower and a VGGT 3D foundation model; the resulting features are aligned before language reasoning, so spatial evidence participates in the model's internal representation rather than being appended as text or an auxiliary depth map.
- **3D Adapter (3DA):** VGGT extracts multi-layer geometric features from the same camera image. `VGGTEmbeddingMerger` resamples the VGGT patch grid to Qwen's image-token grid, applies token merging, normalization, and MLP projection, then injects the result through a cross-attention residual (image tokens query VGGT tokens). This preserves the original sequence length while adding depth, layout, and relative-position cues.
- **Efficient Image-Embodied Adapter (EIEA):** Outputs from embodied tools such as occupancy, 3D boxes, and map cues are distilled into a compact set of TOR embeddings. These embeddings are inserted at designated positions in the MLLM context, providing physical affordance priors without exposing raw tool traces to the language model. This repository ships the inference consumer and precomputed caches.
- **Progressive domain curriculum:** Training proceeds from general multimodal capability to spatial/3D supervision and then embodied driving and manipulation data. The staged recipe reduces catastrophic forgetting and yields consistent gains across the 18 benchmarks reported in the paper.
The released main checkpoint is the 3DA variant under `xiaomi_spatialembodied/3DA/my_qwen3_vggt_xattnv2/`; the connector variants in `xiaomi_spatialembodied/