sumuru789 / DK-VLM

DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language Models 
 Han Su 1,2 , Meng Tian 2 , Tianyu Huang 3 , Mingyu Gao 1 , Lennox Wei 1 , 
 Dongxu Li 1 , Jianhua Han 2 , Hang Xu 2 , Xiaohe Wu 1 , Wangmeng Zuo 1,* 
 1 Harbin Institute of Technology   
 2 Yinwang Intelligent Technology   
 3 Huawei
---
## Abstract
Vision-language models still struggle to understand 3D spatial relations across views and update spatial judgments as the observer moves. A common approach fuses 3D reconstruction features into visual tokens, but aligning these features with pretrained semantic representations remains challenging under limited spatial supervision. We propose **DK-VLM** with *Dual-Kernel Attention*, which computes attention from both a visual token's input position and its geometric position in the scene. A semantic kernel retains the pretrained model's native attention and positional encoding, while a separate geometric kernel uses estimated 3D points and camera poses to directly modulate attention through point displacements and relative view transformations. Then, a learned gate fuses their outputs. Geometry thus enters attention as position rather than as fused reconstruction features. We further introduce Reference-Frame Composition (RC) supervision to train the model to update its reference frame across successive judgments. It provides stepwise turn and heading targets derived from verified single-step annotations. Among the compared methods, DK-VLM-8B achieves the highest average score across six spatial benchmarks and the highest overall scores on VSI-Bench (74.33) and ReVSI-32 (58.28). RC supervision further improves multi-step spatial reasoning, while the full model preserves general video understanding.
---
## 📥 Models
Download the 7B and 8B checkpoints from Hugging Face.
The checkpoints contain LoRA adapters and geometric branch weights. Download the matching base model separately:
| DK-VLM checkpoint | Required base model |
| --- | --- |
| `ckpt/DKVLM_Qwen25_7B` | Qwen2.5-VL-7B-Instruct |
| `ckpt/DKVLM_Qwen3_8B` | Qwen3-VL-8B-Instruct |
Use **`vggt_omega_1b_512.pt`**, available after requesting access to VGGT-Omega. This is the frontend used in our experiments.
---
## 🛠️ Installation
Recommended environment: **Python 3.10.20, PyTorch 2.6.0+cu124, and CUDA 12.4**. See environment setup for installation instructions.
---
## 🚀 Quick Start
Use the included demo video and replace the model paths below:
 [code] 
For 8B, switch to `../ckpt/DKVLM_Qwen3_8B` and the matching Qwen3-VL base model. `--video` also accepts your own video or a directory of image frames.
For VSI-Bench and ReVSI-32 evaluation, see evaluation instructions.
---
## 📊 RC Annotations
Single-step spatial QA typically supervises judgments within a given reference frame, but lacks explicit supervision for updating that frame across successive turns or viewpoint changes. RC composes verified single-step relations into **1.6M examples with 2–4 reasoning steps**, providing