ShuhongLL / SGS-SLAM

SGS-SLAM: Semantic Gaussian Splatting For Neural Dense SLAM 
 ECCV 2024 
 Mingrui Li*, Shuhong Liu*, Heng Zhou, Guohao Zhu, Na Cheng, Tianchen Deng, Hongyu Wang
 (*equal contribution)
 Paper | Video | Replica Dataset 
## Table of Contents
 Overview 
 Abtract 
 Installation 
 Download Dataset 
 Usage 
 Saving and Visualization 
 Logging 
 Acknowledgement 
 Citation 
## Abstract
We present SGS-SLAM, the first semantic visual SLAM system based on 3D Gaussian Splatting. It incorporates appearance, geometry, and semantic features through multi-channel optimization, addressing the oversmoothing limitations of neural implicit SLAM systems in high-quality rendering, scene understanding, and object-level geometry. We introduce a unique semantic feature loss that effectively compensates for the shortcomings of traditional depth and color losses in object optimization. Through a semantic-guided keyframe selection strategy, we prevent erroneous reconstructions caused by cumulative errors. Extensive experiments demonstrate that SGS-SLAM delivers state-of-the-art performance in camera pose estimation, 3D map reconstruction, precise semantic segmentation, and object-level geometric accuracy, while ensuring real-time rendering capabilities.
## Installation
 [code] 
## Download Dataset
DATAROOT is `./data` by default. Please change the `input_folder` path in the scene-specific config files if datasets are stored elsewhere.
### Replica
You can download the Replica dataset with ground-truth semantic masks from this link. The original Replica scenes, created by the Meta research team, are accessible through their official repository. By accessing the dataset via the provided link, you consent to the terms of the license. The ground-truth semantic masks were generated by us refering to the preprocessing procedure of Semantic-NeRF. Additionally, the camera trajectories in the dataset were captured using iMAP.
### ScanNet
Please follow the data downloading procedure on the ScanNet website, and extract color/depth/semantic frames from the `.sens` file using the following preprocessing step:
 [code] 
 [Directory structure of ScanNet (click to expand)] 
 [code] 
We use the following sequences following the previous studies: 
 [code] 
### ScanNet++
Please follow the data downloading and image undistortion procedure on the ScanNet++ website. 
We use the following sequences following SplaTAM: 
 [code] 
To extract the ground-truth semantic masks, please follow the guidelines on its official repo
## Usage
We use the Replica dataset as an example. Similar approaches apply to other datasets as well.
Run the slam system:
 [code] 
Run the post optimization after the slam system:
 [code] 
Visualize the reconstruction in an online mannar:
 [code] 
Visualize the final reconstructed scenes and manipulate the scene:
 [code] 
## Saving and Visualization
By default, the system stores the reconstructed scenes in `.npz` format, which includes both appearance and semantic features. Addition