ShuhongLL / RawVLA

RawVLA: Embodied Neural Image Signal Processor for Robotic Manipulation 
 Shuhong Liu 1,2 ,
 Heng Zhou 2 , Lingfeng Qian 2 , Yuhao Fang 2 ,
 Xianbao Hou 2 , Qianyu Zhou 1 ,
 Lin Gu 4 ,
 Wei Sui 2 , Jianfei Yang 3 ,
 Ziteng Cui 1,5 
 1 The University of Tokyo  
 2 D-Robotics  
 3 NTU  
 4 Tohoku University  
 5 HKUST(GZ)
RAW-VLA is a streaming, illumination-adaptive RAW image frontend for
vision-language-action policies. This repository contains the RAW frontend,
StarVLA integration, RAWVLA-Bench lighting transforms, and LIBERO and RoboTwin
2.0 evaluation adapters.
## 📦 Installation
Clone the repository with its submodules:
 [code] 
Several project-maintained submodules are branches of this repository. Their
branch and pinned commit are documented in SUBMODULES.md.
### Choose an environment
Install only the environment needed for your workflow:
| Benchmark | Training environment | Evaluation environment |
| --- | --- | --- |
| LIBERO | `environment-libero.yml` (`starvla-libero`) | `environment-libero.yml` (`starvla-libero`) |
| RoboTwin 2.0 | `environment-libero.yml` (`starvla-libero`) for RAW-VLA training | `environment-libero.yml` (`starvla-libero`) for the policy + `environment-robotwin2.yml` (`robotwin`) for the simulator |
The `robotwin` environment runs the simulator; RAW-VLA training for both
benchmarks uses the StarVLA environment. RoboTwin π0/π0.5 training recipes are
in `configs/robotwin`.
Standalone OpenVLA-OFT and GR00T have separate, optional environments;
see ENVIRONMENTS.md for their installation and validation.
### LIBERO and RAW-VLA
Use this environment for StarVLA/RAW-VLA training, LIBERO evaluation, and the
StarVLA policy server used alongside RoboTwin:
 [code] 
The last command writes ignored, machine-local simulator paths to
`.local/libero/config.yaml`. Versioned RAW-VLA training recipes are in
`configs/libero`.
### RoboTwin 2.0
Install the simulator for evaluation in its own environment. The asset script
downloads the official assets unless you set `ROBOTWIN_ASSETS_SOURCE` to an
existing asset directory:
 [code] 
Evaluation with a StarVLA policy server also uses the `starvla-libero`
environment above. See
ENVIRONMENTS.md for interpreter selection, host-specific
graphics requirements, and validation commands.
## 🧠 VLA Checkpoints
Download the public StarVLA LIBERO checkpoints and their base models with:
 [code] 
Set `STARVLA_ROOT=/custom/path/to/starVLA` when the StarVLA checkout is not at
the repository default.
## 🗂️ RawVLA-Bench
The train and test resources are hosted separately: paired training
trajectories are on Hugging Face, while the frozen test/evaluation protocols
are JSON manifests in this GitHub repository. No pre-rendered test trajectories
are included in the Hugging Face dataset.
### Train: Paired Trajectories (Hugging Face)
The paired RAW/RGB training trajectories are available directly from
RawVLA-Bench on Hugging Face.
It contains 2,403 successful trajectories: 1,771 from LIBERO and 632 from
RoboTwin 2.0.
 [co