dsta022 / Loop-Engineering-for-VLA

# Loop Engineering for VLAnything
Website: https://dsta022.github.io/Loop-Engineering-for-VLA/.
## Multimodal VLA Dataset Toolkit
An end-to-end toolkit for collecting, merging, auditing, enriching, and
iteratively improving multimodal LeRobot datasets for vision-language-action
models. It supports standard RGB-only LeRobot datasets and RGB-D datasets with
lossless depth sidecars, and it treats RGB video, robot actions, states,
metadata, task semantics, language annotations, and human feedback as one
auditable engineering loop: **Capture · Audit · Enrich · Improve**.
The offline pipeline never imports LeRobot or camera SDKs. The recorder in
`record/` is a self-contained, framework-free data collection layer that talks
to SO-100/SO-101 arms and cameras directly and writes LeRobot v3.0 datasets:
- `record/rgb_record/`: RGB-only recording.
- `record/rgbd_record/`: Orbbec / Intel RealSense RGB-D recording, plus
 human-guided policy feedback.
- `record/visuo-tactile_record/`: visuo-tactile recording, in progress.
Recorded datasets flow straight into the merge, audit, clean, enrich, feedback,
and upload steps below. Three guiding rules run through every stage:
1. **Conservative decisions.** Only objective structural, numerical, or decoding
 failures become `drop`; semantic and threshold-based findings are `review`;
 a check with missing preconditions records `not_checked` instead of passing.
2. **Sidecars, never in-place edits.** Depth, feedback provenance, language,
 training weights, and reports live outside the dataset; source data is
 read-only.
3. **Pluggable, model-free by default.** Evaluators, annotators, triggers, and
 the train/evaluate/deploy trio load by `module:attribute`; the base install
 downloads no weights.
## Dataset
The merged RGB-D VLA dataset is published on Hugging Face as
DerekLX/lerobot_derek_depth.
Its root contains the LeRobot subdirectories directly:
 [code] 
## Dataset Preview
Samples from one `pick_up_cups_dataset` episode with two cups in the scene.
Left to right: global robot collection view, wrist camera view, and front depth visualization.
If the video does not render in your Markdown viewer, open task_demo.mp4 directly.
| Front camera, 9 sampled episodes | Wrist camera, 9 sampled episodes | Front depth, 9 sampled episodes |
| --- | --- | --- |
| | | |
## Repository Layout
 [code] 
## Getting Started
Python 3.12 is recommended. Create an environment, install the project in
editable mode, and add `ffmpeg`/`ffprobe` for video decoding and trimming:
 [code] 
The editable install exposes console commands and lets every script run by
path from any directory:
| Command | Script | Purpose |
| --- | --- | --- |
| `vla-merge` | `pipeline/dataset_merge.py` | merge datasets into one real dataset |
| `vla-audit` | `pipeline/data_quality_audit.py` | conservative quality audit |
| `vla-clean` | `pipeline/dataset_clean_build.py` | build a clean dataset from a drop list |
| `vla-enrich` | `pipeline/trajectory_language_enrichment.py` |