blendi-remade / prompt2rig

# Prompt2Rig
**Type a motion, watch a rigged character perform it — in seconds.**
A text prompt goes to a fast video model on fal — MiniMax H3 Max Turbo (faster than real time) or FLUX 3 Max (higher fidelity, ~real time) — the clip's human motion is recovered with GVHMR on a local GPU, and retargeted live onto a Mixamo skeleton in the browser. Inspired by Motion2MixamoRig, rebuilt for interactive speed.
## How it works
 [code] 
| Stage | Where | Time (5s clip, RTX 4090) |
|---|---|---|
| Video generation | fal · H3 Max Turbo (480P) | ~2–3s wall (0.5s GPU) |
| | fal · FLUX 3 Max (720p) | ~8–9s wall (4.5s GPU) |
| **Live** preview — MediaPipe Pose, retargeted per frame | browser | immediate |
| **HQ** motion — GVHMR → SMPL rotations → Mixamo | local GPU worker | ~3.5–4s (0.7–0.8× real time) |
| **Fingers** — WiLoR hand model (MediaPipe fallback), in parallel with GVHMR | local GPU worker | +~0.3s |
The live pass hides the wait; the HQ pass is the real result. GVHMR sees the whole clip at once, outputs joint *rotations* on a fixed-proportion body, and recovers a gravity-aware root path — so there's no jitter, limb stretching, or twisted forearms.
## Quick start
**Needs:** Node 18+, a fal API key. For the HQ pass: Windows + NVIDIA GPU, `uv`, `ffmpeg` on PATH.
 [code] 
**HQ pass (optional, recommended):**
 [code] 
Then download **SMPL-X v1.1 (NPZ)** from smpl-x.is.tue.mpg.de (free registration) and put `SMPLX_NEUTRAL.npz` in `hq/body_models/smplx/`.
**Finger tracking (optional):** WiLoR needs the MANO hand model — register at mano.is.tue.mpg.de and download *Models & Code* (`mano_v1_2.zip`), then:
 [code] 
Without MANO, fingers fall back to MediaPipe Hands (less accurate depth). Start the worker next to the web server:
 [code] 
## Using it
- **Generate** a prompt or click a preset. By default every clip starts from the same flux-generated *performer* (one person, full body, plain backdrop, static camera — ideal pose-estimation input).
- **Video model** — *H3 Max Turbo* for the snappiest loop and Stream mode; *FLUX 3 Max* for more realistic motion (best for single takes — at ~9s per 5s clip it can't keep a chained stream fed yet).
- **Character** — X Bot, Michelle or Soldier (textured Mixamo rigs from the three.js examples), or **Load Mixamo file…** for any Mixamo `.fbx`/`.glb` (download from mixamo.com, T-pose).
- **Track fingers** — the HQ pass also recovers hands: WiLoR (MANO-based) on the full-resolution clip, assigned to the body's wrists, rotated into the motion's world frame and smoothed; the rig gets palm orientation plus all 15 finger joints. Untracked hands rest in a natural curl.
- **Swap character every loop** — X Bot → Michelle → Soldier, cutting on each loop (or each new clip in Stream mode). The motion lives on one hidden driver rig; the character on stage copies its pose, so swapping is instant.
- **Cinematic camera** — slow arc and push-in around the performer on a studio stage (environment lighting, warm key + cool rim, soft reflectiv