ZeamoxWang / pdmd

## PDMD: Projected Distribution Matching Distillation for Video Diffusion Models
https://github.com/user-attachments/assets/a7482145-3364-46f2-9ce0-439e584de3e6
 Every frame is 4 network evaluations. Music: "Goliath" by Scott Buckley, CC BY 4.0. 
This repo contains training, inference and evaluation code for PDMD on MiniMax-H3-33B joint
video–audio generation. Train a four-step student from the base model, or run the released four-
and two-step checkpoints on a single 24GB or 80GB GPU. More videos, with sound, are on
our project page.
📦 Checkpoints · 🎬 Sampling · 🏋️ Training ·
📊 Evaluation · 📚 BibTeX
> **PDMD: Projected Distribution Matching Distillation for Video Diffusion Models** 
> Zimo Wang, Junkun Yuan, Angtian Wang, Haotian Yang, Canyu Zhang, Siyuan Yuan, Xingchang Huang,
> Bo Liu, Yizhi Wang, Yiding Yang, Chongyang Ma, Gordon Guocheng Qian
> UC San Diego, ByteDance 
Distribution Matching Distillation (DMD) reduces video diffusion models to a few network
evaluations (NFE), but its samples can degrade during training, drifting toward oversaturation and
artifacts. We trace this instability to critic errors that accumulate in successive student
updates, and introduce Projected Distribution Matching Distillation (PDMD), which projects out the
component of the DMD update parallel to the student–critic endpoint residual, an unbiased estimate
of the critic's endpoint error. PDMD is a one-line change to DMD, with no extra loss, network, data,
model pass, or training stage. With Wan2.1, PDMD reaches a VBench total score of 83.73 at 4 NFE;
on MiniMax-H3 joint video–audio generation, it reaches a VideoGen-Eval visual total of 83.17 and
the best score on all six audio metrics among the compared 4-NFE models.
This repository contains:
* 🪐 PDMD LoRA checkpoints at 4 NFE and 2 NFE
* ⚡️ Inference scripts that run them on a single 24GB GPU (`run_low_vram.py`) or a single 80GB GPU (`run_bf16.py`)
* 🏋️ Multi-node training of student and critic LoRAs, with automatic model and training-data downloads
* 📊 Video evaluation: VBench quality and Qwen semantic scoring on 387 VideoGen-Eval prompts
* 🔊 Audio evaluation: PQ, CE, CU, IS, IB and DeSync
> **Note.** The main script, `run_low_vram.py`, is optimized for GPUs with little memory, e.g. an NVIDIA A10
> (24GB of GPU memory, with 128GB of host RAM). The script quantizes the transformer and the Qwen3-VL
> text encoder to int8, and offloads their weights to host memory: the transformer is streamed to the
> GPU one block at a time and the text encoder one layer at a time, and the VAE is moved to the GPU
> only to decode. On 80GB GPUs, `run_bf16.py` runs all components in bf16 instead (see
> Sampling).
## Setup
First, download and set up the repo:
 [code] 
The instructions below set up the Diffusers-based inference workers. For training and the
DiffSynth-based benchmark sampler, use the separate environment in Training setup.
Install the inference dependencies (the versions we tested):
 [code] 
The inference script re