NoizAI / WorldSonus

# WorldSonus: Bringing Sound to Worlds
**WorldSonus brings synchronized, controllable stereo sound to visual worlds.**
Our causal autoregressive diffusion model generates audio in 100 ms chunks,
combining fine-grained visual motion with text instructions that can change
during a sequence.
## Demo
https://github.com/user-attachments/assets/22400b47-b292-476a-89a1-eeaf33372ba2
## Overview
## Installation
Requires Python 3.10+, a CUDA-capable GPU, FFmpeg and ffprobe. The setup script
installs PyTorch with CUDA 12.4 wheels; GPU drivers must already be installed.
 [code] 
## Pretrained models
Weights are available on Hugging Face.
The latest `worldsonus_150k_padtrim.pt` was retrained with audio-padding filtering
and tail trimming, followed by HQ fine-tuning. Its five-second context window is
read automatically from the checkpoint. Update the code and rerun the download
command when upgrading. The decoder and latent normalization are unchanged.
The earlier `worldsonus_150k.pt` remains compatible through `--checkpoint`.
Download the generation model, audio decoder, normalization statistics, and
frozen video/text encoders:
 [code] 
DINOv3 and T5Gemma 2 retain their upstream licenses and access conditions.
Accept their terms and authenticate with Hugging Face if required. Use the
normalization statistics supplied with the checkpoint.
## Video-to-audio generation
Pass a video and an audio description directly. Feature extraction, audio
generation, and decoding run automatically:
 [code] 
The output is stereo 48 kHz floating-point WAV, preserving decoder amplitudes
without integer-PCM clipping. By default the input duration is rounded down
to a complete 100 ms chunk. Use `--start 5 --seconds 10` to select a window, or
omit `--prompt` for video-only conditioning.
## Streaming generation
Add `--stream` to read video incrementally and generate/decode each 100 ms audio
chunk as soon as its three video frames are available:
 [code] 
Visual temporal differences, autoregressive Ring-KV history, and causal decoder
convolution states persist across chunks. Audio is not assembled from
independently decoded clips. Model/encoder loading happens before streaming
starts; this reference implementation does not promise a particular latency.
Offline and streaming modes use the same three-frame generation path. With the
same inputs, seed, device and compilation mode, they produce matching latents;
whole-clip and incremental decoding differ only by floating-point rounding.
CUDA inference is compiled by default in both modes. Add `--no-compile` to
disable compilation; CPU inference remains uncompiled by default.
The first invocation compiles kernels, and the first chunks of a new session
warm CUDA graphs. Expect slower startup; compilation does not change the model
or sampling configuration.
For immediate playback with FFplay, stream raw PCM instead of waiting for a file:
 [code] 
## License
WorldSonus's original contributions are released under CC BY-NC 4.0,
for non-commercia