qlabs-eng / noise

# Noise: Speedrun for pretraining without backpropagation
Noise is a pretraining speedrun for learning algorithms that never run a backward pass. We think replacing
backprop with a much better credit assignment algorithm is inevitable. Deep learning's architectures,
optimizers and hardware have co-evolved with backprop, but as compute grows, the
bitter lesson suggests that more general,
brute-force learning algorithms based on search will win. Dust is a big step
forward: it rivals backprop under SGD and is 1,000 to 10,000x more efficient than EGGROLL, the state-of-the-art
weight-space ES, but still trails backprop under Adam. These methods have also had a small fraction of the
research that went into backprop. We made Noise to close the gap: better estimators, and optimizers co-designed
for them instead of for backprop gradients.
The model, the data and the score are fixed: the 37.7M-parameter transformer of Dust, its 10M-token FineWeb
training set, and the cross-entropy on its validation set. What changes is the learning algorithm, which may
only run the model forward. The architecture is fixed for now, but we will relax this constraint later.
## Tracks
1. **Validation loss.** The lowest validation loss after one epoch on one 8xH100 node, with no time limit.
2. **Speedrun.** The shortest training time on one 8xH100 node to reach validation loss **4.2**.
## Rules
- **No backpropagation.** No autograd (reverse or forward mode) and no hand-written backward pass. Learning
 signal comes from evaluating the model forward: every error signal is measured against a loss by running the
 model forward, and may then take at most one step through a nonlinear part of the model. Steps through
 linear maps don't count, and a signal that has taken a step can't take another, so credit can't be chained
 back through the network block by block. Entries are judged by the spirit of this rule: a method should
 still apply to networks with non-differentiable parts.
- **Fixed model.** Final weights must load into `utils/model.py` with a strict state-dict
 match. How training computes the same function, and how it initializes it, is free.
- **Fixed data.** Train only on `data/train.pt` (4,880 sequences of 2,048 predictions), for at most one
 epoch: each training sequence feeds at most one update. Batch size and order are free.
- **Score.** `utils/eval.py` on the final weights: mean next-token cross-entropy over the
 1,114,112 predictions of `data/val.pt`. No checkpoint selection on validation loss.
- **Timing** (speedrun track; reported on the other). From the start of the first update to the end of the
 last, excluding any validation. An untimed warmup on random tokens may compile kernels, as long as training
 then starts from a fresh state (the baselines rebuild everything from the seed). Runs use seed 0.
## Setup
Requires Linux, an NVIDIA GPU with bfloat16 support and uv.
 [code] 
`utils/prepare_data.py` streams FineWeb `sample-10BT`, tokenizes it with the fixed BPE-4