HazyResearch / ncm

Next-Concept Modeling 
 Fast, Parallel Coarse-to-Fine Text Generation 
Next-Concept Modeling (NCM) is a two-stage framework for parallel text generation. A multiscale residual VQ tokenizer represents a sequence as a hierarchy of discrete codes, and a transformer predicts that hierarchy from coarse to fine. Positions within a level are generated in parallel, so generation requires one forward evaluation per level.
## Highlights
### Coarse-to-fine generation
NCM generates coarse to fine: each level predicts a longer sequence in parallel, and decoding every level shows a sample being refined from a rough sketch to the full output in a few steps.
### Quality at a fraction of the cost
NCM beats or matches strong diffusion baselines on different tasks & datasets while generating 9-31× faster.
## Released models
Weights and tokenized datasets are hosted in the NCM Hugging Face collection. Each released model consists of a frozen tokenizer and a next-concept generator.
 Task Model Tokenizer Generator Config Data 
 XSum – tokenizer.pt generator.pt config.yaml test set 
 WMT14 De–En – tokenizer.pt generator.pt config.yaml test set 
 Sudoku easy tokenizer.pt generator.pt config.yaml test boards 
 medium tokenizer.pt generator.pt config.yaml 
 hard tokenizer.pt generator.pt config.yaml 
 LM1B small tokenizer.pt generator.pt config.yaml tokenized text 
 medium generator.pt config.yaml 
 large generator.pt config.yaml 
 TinyStories – tokenizer.pt generator.pt config.yaml tokenized text 
 OpenWebText – tokenizer.pt generator.pt config.yaml tokenized text 
## Installation
 [code] 
Python 3.10 or newer and a CUDA-capable PyTorch installation are recommended.
## Evaluation
Download the checkpoints and evaluation inputs:
 [code] 
Then run:
 [code] 
This runs one generation shard per visible GPU, merges them, and prints ROUGE-1/2/L and generative perplexity for XSum; cased and lowercased sacreBLEU, chrF and length ratio for WMT14; solved, cell and valid-grid rates for Sudoku; and generative perplexity under GPT-2 large with unigram entropy, over 5 seeds of 256 unconditional samples, for TinyStories, OpenWebText and LM1B, the latter under the FLM scoring protocol. Add `--cfg ` for classifier-free guidance on XSum and WMT14; the guided WMT14 row uses `--cfg 1.0`.
## Repository layout
- `evaluate.py`: evaluation entry point
- `scripts/`: per-GPU generation shard and the merge that scores it
- `config/eval/`: one config per released checkpoint, identical to the `config.yaml` shipped with it
- `models/`, `tokenization/`: generator and tokenizer implementations
## Citation
 [code] 
Released under the MIT license.