datamllab / OpenEBFM

Open-source code for training and scaling Energy-based Foundation Models 
 Keywords: Energy-based Foundation Models, language models, pretraining, scaling laws
 🎉 Updates •
 🚀 Quick Start •
 📈 Scaling Law •
 ⚙️ Pretrained Checkpoints •
 💬 Chat Demo 
 English · 简体中文 
---
## 🎉 Updates
- **2026-10-07** — ✨✨ Full codebase released.
## 🚀 Quick Start
Install the GPU, Hugging Face, and web-chat dependencies once:
 [code] 
Download and tokenize the training data (run once before the first training job;
requires network). Prepares `climbmix`, `dclm`, and `fineweb` by default; use
`--datasets` to prepare only a subset:
 [code] 
Run a complete training job by setting the token budget:
 [code] 
> Data preparation needs network access, so run it once on a networked
> cpu-worker. Training itself is offline, but must run where the data is ready.
## 📈 Scaling Law
On DCLM, validation BPB falls as training tokens and model size increase.
Zero-shot accuracy scales with model size across ClimbMix, DCLM, and FineWeb,
and improves with training tokens.
 DCLM pretraining 
 Parameter scaling 
 Zero-shot model-size scaling 
 Zero-shot token scaling 
Use `configs/train.yaml` as the default configuration and set
`TARGET_TOTAL_TOKENS` to choose the training budget.
## ⚙️ Pretrained Checkpoints
Pretrained models are published in the
EBFM Hugging Face collection.
The collection includes 160M, 520M, and 1B models trained on DCLM, FineWeb, and
ClimbMix; one 160M OWT model; and a FineWeb SFT model named
`EBFM-FineWeb-1B-CHAT`.
Load a published model directly by its Hugging Face ID. The first call downloads
and caches the weights and tokenizer:
 [code] 
Replace the model ID with any repository listed in the collection.
## 💬 Chat Demo
Run the browser chat with the published SFT model:
 [code] 
Open http://localhost:8000 in a browser on the server
machine. If the server runs on a remote GPU node, use your cluster's port
forwarding to expose port 8000 on your local machine.
The PyTorch CPU and CUDA indexes are configured in `pyproject.toml`. Choose
one extra per environment.
## Run
The shell launchers are deliberately thin. They always run from the project
root, so relative paths are stable in local shells and distributed jobs.
 [code] 
Chat, validation evaluation, QA evaluation, and SFT initialization accept
published Hugging Face model IDs. BPB evaluation also needs the local snapshot
directory for `token_bytes.pt`.
Training reads `configs/train.yaml`; evaluation reads `configs/eval.yaml`.
Explicit command-line arguments take precedence over YAML values. Training
length is controlled by `TARGET_TOTAL_TOKENS`. The effective gradient
accumulation is derived from `global_batch_size`, `device_batch_size`, and the
distributed world size, so `max_steps` and `accumulate_grad_batches` are not
independent training controls.
Every run uses the same output layout:
 [code] 
When training finishes, `checkpoints_hf/` holds a clean Hugging Face model
(`config.json` + `model.safetensors`) exporte