datologyai / zephon

Fast, flexible, elastically deterministic data loading for foundation model training 
 zephon.io ·
 User Guide ·
 Launch blog post ·
 Paper ·
 PyPI 
---
Zephon is a high-performance data loading library that separates **what** you train on
from **how** data is loaded and transformed. It supports online processing (tokenization,
sequence packing, filtering, dynamic mixtures) rather than requiring expensive offline
preprocessing, while guaranteeing deterministic, reproducible output
regardless of parallelism or hardware topology.
We built Zephon at DatologyAI because running data
experiments is a lot of what we do, and an experiment that compares two datasets is only
useful if nothing else about the two training runs changes. In practice, the data loader
is often the part that doesn't hold still: if a job gets preempted and comes back on fewer
GPUs (which happens more often than any of us would like), many loaders will either
refuse to resume or quietly start feeding it different data. That's a manageable problem
for datasets you can index into directly. It's much harder for pipelines that pack
sequences, since what ends up in each packed sequence depends on every sample that came
before it. Zephon was designed from the start to keep that kind of pipeline deterministic,
so the number of GPUs you happen to have on a given day is one less thing to worry about.
## Installation
Zephon requires Python 3.10 or newer.
 [code] 
Most of what Zephon can read is behind optional extras, so you only install what you use:
| Extra | What it adds |
|---|---|
| `zephon[cloud]` | Reading from S3, GCS, and Azure (`s3://`, `gs://`, `az://` paths) |
| `zephon[hf]` | Hugging Face datasets (`hf://` paths), plus Parquet |
| `zephon[parquet]` | Parquet shards |
| `zephon[streaming]` | MosaicML Streaming / MDS shards |
| `zephon[litdata]` | LitData shards |
| `zephon[vortex]` | Vortex shards (Python 3.11+) |
| `zephon[lance]` | Lance datasets, one shard per fragment |
| `zephon[ray]` | The experimental Ray runner |
For example, to train on Parquet data in S3: `uv pip install "zephon[cloud,parquet]"`.
Tokenization uses Hugging Face tokenizers, so you'll also want `uv pip install transformers`
if you're following the example below.
## A First Pipeline
A Zephon pipeline has three parts: a `Dataset` that describes your data, a `WorkSource`
that decides what to train on and in what order, and a `Pipeline` of operators that turns
samples into training batches.
 [code] 
Mixing datasets is a matter of adding them to the `WorkSource` with the proportions you
want:
 [code] 
A `Pipeline` is a plain Python iterable, so it works with whatever training loop you
have. It will also run inside a PyTorch `DataLoader` or torchdata `StatefulDataLoader` if
your framework insists on one, although we recommend iterating over it directly.
## Checkpointing and Elastic Resume
You save and restore a pipeline's position alongside your model checkpoint:
 [code] 
Resuming on a different number of GPUs c