zml / zml

Website 
 | Getting Started 
 | Documentation 
 | Discord 
 | Contributing 
[ZML]: https://zml.ai/
[Getting Started]: #getting-started
[Documentation]: ./docs/README.md
[Contributing]: ./CONTRIBUTING.md
[Discord]: https://discord.gg/6y72SN2E7H
# About
ZML is a production inference stack, purpose-built to decouple AI workloads from proprietary hardware.
Any model, many hardwares, one codebase, peak performance.
Compiled directly to NVIDIA, AMD, INTEL, TPU, Trainium for peak hardware performance on any accelerator. No rewriting.
It is built using the
Zig language, MLIR, and Bazel.
# Getting Started
## Prerequisites
We use `bazel` to build ZML and its dependencies. The only prerequisite is
`bazel`, which we recommend installing through `bazelisk`.
### macOS
 [code] 
### Linux
 [code] 
## 30-Second Smoke Test
Run the MNIST example:
 [code] 
This downloads a small pretrained MNIST model, compiles it, loads the weights, and
classifies a random handwritten digit.
## LLM Quickstart
The main LLM example is `//examples/llm`. It currently supports:
- Llama 3.1 / 3.2
- Qwen 3.5
- LFM 2.5
Authenticate with Hugging Face if you want to load gated repos such as Meta
Llama:
 [code] 
Alternatively, set the `HF_TOKEN` environment variable.
Then run a prompt directly:
 [code] 
Open the interactive chat loop by omitting `--prompt`:
 [code] 
You can also load from:
- a local directory: `--model=/var/models/meta-llama/Llama-3.2-1B-Instruct`
- S3: `--model=s3://bucket/path/to/model`
## Running Models on GPU / TPU
Append one or more platform flags when compiling or running:
- NVIDIA CUDA: `--@zml//platforms:cuda=true`
- AMD RoCM: `--@zml//platforms:rocm=true`
- Intel OneAPI: `--@zml//platforms:oneapi=true`
- Google TPU: `--@zml//platforms:tpu=true`
- AWS Trainium / Inferentia 2: `--@zml//platforms:neuron=true`
- Disable CPU compilation: `--@zml//platforms:cpu=false`
Example on CUDA:
 [code] 
Example on ROCm:
 [code] 
Example on Intel OneAPI:
 [code] 
## Run Tests
 [code] 
# Examples
- `examples/llm`: unified LLM CLI for Llama, Qwen, and LFM
- `examples/mnist`: smallest end-to-end model run
- `examples/sharding`: logical mesh, partitioners, shard-local execution, profiler output
- `examples/io`: inspect and load local, `hf://`, `https://`, and `s3://` repositories through the VFS layer
- `examples/benchmark`: measure loading and execution performance
# A Taste Of ZML
 [code] 
For a full walkthrough, see:
- Getting Started
- Writing your first model
- ZML Concepts
- Deploying on a server
# Where To Go Next
- Run more examples in `./examples`
- Read the example-specific notes in `examples/llm/README.md`
- Learn tagged dimensions in `working_with_tensors.md`
- Start building a model with `write_first_model.md`
- Explore deployment in `deploy_on_server.md`
# Contributing
See [here][Contributing].
# License
ZML is licensed under the Apache 2.0 license.
# Thanks To Our Contributors