midagedev / bloomery

bloomery 
 A fast inference engine for mixture-of-experts models, built for offloading. 
 Written in Rust from the HTTP server down to the CUDA kernels. A drop-in
 llama-server .
- **MoE models larger than the card, and larger than the host's RAM.** Routed experts live on the GPU and in host
 RAM; the engine counts its own routing and moves the experts it calls most onto the card while it runs. On one
 RTX 3090 24 GB beside a 256 GB host, DeepSeek-V4.1-Flash `Q3_K_M` (347 GB) serves two requests at 34.4 tok/s in
 total and GLM-5.3-Flash at 26.9. When the RAM cannot hold the host's experts either, Qwen3.8, GLM-5.3-Flash and
 DeepSeek-V4.1-Flash page the rest from the model file on NVMe.
- **A prompt cache that survives other requests.** The Qwen3.8 and GLM-5.3-Flash seats keep saved prompt states in
 host RAM: a request that shares a prefix with an earlier one restores it, even after another request used the
 slot (recorded on tape).
- **Several requests, one pass.** With `--parallel 2` two busy streams run through the model together. With the MTP
 draft off, that gave 25–37 % more tokens a second in total than one stream (0.2.1,
 measured on the A6000). The draft
 is on by default and already fills the pass's rows: in 0.2.10's release check (A6000), two streams of Qwen3.8
 (`--place a`) and GLM-5.3 `UD-IQ4_XS` gave about the total of one stream.
- **llama-server compatible.** The same HTTP API (OpenAI and Anthropic), GGUF files and flag spellings: your client
 works unchanged.
- **Rust all the way down.** Every CUDA kernel is written in Rust and compiled with
 cuda-oxide.
 [code] 
## Status (0.3.6)
0.3.6 answers structured requests with schemajump (the fixed parts of a JSON answer are fed in one call, not decoded
token by token), serves `/v1/systemone` on every generative seat in the same body as the Clef seat, and runs
Solar-Open2-250B with its experts on the card and the host (release notes).
| Area | State |
|---|---|
| Linux x86-64, NVIDIA sm_86 (RTX 3090, RTX A6000, RTX 3060) | Runs. Every number here comes from these cards |
| NVIDIA driver | R580 or newer (CUDA 13); an older driver is refused at the start |
| Ada and Blackwell (sm_89, sm_120) | Blackwell ran on a user's RTX 5090 under WSL2 (#2): the sm_86 PTX compiles for the card at the first start. Ada not yet run. Reports welcome |
| NVIDIA DGX Spark (GB10, aarch64) | Builds and runs from source (a contributor's port, `docs/contrib/dgx-spark.md`): the plan treats the unified memory as one pool, `BLOOMERY_ARCH` sets sm_121, and the quant types without an aarch64 kernel run through a scalar mirror. Two Sparks over RDMA serve one model, a node owning routed experts (#358, #420). The release binaries are x86-64 only |
| Host CPU | x86-64 with AVX2 (x86-64-v3), AMD or Intel. Every number here comes from an 8-channel host; a 2-channel desktop host is not measured here and runs the CPU experts slower |
| macOS, Apple silicon · AMD GPUs, Windows without WSL2 | Not supported yet · Not supported