sybil-solutions / glm-flash-lite

# glm-flash-lite
Run **GLM-5.3-Flash** (125 GB, EXL3 3.05 bpw) on **one 24 GB RTX 3090** with an OpenAI-compatible API, using RAM and
NVMe for the experts that do not fit on the GPU.
| RAM for the server | mode | decode, 1 user | prefill 8k / 32k | output vs stock exllamav3 |
|---|---|---|---|---|
| 218 GiB free | `fast` | 28.2 tok/s | 710 / 951 tok/s | near-exact (decode KL 0.005) |
| **55 GiB cap** + NVMe | **`nvme`** | **19.5 tok/s** | 664 / 969 tok/s | near-exact (decode KL 0.008) |
| 16 GiB cap + NVMe | `nvme` | 14.7 tok/s (lab; 14.38 sweep) | 650 / not run | near-exact (same CPU lane) |
| 55 GiB cap + NVMe | `nvme-exact` | 8.3 tok/s | 564 / 806 tok/s | bit-exact |
| 55 GiB cap + NVMe + 1x Arc Pro B70 | `nvme` + `GLM53_B70=1` | 26.5 tok/s | 658 / 968 tok/s | near-exact (decode KL 0.003) |
Measured on 1x RTX 3090 (PCIe 4.0 x16), AMD EPYC 7443P (24 cores), 8-channel DDR4, and 4x Samsung 9100 PRO in RAID0.
Decode runs each answer to its natural end. Full tables, dates and raw files: docs/results.md.
Image: ghcr.io/sybil-solutions/glm53-flash-offload (name kept for existing installs)
## Run it
**Omarchy Local AI.** Run the line below. In Local AI, pick **GLM-5.3-Flash, EXL3 3.05 bpw (55 GB RAM, experts on
NVMe)** and press **Start**. The plugin downloads the weights, builds the store and starts the server.
 [code] 
**Docker.** Needs the NVIDIA container toolkit, the `hf` CLI, 125 GB for the weights, and 117 GB for the expert store
on a fast local NVMe filesystem (xfs or ext4).
 [code] 
The server is ready in about 80 s at `http://127.0.0.1:30000/v1`, model `glm-5.3-flash`. For 16 GB of RAM use
`--memory 16g --memory-swap 16g`. For bit-exact output use `GLM53_MODE=nvme-exact`. The all-RAM `fast` mode uses an
older image; see docs/results.md.
## Supported recipes
Every local-ai-registry recipe whose launch image is
`ghcr.io/sybil-solutions/glm53-flash-offload` or `ghcr.io/0xsero/glm53-flash-offload`, single-GPU only. Only one
hardware card has such recipes: the RTX 3090 24 GB. The GLM-5.3-Flash recipes on other cards (DGX Spark GB10,
RTX PRO 6000 Blackwell, CMP 170HX, Apple M5 Ultra) use other engines and images (vLLM, SGLang, oMLX). No recipe runs
this image on an Arc Pro B70 alone.
Status: **accepted** = the registry lab passed all six gates (load, chat, reasoning, tools, context, speed) on that
digest; **reported** = the lab ran it and a gate failed, or only the owner's campaign measured it; **candidate** = not
in the plugin catalog. Lab rows: prefill is the context gate (one 93,437-token prompt, tokens / request time), decode is
the speed gate (one stream, first 30 s, natural completion, gate 15 tok/s). Sweep rows: `bench/sweep.py`, see
docs/results.md. "agg" is the aggregate over all streams.
### RTX 3090 24 GB
| prefill size | prefill speed | decode speed | concurrency | kv cache | gpu count | recipe | RAM / storage | status | image digest | source |
|---|---|---|---|---|---|---|---|---|---|---|
| 93,437 | 1,015 tok/s | 20.0 tok/s | 1 | 131,