yibie / strata-mlx

# strata-mlx
Qwen3.8-Flash-Next on Apple Silicon: an MLX engine that reads the GGUF files
Strata uses and speaks Strata's line protocol, so Strata's server and
web page can drive it.
This is an independent, unofficial project. It is not affiliated with Strata or its authors, who have said
that macOS is out of scope for Strata itself. It is experimental and has been run on one machine.
## What it is
Strata runs a 125B-parameter mixture-of-experts model on a 12 GB graphics card by keeping most experts in
host memory. A Mac with unified memory does not need that half of it. This engine takes the other half:
- **The file as it is.** The GSQ-RCO GGUF files load without a weight being changed. Experts stay in the
 file's own blocks, nine block types, and Metal kernels multiply from them. The 28.8 GB n-gram table stays
 on disk.
- **Draft and verify.** The model's own draft head guesses the next tokens and one pass of the model checks
 them. The model decides every token that is written.
- **Prompts.** Read in chunks, the experts a tile at a time under Apple's matrix product for shaders. A
 conversation is read once: a follow-up reads only what is new, a new chat starts after the system prompt it
 shares with the last one, and a conversation can be saved and restored. With `--conversation-cache-mib` a
 conversation that another request interrupts is parked in memory and taken back, not read again.
- **Long conversations**, measured to 128K tokens of context.
- **Less memory than the model.** Experts that do not fit are read from the SSD as tokens ask for them, and
 the dense tensors MLX would hold at twice their size stay the file's blocks, which leaves the experts more.
 A long prompt goes through the layers one at a time, so each layer's experts are read once for it.
- **Strata's line protocol** on stdin and stdout (`strata_mlx/serve.py`).
## Measured
M4 Max, 128 GB, macOS 27, MLX 0.32.3. Greedy decoding. Every number is from this one machine, which slows
down under sustained load, so read them as ranges. The two model files are Q2_0 (66.4 GB) and IQ3_S (83.6 GB).
Answers are written 1.6 to 2.9 times as fast as by llama.cpp or LM Studio, and a long prompt is read 4 to 36%
faster.
**Writing an answer**, tokens a second over four prompts of 256 tokens each (more is better):
| Engine | Q2_0 | IQ3_S |
| :--- | ---: | ---: |
| **strata-mlx (this project)** | **68-111** | **65-95** |
| strata-mlx, draft head off | 44-67 | 43-56 |
| llama.cpp `llama-server` 0.4.1 | 38-39 | 35-36 |
| LM Studio 0.4.25 (llama.cpp runtime) | 43 | 40 |
**Reading a prompt** of 1,180 tokens, tokens a second (more is better):
| Engine | Q2_0 | IQ3_S |
| :--- | ---: | ---: |
| **strata-mlx (this project)** | **570-578** | **510-517** |
| strata-mlx, draft head off | 595-612 | 504-515 |
| llama.cpp `llama-server` 0.4.1 | 494-495 | 488-489 |
| LM Studio 0.4.25 (llama.cpp runtime) | 426-446 | 421-424 |
With the draft head off the engine still copies guesses from earlier in the conversat