StayLameBro / backburner

# Backburner
Plug your iPhone into your MacBook with a 10 Gb/s USB-C cable and it helps run Qwen3.8-27B locally:
- **Faster prefill (up to 64k context).** For every batch of prompt tokens the Mac runs layers 1-40 and the iPhone runs 41-64
 on its GPU, pipelined. Your agent waits less every time it reads a file or a tool result of more than ~512 tokens: 29-44% faster prefill at 16k-48k.
- **More context.** A 24 GB Mac fits 64k tokens of 8-bit context next to the model. The iPhone holds the oldest part past that
 and computes attention over it: its GPU during prefill, its GPU and Neural Engine while writing. The server sizes the total
 from the phone's free memory at startup (196k-229k tokens at 8-bit on an iPhone 17 Pro Max). Tested end to end to 128k at
 8-bit and 140k at 4-bit.
- **Same answers.** Greedy output is token-identical with and without the phone (256/256 tokens at 8k and 32k; 32/32 at 140k).
The engine is a llama.cpp fork (`llama.cpp/`, StayLameBro/backburner-llama.cpp)
with its own Mac kernels (SME2, Metal fusions, DFlash2 speculative decoding). Those speed things up on the Mac alone too; the
numbers below keep the two apart.
Tested on a MacBook Pro M4 Pro (24 GB) with iPhone 17 Pro Max (A19 Pro) and iPhone 16 Pro Max (A18 Pro) phones. Others
have run it on 8 and 18 GB Macs, an M5 and an M1 iPad Pro and an iPhone 18 Pro Max: see "Will it work on my …?" below.
**Where this is going:** one inference machine out of every device you own: Macs, iPhones, iPads, then Linux and Windows
PCs and Android phones. The goal is to run models too big for any one of them, with the same answers a single machine big
enough to hold the model would give.
docs/ROADMAP.md has the plan and what people have asked for.
## Results
Measured on 2026-10-01: Qwen3.8-27B IQ4_XS, MacBook Pro M4 Pro 24 GB, iPhone 17 Pro Max over USB-C. Raw rows are in
`bench/results/*.jsonl`; the scripts that produced them are in `bench/`.
### Reading (prefill): Mac alone vs Mac + iPhone, same build
A 2,000-token file or tool result read into a saved agent session (`bench/turn-bench.py`, two reads per depth):
| Context already in the session | Mac alone | Mac + iPhone | |
|---|---|---|---|
| 16k | 109 tok/s (18.8 s) | 157 tok/s (13.1 s) | +44%, 31% less waiting |
| 32k | 101 tok/s (20.3 s) | 130 tok/s (15.8 s) | +29%, 22% less waiting |
| 48k | 87 tok/s (23.5 s) | 113 tok/s (18.1 s) | +30%, 23% less waiting |
A new omp session (omp's system prompt, project notes and 12 tools, 26,849 tokens, read cold; `bench/session-bench.py`):
| | stock llama.cpp | this fork, Mac alone | this fork + iPhone |
|---|---|---|---|
| first answer | 245 s | 228 s | 168 s |
| later turns (1.3-1.9k-token tool results) | 17.9 s | 19.2 s | 14.5 s |
After the first time, the SSD prompt cache (`scripts/proxy.py`) restores that 27k-token start in 0.3-5 s.
Past 64k the Mac-alone config switches to 4-bit context: 128k at 8-bit measured ~0.3 GB over the GPU's 20 GB memory limit
with the draft model loaded (2026-09-2