sybil-solutions / dsv41-flash-offload

# dsv41-flash-offload
## At a glance
| | |
|---|---|
| Model | DeepSeek-V4.1-Flash, EXL3 3.0 bpw (about 200 GB of experts), 262,144-token context |
| Hardware | 1x RTX 3090 24 GB, 256+ GiB RAM, an AVX2 CPU with 24+ cores, NVMe |
| Default (D141) | decode 19.3 / 24.0 / 26.6 tok/s at 1 / 2 / 4 streams; prefill 690-790 tok/s from 8k to 261k |
| Quality (D141) | 50/50 long-prompt and long-task runs correct at C1 and C4: needle retrieval at 8k/64k/200k, executed code tests, math, long essays. See results/Q001-d141-quality |
| Image | `ghcr.io/0xsero/dsv41-flash-offload:d141-main`, pinned digest and attestation under Run |
| Experimental | Intel Arc B70 expert tier: **41.3 tok/s at 4 streams (+55%)**, crash root-caused and fixed, quality A/B pending. See experimental/b70-tier |
Serve **DeepSeek-V4.1-Flash** (EXL3 3.0 bpw, 384 routed experts x 40 MoE layers, Engram n-gram tables) from **one
24 GB RTX 3090** plus host DDR4 and NVMe, with an OpenAI-compatible API (chat with tool calls and reasoning,
completions). All routed experts are pinned in host RAM. During prefill the copy engine streams them into VRAM slots
in batches, and busy experts run as FP16 GEMMs. During decode a VRAM mirror cache holds the hottest experts and an
AVX2 kernel on the host cores computes the cold misses. The Engram tables stay memory-mapped on NVMe.
This is the code and configuration of run **D119** (= D118 at 262k context) of the FreeToken-EXL3 campaign: D061 (staged-DMA prefill, fat
threshold 32) + the CPU expert tier + the **elastic** VRAM expert cache. Stack: vLLM 0.13 (lazymio/vllm-backport,
DeepSeek-V4.1 support) + Ampere patches + the `vllm_exl3` plugin with exllamav3 kernels, built for sm_86. A few small
source patches go on top, and the image applies them at build time.
Status: **candidate**. The default launch (262,144-token context, 1.5 GiB fp8 KV) was measured end to end as run
**D119** at C1 from 8k to 261k tokens: prefill 589-813 tok/s, decode 16.4-20.9 tok/s (table below). Prefill
fidelity of the staged-DMA path is just outside the campaign's inherited guard (see Quality).
## Current default: D141 (2026-10-03)
D140 + near-full-window requests (agent compaction at ~250k with max_tokens) accepted instead of HTTP 400.
D135 + CPU-assisted prefill for lone 513-2048-token agent turns (6.5-7.8 s instead of 8.3-8.6 s to first token). D135 = D130 below plus a tuned CPU/GPU split (C1/C2/C4 19.8/23.0/25.6) and cross-layer DMA prefetch (prefill 8k/64k/261k
787/769/694 tok/s). See results/D130.md.
### D130
Prefix caching (exact), CPU-tier split path for = 0.988, KL 64 tokens) and refilled with the same experts when decode resumes | whatever is left (8.76 GiB = 709 of 15,360 experts in D107) |
| DDR4 (pinned, GPU-mapped) | all 15,360 routed experts, the home copy that prefill DMA, decode zero-copy reads, the VRAM cache and the CPU tier all read | 190.5 GiB |
| | CPU tier: 22 threads (cores 2-23) compute cold decode misses straight from the pinned copy and add an fp32 partial b