MiaAI-Lab / Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold

Qwen3.8 Flash Next on one DGX Spark (TensorFold) 
 by Mia'a AI Lab 
> **NVIDIA's quantization-aware-distilled (QAD) FP4 checkpoint, close to bf16.** The default model is
> `nvidia/Qwen3.8-Flash-Next-FP4-Local`: NVIDIA trained the
> NVFP4 and MXFP8 weights against the bf16 model as teacher, so the 4-bit weights are tuned to match it. On NVIDIA's own
> numbers it scores 91.5% on GPQA Diamond and 88.7% on ToolEvalBench against 92.2% and 89.4% for bf16 (model card, not
> measured here). It is what `./start.sh` serves; the performance numbers below are for it.
Serve **Qwen3.8 Flash Next** from a single NVIDIA DGX Spark (GB10, 128 GB) through an OpenAI-compatible API. It runs
TensorFold's Zig engine, `tensorfold-native` (branch
`zig-flashnext`), in NVIDIA's PyTorch container. One GPU,
with image and video input (`VISION=1`). `VISION=0` serves text only.
The default checkpoint is NVIDIA's FP4 QAD export: routed experts in NVFP4 (group size 16, CSF-compressed scale factors),
attention and shared-expert linears in MXFP8, the MTP experts and the vision tower quantized too (MXFP8 attention, W4A16
NVFP4 MLP), an FP8 n-gram table, and all 512 routed experts with 10 per token (the INT4 cut keeps 5). The optional INT4-AutoRound checkpoint (`CHECKPOINT=int4`) uses 5 experts per token (4.8B active
parameters) and scores about 10% lower general capability on its authors' own harness (46.6-47.6 against 51.8). Tool use
is unchanged.
- Checkpoint (default): `nvidia/Qwen3.8-Flash-Next-FP4-Local`
 at revision `b9ce18a6887e18c089fdea8509a33e63429a3300` (~96 GiB; loading takes about 81 GiB of memory with vision on, measured).
- Checkpoint (optional, `CHECKPOINT=int4`): `azampatti/Qwen3.8-Flash-Next-125B-A5B-INT4-AutoRound`
 at revision `1464274120d36a4d8fcaa934552334a7d83ce0fd` (~122 GiB). Routed experts are GPTQ int4 (group size 128), the
 n-gram table is FP8 and lives in `ple-table/`, and it is loaded with the weights. There is no SSD n-gram reader and
 no int8 KV cache. KV is FP8 by default (lossy, ~98.8% top-1 agreement with bf16); `KV_DTYPE=bf16` is exact. FP8 works with image and video input.
- API model id: `Qwen3.8-Flash-Next`
- Context: **262,144 tokens** a request (the native window). Up to 5 requests at once (`PARALLEL`). A full window's
 fp8 cache is 4.37 GiB (bf16 is 7.52 GiB). The engine grows each request's cache as it goes and refuses one that would not fit, keeping
 10 GiB free, and another 2 GiB for the vision helper.
- One command: `./start.sh` sets everything up on the first run and starts the server; `./stop.sh` stops it
## Performance
Numbers for the default checkpoint, **`nvidia/Qwen3.8-Flash-Next-FP4-Local`** (NVIDIA's QAD FP4 export), as `./start.sh` serves it: 262,144-token
window, 5 requests at once, fp8 KV, vision on, MTP drafts. One DGX Spark (GB10), measured with
sparkDash's Decode and Prefill benchmarks, running on the same Spark as the server (no network between
them), 400 tokens a request. Aggregate is the sum of the per-request decode sp