jayleaton / glm53-tensorfold-spark

Follow me on X for more updates: https://x.com/jayleaton
Support me here: https://buymeacoffee.com/jayleaton
# GLM-5.3-Flash on TensorFold, 2x NVIDIA DGX Spark
Serve GLM-5.3-Flash (the abliterated EXL3 4-bit checkpoint
`neko-legends/GLM-5.3-Flash-Uncensored-EXL3`)
across two NVIDIA DGX Sparks, tensor-parallel over the 200 Gb/s CX7 link, behind an OpenAI-compatible API. The
engine is TensorFold (pinned, unmodified submodule) plus 77 patches applied
at image build: 4-bit non-expert weights, image input (the checkpoint's own vision tower), structured output, tool-calling fixes, a latent (absorbed MLA) KV cache, fast chunked prefill, a multi-session
state cache with an NVMe tier, batching of up to 4 requests over a shared 1M-token KV pool, FP8 KV storage,
shared system-prompt reuse, a RoCE all-gather, fast restarts, deeper drafting and verify windows, and ops tooling.
Every patch is off by default; the configs in `config/` turn on the measured set, and
`docs/PATCHES.md` records the ones that were measured and not adopted.
> **Work in progress.** This is an experimental setup, measured on one pair of Sparks. Knobs, defaults, APIs and
> numbers may change between commits. Read Limits before relying on it.
Weights attribution (the checkpoint's license requires it): the weights are by **Local Inference Lab, Inc.**
( ), upstream source
 , under the ShapleyMCG License 1.0. They are not
included here. See Licensing.
SPDX-License-Identifier: Apache-2.0 (this project's own code, scripts, benchmarks and docs; see Licensing).
## Tool calling (W21)
Both tool-calling benchmarks from issue #6 on production (image b11, our abliterated checkpoint), with the 0620 fixes
on and off and thinking off / high / low; one run per configuration. Details:
`docs/TOOL-CALLING.md` §7, `docs/RESULTS.md` W21.
| configuration | tool-eval-bench | multi-step chains | spark-bench TrueScore | agentic |
|---|---:|---:|---:|---:|
| fixes off, thinking off | 90 | 8/8 | 94.0 | 90.4 |
| fixes on, thinking off | 90 | 8/8 | 91.4 | 90.4 |
| fixes on, thinking high | 91 | 8/8 | 91.0 | 94.9 |
| fixes on, thinking low (chains + agentic only) | - | 8/8 | - | **97.3** |
For agent clients: **thinking on at effort low** (best agentic score, fastest agentic runs). The fixes stay on; the
fixes-off TrueScore lead is one safety scenario whose history renders differently without them (explained in §7).
The independent tester's run (different checkpoint): tool-eval-bench 90, chains 6/8, TrueScore 92.4, agentic 82.6.
## What's new (W20)
Patch 0620 (tool-calling fixes, 77 patches in total), test window W20 and a new 3-run RigMark. Production config
(`config/prod.env.example`): image b11 = image b10's list + 0620, with `GLM53_TF_TOOL_FIXES=all`. Numbers from
`docs/RESULTS.md` W20.
Host-side tuning outside this repo: +4.8% single-stream decode, +4.3% at 4 streams, more memory headroom (stress-test
memory minimum now ~10.7 GiB). Same bits (reply hashes unchanged).
**Not adopted:** pinning the server containers o