NVIDIA / spark-local-inference
git clone https://github.com/NVIDIA/spark-local-inference.git
Local LLM inference server for NVIDIA SM120/SM121 GPUs, with a Python build step and a Rust serving runtime.
Why you might care
Aimed at long conversations, coding agents, and repeated tool use on workstations and compact systems like DGX Spark. Models are compiled and CUDA graphs captured ahead of time, then served from Rust without Python at runtime. It trades vLLM's model and hardware breadth for a focused serving contract on selected models.
+9
stars, 24h · #4,291
+140
stars, 7 days · #2,040
+695
stars, 30 days · #1,450
695
★ total · +25% this week
75
forks · 63 open issues
Star history (as recorded here)
LanguageRust
LicenseApache-2.0
Created2026-10-20 (8d ago)
Last pushtoday
Watchers1
OwnerOrganization
Links
README
Loading README…
Blurb generated by claude-haiku-5-5 3d ago. Star deltas marked ~ are extrapolated from partial history.