xlite-dev / LeetCUDA
An Open-source book with Modern CUDA learning Notes for Beginners - BF16/FP8/FP4, HGEMM, CuTe, Flash-Attention, etc.
xlite-dev / LeetCUDA
NVlabs / Sana
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformer
FareedKhan-dev / kimi-k3-in-c
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
NVIDIA / TransformerEngine
A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.
NVlabs / LongLive
Long Video Gen / World Model / World Action Model (Embodied) Infrastructure
intel / neural-compressor
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime
Tencent / AngelSlim
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
intel / auto-round
PTQ-Level Efficiency, QAT-Level Accuracy|่ฎญ็ปๅ้ๅ็ๆ็๏ผ้ๅๆ็ฅ่ฎญ็ป็็ฒพๅบฆ
microsoft / Tutel
Tutel MoE: Optimized Mixture-of-Experts Library, Support GLM/Kimi/Gpt/DeepSeek/Qwen using FP8/NVFP4/MXFP4
NVIDIA / cudnn-frontend
cuDNN Frontend is NVIDIA's modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels.
Avarok-Cybersecurity / atlas
Pure Rust Inference Engine
kacper-daftcode / vLLM-Moet
A vLLM patch + handโwritten SM120 SASS kernels: 2โbit MoE experts + an FP4 "delta" cache that recovers precision โ matching the official (NV)FP4 checkpoint's quality on consumer Blackwell cards
MiaAI-Lab / Qwen3.8-Flash-Next-Dual-DGX-Sparks
Qwen3.8-Flash-Next-NVFP4-vLLM-Dual-DGX-Spark
julianmb / q38rocm
Qwen 3.8 27B ROCmFP4 on AMD Strix Halo (Ryzen AI Max+ 395). Up to 36 tok/s via MTP Speculation, TurboQuant & Mesa RADV Wave64.
tonyd2wild / GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark
GLM-5.3-Flash (NVFP4) on 2x NVIDIA DGX Spark - vLLM TP2, 262K context, MTP. World-first deploy recipe: 7 day-0 bugs found and fixed, patched sm121 image, probes and full report.
hasso5703 / dgx-spark-qwen38
One command turns a DGX Spark or ASUS Ascent GX10 into a private AI server: seven measured, pinned AI lanes (text up to 1M tokens, images, video with sound), run from a local cockpit on a laptop or a phone. Nothing leaves the box.
mmastrac / djev-spark
DiffusionGemma NVFP4 structured decisions on a DGX Spark: container recipe
sergqwer / strata-nvfp4
Strata fork: Qwen3.8-Flash-Next 125B MoE in NVFP4 on one RTX 20-50 card (12 GB+) and 64 GB of RAM or more. Setup installs our GPTQ quants (huihui abliterated, OrcaRouter) from Hugging Face. CPU experts on AVX-512 or AVX2 (+AVX-VNNI), all experts in RAM from 92 GB, a low-RAM mode below, vision, W4A8 prompts. Windows / CUDA 13.
GGZ14 / vllm-mxfp4
The fastest vLLM build for AMD R9700's. For the latest updates join the Launch80 discord server. I am on X now also https://x.com/bkuyper
dnv2003 / v100-skinny
Hand-written NVFP4 W4A16 CUDA kernels for Volta
jpezzulli / sglang-rtxpro6000
Optimized SGLang runtime for Qwen3.8-27B FP8 with DFlash2 and Qwen3.8 Flash-Next NVFP4 with FR-Spec on one NVIDIA RTX PRO 6000 Blackwell 96 GB GPU (SM120): 524K context, HiCache and NIXL.
0xSero / deepseek-v4-flash-0731-spark-sparkinfer
Pinned one-DGX-Spark Docker recipe for DeepSeek V4 Flash with EXL3, SparkInfer, and 262K NVFP4 MLA KV cache
truespar / paddock
Native Rust inference server for open models on NVIDIA and Apple Silicon. OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio
tonyd2wild / DeepSeek-V4.1-Flash-vLLM-DGX-Spark
DeepSeek-V4.1-Flash (552B MoE, MXFP4 experts, 1M ctx) on four NVIDIA DGX Sparks with vLLM TP4: Engram-on-disk patch, sm121 kernel build, launchers, measured numbers
tonyd2wild / Qwen3.8-Flash-Next-NVFP4-DGX-Spark
Qwen3.8-Flash-Next (NVFP4) on DGX Spark in vLLM: one Spark 43.9 tok/s with our disk-backed n-gram table patch, staged gather and reduced-vocab MTP draft; TP2 SPEED 53.7; TP4 CONTEXT 9M KV pool. Launchers, patch, 40-prompt harness, KV ledger, credits. SGLang TP2 day-0 lane kept.
alexellis / glm-5.3-flash-4x-dgx-spark-switchless
GLM-5.3-Flash (NVFP4) at TP4 across 4x DGX Spark via a switchless RoCE ring + DFlash2 โ reproducible recipe
MiaAI-Lab / Qwen3.8-27B-NVFP4-RTX-5090
Qwen3.8-27B-NVFP4 on a Single RTX 5090 (vLLM)
julianmb / halofpx
High-performance Lemonade alternative for AMD Strix Halo & Radeon โ latest MoE models (Ling-3.0-Flash, Qwen 3.8 Flash Next, Ornith 1.5), bleeding-edge RDNA 3.5 Wave64/ROCmFP4 kernels, and silicon-tuned profiles.
MiaAI-Lab / GLM-5.3-Flash-NVFP4-Dual-DGX-Spark
2x DGX Spark recipe for LibertAIDAI/GLM-5.3-Flash-NVFP4 (NVFP4, Ray TP=2, multimodal)
kindlingai / glm-5.3-flash-gx10
GLM-5.3-Flash (NVFP4) at TP=4 across 4x ASUS GX10 / GB10 - vLLM + mentat instead of Ray, 524k context, DFlash2. Minimal working recipe.
atbender / deepseek-v41-flash-mac-mini
Run original DeepSeek V4.1 Flash FP4/FP8 weights on a 16 GB M1 Mac mini with bounded SSD streaming in MLX. Recipe, benchmarks, and demo.
MiaAI-Lab / GLM-5.2-NVFP4-AQLM-Triple-DGX-Sparks
GLM-5.2 NVFP4+AQLM on 3ร DGX Spark โ 380k context MTP serve stack
headpiece747 / ninfer-5090-windows
Native Windows port of NInfer engine for RTX 5090. Features Qwen3.8-27B with QUASAR and NInfer models, MTP/DFlash2 with vision and 262,144 context
Enntity / sparkglm
Fast, capable GLM-5.3-Flash FP4-Local on two DGX Sparks: opinionated Atlas serving and reproducible inference research.
0xBakeer / TandemLLM
An environment where open models work as agents: a first-level harness with routing, skills, memory and sandboxed tools, on an engine that drafts and verifies its own tokens.
tonyd2wild / GLM-5.3-Flash-NVFP4-1M-KV-4x-DGX-Spark
GLM-5.3-Flash (320B MoE) at TP4 across four DGX Sparks, same day as the model drop: 36 tok/s, 1.26M-token FP8 KV pool, 262K context, MTP spec decode. First TP4 glm5_next outside B200 hardware. Full patched-image recipe + the GB10 memory study.
0xBakeer / Qwen3.8-27B-4-bit-on-a-single-DGX-Spark
Serving 4-bit Qwen3.8-27B on a single DGX Spark (GB10): 75 tok/s single-stream, 246 tok/s aggregate at 8-way concurrency. NVFP4 vs MixedInt4-AutoRound vs the FP8 baseline, measured on one harness โ including why the quantization advantage collapses to +0.2% by c16.
ormandj / sglang-glm53-flash-sm120
SGLang for GLM-5.3-Flash on two or four RTX PRO 6000 Blackwell GPUs (SM120), with TP2 W4A16 and TP4 NVIDIA NVFP4 profiles.
seanyourhighness / vllm-sm12x-nvfp4-dflash2
All-NVFP4 vLLM + DFlash2 K7 for Blackwell (SM120/SM121): NVFP4 target/draft/KV, 262K context, optional CPU vision sidecar.
knapcio / GLM-5.3-Flash-4x-DGX-Spark-TP4
GLM-5.3-Flash NVFP4 on 4x NVIDIA DGX Spark: vLLM TP4, DFlash2 adaptive draft, measured FP8 vs NVFP4 vs EXL3 vs SGLang
satellitedown / fast-long-context-cinference
One-menu RTX 5090 setup for non-Swift Huihui Qwen3.8-27B NVFP4 with Cinference, MTP-10 and K8V4.
MiaAI-Lab / Ornith-1.5-35B-A3B-DGX-Spark
Serve Ornith-1.5-35B-A3B (NVFP4 + in-checkpoint MTP, 256k ctx) on one DGX Spark โ vLLM b12x recipe plus CUDA-graph fixes.
dg1kjd / sglang-sxm2
Agentic-coding oriented inference engine for N x V100-SXM2, based on SGLang: GLM-5.3-Flash, Qwen3.8-Flash-Next, DeepSeek-V4.1-Flash, MiniMax-H3.
tsaipifong / whirl-llm
WHIRL (Windows HIP Inference for RDNA LLMs): native Windows C++/HIP LLM inference engine for the AMD Ryzen AI Max+ 395 (Radeon 8060S) and Radeon AI PRO R9700. Unzip and run, only the AMD driver.
r0b0tlab / DeepSeek-V4-Flash-DSpark-v026-SM121
DeepSeek-V4-Flash-DSpark optimized vLLM 0.26.0 SM121 dual-GB10 evidence (NVFP4 KV, B12X, DSpark K6)
tonyd2wild / DeepSeek-v4-Flash-0731-Vision-DSpark-1M-NVFP4-KV-2x-DGX-Spark
Give DeepSeek V4 Flash eyes. OpenAI-compatible shim + tiny local VLM. Point any harness at one URL and your text-only model accepts images. Zero dependencies.
Weschera / Qwen3.8-27B-NVFP4-DFlash2-DGX-Spark
Reproducible Qwen3.8-27B NVFP4 + DFlash2 recipe and qualification evidence for one NVIDIA DGX Spark
gabrielolympie / sglang-flashnext-sm120
Qwen3.8-Flash-Next 180B NVFP4 @ 231 tok/s on one RTX PRO 6000 Blackwell (sm120) - sglang patches, launchers, full optimization analysis
othexmr / GLM-5.3-Flash-NVFP4-2x-4x-DGX-Sparks-RiNGSiDE
GLM-5.3-Flash NVFP4 on two or four DGX Sparks cabled as a ring, with no switch: the RiNGSiDE serving recipe
drowzeys / keys-vLLm.0.27.1-GLM-5.3-Flash-NVFP4-NVFP4KV-1M-Context-Abliterated
GLM-5.3-Flash (Z.ai) NVFP4 (LibertAI) + NVFP4 KV, 1M context on 2x DGX Spark GB10 โ vLLM TP2 recipe, walls ledger, benchmarks, needle to 1.04M; abliteration with blackforest_AI next
Hitheshkaranth / Ornith-1.5_A3B_Model_DGX_Spark_Setup
Ornith-1.5-35B-A3B (NVFP4 + FP8 + MTP) on NVIDIA DGX Spark (GB10)
WestWaters / pollard-weights
A tiered-memory system design for workloads that don't fit in RAM: measure the working set, pin the hot tier, stream the cold tier from flash. Ships the residency calculator, measurement harnesses, and the build recipes behind it. Predictions validated against public benchmarks.
gpillon / ignis
OpenAI-compatible inference server for Qwen3.8 on one RTX 5090 / RTX PRO 6000: the 27B at ~170 tok/s (peaks ~370), Flash-Next (125B MoE) at ~100 tok/s, built for local coding agents. Rust + CUDA, Windows and Linux.
nguyenthimy2022kg-alt / Qwen-Flash-SM80-170HX
ๅ SM80 / CMP 170HX ็ Qwen3.8-Flash-Next ็คพๅบ vLLM๏ผNVIDIA NVFP4ใGDSใTEP2ใMTP6๏ผๆฏๆ 1๏ฝ10 ๅนถๅ๏ผ่ฏ่กจไธ PCIe ้ไฟกไผๅใไธญ่ฑๆ้จ็ฝฒไธๅฏๆๅปบๆบ็ ใ
r0b0tlab / qwen38-27b-nvfp4-sm121-vllm
Qwen3.8-27B NVFP4 (W4A16 shipped recipe) + MTP on NVIDIA DGX Spark GB10/SM121 โ vLLM v0.27.2rc0, FP8 KV, 262K context, full reproducibility pack
dolf3131 / dgx-spark-llm-serving
Serving large LLMs on a single DGX Spark (121 GiB): Qwen3.8-Flash-Next on TensorFold/SparkFold EXL3 and vLLM NVFP4, Nemotron 3.5 Super VL on vLLM; what ran, what it measured, what did not work
darkdatter / gb10-repo
Qwen3.8-27B on DGX Spark (GB10): SGLang + NVFP4 + DFlash2. Measured recipe, benchmark harness, and the traps that cost real time.