Soulmate-Halo / heterogeneous-gpu-pd-lab

使用稠密算力卡对稀疏大显存主机进行加速，主机包括 AI Max+ 395、M3 512G、DGX Spark 等。

jadezane6634-ops / dsh-purge-flash

DSH 破甲插件：token 级重试环 + 十片按需叶子（含邻近交付）+ 分级 prefill + 拒答阶梯 L0–L3 + finish_reason 四路分流 + 请求体级整形。零宿主补丁，188 项测试 / 56 条探针。

deepseek-ai / FlashMLA

FlashMLA: Efficient Multi-head Latent Attention Kernels

MaxDevv / lean-qwenflash

Qwen3.8-Flash-Next at full 262k context on 8 GB RAM and 16 GB VRAM—20 tok/s generation and 1,600 tok/s prefill.

Luce-Org / lucebox

LLM speculative inference server for heterogeneous hardware & consumer GPUs

1115714829 / llama.cpp-v100

llama.cpp optimized for NVIDIA Tesla V100 (SM70): Qwen3.8-Flash-Next with MTP (prefill 3402 tok/s at 200K context on 6x V100-16GB; also runs on 2x V100-16GB) and Qwen3.8-27B with DFlash2 (decode up to 220 tok/s); 2-6 GPUs, 262K-512K context, multimodal, concurrent serving, built-in web UI.

jayleaton / deepseek-v41-tensorfold-spark

DeepSeek-V4.1-Flash on 2x NVIDIA DGX Spark with the TensorFold engine: 1.8-1.9x code, 2.5x multi-stream, 1.7-1.95x prefill vs the vLLM kit, exact speculative decoding. Work in progress.

Yinsongxu / LLM2Jev

Turn local language models into Jev-style structured decision models. Get results from text and images with prefill alone—no token-by-token decoding required.

professorpalmer / bonsai-ada-surgery

Bonsai 2 27B: 262k context (trained max) on 12 GB, 54 to 97 tok/s decode, 2x prefill, q8_0 KV recipe, grammar tool calls 9/9. Same ternary weights. Windows bundle + receipts.

sybil-solutions / dsv41-flash-offload

DeepSeek-V4.1-Flash EXL3 on one 24 GB RTX 3090 + DDR4 + NVMe: staged-DMA prefill, AVX2 CPU expert tier, elastic VRAM expert cache, Engram on disk, OpenAI API

SergiioB / intel-arc-pro-b70-inference-cookbook

Open recipes, engine patches, and benchmark harnesses for LLM inference on Intel Arc Pro B60/B70 (Battlemage, Xe2). MoE 35B at 160 t/s decode / 7.5K t/s prefill single-stream, 27B at 50~ t/s decode / 1.7K t/s prefill single stream. vLLM XPU MTP unlocked. Muse Glimmer recipe added!!

DominikBucko / qwen38-flash-next-2x3090

Qwen3.8-Flash-Next on 2× RTX 3090: with 128 GB RAM up to 4,191 tok/s prefill · 111.5 tok/s decode (131K prompt), full 256K window at 2,865 tok/s; with 64 GB RAM 3,410 tok/s prefill · 84 tok/s decode. New: opt-in uncensored mode (runtime abliteration, no new weights).

Nathanw1014 / strix-halo-llamacpp

Performance-tuned llama.cpp for AMD Strix Halo (gfx1151): FA + MoE-prefill fixes with a bundled current Mesa driver. Vulkan and HIP; portable dir, Docker, and distrobox.

architectds / collabosm

A local app that rents a GPU in your own Colab account and serves Qwen3.8 on it: 27B on an A100-40G, or Flash-Next (125B-A6B MoE) on an A100-80G. The cost is shown before it starts; chat with images; Codex or any OpenAI client uses a fixed local /v1. Auto-stop, CU ledger, live prefill/decode. Windows, macOS, Linux.

Rurijian / KimiThinkingPrefill

SillyTavern extension: client-side reasoning_content thinking prefill for Kimi/Moonshot models (kimi-k3-jb patch equivalent, no server modifications)

nisesen / inferenceinquire

Real-time monitoring for llama.cpp inference servers: a read-only host agent plus a web dashboard for hardware, health and inference telemetry (prefill/decode vs context depth, slot occupancy, KV cache, GPU and PCIe).

chadhurley25075-png / pd-bridge

Heterogeneous prefill/decode for DeepSeek-V4-Flash: CUDA prefill (DGX Spark, vLLM) -> Metal decode (Mac Studio, oMLX) over plain 10GbE

allover326 / deepseek-v4-cmp170hx

DeepSeek-V4-Flash-0731 on 4x CMP 170HX (sm_80): 98 tok/s decode, ~5300 tok/s prefill. DSpark speculative decoding under pipeline parallelism, which vLLM does not support upstream.

tonyd2wild / The-Sparky-Command-Center

Plug-and-play, dependency-free web command center for DGX Spark fleets (and any GPU boxes): live per-node GPU/CPU/mem/temps/power, optional RouterOS switch monitoring, and per-model decode/prefill/TTFT/KV cards. Point it at your own hardware.

qhfan / FlashPrefillv2

FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

mohamedhossammohamed / m4-prefill-engine

Low-level Metal inference engine for Apple M4: universal quant formats at MLX parity, 1M-token out-of-core contexts, and speculative decode off flash storage.

NeelM0906 / Mference

Swift + Metal MoE inference for Apple Silicon: Qwen 3.6 35B at 23.5–29.3 tok/s decode with 2.20× faster long-prompt prefill on a 24 GB M5; Gemma 4 26B in ~2 GB, DeepSeek-V4-Flash 284B, Inkling-Small 276B, native Mac app, CLI, and OpenAI-compatible server.