MemTensor / MemT-1.0
Official repository for MemT-1.0
MemTensor / MemT-1.0
justxor / deepseek-local-guide-ru
🐋 Полный гайд на русском: как установить и запустить DeepSeek локально (V4, R1) на ПК, Mac и сервере — Ollama, llama.cpp, LM Studio, vLLM, Open WebUI, API, RAG, агенты, fine-tuning, лучшие фишки
eagle-hq / EAGLE-4
Official Inference Code of EAGLE-4
sgl-project / sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
xorbitsai / inference
Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.
ai-dynamo / dynamo
A Datacenter Scale Distributed Inference Serving Framework
OpenDCAI / DataFlow
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
kvcache-ai / Mooncake
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
gpustack / gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
CalvinXKY / InfraTech
分享AI Infra知识&代码练习：PyTorch、vLLM/SGLang、slime/vime框架入门⚡️、性能加速🚀、大模型基础🧠、AI软硬件🔧等
noonghunna / club-3090
Community recipes for serving LLMs on RTX 3090/4090/5090 CUDA gpus. Multi-engine (vLLM, sglang, llama.cpp and other custom engines) and model-agnostic. Currently shipping Qwen3.8-27B Qwen3.6 35B Gemma 4 26B Gemma 4 31B configs for 1×/2×/4x/8x cards.
SemiAnalysisAI / InferenceX
Open Source AI Accelerator Research Platform Standard / 开源推理研究平台
ovg-project / kvcached
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
sybil-solutions / local-studio
Control panel for VLLM, Sglang, llama.cpp, exllamav3
intel / auto-round
PTQ-Level Efficiency, QAT-Level Accuracy|训练后量化的效率，量化感知训练的精度
sgl-project / sglang-omni
SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.
datawhalechina / zero-to-sglang
Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
ModelCloud / GPTQModel
LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.
sgl-project / SpecForge
Train speculative decoding models effortlessly and port them smoothly to SGLang serving.
OpenMOSS / MOVA
A foundation model that generates synchronized video and audio in a single model
Tencent-Hunyuan / UniRL
UniRL is a Framework for Unified Multimodal Model Reinforcement Learning
patchy631 / time-to-first-token
A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking.
XHToken / Spark-X2.6
Spark-x2.5 open model series. Pushing the Limits of Agentic Capabilities in On-Device Models
smg-project / smg
Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.
XYZ-AI-Lab / axrl
AxisRL is an agentic RL post-training framework built on SGLang rollout, Megatron training, and real-world agent workflows.
spark-arena / sparkrun
sparkrun - launch, manage, and stop LLM inference workloads on NVIDIA DGX Spark systems. Live support: https://discord.com/invite/GH5kRgv6ZD
ome-projects / ome
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
MiaAI-Lab / Qwen3.8-27B-SGLang-DGX-Spark
Qwen3.8 27B on SGLang for DGX Spark
amitshekhariitbhu / llm-inference-engineering
Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.
hasso5703 / dgx-spark-qwen38
One command turns a DGX Spark or ASUS Ascent GX10 into a private AI server: seven measured, pinned AI lanes (text up to 1M tokens, images, video with sound), run from a local cockpit on a laptop or a phone. Nothing leaves the box.
VanGong1999 / FDE
更新中｜收集FDE AI前线部署工程师相关全部内容｜从0到1｜ Ontology|AI Agent，Workflows，工作流 ,RAG,知识库，推理优化与部署，vllm与sglang，workflow,graph/loop/conext/prompt工程
kunpengtalk / OmniStudio
OmniStudio 是一个本地大模型一体化桌面工作台，集模型市集下载、llama.cpp/vLLM/SGLang 三引擎推理管理，以及对话、语音合成、ASR语音识别、图片生成、视频生成、OCR 等多种大模型应用于一体，全程本地优先。
jpezzulli / sglang-rtxpro6000
Optimized SGLang runtime for Qwen3.8-27B FP8 with DFlash2 and Qwen3.8 Flash-Next NVFP4 with FR-Spec on one NVIDIA RTX PRO 6000 Blackwell 96 GB GPU (SM120): 524K context, HiCache and NIXL.
Gxj230958 / TritonAscendBench
A migration benchmark that measures whether production GPU Triton kernels — extracted from vLLM and SGLang inference paths — can be ported to the Huawei Ascend 910B (Atlas A2) via `triton-ascend` without losing correctness or performance.
rednote-machine-learning / Inference-autopilot
Evidence-driven SGLang inference deployment optimization.
yanght27 / GPU-Perf-Playground
GPU 性能与 AI Infra 学习项目：CUDA/Triton 算子、NCU/NSYS、vLLM/SGLang/TRT-LLM/ms-swift、PyTorch/DeepSpeed/ms-swift 训练、并行架构
tonyd2wild / Qwen3.8-Flash-Next-NVFP4-DGX-Spark
Qwen3.8-Flash-Next (NVFP4) on DGX Spark in vLLM: one Spark 43.9 tok/s with our disk-backed n-gram table patch, staged gather and reduced-vocab MTP draft; TP2 SPEED 53.7; TP4 CONTEXT 9M KV pool. Launchers, patch, 40-prompt harness, KV ledger, credits. SGLang TP2 day-0 lane kept.
RH-RunningHub / MiniMax-H3-MultiGPU-Lightning
MiniMax-H3 multi-GPU inference acceleration: ~12x on 8x RTX 6000D (step distillation + SageAttention2 + Cache-DiT + torch.compile, TP2+Ulysses4 on sglang)
azrtydxb / Fastllm-proxy
The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLLM/SGLang — 0.76 µs per request, no I/O on the request path, cache-affinity routing, RBAC, budgets and a 13-screen UI in the binary.
htobty / ReadyLLM
Deploy, monitor, and tune LLM through a visual interface — plus image / video generation on top of ComfyUI.
modelsphere / modelsphere
ModelSphere is an open-source LLM inference infrastructure designed to make production-grade model serving simple, efficient, and continuously optimized.
juliankang4 / spark-scope
Read-only dashboard and rack panel for NVIDIA DGX Spark-class nodes, other NVIDIA GPU Linux machines and Apple Silicon Macs running vLLM, SGLang, TensorFold, llama.cpp, Strata or oMLX
sybil-solutions / qwen-serve
Fastest validated sglang recipe for Qwen3.8-27B (AWQ-INT4 + DSpark) on RTX 3090s — 1×/2×/4× GPU presets, image+video, full benchmarks
knapcio / DeepSeek-V4.1-Flash-4x-DGX-Spark-TP4
Tuned TP4 serving profile for DeepSeek-V4.1-Flash on 4x NVIDIA DGX Spark (SGLang, DSpark), built on the MiaAI-Lab recipe
MiaAI-Lab / Qwen3.8-27B-RTX-6000-PRO-SGLang-DSpark
Qwen3.8-27B on SGLang for NVIDIA RTX PRO 6000 (96GB, SM120): DSpark speculative decoding, full 256K context, 8 concurrent requests
baidu-baige / sglang-kunlun
SGLang Kunlun (sglang-kunlun) is a community-maintained hardware plugin designed to seamlessly run SGLang on the Kunlun XPU.
infercrane / infercrane
Open-source infrastructure for the full inference lifecycle: deploy, observe, scale, optimize, and safely release self-hosted models behind one endpoint.
openfabric-systems / simllm
Network-faithful simulation of LLM serving and training deployments
bitcloud-ai / Bittune
Local-first engineering agent that deploys, benchmarks and tunes vLLM/SGLang inference services on your own GPUs, with auditable run evidence.
ormandj / sglang-glm53-flash-sm120
SGLang for GLM-5.3-Flash on two or four RTX PRO 6000 Blackwell GPUs (SM120), with TP2 W4A16 and TP4 NVIDIA NVFP4 profiles.
huoji120 / QWEN-EXO-booster
QWEN-BASE model-native memory and continuous learning system,base on SGLang
intel / gpu-ai-skills
Agent Skills for running, benchmarking, and profiling Hugging Face models on Intel GPUs — PyTorch, vLLM-XPU, and SGLang-XPU
knapcio / GLM-5.3-Flash-4x-DGX-Spark-TP4
GLM-5.3-Flash NVFP4 on 4x NVIDIA DGX Spark: vLLM TP4, DFlash2 adaptive draft, measured FP8 vs NVFP4 vs EXL3 vs SGLang
ormandj / sglang-deepseek-v4-flash-sm120
Reproducible SGLang image for DeepSeek-V4-Flash-0731 on SM120
pangoleen / qwen3.8-27b-dgx-spark-dflash2
Qwen3.8-27B on one DGX Spark: SGLang + DFlash2 recipe, engine image, benchmarks, data and charts. By @redp314
DocPang / qwen38-k100ai-int8-optimization
Qwen3.8-27B INT8/W8A8 optimization for Hygon K100AI: SGLang, TP1/TP4, DFlash2 and long-context Agent workloads
dg1kjd / sglang-sxm2
Agentic-coding oriented inference engine for N x V100-SXM2, based on SGLang: GLM-5.3-Flash, Qwen3.8-Flash-Next, DeepSeek-V4.1-Flash, MiniMax-H3.
gabrielolympie / sglang-flashnext-sm120
Qwen3.8-Flash-Next 180B NVFP4 @ 231 tok/s on one RTX PRO 6000 Blackwell (sm120) - sglang patches, launchers, full optimization analysis
luxingcom / LuZ-0.1.7-DeepSeek-v4.1-Flash-DGXspark-TP4-Ring
DeepSeek-V4.1-Flash on 4x DGX Spark TP4 switchless RoCE ring - SGLang production recipe, 600K context, DSpark MTP, benchmark archive
darkdatter / gb10-repo
Qwen3.8-27B on DGX Spark (GB10): SGLang + NVFP4 + DFlash2. Measured recipe, benchmark harness, and the traps that cost real time.