justxor / deepseek-local-guide-ru
🐋 Полный гайд на русском: как установить и запустить DeepSeek локально (V4, R1) на ПК, Mac и сервере — Ollama, llama.cpp, LM Studio, vLLM, Open WebUI, API, RAG, агенты, fine-tuning, лучшие фишки
justxor / deepseek-local-guide-ru
Elevatormusic / DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark
DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw on 2x NVIDIA DGX Spark (GB10) with TensorFold: 2.1-2.3x the vLLM kit's decode speed.
eagle-hq / EAGLE-4
Official Inference Code of EAGLE-4
vllm-project / vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
BerriAI / litellm
The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM]
modelscope / FunASR
Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
budtmo / docker-android
Android in docker solution with noVNC supported, video recording, mcp server and AI-agent
Orchestra-Research / AI-Research-SKILLs
Comprehensive open-source library of AI research and engineering skills for any AI model. Package the skills and your claude code/codex/gemini agent will be an AI research agent with full horsepower. Maintained by Orchestra Research.
halfrost / Halfrost-Field
✍🏻 Source Code Deep Dives, System Design & Engineering Blogs | Halfrost-Field 冰霜之地：源码解析、系统设计与工程实践笔记
LMCache / LMCache
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
OpenRLHF / OpenRLHF
An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)
xorbitsai / inference
Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.
ai-dynamo / dynamo
A Datacenter Scale Distributed Inference Serving Framework
OpenDCAI / DataFlow
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
vllm-project / vllm-omni
A framework for efficient model inference with omni-modality models
kvcache-ai / Mooncake
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
vllm-project / semantic-router
An open, programmable decision layer for models and compute.
kserve / kserve
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
mostlygeek / llama-swap
Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc
gpustack / gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
OpenBMB / UltraRAG
A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines
xlite-dev / Awesome-LLM-Inference
📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉
NVIDIA / Model-Optimizer
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
katanaml / sparrow
Structured data extraction, instruction calling and agentic workflows with ML, LLM and Vision LLM
vllm-project / aibrix
Cost-efficient and pluggable Infrastructure components for GenAI inference
tencentmusic / cube-studio
cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库，AI模型市场，支持国产cpu/gpu/npu 昇腾生态，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano等分布式
skyzh / tiny-llm
learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen
CalvinXKY / InfraTech
分享AI Infra知识&代码练习：PyTorch、vLLM/SGLang、slime/vime框架入门⚡️、性能加速🚀、大模型基础🧠、AI软硬件🔧等
lemony-ai / cascadeflow
Cascading runtime for AI agents. Optimize cost, latency, quality, and policy decisions inside the agent loop.
vllm-project / llm-compressor
State-of-the-art LLM compression, built for production inference with vLLM
PaddlePaddle / FastDeploy
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
containers / ramalama
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
vllm-project / vllm-ascend
Community maintained hardware plugin for vLLM on Huawei Ascend
vllm-project / production-stack
vLLM’s reference system for K8S-native cluster-wide deployment with community-driven performance optimization
data-infra / cube-studio
cubestudio开源云原生一站式机器学习/深度学习/大模型AI平台/MaaS/mlops/人工智能平台/训推平台，算法全链路流程，多租户，算力租赁平台，token中转，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务，VGPU虚拟化，云边端协同，边缘计算，自动化标注平台，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库llmops智能体，AI模型市场，支持国产异构算力调度,昇腾/寒武纪/海光/摩尔/沐曦等，支持ib/roce/RDMA，信创支持
eugr / spark-vllm-docker
Docker configuration for running VLLM on dual DGX Sparks
noonghunna / club-3090
Community recipes for serving LLMs on RTX 3090/4090/5090 CUDA gpus. Multi-engine (vLLM, sglang, llama.cpp and other custom engines) and model-agnostic. Currently shipping Qwen3.8-27B Qwen3.6 35B Gemma 4 26B Gemma 4 31B configs for 1×/2×/4x/8x cards.
syv-ai / HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
vllm-project / vllm-metal
Community maintained hardware plugin for vLLM on Apple Silicon
SemiAnalysisAI / InferenceX
Open Source AI Accelerator Research Platform Standard / 开源推理研究平台
ovg-project / kvcached
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
sybil-solutions / local-studio
Control panel for VLLM, Sglang, llama.cpp, exllamav3
vllm-project / guidellm
Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
intel / auto-round
PTQ-Level Efficiency, QAT-Level Accuracy|训练后量化的效率，量化感知训练的精度
waybarrios / vllm-mlx
High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.
QwenAudio / Fun-ASR
Fun-ASR speech recognition models, with native Hugging Face Transformers support for Fun-ASR-Nano and separate FunASR, vLLM and llama.cpp deployment paths.
nokia-applied-research / AnyJev
Turn any LLM into a Jev-style decision model: typed decisions, real probabilities, no training. (continue updating, welcome any issue and PR request)
1CatAI / 1Cat-vLLM
V100 / SM70-focused vLLM engineering fork for modern LLM inference.
verl-project / verl-omni
Multimodal RL training framework for diffusion & omni models
kubeai-project / kubeai
AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.
ModelCloud / GPTQModel
LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.
ENTERPILOT / GoModel
AI gateway / AI control plane / AI proxy written in Go. Unified OpenAI-compatible and Anthropic-compatible API for OpenAI, Anthropic, Gemini, Groq, xAI, Ollama, vLLM and more. A LiteLLM alternative with observability, guardrails, streaming, cost tracking, intelligent routing, sticky sessions, failover, real-time logs and usage tracking. Prod ready.
weicj / vLLM-2080Ti-Definitive
The definitive vLLM runtime for dual RTX 2080 Ti 22GB + NVLink, delivering Qwen 27B local inference with maximum 200+ tok/s single-request decode with support of FP8 weight ( Join Discord :https://discord.gg/VFqVVySdMS )
jmaczan / tiny-vllm
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
vllm-project / recipes
Common recipes to run vLLM
Tencent-Hunyuan / UniRL
UniRL is a Framework for Unified Multimodal Model Reinforcement Learning
patchy631 / time-to-first-token
A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking.
CurvineIO / curvine
AI-Native & Cloud-Native FS: A high-performance file semantic layer for cloud object storage, integrated with high-speed cache. CNCF Sandbox Project.
StartLuxLabs / StartLux-Decision
Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.
vllm-project / speculators
A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM