deepseek-ai / DeepInfer
DeepInfer: High-performance, large-scale inference framework for DeepSeek models
deepseek-ai / DeepInfer
Comfy-Org / ComfyUI
Node-graph GUI and Python backend for running diffusion models to generate images, video, 3D, and audio locally.
lyogavin / airllm
Python library that runs large language model inference layer by layer, so 70B-class models fit on a single 4GB GPU without quantization.
Niko1221 / Strata
Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
vllm-project / vllm
A high-throughput LLM inference and serving engine featuring PagedAttention, continuous batching, and an OpenAI-compatible API server.
cheaper-inference / OmniRoute
Never stop coding. Free MIT AI gateway: one endpoint, 375 providers (150+ free), 1,400+ models Kimi, Claude, GPT, Gemini, GLM, DeepSeek, MiniMax. Works with Claude Code, Codex, Cursor, OpenCode, Cline & Copilot. Quota-aware auto-fallback, RTK+Caveman compression saves 15-95% tokens, MCP/A2A, Desktop/PWA. Built by 900+ contributors
mw00 / project-maya
Project Maya - run GLM-5.3-Flash (321B MoE) and DeepSeek-V4.1-Flash-1017 (748B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.
StartLuxLabs / StartLux-Decision
Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.
securetron-gh / aifw
AI Agent Identity and Access Management and Inference Firewall
eagle-hq / EAGLE-4
Official Inference Code of EAGLE-4
surya-koritala / Glyd
Lossless AI compression: 33% less GPU memory, bit for bit.
ggml-org / llama.cpp
LLM inference in C/C++
gitleaks / gitleaks
Scans git repos, files, and stdin for hardcoded secrets such as passwords, API keys, and tokens.
incoai / splash
A local inference engine for Apple silicon, built around the model.
mnfst / awesome-free-llm-apis
A curated list of LLM APIs that offer permanent free tiers for text inference, with base URLs, models, and rate limits.
sgl-project / sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
NVIDIA / flare-local-inference
—
trymirai / uzu
Inference engine for running AI models locally inside apps, with Rust core and Swift, Python, and TypeScript bindings.
hanzohanzhe / Grandet
Find the cheapest API for any model and connect it from your AI agent: Grandet Skill and an anonymous MCP server, backed by listed prices from 1,800+ official, inference and relay providers on grandet.ai.
nace-ai / kapitsa
Blazing fast Apple Silicon optimized inference engine built for Qwen and Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai
Indras-Mirror / Xolotl
Xolotl (formerly Strata-DSV4.1): DeepSeek-V4.1-Flash, all 552B, on one gaming PC (RTX 4090 + 90 GB RAM) at ~27 tok/s, plus GLM-5.3-Flash, Qwen3.8-Flash-Next and Qwen3.8-27B. A heavily extended fork of Strata: MoE experts tiered across VRAM, pinned RAM and NVMe; OpenAI + Anthropic API.
ddalcu / mlx-serve
Native macOS and Apple Silicon inference server exposing OpenAI, Anthropic and Ollama APIs, with a Swift app, written in Zig.
bojieli / ai-infra-book
《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
google / agentmesh
Agent Mesh is a private network that lets AI agents publish and discover tools and models, with verifiable node identities.
justxor / deepseek-local-guide-ru
🐋 Полный гайд на русском: как установить и запустить DeepSeek локально (V4, R1) на ПК, Mac и сервере — Ollama, llama.cpp, LM Studio, vLLM, Open WebUI, API, RAG, агенты, fine-tuning, лучшие фишки
black-forest-labs / flux-3-image-klein
FLUX 3 Image Klein: reference inference code
apple-aiml-research / ml-rsi-inference-bench
A Long-Horizon Benchmark for Self-Improving Inference Systems
NVIDIA / TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
ChenLiu-1996 / figures4papers
My Python scripts to make high-quality figures for publications in top AI conferences and journals.
magnitudedev / magnitude
Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, or just a CPU.
hao-ai-lab / FastVideo
A unified inference and post-training framework for accelerated video generation.
amitshekhariitbhu / ai-engineering-interview-questions
Your Cheat Sheet for AI Engineering Interview – Questions and Answers.
ovg-project / kvcached
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
T-Crypt / berth
Decides who gets your GPU. Chat, image and voice engines lease the card in turn instead of grabbing it, so two of them never load at once and crash each other. One Go binary, nothing else to install.
huggingface / transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
FlashML-org / FreeToken
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
FlashML-org / FreeVideo
Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.
fudan-generative-vision / Remory
Remory: Learning Residual Memory for Context Compaction
Lightricks / LTX-2
Official Python inference and LoRA trainer package for the LTX-2 audio–video generative model.
Human-Agent-Society / reef
Infrastructure for continually self‑improving agents
ggml-org / whisper.cpp
Port of OpenAI's Whisper model in C/C++
drumih / turbo-fieldfare
Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook
llm-d / llm-d
State of the art inference performance with modern accelerators on Kubernetes
syv-ai / HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
SYSTRAN / faster-whisper
Faster Whisper transcription with CTranslate2
modelscope / FunASR
Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
HalfVulpes / Deepinfer-SM80
Deepinfer inspired implemetation on SM80 Ampere architecture. Deepest respect for DeepSeek opensource this work!
sgl-project / sglang-omni
High Performance Serving Runtime For Interactive Super Intelligence System
facebookresearch / segment-anything
Inference code, checkpoints, and notebooks for Meta's Segment Anything Model (SAM), which produces object masks from points, boxes, or automatically for whole images.
facebookresearch / audiocraft
PyTorch library for audio generation research, providing inference and training code for MusicGen, AudioGen, and EnCodec.
ashhart / TensorFold
Serves language models from a Zig binary on Apple Silicon and NVIDIA GPUs using Metal and CUDA kernels.
VecSzn / PolyStrata
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.
microsoft / BitNet
Official C++ inference framework for 1-bit (1.58-bit) LLMs, providing optimized CPU and GPU kernels.
google-ai-edge / mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
antirez / ds4
DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm
omnimind-ai / OmniInfer
Unified, efficient, and easy-to-use inference infrastructure for edge AI || 面向端侧 AI 的统一、高效、易用的推理基础设施
Lore-Hex / token-exchange-neocloud-in-a-box
Open-source confidential GPU inference cloud in a box: Intel TDX + NVIDIA confidential-computing nodes for DeepSeek-V4.1-Flash and GLM-5.3, verifiable and listed on TrustedRouter
IPADS-SAI / Mobiinfer-Android
MobiInfer runs 35B-A3B-class MoE models fully on-device on the Qualcomm Hexagon NPU, streaming experts from flash with lookahead scheduling (LES) and DSpark speculation.
Elevatormusic / DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark
DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw on 2x NVIDIA DGX Spark (GB10) with TensorFold: 2.1-2.3x the vLLM kit's decode speed.
jundot / omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar