deepseek-ai / DeepInfer
DeepInfer: High-performance, large-scale inference framework for DeepSeek models
deepseek-ai / DeepInfer
Comfy-Org / ComfyUI
Node-graph GUI and Python backend for running diffusion models to generate images, video, 3D, and audio locally.
Niko1221 / Strata
Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
cheaper-inference / OmniRoute
Never stop coding. Free MIT AI gateway: one endpoint, 375 providers (150+ free), 1,400+ models Kimi, Claude, GPT, Gemini, GLM, DeepSeek, MiniMax. Works with Claude Code, Codex, Cursor, OpenCode, Cline & Copilot. Quota-aware auto-fallback, RTK+Caveman compression saves 15-95% tokens, MCP/A2A, Desktop/PWA. Built by 900+ contributors
lyogavin / airllm
Python library that runs large language model inference layer by layer, so 70B-class models fit on a single 4GB GPU without quantization.
vllm-project / vllm
A high-throughput LLM inference and serving engine featuring PagedAttention, continuous batching, and an OpenAI-compatible API server.
StartLuxLabs / StartLux-Decision
Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.
eagle-hq / EAGLE-4
Official Inference Code of EAGLE-4
colinhacks / reylo
Import .sql files into TypeScript with full type inference and zero codegen.
notinferred / keypaste
A KeePass-compatible password and secrets manager for people who work with AI agents.
mw00 / project-maya
Project Maya - run GLM-5.3-Flash (321B MoE) and DeepSeek-V4.1-Flash-1017 (748B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.
Generated-Computers / Locals
Run many AI agents at once on your Mac, up to 3x faster.
NVIDIA / flare-local-inference
—
ggml-org / llama.cpp
LLM inference in C/C++
mnfst / awesome-free-llm-apis
A curated list of LLM APIs that offer permanent free tiers for text inference, with base URLs, models, and rate limits.
sgl-project / sglang
SGLang is an open-source inference framework for serving large language, vision-language, and diffusion models.
lightseekorg / tokenspeed
TokenSpeed is an LLM inference engine built for agentic workloads, with a C++ scheduler control plane and Python execution plane.
incoai / splash
A local inference engine for Apple silicon, built around the model.
nace-ai / kapitsa
Blazing fast Apple Silicon optimized inference engine built for Qwen & Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai
hanzohanzhe / Grandet
Find the cheapest API for any model and connect it from your AI agent: Grandet Skill and an anonymous MCP server, backed by listed prices from 1,800+ official, inference and relay providers on grandet.ai.
gitleaks / gitleaks
Scans git repos, files, and stdin for hardcoded secrets such as passwords, API keys, and tokens.
bojieli / ai-infra-book
《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
huggingface / transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
VecSzn / PolyStrata
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.
llm-d / llm-d
State of the art inference performance with modern accelerators on Kubernetes
trymirai / uzu
Inference engine for running AI models locally inside apps, with Rust core and Swift, Python, and TypeScript bindings.
TrenTorch / TrenTorch
Don't memorize ML. Understand it from first principles including Inference, Cuda, Deep Learning, Data Science and Reinforcement Learning
fudan-generative-vision / Remory
Remory: Learning Residual Memory for Context Compaction
FlashML-org / FreeVideo
Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.
securetron-gh / aifw
AI Agent Identity and Access Management and Inference Firewall
SYSTRAN / faster-whisper
Faster Whisper transcription with CTranslate2
ggml-org / whisper.cpp
Port of OpenAI's Whisper model in C/C++
Indras-Mirror / Xolotl
Xolotl (formerly Strata-DSV4.1): DeepSeek-V4.1-Flash, all 552B, on one gaming PC (RTX 4090 + 90 GB RAM) at ~27 tok/s, plus GLM-5.3-Flash, Qwen3.8-Flash-Next and Qwen3.8-27B. A heavily extended fork of Strata: MoE experts tiered across VRAM, pinned RAM and NVMe; OpenAI + Anthropic API.
recursive-oss / rec
Our lightning-fast inference engine.
FlashML-org / FreeToken
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
magnitudedev / magnitude
Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, or just a CPU.
syv-ai / HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
justxor / deepseek-local-guide-ru
🐋 Полный гайд на русском: как установить и запустить DeepSeek локально (V4, R1) на ПК, Mac и сервере — Ollama, llama.cpp, LM Studio, vLLM, Open WebUI, API, RAG, агенты, fine-tuning, лучшие фишки
antirez / ds4
DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm
hao-ai-lab / FastVideo
A unified inference and post-training framework for accelerated video generation.
Human-Agent-Society / reef
Infrastructure for continually self‑improving agents
apple-aiml-research / ml-rsi-inference-bench
A Long-Horizon Benchmark for Self-Improving Inference Systems
T-Crypt / berth
Decides who gets your GPU. Chat, image and voice engines lease the card in turn instead of grabbing it, so two of them never load at once and crash each other. One Go binary, nothing else to install.
ZiyuanDu / deepinfer-unpacked
Bringing thousands of learners together to unpack DeepInfer through peer-reviewed explanations, documentation, and hands-on examples.
NVIDIA / TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
ddalcu / mlx-serve
Native macOS and Apple Silicon inference server exposing OpenAI, Anthropic and Ollama APIs, with a Swift app, written in Zig.
Phocinae / Phocinae-Largha-150M-v1
斑海豹 — Phocinae-Largha-150M-v1: a 144.3M bilingual decision model. Tiny model. Big decisions. 小海豹，大决断。
Lightricks / LTX-2
Official Python inference and LoRA trainer package for the LTX-2 audio–video generative model.
IPADS-SAI / Mobiinfer-Android
MobiInfer runs 35B-A3B-class MoE models fully on-device on the Qualcomm Hexagon NPU, streaming experts from flash with lookahead scheduling (LES) and DSpark speculation.
0xShug0 / audio.cpp
An all-in-one, pure C++ inference engine for audio models, powered by ggml. Supports TTS, STT, VAD, voice conversion, music generation, and more, with highly optimized performance. No Python dependency.
meta-pytorch / dist_moe
Dist-MoE implements Speed-of-Light distributed Mixture-of-Experts forward and backward kernels. They are highly optimized and fused comm-compute kernels for training and inference on SM100+ GPUs
jundot / omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
florinp93 / hells-gate-recomp
Xbox 360 to PC port of Dante's Inferno using ReXGlue (static recompilation)
deepseek-ai / FlashMLA
Optimized GPU and NPU attention kernels from DeepSeek for sparse prefill and FP8/FP4 KV-cache decoding in DeepSeek-V4.1 inference.
ashhart / TensorFold
Serves language models from a Zig binary on Apple Silicon and NVIDIA GPUs using Metal and CUDA kernels.
flashrt-project / FlashRT
FlashRT is a high-performance realtime inference engine for small-batch, latency-sensitive AI workloads. The flagship integration is production VLA control for Pi0, Pi0.5, GROOT N1.6, and Pi0-FAST. Also support llm e.g, qwen3.6-27B
surya-koritala / Glyd
Lossless AI compression: 33% less GPU memory, bit for bit.
vllm-project / vllm-omni
A framework for efficient model inference with omni-modality models
facebookresearch / sam3
The repository provides code for running inference and finetuning with the Meta Segment Anything Model 3 (SAM 3), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.
FareedKhan-dev / kimi-k3-in-c
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.