nace-ai / kapitsa
Blazing fast Apple Silicon optimized inference engine built for Qwen and Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai
nace-ai / kapitsa
ggml-org / llama.cpp
LLM inference in C/C++
ggml-org / whisper.cpp
Port of OpenAI's Whisper model in C/C++
microsoft / BitNet
Official inference framework for 1-bit LLMs
google-ai-edge / mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
Niko1221 / Strata
Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
mozilla-ai / llamafile
Distribute and run LLMs with a single file.
Tencent / ncnn
ncnn is a high-performance neural network inference framework optimized for the mobile platform
alibaba / MNN
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.
NVIDIA / TensorRT
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
openvinotoolkit / openvino
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
Const-me / Whisper
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
RunanywhereAI / runanywhere-sdks
Production ready toolkit to run AI locally
leejet / stable-diffusion.cpp
Diffusion model(SD,Flux,Wan,Qwen Image,Z-Image,...) inference in pure C/C++
google / gemma.cpp
lightweight, standalone C++ inference engine for Google's Gemma models.
kvcache-ai / Mooncake
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
google-ai-edge / LiteRT-LM
LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.
cactus-compute / cactus
Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.
lemonade-sdk / lemonade
Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk
NVIDIA / DALI
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
OpenNMT / CTranslate2
Fast inference engine for Transformer models
uxlfoundation / oneDNN
Open-source library of optimized deep learning operations (matmul, convolution, attention) for CPUs (x64, AArch64, RISC-V) and Intel GPUs. Used by PyTorch, TensorFlow, OpenVINO, and ONNX Runtime.
0xShug0 / audio.cpp
An all-in-one, pure C++ inference engine for audio models, powered by ggml. Supports TTS, STT, VAD, voice conversion, music generation, and more, with highly optimized performance. No Python dependency.
b4rtaz / distributed-llama
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
Neroued / ninfer
High-performance single-GPU inference for selected model checkpoints and GPUs.
Luce-Org / lucebox
LLM speculative inference server for heterogeneous hardware & consumer GPUs
stan-dev / stan
Stan development repository. The master branch contains the current release. The develop branch contains the latest stable development. See the Developer Process Wiki for details.
microsoft / foundry-local
—
jolibrain / deepdetect
Deep Learning Server and CLI for Torch and TensorRT
handy-computer / transcribe.cpp
ggml speech-to-text inference for 16+ model families
microsoft / onnxruntime-inference-examples
Examples for using ONNX Runtime for machine learning inferencing.
xLLM-AI / xllm
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
incoai / splash
A local inference engine for Apple silicon, built around the model.
lean-dojo / LeanCopilot
LLMs as Copilots for Theorem Proving in Lean
ai-dynamo / nixl
NVIDIA Inference Xfer Library (NIXL)
mw00 / project-maya
Project Maya - run GLM-5.3-Flash (321B MoE) and DeepSeek-V4.1-Flash-1017 (748B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.
jmaczan / tiny-vllm
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
aiptimizer / TurboOCR
TurboOCR, >200 img/s OmnidocBench. TensorRT FP16, PP-OCRv6, HTTP + gRPC
kvmem / kvmem
KVMem virtualizes KV memory across inference backends to make ultra-long-context local AI accessible on limited hardware.
feder-cr / jev
jevos: an open-source alternative to Jev and Microsoft-Decision-1. Yes/no decisions, choices and scores with probabilities, on your laptop's CPU.
openvinotoolkit / model_server
A scalable inference server for models optimized with OpenVINO™
foldl / chatllm.cpp
Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)
spiritbuun / buun-llama-cpp
Experimental llama.cpp fork for inference research and development
flashrt-project / FlashRT
FlashRT is a high-performance realtime inference engine for small-batch, latency-sensitive AI workloads. The flagship integration is production VLA control for Pi0, Pi0.5, GROOT N1.6, and Pi0-FAST. Also support llm e.g, qwen3.6-27B
triton-inference-server / client
Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.
triton-inference-server / python_backend
Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.
uxlfoundation / oneDAL
oneAPI Data Analytics Library (oneDAL)
open-edge-platform / dlstreamer
Deep Learning Streamer (DL Streamer) Pipeline Framework is an open-source streaming media analytics framework, based on GStreamer* multimedia framework, for creating complex media analytics pipelines for the Cloud or at the Edge.
Helldez / BigMoeOnEdge
Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.
brontoguana / krasis
Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware
FAST-Imaging / FAST
A framework for high-performance medical image processing, neural network inference and visualization
InfiniTensor / InfiniTensor
InfiniTensor is a high-performance inference engine tailored for GPUs and AI accelerators. Its design focuses on effective deployment and swift academic validation.
NVIDIA-ISAAC-ROS / isaac_ros_pose_estimation
Deep learned, NVIDIA-accelerated 3D object pose estimation
VecSzn / PolyStrata
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.
deepseek-ai / DeepEP-Ascend
A high-performance communication library for machine learning training and inference on Huawei Ascend NPUs.
Indras-Mirror / Xolotl
Xolotl (formerly Strata-DSV4.1): DeepSeek-V4.1-Flash, all 552B, on one gaming PC (RTX 4090 + 90 GB RAM) at ~27 tok/s, plus GLM-5.3-Flash, Qwen3.8-Flash-Next and Qwen3.8-27B. A heavily extended fork of Strata: MoE experts tiered across VRAM, pinned RAM and NVMe; OpenAI + Anthropic API.
JakeATX / llamAmpere
llama.cpp fork for significantly improved performance on Ampere (especially RTX 3090 / 3090 Ti): example: 95+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx
sergqwer / strata-nvfp4
Strata fork: Qwen3.8-Flash-Next 125B MoE in NVFP4 on one RTX 20-50 card (12 GB+) and 64 GB of RAM or more. Setup installs our GPTQ quants (huihui abliterated, OrcaRouter) from Hugging Face. CPU experts on AVX-512 or AVX2 (+AVX-VNNI), all experts in RAM from 92 GB, a low-RAM mode below, vision, W4A8 prompts. Windows / CUDA 13.
lkarlslund / laya.cpp
C++ inference for Laya typed decisions - supports CUDA, Vulkan, Core ML, CPU
Wallawalla47 / Infernix
C++/CUDA single-GPU inference engine for Qwen3.8 (incl. Qwen3.8-Flash-Next with offloaded experts) on the RTX 5090; grew from NInfer