triton-inference-server / server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

openvinotoolkit / openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

aws / amazon-sagemaker-examples

Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.

Const-me / Whisper

High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model

ultralytics / yolov3

PyTorch implementation of YOLOv3, YOLOv3-SPP, and YOLOv3-tiny for real-time object detection with training, validation, inference, and multi-format export.

RunanywhereAI / runanywhere-sdks

Production ready toolkit to run AI locally

pymc-devs / pymc

Bayesian Modeling and Probabilistic Programming in Python

Lightricks / LTX-2

Official Python inference and LoRA trainer package for the LTX-2 audio–video generative model.

xorbitsai / inference

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

oumi-ai / oumi

Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!

FareedKhan-dev / kimi-k3-in-c

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

pyro-ppl / pyro

Deep universal probabilistic programming with Python and PyTorch

open-circle / valibot

The modular and type safe schema library for validating structural data 🤖

bentoml / BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

google-deepmind / alphafold3

AlphaFold 3 inference pipeline.

py-why / dowhy

DoWhy is a Python library for causal inference that supports explicit modeling and testing of causal assumptions. DoWhy is based on a unified language for causal inference, combining causal graphical models and potential outcomes frameworks.

ai-dynamo / dynamo

A Datacenter Scale Distributed Inference Serving Framework

InternLM / lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

EricLBuehler / mistral.rs

Fast, flexible LLM inference

leejet / stable-diffusion.cpp

Diffusion model(SD,Flux,Wan,Qwen Image,Z-Image,...) inference in pure C/C++

flyteorg / flyte

Type-safe, distributed orchestration of agents, ML pipelines, and real-time inference on your k8s — in pure Python with async/await, also other languages (rust, go and ts)

Human-Agent-Society / reef

Infrastructure for continually self‑improving agents

algorithmicsuperintelligence / openevolve

Open-source implementation of AlphaEvolve

vllm-project / vllm-omni

A framework for efficient model inference with omni-modality models

magnitudedev / magnitude

Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, or just a CPU.

katanemo / plano

Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.

google / gemma.cpp

lightweight, standalone C++ inference engine for Google's Gemma models.

mizorewww / laya-mlx

Native MLX runtime for Laya typed decision models — 7–14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.

drumih / turbo-fieldfare

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

kvcache-ai / Mooncake

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

bojieli / ai-infra-book

《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

flashinfer-ai / flashinfer

FlashInfer: Kernel Library for LLM Serving

google-ai-edge / LiteRT-LM

LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.

argmaxinc / argmax-oss-swift

On-device Speech AI for Apple Silicon

Trusted-AI / adversarial-robustness-toolbox

Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams

open-edge-platform / anomalib

An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.

vllm-project / semantic-router

An open, programmable decision layer for models and compute.

davidhalter / jedi

Awesome autocompletion, static analysis and refactoring library for python

kserve / kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

cactus-compute / cactus

Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.

uber / causalml

Uplift modeling and causal inference with machine learning algorithms

Michael-A-Kuykendall / shimmy

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

lemonade-sdk / lemonade

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk

vimeo / psalm

A PHP static analysis tool for finding errors and security vulnerabilities in PHP applications

gpustack / gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

NVIDIA / DALI

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

alexzhang13 / rlm

General plug-and-play inference library for Recursive Language Models (RLMs), supporting various sandboxes.

Blaizzy / mlx-vlm

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

obss / sahi

Framework agnostic sliced/tiled inference + interactive ui + error analysis plots

xlite-dev / Awesome-LLM-Inference

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

rasbt / reasoning-from-scratch

Implement a reasoning LLM in PyTorch from scratch, step by step

NVIDIA / Model-Optimizer

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

vllm-project / aibrix

Cost-efficient and pluggable Infrastructure components for GenAI inference

tencentmusic / cube-studio

cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库，AI模型市场，支持国产cpu/gpu/npu 昇腾生态，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano等分布式

huggingface / text-embeddings-inference

A blazing fast inference solution for text embeddings models

llm-d / llm-d

Achieve state of the art inference performance with modern accelerators on Kubernetes

py-why / EconML

ALICE (Automated Learning and Intelligence for Causation and Economics) is a Microsoft Research project aimed at applying Artificial Intelligence concepts to economic decision making. One of its goals is to build a toolkit that combines state-of-the-art machine learning techniques with econometrics in order to bring automation to complex causal inference problems. To date, the ALICE Python SDK (econml) implements orthogonal machine learning algorithms such as the double machine learning work of Chernozhukov et al. This toolkit is designed to measure the causal effect of some treatment variable(s) t on an outcome variable y, controlling for a set of features x.

skyzh / tiny-llm

learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen

facebookincubator / AITemplate

AITemplate is a Python framework which renders neural network into high performance CUDA/HIP C++ code. Specialized for FP16 TensorCore (NVIDIA GPU) and MatrixCore (AMD GPU) inference.

OpenNMT / CTranslate2

Fast inference engine for Transformer models