deepseek-ai / DeepInfer

DeepInfer: High-performance, large-scale inference framework for DeepSeek models

Comfy-Org / ComfyUI

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.

black-forest-labs / flux-3-image-klein

FLUX 3 Image Klein: reference inference code

apple-aiml-research / ml-rsi-inference-bench

A Long-Horizon Benchmark for Self-Improving Inference Systems

Lore-Hex / token-exchange-neocloud-in-a-box

Open-source confidential GPU inference cloud in a box: Intel TDX + NVIDIA confidential-computing nodes for DeepSeek-V4.1-Flash and GLM-5.3, verifiable and listed on TrustedRouter

Elevatormusic / DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark

DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw on 2x NVIDIA DGX Spark (GB10) with TensorFold: 2.1-2.3x the vLLM kit's decode speed.

j1s4nn / jev-adaptive-granularity-WM

Official code, locked protocols, and audit artifacts for "Jev-Routed Temporal Granularity with Recovery and Hysteresis for Causal Video Generation": per-segment fine/coarse/hold routing for Causal-rCM (Wan2.1-1.3B) via the Jev oracle, with two-GPU (T4/A100) feasibility audits.

liangkaihua123456 / Latticor

Latticor: Optimising Edge Inference through Load-Aware Diffusion Tree Orchestration

surya-koritala / condor

Qwen3.8-Flash-Next (180B MoE) on one 12-16 GB NVIDIA GPU, Linux or Windows. Own engine, own compression.

qnguyen3 / QUAN-Transcribe

Inference code for QUAN-Transcribe: fast Vietnamese speech recognition on EmbeddingGemma 2

eagle-hq / EAGLE-4

Official Inference Code of EAGLE-4

huggingface / transformers

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

ultralytics / yolov5

Ultralytics YOLOv5 in PyTorch for object detection, instance segmentation, classification, training, and export.

zylon-ai / private-gpt

Complete API layer for private AI applications on local models: RAG, skills, tools, MCP, text-to-sql, and more. Works with any OpenAI-compatible inference server.

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

deepspeedai / DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

hpcaitech / ColossalAI

Making large AI models cheaper, faster and more accessible

sgl-project / sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

huggingface / pytorch-image-models

The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNetV4, MobileNet-V3 & V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more

SYSTRAN / faster-whisper

Faster Whisper transcription with CTranslate2

jundot / omlx

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

modelscope / FunASR

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

kvcache-ai / ktransformers

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

stas00 / ml-engineering

Machine Learning Engineering Open Book

NVIDIA / TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

FlashML-org / FreeToken

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

Lightning-AI / litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

bentoml / OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

LMCache / LMCache

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

facebookresearch / sam3

The repository provides code for running inference and finetuning with the Meta Segment Anything Model 3 (SAM 3), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

triton-inference-server / server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

ultralytics / yolov3

PyTorch implementation of YOLOv3, YOLOv3-SPP, and YOLOv3-tiny for real-time object detection with training, validation, inference, and multi-format export.

pymc-devs / pymc

Bayesian Modeling and Probabilistic Programming in Python

Lightricks / LTX-2

Official Python inference and LoRA trainer package for the LTX-2 audio–video generative model.

xorbitsai / inference

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

oumi-ai / oumi

Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!

pyro-ppl / pyro

Deep universal probabilistic programming with Python and PyTorch

bentoml / BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

google-deepmind / alphafold3

AlphaFold 3 inference pipeline.

py-why / dowhy

DoWhy is a Python library for causal inference that supports explicit modeling and testing of causal assumptions. DoWhy is based on a unified language for causal inference, combining causal graphical models and potential outcomes frameworks.

InternLM / lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Human-Agent-Society / reef

Infrastructure for continually self‑improving agents

algorithmicsuperintelligence / openevolve

Open-source implementation of AlphaEvolve

vllm-project / vllm-omni

A framework for efficient model inference with omni-modality models

mizorewww / laya-mlx

Native MLX runtime for Laya typed decision models — 7–14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.

bojieli / ai-infra-book

《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

Trusted-AI / adversarial-robustness-toolbox

Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams

open-edge-platform / anomalib

An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.

davidhalter / jedi

Awesome autocompletion, static analysis and refactoring library for python

uber / causalml

Uplift modeling and causal inference with machine learning algorithms

gpustack / gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

alexzhang13 / rlm

General plug-and-play inference library for Recursive Language Models (RLMs), supporting various sandboxes.

Blaizzy / mlx-vlm

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

obss / sahi

Framework agnostic sliced/tiled inference + interactive ui + error analysis plots

xlite-dev / Awesome-LLM-Inference

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

NVIDIA / Model-Optimizer

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

llm-d / llm-d

Achieve state of the art inference performance with modern accelerators on Kubernetes

skyzh / tiny-llm

learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen

facebookincubator / AITemplate

AITemplate is a Python framework which renders neural network into high performance CUDA/HIP C++ code. Specialized for FP16 TensorCore (NVIDIA GPU) and MatrixCore (AMD GPU) inference.