hao-ai-lab / FastVideo
A unified inference and post-training framework for accelerated video generation.
hao-ai-lab / FastVideo
openvinotoolkit / open_model_zoo
Pre-trained Deep Learning models and demos (high quality and extremely fast)
ModelTC / LightLLM
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
algorithmicsuperintelligence / optillm
Optimizing inference proxy for LLMs
wafer-ai / gpu-perf-engineering-resources
A curated resource list for learning AI performance engineering, from GPU fundamentals to production inference.
raullenchai / Rapid-MLX
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
Lightning-AI / LitServe
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
vllm-project / llm-compressor
State-of-the-art LLM compression, built for production inference with vLLM
PaddlePaddle / FastDeploy
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
NVIDIA / TransformerEngine
A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.
huggingface / optimum
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
superlinked / sie
Open-source inference server and production cluster for all the models your agent needs.
pgmpy / pgmpy
Python Toolkit for Causal and Probabilistic Reasoning
containers / ramalama
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
pytorch / ao
PyTorch native quantization for training and inference
vllm-project / vllm-ascend
Community maintained hardware plugin for vLLM on Huawei Ascend
thu-pacman / chitu
High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.
Tiger3807861189 / J-Space-Cognition-Suite
J-Space Cognition Suite — a model-agnostic inference-time control suite for deep reasoning, long-horizon work, verification, and recovery. Based on Anthropic's J-space global workspace research.
ModelTC / LightX2V
Lightweight Image Video Action Generation Inference Framework
pnnbao97 / VieNeu-TTS
Vietnamese TTS with instant voice cloning • On-device • Real-time CPU inference • 48kHz audio quality • Chuyển văn bản thành giọng nói tiếng Việt • Text to speech tiếng Việt • TTS tiếng Việt
illuin-tech / colpali
The code used to train and run inference with the ColVision models, e.g. ColPali, ColQwen2, and ColSmol.
autodistill / autodistill
Images to inference with no labeling (use foundation models to train supervised models).
pyro-ppl / numpyro
Probabilistic programming with NumPy powered by JAX for autograd and JIT compilation to GPU/TPU/CPU.
xdit-project / xDiT
xDiT: A Scalable Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism
lightseekorg / tokenspeed
TokenSpeed is a speed-of-light LLM inference engine.
youssofal / MTPLX
The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.
data-infra / cube-studio
cubestudio开源云原生一站式机器学习/深度学习/大模型AI平台/MaaS/mlops/人工智能平台/训推平台，算法全链路流程，多租户，算力租赁平台，token中转，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务，VGPU虚拟化，云边端协同，边缘计算，自动化标注平台，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库llmops智能体，AI模型市场，支持国产异构算力调度,昇腾/寒武纪/海光/摩尔/沐曦等，支持ib/roce/RDMA，信创支持
roboflow / inference
Turn any computer or edge device into a command center for your computer vision projects.
guy-hartstein / company-research-agent
An agentic company research tool powered by LangGraph and Tavily that conducts deep diligence on companies using a multi-agent framework. It leverages OpenRouter on the backend for rapid inference with any leading model.
dstackai / dstack
A unified orchestration layer for heterogeneous AI compute. It standardizes how to manage compute and run training and inference on GPU clouds, Kubernetes, VMs, or bare-metal clusters.
mozilla-ai / any-llm
Communicate with an LLM provider using a single interface
FlashML-org / FreeVideo
Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.
syv-ai / HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
cfregly / ai-performance-engineering
Code, labs, and resources for O'Reilly AI Systems Performance Engineering: GPU optimization, distributed training, inference scaling, and full-stack tuning.
GPflow / GPflow
Gaussian processes in TensorFlow
SemiAnalysisAI / InferenceX
Open Source AI Accelerator Research Platform Standard / 开源推理研究平台
ovg-project / kvcached
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
dphnAI / sonar
Large-scale LLM inference engine
tile-ai / TileRT
Tile-Based Runtime for Ultra-Low-Latency LLM Inference
turboderp-org / exllamav3
An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs
vllm-project / guidellm
Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
CoinCheung / BiSeNet
Add bisenetv2. My implementation of BiSeNet
waybarrios / vllm-mlx
High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.
sgl-project / sglang-omni
SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.
datawhalechina / zero-to-sglang
Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
Lightning-AI / lightning-thunder
PyTorch compiler that accelerates training and inference. Get built-in optimizations for performance, memory, parallelism, and easily write your own.
1CatAI / 1Cat-vLLM
V100 / SM70-focused vLLM engineering fork for modern LLM inference.
alibaba / rtp-llm
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
open-edge-platform / geti
Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel XPU (CPU-only and CUDA also supported).
uxlfoundation / scikit-learn-intelex
Extension for Scikit-learn is a seamless way to speed up your Scikit-learn application
Capsize-Games / airunner
Offline inference engine for art, real-time voice conversations, LLM powered chatbots and automated workflows
bambinos / bambi
BAyesian Model-Building Interface (Bambi) in Python.
vipshop / cache-dit
A PyTorch-native inference engine with cache, parallelism, quantization and cpu offload for DiTs.
Arize-ai / openinference
OpenTelemetry Instrumentation for AI Observability
NVIDIA / kvpress
LLM KV cache compression made easy
qualcomm / ai-hub-models
Qualcomm® AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.
microsoft / MInference
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
basetenlabs / truss
The simplest way to serve AI/ML models in production
openvinotoolkit / nncf
Neural Network Compression Framework for enhanced OpenVINO™ inference
pymc-labs / CausalPy
A Python package for causal inference in quasi-experimental settings