vllm-project / vllm

A high-throughput LLM inference and serving engine featuring PagedAttention, continuous batching, and an OpenAI-compatible API server.

mylc2010 / sci-translation-skill

An evidence-preserving Chinese-to-English manuscript translation and review workflow for computing, medicine and other STEM disciplines.

sgl-project / sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

bojieli / ai-infra-book

《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

NVIDIA / TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

ovg-project / kvcached

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

FlashML-org / FreeToken

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

rednote-machine-learning / RedKnot

Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

modelscope / FunASR

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

sgl-project / sglang-omni

SGLang-Omni is a high-performance serving runtime for interactive intelligence model systems.

vllm-project / vllm-omni

A framework for efficient model inference with omni-modality models

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

mode-io / vllm-jev

Native vLLM serving for Jev decision models

noonghunna / club-3090

Community recipes for serving LLMs on RTX 3090/4090/5090 CUDA gpus. Multi-engine (vLLM, sglang, llama.cpp and other custom engines) and model-agnostic. Currently shipping Qwen3.8-27B Qwen3.6 35B Gemma 4 26B Gemma 4 31B configs for 1×/2×/4x/8x cards.

bentoml / BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

gpustack / gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

skyzh / tiny-llm

learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen

vllm-project / vllm-ascend

Community maintained hardware plugin for vLLM on Huawei Ascend

wxyhgk / retain-pdf

在保留版面、公式与结构的前提下进行 PDF 翻译，适用于科研与技术文档

SemiAnalysisAI / InferenceX

Open Source AI Accelerator Research Platform Standard / 开源推理研究平台

NVIDIA / flashdreams

high-performance inference and serving library for interactive autoregressive video and world models

lithos-ai / lithos-metal

lithos-metal: high-performance LLM inference kernels and serving for Apple silicon.

FujitsuPolycom / sparkring

Low-latency collective transport and distributed LLM serving stack for switchless GB10 clusters.

knapcio / DeepSeek-V4.1-Flash-4x-DGX-Spark-TP4

Tuned TP4 serving profile for DeepSeek-V4.1-Flash on 4x NVIDIA DGX Spark (SGLang, DSpark), built on the MiaAI-Lab recipe

SatoDive / ComfyUI-H3-IMG-Gen-SatoDive

Simple ComfyUI nodes for MiniMax H3: exact output resolution, multi-reference images, single-pass generation, optional resolution-preserving detail refine.

bentoml / OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

skypilot-org / skypilot

The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.

InternLM / lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

clearml / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

ModelTC / LightLLM

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

Lightning-AI / LitServe

A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

thu-pacman / chitu

High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.

evansd / whitenoise

Radically simplified static file serving for Python web apps

future-agi / future-agi

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

AMAP-ML / LongHorizon-Harness

The long-horizon computer-use harness. Run AI agents across desktop apps and the CLI for extended periods while preserving task state and making reliable progress on complex workflows. Features fresh-context execution, durable verified state, independent auditing, recoverable progress, and native Claude Code / Codex / OpenClaw integration.

alibaba / rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

basetenlabs / truss

The simplest way to serve AI/ML models in production

pytroll / satpy

Python package for earth-observing satellite data processing

ARahim3 / kaggle-tpu-lab

Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

PaddlePaddle / FastDeploy

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

containers / ramalama

RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.

meta-pytorch / opacus

Training PyTorch models with differential privacy

mlrun / mlrun

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

pixeltable / pixeltable

The backend agents build with - Multimodal database, orchestration, and serving in one file

sgl-project / SpecForge

Train speculative decoding models effortlessly and port them smoothly to SGLang serving.

a1ex4 / ownfoil

Nintendo Switch library manager, with automated management tasks, serving your library to multiple supported clients on your Switch, with shop customization and multi user authentication.

XTraceAI / cuhepy

GPU-accelerated homomorphic encryption in Python: Paillier and BFV, with encrypted nearest-neighbour search. Experimental research code.

mosecorg / mosec

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

SeldonIO / MLServer

An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more

KristenPan / pickoo-your-favorites

Local-first creator favorites workbench with evidence-preserving Markdown capture and a complete browser UI.

python-poetry / tomlkit

Style-preserving TOML library for Python

RichmondAlake / memorizz

MemoRizz: A Python library serving as a memory layer for AI applications. Leverages popular databases and storage solutions to optimize memory usage. Provides utility classes and methods for efficient data management.

lbouaraba / comfyui-krea2edit

Instruction-based, identity-preserving image editing for Krea 2 in ComfyUI — nodes + workflows for the Krea 2 Identity Edit LoRA

zengxiao-he / tessera

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

dnv2003 / v100-skinny

Hand-written NVFP4 W4A16 CUDA kernels for Volta

heise3 / academic-deai

Model-agnostic Chinese and English academic editing for the latest models across Agent platforms. Reduce formulaic prose while preserving evidence and author voice.

Blackwellboy / model-serving-minefield

Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each.

MrZoyo / deslop-GPT

Deletion-first Agent Skill for removing test bloat, verification theater, and speculative fallbacks while preserving behavior.

peonist-ai / halogen-server

The fastest way to run Qwen3.8 27B on Strix Halo (gfx1151)

StoKou / FACET-Terminal

FACET synthesizes coherent and verifiable terminal tasks from reusable agent skills by preserving source intent and grounding all artifacts in a shared executable state.