dragosmiron / immich-video-optimizer
Interactive terminal client to reclaim gigabytes on your Immich server by re-encoding videos to hardware-accelerated HEVC while preserving 100% of metadata
dragosmiron / immich-video-optimizer
vllm-project / vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
ray-project / ray
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
sgl-project / sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
liguodongiot / llm-action
本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）
EthicalML / awesome-production-machine-learning
A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning
modelscope / FunASR
Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
hyperledger / fabric
Hyperledger Fabric is an enterprise-grade permissioned distributed ledger framework for developing solutions and applications. Its modular and versatile design satisfies a broad range of industry use cases. It offers a unique approach to consensus that enables performance at scale while preserving privacy.
NVIDIA / TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
FlashML-org / FreeToken
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
bentoml / OpenLLM
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
sigoden / dufs
A file server that supports static serving, uploading, searching, accessing control, webdav...
skypilot-org / skypilot
The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.
bentoml / BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
TheAssassin / AppImageLauncher
Helper application for Linux distributions serving as a kind of "entry point" for running and integrating AppImages
ai-dynamo / dynamo
A Datacenter Scale Distributed Inference Serving Framework
InternLM / lmdeploy
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
flyteorg / flyte
Type-safe, distributed orchestration of agents, ML pipelines, and real-time inference on your k8s — in pure Python with async/await, also other languages (rust, go and ts)
Moriafly / SaltPlayerSource
A multi-platform local music player in continuous development since 2020, serving over one million users. This repository is used for issue tracking and official Android package releases.
vllm-project / vllm-omni
A framework for efficient model inference with omni-modality models
vespa-engine / vespa
The AI search platform
clearml / clearml
ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
kvcache-ai / Mooncake
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
bojieli / ai-infra-book
《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
flashinfer-ai / flashinfer
FlashInfer: Kernel Library for LLM Serving
Lymphatus / caesium-image-compressor
Caesium is an image compression software that helps you store, send and share digital pictures, supporting JPG, PNG, WebP and TIFF formats. You can quickly reduce the file size (and resolution, if you want) by preserving the overall quality of the image.
tensorflow / serving
A flexible, high-performance serving system for machine learning models
kserve / kserve
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
knative / serving
Kubernetes-based, scale-to-zero, request-driven compute
volcano-sh / volcano
A Cloud Native Batch System (Project under CNCF)
lemonade-sdk / lemonade
Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk
gpustack / gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
qpdf / qpdf
qpdf: A content-preserving PDF document transformer
beclab / Olares
Open-Source Personal Cloud OS for Always-On Agents
brianfrankcooper / YCSB
Yahoo! Cloud Serving Benchmark
microsoft / SPTAG
A distributed approximate nearest neighborhood search (ANN) library which provides a high quality vector index build, search and distributed online serving toolkits for large scale vector search scenario.
skyzh / tiny-llm
learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen
SPLWare / esProc
esProc SPL is a JVM-based programming language designed for structured data computation, serving as both a data analysis tool and an embedded computing engine.
ModelTC / LightLLM
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
Lightning-AI / LitServe
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
PaddlePaddle / FastDeploy
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
nuwave / lighthouse
A framework for serving GraphQL from Laravel
NVIDIA-NeMo / Switchyard
Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
containers / ramalama
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
pulumi / kubespy
Tools for observing Kubernetes resources in real time, powered by Pulumi.
vllm-project / vllm-ascend
Community maintained hardware plugin for vLLM on Huawei Ascend
thu-pacman / chitu
High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.
michaelfeil / infinity
Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali
evansd / whitenoise
Radically simplified static file serving for Python web apps
kboghdady / youTube_ads_4_pi-hole
YouTube ad-serving domain blocklist for Pi-hole (v5 & v6), AdGuard Home, pfBlockerNG and other DNS blockers - add one URL and block YouTube ads on every device on your network. Updated daily.
wxyhgk / retain-pdf
在保留版面、公式与结构的前提下进行 PDF 翻译，适用于科研与技术文档
openlake-project / openlake
OpenLake is a high performance storage engine for efficient LLM inference and GPU Training
WalletWasabi / WalletWasabi
Open-source, non-custodial, privacy preserving Bitcoin wallet for Windows, Linux, and Mac.
rednote-machine-learning / RedKnot
Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
noonghunna / club-3090
Community recipes for serving LLMs on RTX 3090/4090/5090 CUDA gpus. Multi-engine (vLLM, sglang, llama.cpp and other custom engines) and model-agnostic. Currently shipping Qwen3.8-27B Qwen3.6 35B Gemma 4 26B Gemma 4 31B configs for 1×/2×/4x/8x cards.
static-web-server / static-web-server
A cross-platform, high-performance and asynchronous web server for static files-serving. ⚡
tensorchord / envd
🏕️ Reproducible development environment for humans and agents
future-agi / future-agi
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
matrixorigin / matrixone
AI-native HTAP database with Git-for-Data and built-in vector search, serving as the data and memory backbone for intelligent agents and applications.
webp-sh / webp_server_go
Go version of WebP Server. A tool that will serve your JPG/PNG/BMP/SVGs as WebP/AVIF format with compression, on-the-fly.