jundot / omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
jundot / omlx
graphql / dataloader
DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
raullenchai / Rapid-MLX
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
Lightning-AI / LitServe
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
waybarrios / vllm-mlx
High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.
jmaczan / tiny-vllm
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
mosecorg / mosec
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
TanStack / pacer
Utilities for debouncing, throttling, rate limiting, and queueing. Non-async and Async.
zengxiao-he / tessera
From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
amitshekhariitbhu / llm-inference-engineering
Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.
chengyi-ai / douyin-image-post-scheduler
Codex Skill for batching, scheduling, and verifying Douyin image posts.
lora-sys / nano-vllm-interactive-guide
面向开发者与初学者的 nano-vLLM 交互式源码教程 - 通过 13 个 HTML 互动实验 + 13 章中文教程理解大模型推理引擎