jundot / omlx

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

raullenchai / Rapid-MLX

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.

Lightning-AI / LitServe

A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

waybarrios / vllm-mlx

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

mosecorg / mosec

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

zengxiao-he / tessera

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

chengyi-ai / douyin-image-post-scheduler

Codex Skill for batching, scheduling, and verifying Douyin image posts.