google / XNNPACK
High-efficiency floating-point neural network inference operators for mobile, server, and Web
google / XNNPACK
syv-ai / HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
brontoguana / krasis
Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware
amitshekhariitbhu / llm-inference-engineering
Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.
nari-labs / nari-qwen3-tts
Ultrafast Qwen3-TTS: sub-50 ms time-to-first-audio at 10 requests per second.
truespar / paddock
Native Rust inference server for open models on NVIDIA and Apple Silicon. OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio
htobty / ReadyLLM
Deploy, monitor, and tune LLM through a visual interface — plus image / video generation on top of ComfyUI.