jundot / omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
jundot / omlx
triton-inference-server / server
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
Michael-A-Kuykendall / shimmy
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
raullenchai / Rapid-MLX
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
superlinked / sie
Open-source inference server and production cluster for all the models your agent needs.
containers / ramalama
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
roboflow / inference
Turn any computer or edge device into a command center for your computer vision projects.
CoinCheung / BiSeNet
Add bisenetv2. My implementation of BiSeNet
waybarrios / vllm-mlx
High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.
basetenlabs / truss
The simplest way to serve AI/ML models in production
aiptimizer / TurboOCR
TurboOCR, >200 img/s OmnidocBench. TensorRT FP16, PP-OCRv6, HTTP + gRPC
triton-inference-server / tensorrtllm_backend
The Triton TensorRT-LLM Backend
triton-inference-server / tutorials
This repository contains tutorials and examples for Triton Inference Server
triton-inference-server / client
Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.
triton-inference-server / python_backend
Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.
triton-inference-server / model_analyzer
Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Server models.
ZinYY / AdaFlash
[COLMW'26 Spotlight] AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters
alvarobartt / sys1
Blazing fast, self-hosted structured decisions for open models with a TypeSafe AI compatible API, written in Rust.
jitllm / jitllm
Smart inference engine that adapts to your hardware to run models faster, even when they don’t fit in memory
pegainfer-project / pega-omni
OpenAI-compatible speech serving in Rust: streaming TTS and full-duplex voice over GPT-Live. 128 concurrent PersonaPlex sessions on one GPU.
zoompilot / jetlink
Runs openpilot's large driving model on Jetson Orin Nano, macOS, iOS, Android, CUDA laptops with only your Comma, and a USB3 cable
otheru-ai / ember
DeepSeek-V4-Flash inference for AMD Strix Halo