lyogavin / airllm

Python library that runs large language model inference layer by layer, so 70B-class models fit on a single 4GB GPU without quantization.

trymirai / uzu

Inference engine for running AI models locally inside apps, with Rust core and Swift, Python, and TypeScript bindings.

deepseek-ai / FlashMLA

Optimized GPU and NPU attention kernels from DeepSeek for sparse prefill and FP8/FP4 KV-cache decoding in DeepSeek-V4.1 inference.

NVIDIA / Model-Optimizer

A library of model optimization techniques (quantization, pruning, distillation, NAS, speculative decoding, sparsity) that exports compressed checkpoints for inference frameworks.

FluidInference / FluidAudio

Swift SDK running open-source speech models (ASR, TTS, VAD, speaker diarization) locally on Apple devices via CoreML and the Neural Engine.

facebookresearch / segment-anything

Inference code, checkpoints, and notebooks for Meta's Segment Anything Model (SAM), which produces object masks from points, boxes, or automatically for whole images.

facebookresearch / audiocraft

PyTorch library for audio generation research, providing inference and training code for MusicGen, AudioGen, and EnCodec.

facebookresearch / sam2

Meta's Segment Anything Model 2 for promptable segmentation in images and videos, with inference code, checkpoints, and example notebooks.

espressif / esp-dl

Lightweight neural network inference framework for running AI models on Espressif ESP series chips.

zml / zml

A Zig inference framework that compiles ML models through MLIR and OpenXLA to run on NVIDIA, AMD, Intel, TPU, and Trainium hardware.

google-coral / coralnpu

Coral NPU is an open-source RISC-V neural processing unit IP core for low-power ML inference on wearables and edge SoCs.