deepseek-ai / DeepInfer

DeepInfer: High-performance, large-scale inference framework for DeepSeek models

Comfy-Org / ComfyUI

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.

lyogavin / airllm

AirLLM 2.8T LLM inference with single 4GB GPU

gitleaks / gitleaks

Find secrets with Gitleaks 🔑

trymirai / uzu

A high-performance inference engine for AI models

nace-ai / kapitsa

Blazing fast Apple Silicon optimized inference engine built for Qwen and Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai

ddalcu / mlx-serve

Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Zig backend, Swift frontend macOS app with chat, music, voice, video generation.

google / agentmesh

Agent Mesh

justxor / deepseek-local-guide-ru

🐋 Полный гайд на русском: как установить и запустить DeepSeek локально (V4, R1) на ПК, Mac и сервере — Ollama, llama.cpp, LM Studio, vLLM, Open WebUI, API, RAG, агенты, fine-tuning, лучшие фишки

black-forest-labs / flux-3-image-klein

FLUX 3 Image Klein: reference inference code

apple-aiml-research / ml-rsi-inference-bench

A Long-Horizon Benchmark for Self-Improving Inference Systems

facebookresearch / segment-anything

The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

Lore-Hex / token-exchange-neocloud-in-a-box

Open-source confidential GPU inference cloud in a box: Intel TDX + NVIDIA confidential-computing nodes for DeepSeek-V4.1-Flash and GLM-5.3, verifiable and listed on TrustedRouter

Elevatormusic / DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark

DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw on 2x NVIDIA DGX Spark (GB10) with TensorFold: 2.1-2.3x the vLLM kit's decode speed.

j1s4nn / jev-adaptive-granularity-WM

Official code, locked protocols, and audit artifacts for "Jev-Routed Temporal Granularity with Recovery and Hysteresis for Causal Video Generation": per-segment fine/coarse/hold routing for Causal-rCM (Wan2.1-1.3B) via the Jev oracle, with two-GPU (T4/A100) feasibility audits.

liangkaihua123456 / Latticor

Latticor: Optimising Edge Inference through Load-Aware Diffusion Tree Orchestration

surya-koritala / condor

Qwen3.8-Flash-Next (180B MoE) on one 12-16 GB NVIDIA GPU, Linux or Windows. Own engine, own compression.

qnguyen3 / QUAN-Transcribe

Inference code for QUAN-Transcribe: fast Vietnamese speech recognition on EmbeddingGemma 2

hnlyf168 / SimdYolo

Pure C#/.NET YOLO inference: SIMD CPU, Vulkan/Metal GPU, direct .pt and ONNX; detection, segmentation and pose.

eagle-hq / EAGLE-4

Official Inference Code of EAGLE-4

llm-d / llm-d-router

llm-d Router: The intelligent entry point for inference requests

huggingface / transformers

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

ggml-org / llama.cpp

LLM inference in C/C++

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

ultralytics / yolov5

Ultralytics YOLOv5 in PyTorch for object detection, instance segmentation, classification, training, and export.

zylon-ai / private-gpt

Complete API layer for private AI applications on local models: RAG, skills, tools, MCP, text-to-sql, and more. Works with any OpenAI-compatible inference server.

ggml-org / whisper.cpp

Port of OpenAI's Whisper model in C/C++

colinhacks / zod

TypeScript-first schema validation with static type inference

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

deepspeedai / DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

hpcaitech / ColossalAI

Making large AI models cheaper, faster and more accessible

microsoft / BitNet

Official inference framework for 1-bit LLMs

sgl-project / sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

huggingface / pytorch-image-models

The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNetV4, MobileNet-V3 & V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more

Niko1221 / Strata

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

mozilla-ai / llamafile

Distribute and run LLMs with a single file.

SYSTRAN / faster-whisper

Faster Whisper transcription with CTranslate2

liguodongiot / llm-action

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

Tencent / ncnn

ncnn is a high-performance neural network inference framework optimized for the mobile platform

antirez / ds4

DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

jundot / omlx

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

NVIDIA / NemoClaw

Run agents like Hermes, LangChain Deep Agents, and OpenClaw more securely inside NVIDIA OpenShell with managed inference

modelscope / FunASR

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

facebookresearch / sam2

The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

kvcache-ai / ktransformers

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

stas00 / ml-engineering

Machine Learning Engineering Open Book

mlc-ai / web-llm

High-performance In-browser LLM Inference Engine

alibaba / MNN

MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.

gvergnaud / ts-pattern

🎨 The exhaustive Pattern Matching library for TypeScript, with smart type inference.

NVIDIA / TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

FlashML-org / FreeToken

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

Lightning-AI / litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

NVIDIA / TensorRT

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

halfrost / Halfrost-Field

✍🏻 Source Code Deep Dives, System Design & Engineering Blogs | Halfrost-Field 冰霜之地：源码解析、系统设计与工程实践笔记

bentoml / OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

ubicloud / ubicloud

Open source alternative to AWS. Elastic compute, block storage (non replicated), firewall and load balancer, managed Postgres, K8s, AI inference, and IAM services.

LMCache / LMCache

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

facebookresearch / sam3

The repository provides code for running inference and finetuning with the Meta Segment Anything Model 3 (SAM 3), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

loft-sh / vcluster

vCluster creates tenant clusters: fully isolated environments delivered as managed Kubernetes, or as the foundation for Slurm, Ray, Run:ai and inference clusters. Each gets its own API server, CRDs and RBAC, and runs on an existing cluster or standalone on bare metal. CNCF Certified Kubernetes.