gitleaks / gitleaks
Find secrets with Gitleaks 🔑
gitleaks / gitleaks
justxor / deepseek-local-guide-ru
🐋 Полный гайд на русском: как установить и запустить DeepSeek локально (V4, R1) на ПК, Mac и сервере — Ollama, llama.cpp, LM Studio, vLLM, Open WebUI, API, RAG, агенты, fine-tuning, лучшие фишки
Elevatormusic / DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark
DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw on 2x NVIDIA DGX Spark (GB10) with TensorFold: 2.1-2.3x the vLLM kit's decode speed.
eagle-hq / EAGLE-4
Official Inference Code of EAGLE-4
ray-project / ray
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
liguodongiot / llm-action
本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）
Lightning-AI / litgpt
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
bentoml / OpenLLM
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
openvinotoolkit / openvino
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
FareedKhan-dev / kimi-k3-in-c
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
bentoml / BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
ai-dynamo / dynamo
A Datacenter Scale Distributed Inference Serving Framework
InternLM / lmdeploy
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
algorithmicsuperintelligence / openevolve
Open-source implementation of AlphaEvolve
katanemo / plano
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
drumih / turbo-fieldfare
Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook
bojieli / ai-infra-book
《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
flashinfer-ai / flashinfer
FlashInfer: Kernel Library for LLM Serving
kserve / kserve
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
cactus-compute / cactus
Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.
Michael-A-Kuykendall / shimmy
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
lemonade-sdk / lemonade
Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk
gpustack / gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
xlite-dev / Awesome-LLM-Inference
📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉
ruvnet / RuVector
RuVector provides High Performance, Real-Time decisions and agent memory , Self-Learning Ai, Vector GNN DB built in Rust.
algorithmicsuperintelligence / optillm
Optimizing inference proxy for LLMs
raullenchai / Rapid-MLX
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
hemansnation / AI-Engineer-Headquarters
A collection of scientific methods, processes, algorithms, and systems to build stories & models.
amitshekhariitbhu / ai-engineering-interview-questions
Your Cheat Sheet for AI Engineering Interview – Questions and Answers.
b4rtaz / distributed-llama
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
spiceai / spiceai
Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.
youssofal / MTPLX
The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.
Nano-Collective / nanocoder
An open coding agent for your terminal, built by a community collective rather than a company. Bring your own model, keep your code on your machine, and owe nothing to anyone.
neuron-core / neuron-ai
The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect components (LLMs, Tools, vector DBs, memory) to agents that interact with your data and UI.
syv-ai / HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
SemiAnalysisAI / InferenceX
Open Source AI Accelerator Research Platform Standard / 开源推理研究平台
ovg-project / kvcached
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
beam-cloud / beta9
Ultrafast serverless GPU inference, sandboxes, and background jobs
AtomicBot-ai / Atomic-Chat
Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your computer. Join our Discord: https://discord.com/invite/8wGSsvmg4V
software-mansion / react-native-executorch
High-performance, privacy-first on-device AI inference library for React Native, powered by PyTorch's ExecuTorch runtime
pallavi-shekhar / ai-engineering-interview-questions-company-wise
Your Cheat Sheet For AI Engineering Interviews at Top AI Companies - Questions and Answers.
xLLM-AI / xllm
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
incoai / splash
A local inference engine for Apple silicon, built around the model.
datawhalechina / zero-to-sglang
Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
jax-ml / scaling-book
Home for "How To Scale Your Model", a short blog-style textbook about scaling LLMs on TPUs
felladrin / awesome-ai-web-search
List of software that allows searching the web with the assistance of AI: https://hf.co/spaces/felladrin/awesome-ai-web-search
ashhart / TensorFold
LLM Inference Engine for Metal and CUDA.
lean-dojo / LeanCopilot
LLMs as Copilots for Theorem Proving in Lean
mw00 / project-maya
Project Maya - run GLM-5.3-Flash (321B MoE) and DeepSeek-V4.1-Flash-1017 (748B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.
jmaczan / tiny-vllm
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
StayLameBro / backburner
Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable
patchy631 / time-to-first-token
A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking.
tingaicompass / AI-Compass
“AI-Compass”将为社区指引在 AI 技术海洋中航行的方向，无论你是初学者还是进阶开发者，都能在这里找到通往 AI 各大方向的路径。旨在帮助开发者系统性地了解 AI 的核心概念、主流技术、前沿趋势，并通过实践掌握从理论到落地的全过程。
feder-cr / jev
jevos: an open-source alternative to Jev and Microsoft-Decision-1. Yes/no decisions, choices and scores with probabilities, on your laptop's CPU.
dalisoft / awesome-hosting
List of awesome hosting sorted by minimal plan price
foldl / chatllm.cpp
Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)
kubernetes-sigs / lws
LeaderWorkerSet & DisaggregatedSet: Kubernetes APIs for Distributed and Disaggregated AI Serving
v-modal / awesome-jev-tools
A curated list of tools built for Jev — TypeSafe AI's System One model for typed decisions.
pegainfer-project / pegainfer
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
warpfront / hipfire
RDNA-native LLM inference engine in Rust.