weicj / vLLM-2080Ti-Definitive
The definitive vLLM runtime for dual RTX 2080 Ti 22GB + NVLink, delivering Qwen 27B local inference with maximum 200+ tok/s single-request decode with support of FP8 weight ( Join Discord :https://discord.gg/VFqVVySdMS )
usamahz / cpu-performance-engineering
A reading path for CPU performance engineering, from one instruction to production inference. Primary sources only, with a runnable benchmark for every section.
local-inference-lab / rtx6kpro
RTX 6000 Pro Wiki — Running Large LLMs (Qwen3.5-397B, Kimi-K2.5, GLM-5) on PCIe GPUs without NVLink
huawei-noah / trustworthyAI
Trustworthy AI related projects
blackjax-devs / blackjax
BlackJAX is a Bayesian Inference library designed for ease of use, speed and modularity.
huggingface / search-and-learn
Recipes to scale inference-time compute of open models
StayLameBro / backburner
Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable
Xilinx / finn
Dataflow compiler for QNN inference on FPGAs
StartLuxLabs / StartLux-Decision
Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.
tingaicompass / AI-Compass
“AI-Compass”将为社区指引在 AI 技术海洋中航行的方向，无论你是初学者还是进阶开发者，都能在这里找到通往 AI 各大方向的路径。旨在帮助开发者系统性地了解 AI 的核心概念、主流技术、前沿趋势，并通过实践掌握从理论到落地的全过程。
mistralai / mistral-common
Official inference library for pre-processing of Mistral models
vllm-project / speculators
A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM
SeldonIO / MLServer
An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more
microsoft / bioemu
Inference code for scalable emulation of protein equilibrium ensembles with generative deep learning
sbi-dev / sbi
sbi is a Python package for simulation-based inference, designed to meet the needs of both researchers and practitioners. Whether you need fine-grained control or an easy-to-use interface, sbi has you covered.
triton-inference-server / tutorials
This repository contains tutorials and examples for Triton Inference Server
insight-platform / Savant
Python Computer Vision & Video Analytics Framework With Batteries Included
ai-dynamo / aiperf
AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution.
DoubleML / doubleml-for-py
DoubleML - Double Machine Learning in Python
bayesflow-org / bayesflow
A Python library for simulation-based inference with deep learning
fluxions-ai / vui
Vui Nano — a small, context-aware text-to-speech model trained on real conversations. 219M active params (305M total), Apache 2.0, voice cloning, streaming, runs on CPU (dependency-free C build). Ships with a full real-time voice assistant: WebRTC, ASR, local LLM, OpenAI Realtime API compatible.
infer-actively / pymdp
A Python implementation of active inference for Markov Decision Processes
ARahim3 / mlx-dspark
Up to 4× faster LLM decoding on Apple Silicon, lossless. Native MLX port of DeepSeek's DSpark & z-lab's DFlash speculative decoding — Gemma-4, Qwen3.8, Muse-Glimmer, Nemotron, LFM2.5, Ornith-1.0, ternary Bonsai-27B.
Picovoice / rhino
On-device Speech-to-Intent engine powered by deep learning
ARahim3 / kaggle-tpu-lab
Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.
WillianFuks / tfcausalimpact
Python Causal Impact Implementation Based on Google's R Package. Built using TensorFlow Probability.
QuantClimate / GPJax
Gaussian processes in JAX and Equinox.
pymc-devs / pytensor
PyTensor allows you to define, optimize, and efficiently evaluate mathematical expressions involving multi-dimensional arrays.
breezeblue-ai / breeze-tts
Official PyTorch inference for Breeze TTS 2
locuslab / open-unlearning
[NeurIPS D&B '25] The one-stop repository for LLM unlearning
thousandbrainsproject / tbp.monty
Monty is a sensorimotor learning framework based on the thousand brains theory of the neocortex.
NVIDIA / TensorRT-Edge-LLM
High-performance, light-weight C++ LLM and VLM Inference Software for Physical AI
pylint-dev / astroid
A common base representation of python source code for pylint and other projects
TrenTorch / TrenTorch
Don't memorize ML. Understand it from first principles including Inference, Cuda, Deep Learning, Data Science and Reinforcement Learning
SigmanticAI / apex-inference-chip
An inference chip design that runs a real LLM (Qwen2.5-0.5B) on FPGA — one transformer decoder layer in RTL, every silicon value bit-exact against a golden model. 0.56 tok/s measured, a 140× climb, full evidence trail.
laurencium / Causalinference
Causal Inference in Python
jaylfc / taOS
Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).
SakanaAI / treequest
A Tree Search Library with Flexible API for LLM Inference-Time Scaling
NVIDIA / cosmos-framework
Our inference and training framework to run on the Cosmos Models
evelyyyyynnnnn / 4.0-Decision-Intelligence-Framework
A research-oriented framework for building optimization-driven decision intelligence systems, integrating causal inference, risk modeling, and AI pipelines for large-scale socio-technical systems
rockmatrix-ai / EdgeRock
EdgeRock: Strong Reasoning at Constant Memory. Multimodal models with O(N) generation and O(1) persistent history. Inference and evaluation code.
zengxiao-he / tessera
From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
thu-ml / SLA
SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse–Linear Attention
aleximmer / Laplace
Laplace approximations for Deep Learning.
spark-arena / sparkrun
sparkrun - launch, manage, and stop LLM inference workloads on NVIDIA DGX Spark systems. Live support: https://discord.com/invite/GH5kRgv6ZD
SearchSavior / OpenArc
Inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Rerank models over OpenAI endpoints.
triton-inference-server / model_analyzer
Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Server models.
AlibabaResearch / SparkDiffusion
SparkDiffusion: Accelerating DiT video generation by 265× via joint Sparsity, Distillation, Quantization, etc.
NVIDIA / flashdreams
high-performance inference and serving library for interactive autoregressive video and world models
nchopin / particles
Sequential Monte Carlo in python
expectedparrot / edsl
Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.
fudan-generative-vision / Remory
Remory: Learning Residual Memory for Context Compaction
framepipe-dev / media-inference-worker
Small client for media model inference
hr98w / jev-visual
An educational Jev-like visual inference experiment on Apple Silicon: shared context, direct candidate scoring, and local visual demos.
zzzz7788990213-ops / EvoVLM
Evolving Multimodal Inference Programs: LLM-guided evolutionary search for accurate and efficient vision-language inference.
lithos-ai / lithos-metal
lithos-metal: high-performance LLM inference kernels and serving for Apple silicon.
Tencent / AngelSpec
A unified, torch-native training framework for MTP and block-parallel speculative decoding.
securetron-gh / aifw
AI Agent Identity and Access Management and Inference Firewall
hasso5703 / dgx-spark-qwen38
One command turns a DGX Spark or ASUS Ascent GX10 into a private AI server: seven measured, pinned AI lanes (text up to 1M tokens, images, video with sound), run from a local cockpit on a laptop or a phone. Nothing leaves the box.
Phocinae / Phocinae-Largha-150M-v1
斑海豹 — Phocinae-Largha-150M-v1: a 144.3M bilingual decision model. Tiny model. Big decisions. 小海豹，大决断。