egotoadcitadel / palette-from-image-ai
AI coloring palette-from-image generator for designers and pixel artists — pull a clean named palette straight from any image, no subscription or cloud upload.
egotoadcitadel / palette-from-image-ai
justxor / deepseek-local-guide-ru
🐋 Полный гайд на русском: как установить и запустить DeepSeek локально (V4, R1) на ПК, Mac и сервере — Ollama, llama.cpp, LM Studio, vLLM, Open WebUI, API, RAG, агенты, fine-tuning, лучшие фишки
UFund-Me / Qbot
[🔥updating ...] AI 自动量化交易机器人(完全本地部署) AI-powered Quantitative Investment Research Platform. 📃 online docs: https://ufund-me.github.io/Qbot ✨ :news: qbot-mini: https://github.com/Charmve/iQuant
hiyouga / LlamaFactory
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
SYSTRAN / faster-whisper
Faster Whisper transcription with CTranslate2
RyanCodrai / turbovec
A vector index built on TurboQuant, written in Rust with Python bindings
FareedKhan-dev / kimi-k3-in-c
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
bitsandbytes-foundation / bitsandbytes
Accessible large language models via k-bit quantization for PyTorch.
cactus-compute / cactus
Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.
NVIDIA / Model-Optimizer
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
OpenNMT / CTranslate2
Fast inference engine for Transformer models
lucidrains / vector-quantize-pytorch
Vector (and Scalar) Quantization, in Pytorch
vllm-project / llm-compressor
State-of-the-art LLM compression, built for production inference with vLLM
huggingface / optimum
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
amitshekhariitbhu / ai-engineering-interview-questions
Your Cheat Sheet for AI Engineering Interview – Questions and Answers.
pytorch / ao
PyTorch native quantization for training and inference
qualcomm / aimet
AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.
intel / neural-compressor
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime
stochasticai / xTuring
Build, personalize and control your own LLMs. From data pre-processing to fine-tuning, xTuring provides an easy way to personalize open-source LLMs. Join our discord community: https://discord.gg/TgHXuSJEk6
AI-Efficiency / Awesome-Model-Quantization
A curated collection of papers, benchmarks, surveys, and tools for model quantization, covering low-bit networks, LLMs, multimodal and generative models, vector and lattice quantization, and efficient deployment.
microsoft / Olive
Olive: Simplify ML Model Finetuning, Conversion, Quantization, and Optimization for CPUs, GPUs and NPUs.
syv-ai / HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
Tencent / AngelSlim
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
turboderp-org / exllamav3
An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs
intel / auto-round
PTQ-Level Efficiency, QAT-Level Accuracy|训练后量化的效率，量化感知训练的精度
JustGlowing / minisom
:red_circle: MiniSom is a minimalistic implementation of the Self Organizing Maps
Xilinx / brevitas
Brevitas: neural network quantization in PyTorch
tensorflow / model-optimization
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
open-edge-platform / geti
Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel XPU (CPU-only and CUDA also supported).
vipshop / cache-dit
A PyTorch-native inference engine with cache, parallelism, quantization and cpu offload for DiTs.
ModelCloud / GPTQModel
LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.
mw00 / project-maya
Project Maya - run GLM-5.3-Flash (321B MoE) and DeepSeek-V4.1-Flash-1017 (748B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.
openvinotoolkit / nncf
Neural Network Compression Framework for enhanced OpenVINO™ inference
Xilinx / finn
Dataflow compiler for QNN inference on FPGAs
huggingface / optimum-quanto
A pytorch quantization backend for optimum
patchy631 / time-to-first-token
A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking.
whitecircle / halo
Halo is an open-source framework built by White Circle for training large language and multimodal models
hailo-ai / hailo_model_zoo
The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment
warpfront / hipfire
RDNA-native LLM inference engine in Rust.
Helldez / BigMoeOnEdge
Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.
wanshuiyin / ARIS-in-AI-Offer
Bilingual (中文+EN) ML / LLM / diffusion / agent interview cheat sheets for AI 秋招 — generated by ARIS /interview-cheatsheet, rendered by /render-html into single-file HTML, reads anywhere — plus a CV→DBLP-fact-checked academic homepage generator and hand-authored long-form blogs 🌱
Continuum-AI-Corp / OrcaBonsai-27B-Uncensored
Runtime behavioral ablation for compressed LLMs. First target: Ternary Bonsai 2 27B — no weight modification or re-quantization. by OrcaRouter team
jaylfc / taOS
Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).
FutureMLS-Lab / OSCAR
OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization
zengxiao-he / tessera
From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
AlibabaResearch / SparkDiffusion
SparkDiffusion: Accelerating DiT video generation by 265× via joint Sparsity, Distillation, Quantization, etc.
prabanta-dev / janas
37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
fermionresearch / phonon
Phonon: open speech recognition models (Phonon-2, Phonon-1) — CLI, CPU and CUDA images
Low-Zi-Hong / ESP32s3-LLM-Cluster
7-node ESP32-S3 cluster running a 0.4B LLM via 1.58-bit (BitNet) ternary quantization over SPI daisy-chain
Fazmin / AILearningGuide
Interactive desktop app with 51 labs covering ML fundamentals, transformers, RAG, agents, fine-tuning and quantization and more.
RadixRootMind / Awesome-Quant-VLA-WAM
Curated quantization research for VLA & WAM models — 180+ papers, a five-category taxonomy, a roadmap, and a companion PTQ + LIBERO toolkit (quant-vla).
pyq623 / QuaNAG
Official implementation of QuaNAG: LLM-driven Memory-bounded Neural Architecture Generation via Quantization-branched Exploration
zwhy149 / bead-grid-studio
拼豆图纸生成器｜图片转拼豆、拼豆像素画与可编辑施工图｜逐格色号、辅助线、底板布局、用量统计｜Local-first fuse-bead pattern generator
Yamz-Labs / kyojin
Kyojin: the Yamz inference engine for AMD Strix Halo (ROCm, gfx1151), built on ExLlamaV3. Runs 300B-class MoE models on one 128 GB mini PC.
JakeATX / llamAmpere
llama.cpp fork for significantly improved performance on Ampere (especially RTX 3090 / 3090 Ti): example: 95+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx
dnv2003 / v100-skinny
Hand-written NVFP4 W4A16 CUDA kernels for Volta
Blackwellboy / model-serving-minefield
Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each.
DominikBucko / qwen38-flash-next-2x3090
Qwen3.8-Flash-Next on 2× RTX 3090: with 128 GB RAM up to 4,191 tok/s prefill · 111.5 tok/s decode (131K prompt), full 256K window at 2,865 tok/s; with 64 GB RAM 3,410 tok/s prefill · 84 tok/s decode. New: opt-in uncensored mode (runtime abliteration, no new weights).
mattn / tensai
A tiny neural-network framework in pure Go with AVX2 SIMD kernels (GOEXPERIMENT=simd)
Jit-Roy / WeeLLM
WeeLLM runs large diffusion models with as little as 4 GB of VRAM, without any quantization. It dynamically determines how many layers can fit within the available VRAM and streams the text encoder and transformer layers to the GPU layer by layer, enabling inference on hardware with limited VRAM. It supports both safetensors and GGUF models.