egotoadcitadel / palette-from-image-ai
AI coloring palette-from-image generator for designers and pixel artists — pull a clean named palette straight from any image, no subscription or cloud upload.
egotoadcitadel / palette-from-image-ai
justxor / deepseek-local-guide-ru
🐋 Полный гайд на русском: как установить и запустить DeepSeek локально (V4, R1) на ПК, Mac и сервере — Ollama, llama.cpp, LM Studio, vLLM, Open WebUI, API, RAG, агенты, fine-tuning, лучшие фишки
UFund-Me / Qbot
[🔥updating ...] AI 自动量化交易机器人(完全本地部署) AI-powered Quantitative Investment Research Platform. 📃 online docs: https://ufund-me.github.io/Qbot ✨ :news: qbot-mini: https://github.com/Charmve/iQuant
hiyouga / LlamaFactory
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
SYSTRAN / faster-whisper
Faster Whisper transcription with CTranslate2
RyanCodrai / turbovec
A vector index built on TurboQuant, written in Rust with Python bindings
FareedKhan-dev / kimi-k3-in-c
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
bitsandbytes-foundation / bitsandbytes
Accessible large language models via k-bit quantization for PyTorch.
cactus-compute / cactus
Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.
NVIDIA / Model-Optimizer
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
OpenNMT / CTranslate2
Fast inference engine for Transformer models
lucidrains / vector-quantize-pytorch
Vector (and Scalar) Quantization, in Pytorch
vllm-project / llm-compressor
State-of-the-art LLM compression, built for production inference with vLLM
huggingface / optimum
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
amitshekhariitbhu / ai-engineering-interview-questions
Your Cheat Sheet for AI Engineering Interview – Questions and Answers.
pytorch / ao
PyTorch native quantization for training and inference
qualcomm / aimet
AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.
intel / neural-compressor
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime
stochasticai / xTuring
Build, personalize and control your own LLMs. From data pre-processing to fine-tuning, xTuring provides an easy way to personalize open-source LLMs. Join our discord community: https://discord.gg/TgHXuSJEk6
AI-Efficiency / Awesome-Model-Quantization
A curated collection of papers, benchmarks, surveys, and tools for model quantization, covering low-bit networks, LLMs, multimodal and generative models, vector and lattice quantization, and efficient deployment.
microsoft / Olive
Olive: Simplify ML Model Finetuning, Conversion, Quantization, and Optimization for CPUs, GPUs and NPUs.
syv-ai / HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
harry7557558 / spirula-studio
Cross-vendor 3D Gaussian Splatting trainer - video to splat to mesh, Vulkan or CUDA.
Tencent / AngelSlim
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
turboderp-org / exllamav3
An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs
intel / auto-round
PTQ-Level Efficiency, QAT-Level Accuracy|训练后量化的效率，量化感知训练的精度
JustGlowing / minisom
:red_circle: MiniSom is a minimalistic implementation of the Self Organizing Maps
Xilinx / brevitas
Brevitas: neural network quantization in PyTorch
tensorflow / model-optimization
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
open-edge-platform / geti
Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel XPU (CPU-only and CUDA also supported).
vipshop / cache-dit
A PyTorch-native inference engine with cache, parallelism, quantization and cpu offload for DiTs.
ModelCloud / GPTQModel
LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.
mw00 / project-maya
Project Maya - run GLM-5.3-Flash (321B MoE) and DeepSeek-V4.1-Flash-1017 (748B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.
openvinotoolkit / nncf
Neural Network Compression Framework for enhanced OpenVINO™ inference
Xilinx / finn
Dataflow compiler for QNN inference on FPGAs
huggingface / optimum-quanto
A pytorch quantization backend for optimum
patchy631 / time-to-first-token
A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking.
whitecircle / halo
Halo is an open-source framework built by White Circle for training large language and multimodal models
hailo-ai / hailo_model_zoo
The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment
warpfront / hipfire
RDNA-native LLM inference engine in Rust.
Helldez / BigMoeOnEdge
Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.
wanshuiyin / ARIS-in-AI-Offer
Bilingual (中文+EN) ML / LLM / diffusion / agent interview cheat sheets for AI 秋招 — generated by ARIS /interview-cheatsheet, rendered by /render-html into single-file HTML, reads anywhere — plus a CV→DBLP-fact-checked academic homepage generator and hand-authored long-form blogs 🌱
Continuum-AI-Corp / OrcaBonsai-27B-Uncensored
Runtime behavioral ablation for compressed LLMs. First target: Ternary Bonsai 2 27B — no weight modification or re-quantization. by OrcaRouter team
jaylfc / taOS
Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).
FutureMLS-Lab / OSCAR
OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization
zengxiao-he / tessera
From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
AlibabaResearch / SparkDiffusion
SparkDiffusion: Accelerating DiT video generation by 265× via joint Sparsity, Distillation, Quantization, etc.
prabanta-dev / janas
37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
fermionresearch / phonon
Phonon: open speech recognition models (Phonon-2, Phonon-1) — CLI, CPU and CUDA images
Low-Zi-Hong / ESP32s3-LLM-Cluster
7-node ESP32-S3 cluster running a 0.4B LLM via 1.58-bit (BitNet) ternary quantization over SPI daisy-chain
Fazmin / AILearningGuide
Interactive desktop app with 51 labs covering ML fundamentals, transformers, RAG, agents, fine-tuning and quantization and more.
RadixRootMind / Awesome-Quant-VLA-WAM
Curated quantization research for VLA & WAM models — 180+ papers, a five-category taxonomy, a roadmap, and a companion PTQ + LIBERO toolkit (quant-vla).
pyq623 / QuaNAG
Official implementation of QuaNAG: LLM-driven Memory-bounded Neural Architecture Generation via Quantization-branched Exploration
zwhy149 / bead-grid-studio
拼豆图纸生成器｜图片转拼豆、拼豆像素画与可编辑施工图｜逐格色号、辅助线、底板布局、用量统计｜Local-first fuse-bead pattern generator
Yamz-Labs / kyojin
Kyojin: the Yamz inference engine for AMD Strix Halo (ROCm, gfx1151), built on ExLlamaV3. Runs 300B-class MoE models on one 128 GB mini PC.
JakeATX / llamAmpere
llama.cpp fork for significantly improved performance on Ampere (especially RTX 3090 / 3090 Ti): example: 95+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx
dnv2003 / v100-skinny
Hand-written NVFP4 W4A16 CUDA kernels for Volta
Blackwellboy / model-serving-minefield
Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each.
DominikBucko / qwen38-flash-next-2x3090
Qwen3.8-Flash-Next on 2× RTX 3090: with 128 GB RAM up to 4,191 tok/s prefill · 111.5 tok/s decode (131K prompt), full 256K window at 2,865 tok/s; with 64 GB RAM 3,410 tok/s prefill · 84 tok/s decode. New: opt-in uncensored mode (runtime abliteration, no new weights).
mattn / tensai
A tiny neural-network framework in pure Go with AVX2 SIMD kernels (GOEXPERIMENT=simd)