egotoadcitadel / palette-from-image-ai
AI coloring palette-from-image generator for designers and pixel artists — pull a clean named palette straight from any image, no subscription or cloud upload.
egotoadcitadel / palette-from-image-ai
hiyouga / LlamaFactory
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
SYSTRAN / faster-whisper
Faster Whisper transcription with CTranslate2
bitsandbytes-foundation / bitsandbytes
Accessible large language models via k-bit quantization for PyTorch.
NVIDIA / Model-Optimizer
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
lucidrains / vector-quantize-pytorch
Vector (and Scalar) Quantization, in Pytorch
vllm-project / llm-compressor
State-of-the-art LLM compression, built for production inference with vLLM
huggingface / optimum
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
pytorch / ao
PyTorch native quantization for training and inference
qualcomm / aimet
AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.
intel / neural-compressor
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime
stochasticai / xTuring
Build, personalize and control your own LLMs. From data pre-processing to fine-tuning, xTuring provides an easy way to personalize open-source LLMs. Join our discord community: https://discord.gg/TgHXuSJEk6
microsoft / Olive
Olive: Simplify ML Model Finetuning, Conversion, Quantization, and Optimization for CPUs, GPUs and NPUs.
syv-ai / HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
Tencent / AngelSlim
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
turboderp-org / exllamav3
An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs
intel / auto-round
PTQ-Level Efficiency, QAT-Level Accuracy|训练后量化的效率，量化感知训练的精度
JustGlowing / minisom
:red_circle: MiniSom is a minimalistic implementation of the Self Organizing Maps
Xilinx / brevitas
Brevitas: neural network quantization in PyTorch
tensorflow / model-optimization
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
open-edge-platform / geti
Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel XPU (CPU-only and CUDA also supported).
vipshop / cache-dit
A PyTorch-native inference engine with cache, parallelism, quantization and cpu offload for DiTs.
ModelCloud / GPTQModel
LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.
openvinotoolkit / nncf
Neural Network Compression Framework for enhanced OpenVINO™ inference
Xilinx / finn
Dataflow compiler for QNN inference on FPGAs
huggingface / optimum-quanto
A pytorch quantization backend for optimum
whitecircle / halo
Halo is an open-source framework built by White Circle for training large language and multimodal models
hailo-ai / hailo_model_zoo
The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment
wanshuiyin / ARIS-in-AI-Offer
Bilingual (中文+EN) ML / LLM / diffusion / agent interview cheat sheets for AI 秋招 — generated by ARIS /interview-cheatsheet, rendered by /render-html into single-file HTML, reads anywhere — plus a CV→DBLP-fact-checked academic homepage generator and hand-authored long-form blogs 🌱
Continuum-AI-Corp / OrcaBonsai-27B-Uncensored
Runtime behavioral ablation for compressed LLMs. First target: Ternary Bonsai 2 27B — no weight modification or re-quantization. by OrcaRouter team
jaylfc / taOS
Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).
FutureMLS-Lab / OSCAR
OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization
zengxiao-he / tessera
From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
AlibabaResearch / SparkDiffusion
SparkDiffusion: Accelerating DiT video generation by 265× via joint Sparsity, Distillation, Quantization, etc.
fermionresearch / phonon
Phonon: open speech recognition models (Phonon-2, Phonon-1) — CLI, CPU and CUDA images
pyq623 / QuaNAG
Official implementation of QuaNAG: LLM-driven Memory-bounded Neural Architecture Generation via Quantization-branched Exploration
Yamz-Labs / kyojin
Kyojin: the Yamz inference engine for AMD Strix Halo (ROCm, gfx1151), built on ExLlamaV3. Runs 300B-class MoE models on one 128 GB mini PC.
dnv2003 / v100-skinny
Hand-written NVFP4 W4A16 CUDA kernels for Volta
Blackwellboy / model-serving-minefield
Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each.
DominikBucko / qwen38-flash-next-2x3090
Qwen3.8-Flash-Next on 2× RTX 3090: with 128 GB RAM up to 4,191 tok/s prefill · 111.5 tok/s decode (131K prompt), full 256K window at 2,865 tok/s; with 64 GB RAM 3,410 tok/s prefill · 84 tok/s decode. New: opt-in uncensored mode (runtime abliteration, no new weights).
Jit-Roy / WeeLLM
WeeLLM runs large diffusion models with as little as 4 GB of VRAM, without any quantization. It dynamically determines how many layers can fit within the available VRAM and streams the text encoder and transformer layers to the GPU layer by layer, enabling inference on hardware with limited VRAM. It supports both safetensors and GGUF models.
htobty / ReadyLLM
Deploy, monitor, and tune LLM through a visual interface — plus image / video generation on top of ComfyUI.
Dreamer-Toby / STEPQuant
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Saganaki22 / ComfyUI-Breeze-TTS-2
Breeze TTS 2 nodes for ComfyUI: voice clone, voice design, and voice direction with bilingual (EN/ZH) speech, inline vocal events, INT8 ConvRot quantized builds, and full ComfyUI / AIMDO dynamic-VRAM integration.
julianmb / halofpx
High-performance Lemonade alternative for AMD Strix Halo & Radeon — latest MoE models (Ling-3.0-Flash, Qwen 3.8 Flash Next, Ornith 1.5), bleeding-edge RDNA 3.5 Wave64/ROCmFP4 kernels, and silicon-tuned profiles.
professorpalmer / bonsai-ada-surgery
Bonsai 2 27B: 262k context (trained max) on 12 GB, 54 to 97 tok/s decode, 2x prefill, q8_0 KV recipe, grammar tool calls 9/9. Same ternary weights. Windows bundle + receipts.
ayutaz / sanoTTS-jp
559 K パラメータの日本語 TTS を ESP32-S3 で実時間合成。漢字かな交じり文の形態素解析・アクセント推定まで端末内で走る（M5Stack CoreS3 実機で確認）。推論は依存ゼロの C99、ブラウザ demo あり。arXiv:2608.21378 sanoTTS の日本語 clean-room 再実装。⚠️ コードは MIT ですが、配布モデルの重みは MIT ではありません（LICENSE-MODEL.md。出力に用途制限が伝播します）
ForeverBlue816 / PrismQuant
PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers
0xBakeer / TandemLLM
An environment where open models work as agents: a first-level harness with routing, skills, memory and sandboxed tools, on an engine that drafts and verifies its own tokens.
chilly23 / RP2040-and-ESP32-AI
Run a 28.9M-parameter TinyLM on ESP32-S3 with an RP2040 OLED display node for fully local embedded AI inference.
0xBakeer / Qwen3.8-27B-4-bit-on-a-single-DGX-Spark
Serving 4-bit Qwen3.8-27B on a single DGX Spark (GB10): 75 tok/s single-stream, 246 tok/s aggregate at 8-way concurrency. NVFP4 vs MixedInt4-AutoRound vs the FP8 baseline, measured on one harness — including why the quantization advantage collapses to +0.2% by c16.
paicat1 / Bonsai-27B-NInfer
Bonsai-27B x NInfer local deployment project (RTX 5080, ternary quantization)
RadixRootMind / Radix-Embodied-Stack
Radix Embodied Stack: Universal infrastructure for VLA model quantization, inference, and deployment on domestic chips . Cross-model, cross-chip, cross-platform.
drJhonatan00 / Game_Translation_Pipeline
A high-performance, offline game translation framework built in Python. Powered by CTranslate2 (INT8 quantization) and Helsinki-NLP neural models, this tool automates multi-language translation for game scripts while strictly protecting low-level syntax, variables, and memory pointers.
PedroMarinhoDev / ComfyUI-HunyuanImage3
Run Tencent's HunyuanImage-3.0 (80B MoE) natively in ComfyUI on a 24 GB GPU: base, Instruct and Instruct-Distil, prompt rewriting, image editing, Spectrum speed-up.
sudoingX / dgx-spark-deepseek-v41
DeepSeek-V4.1-Flash on two NVIDIA DGX Sparks: EXL3 weights from the official checkpoint, upstream vLLM with our patches and decode kernels
RightNow-AI / local-kimi
Optimized local serving engine for Kimi-Linear-48B: INT4 quantizer, fused decode kernels for a measured 3.18x, and an OpenAI-compatible server. Ships with k3, a bridge that detects the client per request so Claude Code, Codex, Cline, Aider and opencode all work unchanged.
sepiablue-ai / ComfyUI-H3-X2-Stream
ComfyUI nodes for MiniMax H3: INT8 X2 VAE and asynchronous NVENC saving. 5-second 1088x1920 video with audio in about 60 s on an RTX 4070 12GB, about 30% faster than the FP16 X2 VAE.
ExVRAM / exvram-lab
Compute-for-memory research lab for running larger LLMs on memory-constrained consumer GPUs. Official website exvram.pro
yzfly / edgejev
离线可用的本地类型化决策：4 核 CPU 单题 15.6ms。Local & offline Jev / System One inference on CPU — ONNX + INT8, no torch at runtime. 支持 laya / kev / PlayJev