egotoadcitadel / palette-from-image-ai

AI coloring palette-from-image generator for designers and pixel artists — pull a clean named palette straight from any image, no subscription or cloud upload.

hiyouga / LlamaFactory

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

SYSTRAN / faster-whisper

Faster Whisper transcription with CTranslate2

bitsandbytes-foundation / bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.

NVIDIA / Model-Optimizer

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

lucidrains / vector-quantize-pytorch

Vector (and Scalar) Quantization, in Pytorch

vllm-project / llm-compressor

State-of-the-art LLM compression, built for production inference with vLLM

huggingface / optimum

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

pytorch / ao

PyTorch native quantization for training and inference

qualcomm / aimet

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.

intel / neural-compressor

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

stochasticai / xTuring

Build, personalize and control your own LLMs. From data pre-processing to fine-tuning, xTuring provides an easy way to personalize open-source LLMs. Join our discord community: https://discord.gg/TgHXuSJEk6

microsoft / Olive

Olive: Simplify ML Model Finetuning, Conversion, Quantization, and Optimization for CPUs, GPUs and NPUs.

syv-ai / HyperQwen

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

Tencent / AngelSlim

Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

turboderp-org / exllamav3

An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs

intel / auto-round

PTQ-Level Efficiency, QAT-Level Accuracy|训练后量化的效率，量化感知训练的精度

JustGlowing / minisom

:red_circle: MiniSom is a minimalistic implementation of the Self Organizing Maps

Xilinx / brevitas

Brevitas: neural network quantization in PyTorch

tensorflow / model-optimization

A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.

open-edge-platform / geti

Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel XPU (CPU-only and CUDA also supported).

vipshop / cache-dit

A PyTorch-native inference engine with cache, parallelism, quantization and cpu offload for DiTs.

ModelCloud / GPTQModel

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

openvinotoolkit / nncf

Neural Network Compression Framework for enhanced OpenVINO™ inference

Xilinx / finn

Dataflow compiler for QNN inference on FPGAs

huggingface / optimum-quanto

A pytorch quantization backend for optimum

whitecircle / halo

Halo is an open-source framework built by White Circle for training large language and multimodal models

hailo-ai / hailo_model_zoo

The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment

wanshuiyin / ARIS-in-AI-Offer

Bilingual (中文+EN) ML / LLM / diffusion / agent interview cheat sheets for AI 秋招 — generated by ARIS /interview-cheatsheet, rendered by /render-html into single-file HTML, reads anywhere — plus a CV→DBLP-fact-checked academic homepage generator and hand-authored long-form blogs 🌱

Continuum-AI-Corp / OrcaBonsai-27B-Uncensored

Runtime behavioral ablation for compressed LLMs. First target: Ternary Bonsai 2 27B — no weight modification or re-quantization. by OrcaRouter team

jaylfc / taOS

Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).

FutureMLS-Lab / OSCAR

OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

zengxiao-he / tessera

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

AlibabaResearch / SparkDiffusion

SparkDiffusion: Accelerating DiT video generation by 265× via joint Sparsity, Distillation, Quantization, etc.

fermionresearch / phonon

Phonon: open speech recognition models (Phonon-2, Phonon-1) — CLI, CPU and CUDA images

pyq623 / QuaNAG

Official implementation of QuaNAG: LLM-driven Memory-bounded Neural Architecture Generation via Quantization-branched Exploration

Yamz-Labs / kyojin

Kyojin: the Yamz inference engine for AMD Strix Halo (ROCm, gfx1151), built on ExLlamaV3. Runs 300B-class MoE models on one 128 GB mini PC.

dnv2003 / v100-skinny

Hand-written NVFP4 W4A16 CUDA kernels for Volta

Blackwellboy / model-serving-minefield

Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each.

DominikBucko / qwen38-flash-next-2x3090

Qwen3.8-Flash-Next on 2× RTX 3090: with 128 GB RAM up to 4,191 tok/s prefill · 111.5 tok/s decode (131K prompt), full 256K window at 2,865 tok/s; with 64 GB RAM 3,410 tok/s prefill · 84 tok/s decode. New: opt-in uncensored mode (runtime abliteration, no new weights).

Jit-Roy / WeeLLM

WeeLLM runs large diffusion models with as little as 4 GB of VRAM, without any quantization. It dynamically determines how many layers can fit within the available VRAM and streams the text encoder and transformer layers to the GPU layer by layer, enabling inference on hardware with limited VRAM. It supports both safetensors and GGUF models.

htobty / ReadyLLM

Deploy, monitor, and tune LLM through a visual interface — plus image / video generation on top of ComfyUI.

Dreamer-Toby / STEPQuant

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Saganaki22 / ComfyUI-Breeze-TTS-2

Breeze TTS 2 nodes for ComfyUI: voice clone, voice design, and voice direction with bilingual (EN/ZH) speech, inline vocal events, INT8 ConvRot quantized builds, and full ComfyUI / AIMDO dynamic-VRAM integration.

julianmb / halofpx

High-performance Lemonade alternative for AMD Strix Halo & Radeon — latest MoE models (Ling-3.0-Flash, Qwen 3.8 Flash Next, Ornith 1.5), bleeding-edge RDNA 3.5 Wave64/ROCmFP4 kernels, and silicon-tuned profiles.

professorpalmer / bonsai-ada-surgery

Bonsai 2 27B: 262k context (trained max) on 12 GB, 54 to 97 tok/s decode, 2x prefill, q8_0 KV recipe, grammar tool calls 9/9. Same ternary weights. Windows bundle + receipts.

ayutaz / sanoTTS-jp

559 K パラメータの日本語 TTS を ESP32-S3 で実時間合成。漢字かな交じり文の形態素解析・アクセント推定まで端末内で走る（M5Stack CoreS3 実機で確認）。推論は依存ゼロの C99、ブラウザ demo あり。arXiv:2608.21378 sanoTTS の日本語 clean-room 再実装。⚠️ コードは MIT ですが、配布モデルの重みは MIT ではありません（LICENSE-MODEL.md。出力に用途制限が伝播します）

ForeverBlue816 / PrismQuant

PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers

0xBakeer / TandemLLM

An environment where open models work as agents: a first-level harness with routing, skills, memory and sandboxed tools, on an engine that drafts and verifies its own tokens.

chilly23 / RP2040-and-ESP32-AI

Run a 28.9M-parameter TinyLM on ESP32-S3 with an RP2040 OLED display node for fully local embedded AI inference.

0xBakeer / Qwen3.8-27B-4-bit-on-a-single-DGX-Spark

Serving 4-bit Qwen3.8-27B on a single DGX Spark (GB10): 75 tok/s single-stream, 246 tok/s aggregate at 8-way concurrency. NVFP4 vs MixedInt4-AutoRound vs the FP8 baseline, measured on one harness — including why the quantization advantage collapses to +0.2% by c16.

paicat1 / Bonsai-27B-NInfer

Bonsai-27B x NInfer local deployment project (RTX 5080, ternary quantization)

Saganaki22 / ComfyUI-AuK

Local AuK Base and AuK-Flash speech generation, voice cloning, editing, enhancement and separation. Uses ComfyUI model management, attention and quantized operations.

RadixRootMind / Radix-Embodied-Stack

Radix Embodied Stack: Universal infrastructure for VLA model quantization, inference, and deployment on domestic chips . Cross-model, cross-chip, cross-platform.

drJhonatan00 / Game_Translation_Pipeline

A high-performance, offline game translation framework built in Python. Powered by CTranslate2 (INT8 quantization) and Helsinki-NLP neural models, this tool automates multi-language translation for game scripts while strictly protecting low-level syntax, variables, and memory pointers.

PedroMarinhoDev / ComfyUI-HunyuanImage3

Run Tencent's HunyuanImage-3.0 (80B MoE) natively in ComfyUI on a 24 GB GPU: base, Instruct and Instruct-Distil, prompt rewriting, image editing, Spectrum speed-up.

sudoingX / dgx-spark-deepseek-v41

DeepSeek-V4.1-Flash on two NVIDIA DGX Sparks: EXL3 weights from the official checkpoint, upstream vLLM with our patches and decode kernels

RightNow-AI / local-kimi

Optimized local serving engine for Kimi-Linear-48B: INT4 quantizer, fused decode kernels for a measured 3.18x, and an OpenAI-compatible server. Ships with k3, a bridge that detects the client per request so Claude Code, Codex, Cline, Aider and opencode all work unchanged.

sepiablue-ai / ComfyUI-H3-X2-Stream

ComfyUI nodes for MiniMax H3: INT8 X2 VAE and asynchronous NVENC saving. 5-second 1088x1920 video with audio in about 60 s on an RTX 4070 12GB, about 30% faster than the FP16 X2 VAE.

ExVRAM / exvram-lab

Compute-for-memory research lab for running larger LLMs on memory-constrained consumer GPUs. Official website exvram.pro