CSUNX233 / qwen3.8-27b-ninfer-5070tilaptop-kvmem-local-agent

RTX 5070 Ti Laptop 12GB 跑 Qwen3.8-27B：NInfer + KVMem、256K 逻辑上下文、RAM KV、MTP、快速缓存续接与 dsh 本地 Agent / 搜索 / 浏览器。含源码、Windows 部署及实测。

ggml-org / llama.cpp

LLM inference in C/C++

microsoft / BitNet

Official inference framework for 1-bit LLMs

mozilla-ai / llamafile

Distribute and run LLMs with a single file.

alibaba / MNN

MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.

alibaba / zvec

A lightweight, lightning-fast, in-process vector database

justcallmekoko / ESP32Marauder

A suite of WiFi/Bluetooth offensive and defensive tools for the ESP32

openvinotoolkit / openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

RunanywhereAI / runanywhere-sdks

Production ready toolkit to run AI locally

kvcache-ai / Mooncake

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

google-ai-edge / LiteRT-LM

LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.

cactus-compute / cactus

Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.

lemonade-sdk / lemonade

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk

pbek / QOwnNotes

QOwnNotes is a plain-text file notepad and todo-list manager with Markdown support and Nextcloud / ownCloud integration.

ztxz16 / fastllm

fastllm是后端无依赖的高性能大模型推理库。同时支持张量并行推理稠密模型和混合模式推理MOE模型，任意10G以上显卡即可推理满血DeepSeek。双路9004/9005服务器+单显卡部署DeepSeek满血满精度原版模型，单并发20tps；INT4量化模型单并发30tps，多并发可达60+。

infiniflow / infinity

The AI-native database built for LLM applications, providing incredibly fast hybrid search of dense vector, sparse vector, tensor (multi-vector), and full-text.

uxlfoundation / oneDNN

Open-source library of optimized deep learning operations (matmul, convolution, attention) for CPUs (x64, AArch64, RISC-V) and Intel GPUs. Used by PyTorch, TensorFlow, OpenVINO, and ONNX Runtime.

b4rtaz / distributed-llama

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

Luce-Org / lucebox

LLM speculative inference server for heterogeneous hardware & consumer GPUs

ROCm / FastFlowLM

Run LLMs on AMD Ryzen™ AI NPUs in minutes; purpose-built and deeply optimized for the AMD NPUs.

hyperfield / ai-file-sorter

Cross-platform desktop application for content-aware file organization and renaming. Supports local and remote LLMs, preview-based workflows, and fully user-controlled changes.

UbiquitousLearning / mllm

Fast Multimodal LLM on Mobile Devices

xLLM-AI / xllm

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

incoai / splash

A local inference engine for Apple silicon, built around the model.

uccl-project / uccl

UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)

lean-dojo / LeanCopilot

LLMs as Copilots for Theorem Proving in Lean

mw00 / project-maya

Project Maya - run GLM-5.3-Flash (321B MoE) and DeepSeek-V4.1-Flash-1017 (748B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.

AGenUI / AGenUI

Native A2UI Renderer for iOS/Android/HarmonyOS. High performance streaming Generative UI. Custom Components, Styles and more.

jmaczan / tiny-vllm

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

orailnoor / cross-platform-llm-client

A unified cross-platform AI client supporting seamless transitions between standard cloud APIs and on-device, offline execution of custom and uncensored language models.

Felix3322 / PotPlayer_ChatGPT_Translate

[support ollama/other model]【兼容ollama和其他模型】将任何具有OpenAI API调用方法的模型集成到PotPlayer中。它使你在观看视频时能够实时翻译字幕，从而打破语言障碍，提升你的观看体验。 This real-time subtitle translation plugin integrates OpenAI's ChatGPT API (or any model with the same API calling method) into PotPlayer. It enables you to translate subtitles on-the-fly while watching videos.

OpenSparX / MasterAgent

Build AI agents that run 100% on-device. Sub-100ms latency on Qualcomm NPU. Zero cloud dependency.

sophgo / tpu-mlir

Machine learning compiler based on MLIR for Sophgo TPU.

feder-cr / jev

jevos: an open-source alternative to Jev and Microsoft-Decision-1. Yes/no decisions, choices and scores with probabilities, on your laptop's CPU.

foldl / chatllm.cpp

Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)

epsilla-cloud / vectordb

A high performance Vector Database leveraging parallel graph computing

invergent-ai / surogate

Train and serve LLMs at extreme speed and massive throughput.

gufo-org / gufo

The Open Source Strix Halo AI engine. Qwen 3.8 Flash Next Q4_K_XL: 1,801.01 pp, 73.17 tg single user, 194.37 tok/s 8 users; Qwen 3.8 27B Q4_K_XL: 656.33pp, 70.56tg tok/s single user with DFlash2

OpenMOSS / MOSS-VL

An open-weight 11B model series for long-form and real-time video understanding

flashrt-project / FlashRT

FlashRT is a high-performance realtime inference engine for small-batch, latency-sensitive AI workloads. The flagship integration is production VLA control for Pi0, Pi0.5, GROOT N1.6, and Pi0-FAST. Also support llm e.g, qwen3.6-27B

Helldez / BigMoeOnEdge

Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.

agentty-org / agentty

AI pair programming in your terminal — one static binary, sub-ms startup, any model

openxlings / xlings

Universal package infrastructure with OS-like SubOS isolation - Multi-version · Rootless · Decentralized Index · Agent-ready.

cobanov / PS5LM

llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8

xifan2333 / fcitx5-vinput

Voice input for Fcitx5 — local and cloud ASR, LLM rewriting, cross-distro packages

intel / llm-scaler

—

brontoguana / krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

VecSzn / PolyStrata

Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.

Low-Zi-Hong / ESP32s3-LLM-Cluster

7-node ESP32-S3 cluster running a 0.4B LLM via 1.58-bit (BitNet) ternary quantization over SPI daisy-chain

Indras-Mirror / Xolotl

Xolotl (formerly Strata-DSV4.1): DeepSeek-V4.1-Flash, all 552B, on one gaming PC (RTX 4090 + 90 GB RAM) at ~27 tok/s, plus GLM-5.3-Flash, Qwen3.8-Flash-Next and Qwen3.8-27B. A heavily extended fork of Strata: MoE experts tiered across VRAM, pinned RAM and NVMe; OpenAI + Anthropic API.

JakeATX / llamAmpere

llama.cpp fork for significantly improved performance on Ampere (especially RTX 3090 / 3090 Ti): example: 95+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx

sergqwer / strata-nvfp4

Strata fork: Qwen3.8-Flash-Next 125B MoE in NVFP4 on one RTX 20-50 card (12 GB+) and 64 GB of RAM or more. Setup installs our GPTQ quants (huihui abliterated, OrcaRouter) from Hugging Face. CPU experts on AVX-512 or AVX2 (+AVX-VNNI), all experts in RAM from 92 GB, a low-RAM mode below, vision, W4A8 prompts. Windows / CUDA 13.

robertkarl / UnbiasedStrata

Strata minus the bullshit. clone, rsync to prod, compile and go

sq8l-revival / SQ8L

Faithful 64-bit port of SQ8L 0.91b, Siegfried Kullmann's Ensoniq SQ-80 emulation, for macOS and Windows (VST2/VST3/AU/CLAP)

IIIIIllllIIIIIlllll / Qwenox

An inference framework customized for AI MAX 395, compatible with both Linux and Windows, and supports deploying only qwen3.8-flash-next. This project is for reference and learning purposes only. 为AI MAX 395定制的推理框架，Linux和Windows通用，只能部署qwen3.8-flash-next。本项目仅供参考学习使用。

poisonxa16 / pxa

The set-and-forget LLM engine for Pascal and Volta: PXQ codec + kernels, auto-tuned per card. Ready-to-run PXQ models in MODELS.md; benchmarks vs llama.cpp in the README.

Tylogi / MFQ

Get more intelligence from every bit. Better quantization formats and smarter calibration let larger, stronger models run smoothly on the hardware you already own.

headpiece747 / ninfer-5090-windows

Native Windows port of NInfer engine for RTX 5090. Features Qwen3.8-27B with QUASAR and NInfer models, MTP/DFlash2 with vision and 262,144 context

shawnshekari / strixite

LLM inference engine written from scratch in HIP for AMD Strix Halo (gfx1151) and Qwen3.8-Flash-Next - ~50 tokens/s out to 492k context

quatrixone / ProsperoAI

A simple LLM-based chat app that runs on the PS5 GPU