BerriAI / litellm

Python SDK and self-hostable proxy server that calls 100+ LLM providers through a unified OpenAI-compatible API.

langfuse / langfuse

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

nullmoth / nvidia-macos-driver

Metal driver for NVIDIA GeForce cards on macOS Sequoia

promptfoo / promptfoo

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

respanai / respan-gateway

One endpoint for every LLM vendor. Any SDK, any model, any tool. One Rust binary.

MakazhanAlpamys / Soup

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

nullmoth / 1401

Install macOS on your PC from Windows

leopard627 / fire-your-seo-agency

Fire your SEO·GEO agency 🔥 A Claude Code skill that audits and optimizes SEO·AEO·GEO·LLMO·NEO(Naver) by itself — 월 50~350만 원짜리 'AI 검색 최적화' 대행, AI 에이전트가 대체합니다

comet-ml / opik

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

ComposioHQ / composio

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

mlflow / mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

GokuMohandas / Made-With-ML

A free course teaching how to design, develop, deploy, and iterate on production-grade machine learning applications in Python.

Arize-ai / phoenix

AI Observability & Evaluation

ShenSeanChen / waku-agent

Waku Waku! Waku Agent is a local-first AI agent harness you actually own, including loop, memory, eval, all in code built to stay legible as it grows.

dipakkr / awesome-ai-engineering

🚀 A curated collection of AI engineering resources, roadmaps, tools, and projects covering LLMs, RAG, AI Agents, MCP, fine-tuning, evaluations, and production AI systems.

0xPlaygrounds / rig

⚙️🦀 Build modular and scalable LLM Applications in Rust

mozilla-ai / otari

Open-source, OpenAI-compatible LLM gateway you run yourself. One endpoint for 40+ providers, with virtual keys, budgets, and usage tracking.

heaven999b / hello-agent-system

👋 Hello Agent System — learn enterprise AI agent system design by building every production layer from scratch. 17 bilingual lessons (EN/中文) with tested exercises: tools, architectures, reliability, security, evals, distributed execution, cost, RAG, release ops. 企业级 Agent 系统设计训练营

liguodongiot / llm-action

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

future-agi / future-agi

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

lmnr-ai / lmnr

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

overmind-core / overmind

The platform for continuously improving AI agents.

dataelement / bisheng

BISHENG is an open LLM devops platform for next generation Enterprise AI applications. Powerful and comprehensive features include: GenAI workflow, RAG, Agent, Unified model management, Evaluation, SFT, Dataset Management, Enterprise-level System Management, Observability and more.

blixvip / NullMotion

Play a finished motion-graphics ad in sync with the rough drafts it grew from, and export the breakdown as MP4.

openlit / openlit

OpenLIT is the open-source agent harness engineering platform: trace, evaluate, guard, and improve everything around the model in your AI agents, on OTEL.

Abilityai / trinity

Self-hosted AI Agents Platform for AI Native companies. Claude Code, Codex, Gemini, OpenCode agents. Apache 2.0.

langwatch / langwatch

Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.

Arize-ai / openinference

OpenTelemetry Instrumentation for AI Observability

Helicone / helicone

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

Giskard-AI / giskard-oss

🐢 Open-Source Evaluation & Testing library for LLM Agents

coze-dev / coze-loop

Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring.

truera / trulens

Evaluation and Tracking for LLM Experiments and AI Agents

vllm-project / vllm-ascend

Community maintained hardware plugin for vLLM on Huawei Ascend

data-infra / cube-studio

cubestudio开源云原生一站式机器学习/深度学习/大模型AI平台/MaaS/mlops/人工智能平台/训推平台，算法全链路流程，多租户，算力租赁平台，token中转，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务，VGPU虚拟化，云边端协同，边缘计算，自动化标注平台，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库llmops智能体，AI模型市场，支持国产异构算力调度,昇腾/寒武纪/海光/摩尔/沐曦等，支持ib/roce/RDMA，信创支持

Netflix / metaflow

Build, Manage and Deploy AI/ML Systems

katanemo / plano

Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.

clearml / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

Fast-Editor / Lynkr

Streamline your workflow with Lynkr, a CLI tool that acts as an HTTP proxy for efficient code interactions using Claude Code CLI.

tensorchord / Awesome-LLMOps

An awesome & curated list of best LLMOps tools for developers

zenml-io / zenml

ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.

jwuthri-cloud / Tracely-ai

Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable.

mikehasa / agentacct

See what your coding agents did and what it cost. Breaks each task down into work steps — tools used, files changed, tests run, time and tokens spent. Local-first dashboard for Claude Code, Codex, OpenCode, and more. No login, no telemetry.

calmrocks / ai-engineer-notebooks

Hands-on, framework-free Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set — model APIs, structured output, tool calling, RAG, evals-as-the-spine, agents (loop from scratch, tool design, guardrails, MCP, Skills), fine-tuning vs LoRA, prompt-injection/security, LLMOps, and customer craft. Runs on the free Groq API.

bentoml / OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

bentoml / BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

apache / hamilton

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

wanmol / goal-flow

Graph-Orchestrated Agent Loop — a production-grade framework on LangGraph. Combine workflow graphs and agent loops, transpile Dify DSL to runnable code, swap wire protocols (Dify/OpenAI).

nullmoth / 1401-probe

Free, offline hardware scanner on a USB stick. Scans a PC's hardware and saves one report; never touches your drives or the network.

francordel / myrtia

Myrtia — agent-native red teaming for LLMs and AI agents: a deliberating council of attacker models (propose, cross-critique, Borda vote, chairman synthesis), 13 attack strategies, live turn-by-turn UI and ASR analytics. Open-source, built on IBM ARES.

AixleHQ / flow

Aixle Flow — orchestrate coding agents through durable, inspectable workflows.

evidentlyai / evidently

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

ianarawjo / ChainForge

An open-source visual programming environment for battle-testing prompts to LLMs.

apache / burr

Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.

bionic-gpt / bionic-gpt

Bionic is sovereign Agentic AI for the enterprise — Runs on-premise and can securely work with your sensitive data and systems.

alibaba / rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

bosun-ai / swiftide

Fast, streaming indexing, query, and agentic LLM applications in Rust

superlinked / VectorHub

Deprecated historical repo. Superlinked now develops SIE, a self-hosted inference engine for embeddings, reranking, OCR, extraction, and document processing.

limelit-co / open

Self-hosted AI visibility tracking (AEO / GEO / LLMO). Track how ChatGPT, Claude, Perplexity, Gemini and Google AI Overviews mention and cite your brand. One Go binary, SQLite, MCP-first, bring your own keys.

apowerb / apowerb

The open-source agentic framework to build, orchestrate, and operate production AI agents.

talhaanwarch / awesome-llmops-platforms

Curated list of open-source LLMOps tools: tracing, evals, AI gateways, guardrails and prompt management. Sorted by stars, refreshed weekly.