langfuse / langfuse
🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
langfuse / langfuse
mastra-ai / mastra
Mastra is the modern TypeScript framework for AI-powered applications and agents.
harbor-framework / harbor
Framework for evaluating and improving agents
ai-boost / awesome-harness-engineering
Awesome list for AI agent harness engineering: tools, patterns, evals, memory, MCP, permissions, observability, and orchestration.
LilMGenius / paperthin
Low-level agentic design patterns. Turning old engineering wisdom into reflexes your agent reaches for on its own—on any agent.
ombharatiya / ai-system-design-guide
AI system design guide for engineers building production AI systems and evals.
Arize-ai / phoenix
AI Observability & Evaluation
Kiln-AI / Kiln
Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
heaven999b / hello-agent-system
👋 Hello Agent System — learn enterprise AI agent system design by building every production layer from scratch. 17 bilingual lessons (EN/中文) with tested exercises: tools, architectures, reliability, security, evals, distributed execution, cost, RAG, release ops. 企业级 Agent 系统设计训练营
modelscope / evalscope
A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.
ShenSeanChen / waku-agent
Waku Waku! Waku Agent is a local-first AI agent harness you actually own, including loop, memory, eval, all in code built to stay legible as it grows.
neatlogs / neatroots
neatroots: see, check and replay your AI agent's runs on your machine while you build it. Local-first, by Neatlogs.
thetahealth / mirobody-env
HAEnv — synthetic longitudinal patients with code-derived gold, for evaluating health agents over time
langwatch / langwatch
Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.
lmnr-ai / lmnr
Laminar - open-source observability platform purpose-built for AI agents. YC S24.
MCPJam / inspector
Testing and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.
ai-evals-course / evals-skills
Skills that guide AI coding agents to help you build product-specific AI evals.
robocurve / inspect-robots
Open source evals for physical AI. Run any LLM/VLA on any arm/humanoid against any real/sim benchmark.
ibragim-bad / evals-from-sessions
Turn your coding-agent sessions into private evals and find the cheapest setup that still solves your tasks.
PrimeIntellect-ai / verifiers
Our library for RL environments + evals
evalstate / fast-agent
Code, Build and Evaluate agents - excellent Model and Skills/MCP/ACP/A2A Support
future-agi / future-agi
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
openai / frontier-evals
OpenAI Frontier Evals
heymrun / heym
Build agentic systems. Run them with confidence. Orchestrate agents, automate business processes, inspect every execution, and keep humans in control. Deploy Heym on your own infrastructure.
langchain-ai / openevals
Readymade evaluators for your LLM apps
darkrishabh / agent-skills-eval
A test runner for agentskills.io-style AI agent skills
decodingai-magazine / building-a-coding-agent-from-scratch-course
Learn harness engineering by building Claude Code from scratch. Free, open-source course: 8 articles, 6 videos, 1 codebase.
overmind-core / overmind
The platform for continuously improving AI agents.
NVIDIA / SkillEvaluator
Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.
browser-use / browser-use-pi
A tiny TypeScript web agent, hill climbed on real browser evals.
ZSeven-W / craft-skills
Research-backed, eval-driven skills for AI agents
donvito / local-evals
An eval app which runs locally and uses local models or any OpenAI-compatible API
FailproofAI / failproofai
Observability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement.
pydantic / logfire
AI observability platform for production LLM and agent systems.
truera / trulens
Evaluation and Tracking for LLM Experiments and AI Agents
superlinear-ai / raglite
🥤 RAGLite is a Python toolkit for Retrieval-Augmented Generation (RAG) with DuckDB or PostgreSQL
JudgmentLabs / judgeval
The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.
braintrustdata / autoevals
AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices.
benchflow-ai / awesome-evals
A curated, non-BS library of the best resources for building and evaluating AI agents — papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow.
chirpz-ai / pandaprobe
open source agent engineering platform: traces, evals, and metrics to debug and improve your AI agents. Integrates with LangGraph, CrewAI, Claude Agent SDK, and more.
calmrocks / ai-engineer-notebooks
Hands-on, framework-free Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set — model APIs, structured output, tool calling, RAG, evals-as-the-spine, agents (loop from scratch, tool design, guardrails, MCP, Skills), fine-tuning vs LoRA, prompt-injection/security, LLMOps, and customer craft. Runs on the free Groq API.
UKGovernmentBEIS / inspect_evals
Collection of evals for Inspect AI
keyuchen21 / agentic-engineering-handbook
The definitive OpenAI, Claude, MCP, Harness, Evals, and Production Agent Systems learning roadmap.
frontier-harness-eval / eval
Public results and task definitions for FrontierHarness Eval
1173591564 / Dynamics-memory
LLM agent 的有界自纠错长期记忆层：把连续、含矛盾的项目交互日志蒸馏成有界、可自更新的记忆库。Bounded, self-correcting long-term memory for LLM agents — candidate pool + value dynamics + tension adjudication + causal-replay evals.
weiwei966 / awesome-ai-harness
The model is the engine; the harness is the car. Curated knowledge on harness engineering — context management, tool design, agent loops, memory, sandboxing, evals. EN/中文
rails / ai-evals
Evals and benchmarks for AI models
AgentX-ai / AgentX-Trace-Eval
The trust layer for your AI agent on any platform. Trace and evaluate your agent. Full Observability. Run Model portability analysis. Autotune your agent.
openlayer-ai / jevals
Agent evals and guardrails as Jev decisions: one request per trace, a fraction of a cent, fast enough for the agent loop. Runs locally with Kev or Laya.
thiientv / godmode
Production-grade Agent Skills for AI coding agents—composable workflows for planning, TDD, debugging, review, UI/UX, releases, incidents, and evals.
sun461941-hub / ai-project-copilot
🚀 Turn ideas and repositories into showcase-ready AI projects with agents, RAG, multimodal AI, local models, evals, safety checks, and polished demos.
ai-evals-course / cartwheel-homeworks
Homework assignments for Evaluating and Improving AI Agents
rails / lemans
A Ruby harness for running agent benchmarks
ZJU-REAL / ageval
Agent eval on one running base. Swap the agent under test with plugins; run the same dataset anywhere.
Git-on-my-level / self-improving-agent-ecosystem
Reference architecture, contracts, skills, and fixtures for safe evaluator-driven self-improving ecosystems.
AI-Safety-Institute / transect
Transcript analysis for long-horizon agent evaluations
talhaanwarch / awesome-llmops-platforms
Curated list of open-source LLMOps tools: tracing, evals, AI gateways, guardrails and prompt management. Sorted by stars, refreshed weekly.
jaz-lang / jaz-evals
Reproduction package for "Harness as a Language: A Minimalist Agent Framework With Maximal Expressivity" — ten agent arms across AppWorld and StuLife
Ed-Marcavage / awesome-security-agent-harnesses
AI agents for pentesting, code audit, fuzzing, vulnerability discovery, and reverse engineering: harnesses, sandboxes, security MCP servers, benchmarks, and evals.
smkrv / jev-calibrate
Calibrate Jev questions against your own labels: tune criteria on labelled examples, confirm on a held-out set, get a verdict per question. Unofficial.