confident-ai / deepeval
The LLM Evaluation Framework
confident-ai / deepeval
comet-ml / opik
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
scaleapi / agentenv-framework
Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments tools, dynamism of the environment, reproducibility, and more. There is no open source framework for building these environments effectively. Until now.
superwesleyhys-ux / factcircuit
FactCircuit: an auditable claim-and-evidence verification loop with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.
mlflow / mlflow
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
Tencent / AI-Infra-Guard
A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.
jwuthri-cloud / Tracely-ai
Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable.
liguozheng-ac / j-gamelqa
Hybrid rule-based and context-aware LLM evaluation system for Chinese-to-Japanese game localization QA.
Arize-ai / phoenix
AI Observability & Evaluation
TIGER-AI-Lab / ClawBench
Open-source benchmark for browser AI agents on daily tasks.
xinxuxin / keystone-bench
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
Q00 / ouroboros
Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI, Gemini CLI, OpenCode, Copilot, Kiro and more.
thetahealth / mirobody-env
HAEnv — synthetic longitudinal patients with code-derived gold, for evaluating health agents over time
NVIDIA / garak
the LLM vulnerability scanner
Danielxu0208 / rsi_eval
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.
Giskard-AI / giskard-oss
🐢 Open-Source Evaluation & Testing library for LLM Agents
future-agi / future-agi
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
decodingai-magazine / building-a-coding-agent-from-scratch-course
Learn harness engineering by building Claude Code from scratch. Free, open-source course: 8 articles, 6 videos, 1 codebase.
overmind-core / overmind
The platform for continuously improving AI agents.
whitecircle / rise
RISE: Red-teaming via Iterative Strategy Evolution (NeurIPS 2026)
whichmen / dxl-commerce-agent
OpenClaw + 大模型驱动的完整电商客服自动化系统：覆盖淘宝、拼多多、抖音、快手、小红书、视频号，包含浏览器/手机 Worker、订单物流与售后工具、记忆、回执、恢复和 Eval。
oumi-ai / oumi
Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!
open-compass / opencompass
OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
EvolvingLMMs-Lab / lmms-eval
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
truera / trulens
Evaluation and Tracking for LLM Experiments and AI Agents
msoedov / agentic_security
Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪
cvs-health / uqlm
[JMLR 2026] UQLM: A Python package for LLM hallucination detection using uncertainty quantification
JudgmentLabs / judgeval
The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.
EverMind-AI / SkillCorpus
Open-source infrastructure that turns scattered SKILL.md files into curated, retrieval-ready agent-skill corpora—with retrieval and evaluation tooling included.
HowieHwong / TrustLLM
[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.
locuslab / open-unlearning
[NeurIPS D&B '25] The one-stop repository for LLM unlearning
OmniJev / awesome-jev-gallery
🔥🔥 Awesome Jev: open-source models, projects, benchmarks, and independent evaluations for Jev and System One models.
aaron-for-value / VeriRun
Evidence-first infrastructure for reproducible, isolated executable evaluation and online rewards.
ktwu01 / benchmark-radar
Track 33,707+ AI benchmark, eval, dataset, and data-quality records from 40 public sources, with linked evidence and daily updates.
jinzijian / EvoTrace
Compile real-world Claude Code and Codex trajectories into verified, tradable post-training assets.
i3T4AN / KADATH
Evolutionary multi-agent runtime that breeds, evaluates, and improves autonomous agents across reproducible epochs to converge on optimization of a goal.
jsdhwfmax / EvalForge
Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.
solis-team / XRepoTest
[EMNLP 2026] A multilingual benchmark for repository-level unit test generation, evaluating LLMs across 5 programming languages and real-world code under reproducible, Docker-based execution.
MrZoyo / deslop-GPT
Deletion-first Agent Skill for removing test bloat, verification theater, and speculative fallbacks while preserving behavior.
openlayer-ai / jevals
Agent evals and guardrails as Jev decisions: one request per trace, a fraction of a cent, fast enough for the agent loop. Runs locally with Kev or Laya.
To3akaRin / VLMForge
Evaluate visual models on your own images, JSON Schema, and production constraints with LangGraph, Pareto analysis, and a no-key Replay demo.
NocodeMrLi / mini-program-engineering-skill-suite
Mini Program Engineering Skill Suite is an Agent Skill suite for evidence‑first mini‑program development. It helps agents bring WeChat or other mini‑program projects from vague intent to reliable engineering work: project intake, product specification, architecture, implementation, UI & device adaptation, debugging, verification, release readiness.
harish-gaggar / mcp-server-gateway
MCP gateway & OAuth MCP servers. LangGraph agent: memory, eval/tracking, PII guard, context trimming. Docker quick start for enterprise agents.
OpenMOSS / SWE-bench-Science
A benchmark for testing whether coding agents can resolve engineering tasks in scientific software
wanqili857-byte / fictionforge
多智能体小说写作框架 | Multi-agent novel-writing framework — 每个角色都有自己的脑子,每一章都有质量门禁
pipelinelabai / paperbench-expanded
PaperBench High-Difficulty and Expanded Tasks: paper reproduction beyond machine learning. 93 tasks in 12 research areas, run in domain-native software and graded by clean replay. 12 public tasks.
PyModel / jev-judge-mcp
Typed judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate, and score: the model judges, policy decides auto, review, or escalate.
datamllab / A2E
—
zjunlp / AutoSciRub
Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
nickname21kmr / gameops-investigator
Evidence-first game operations incident investigation agent with read-only MCP tools and reproducible evaluations.
talhaanwarch / awesome-llmops-platforms
Curated list of open-source LLMOps tools: tracing, evals, AI gateways, guardrails and prompt management. Sorted by stars, refreshed weekly.
Gen-Verse / PAST-Bench
PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents
gwendolynashe / llm-evaluation-workbench
Argolink-focused LLM evaluation and regression workflow for GPT-6 Sol.
Zaious / jev-capability-atlas
Independent, evidence-based map of when TypeSafe's Jev actually holds up vs. breaks down — real API-call receipts, not a leaderboard. 中文為主的雙語 repo。
francordel / myrtia
Myrtia — agent-native red teaming for LLMs and AI agents: a deliberating council of attacker models (propose, cross-critique, Borda vote, chairman synthesis), 13 attack strategies, live turn-by-turn UI and ASR analytics. Open-source, built on IBM ARES.
sumleo / RLCDAlignBench
RLCDAlignBench: 44 alignment-failure detection benchmarks and code for 'Just Ask Jev' (RLCD zero-shot detector of AI alignment failures)
parthmax2 / ghostrun
pytest for AI Apps — Fast, free, offline testing and self-improving prompts for Python developers.
NEDONION / evalhub
An LLM Benchmark evaluation platform for automated model testing and analysis. 面向大语言模型的 Benchmark 自动化评测平台，支持模型测试与结果分析