ifixai-ai / iFixAi

Python CLI and agent plugin that audits whether an AI agent does the job it is supposed to do, using adversarial and operational tests.

AgentSafeLabs / safelabs-eval

Red-teaming and evaluation framework for AI agents, built around an OWASP-inspired ASI01–ASI10 taxonomy

confident-ai / deepeval

The LLM Evaluation Framework

OpenDCAI / DataFlex

[NeurIPS 2026] DataFlex: A Unified Benchmark and Evaluation Platform for Data-Centric Training of Large Language Models

Sanssssssssssssssss / ERP-agent

A Windows workbench and native Python Pi harness for Odoo 19: approved business actions, state readback, traceable runs, and ERP-Bench evaluation.

JEen-wang / Adaptive-rag

按问题复杂度自动分流的电商客服 Agent：简单问题直接回答；复杂问题经过多跳检索并自我纠错；查单、退改通过工具调用完成，且高风险写操作必须先经人工确认。基于 FastAPI 和 LangGraph，混合检索并附带引用，内置离线评测与 Prometheus 指标。

mll-lab-nu / embodied-first-exam

The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.

comet-ml / opik

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

scaleapi / agentenv-framework

Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments tools, dynamism of the environment, reproducibility, and more. There is no open source framework for building these environments effectively. Until now.

groundingpi / GroundingPI

A 4B grounding foundation model for physical intelligence. Unifies detection, referring, pointing, OCR, GUI, and layout grounding, achieving SOTA overall across 34 benchmarks and improving robot generalization and data efficiency in controlled evaluations.

superwesleyhys-ux / factcircuit

FactCircuit: an auditable claim-and-evidence verification loop with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.

mlflow / mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

dataelement / bisheng

BISHENG is an open LLM devops platform for next generation Enterprise AI applications. Powerful and comprehensive features include: GenAI workflow, RAG, Agent, Unified model management, Evaluation, SFT, Dataset Management, Enterprise-level System Management, Observability and more.

Tencent / AI-Infra-Guard

A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.

jwuthri-cloud / Tracely-ai

Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable.

liguozheng-ac / j-gamelqa

Hybrid rule-based and context-aware LLM evaluation system for Chinese-to-Japanese game localization QA.

Mohamad-Hachem / LangChain_Agent_Evaluation_System

—

Arize-ai / phoenix

AI Observability & Evaluation

Kiln-AI / Kiln

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

TIGER-AI-Lab / ClawBench

Open-source benchmark for browser AI agents on daily tasks.

xinxuxin / keystone-bench

A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.

DocAILab / XRAG

XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation

fstandhartinger / jevbench

JevBench by Benchmark Heaven: benchmark and leaderboard for AI decision models, measuring typed decision accuracy, calibration, latency and cost.

wanghui2323 / specialist-model-studio

Conversation-native studio for evidence-backed specialist-model research, training, evaluation, and delivery.

appletea233 / OneSearch-VL

A unified multimodal deep research agent for images and videos, with open models, datasets, and training and evaluation code.

FirmwareVersion / choravox

Offline multi-speaker meeting-speech toolkit: planted-ground-truth synthetic meeting factory, VAD, speaker embeddings, diarization (AHC/spectral/VB-HMM), overlap detection, CTC speaker-attributed ASR, cpWER/DER/JER evaluation, online streaming mode, deterministic reports. NumPy core, CPU-only torch extra.

Q00 / ouroboros

Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI, Gemini CLI, OpenCode, Copilot, Kiro and more.

modelscope / evalscope

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

bespokelabsai / nimble

Local typed decisions, contrastive data curation, and model evaluation.

thetahealth / mirobody-env

HAEnv — synthetic longitudinal patients with code-derived gold, for evaluating health agents over time

NVIDIA / garak

the LLM vulnerability scanner

Barca0412 / Introduction-to-Quantitative-Finance

AI+金融（量化）：1.多因子股票量化框架开源教程 2.学界和业界的经典资料收录 3.AI + 金融的相关工作，包括LLM, Agent, benchmark(evaluation), etc.

robocurve / inspect-robots

Open source evals for physical AI. Run any LLM/VLA on any arm/humanoid against any real/sim benchmark.

paper-instruments / feather-train

Feather agent harness, training integrations, and evaluation tools for knowledge work

Jaasssoooonnnnn / FirstLight_CR

Offline Clash Royale AI agent research: native simulation, imitation learning, PPO, and interactive evaluation. 离线对局模拟、模仿学习、强化学习与交互评测。

Danielxu0208 / rsi_eval

A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.

Giskard-AI / giskard-oss

🐢 Open-Source Evaluation & Testing library for LLM Agents

Jia-Ethan / codex-keysmith

Versioned Codex instruction deployment with preview, ownership manifests, hook isolation, scenario evaluation, and recovery.

future-agi / future-agi

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

Topdu / OpenOCR

OpenOCR: An Open-Source Toolkit for General-OCR Research and Applications, integrates a unified training and evaluation benchmark, commercial-grade OCR and Document Parsing systems, and faithful reproductions of the core implementations from a wide range of academic papers.

mozilla-ai / any-agent

A single interface to use and evaluate different agent frameworks

agentscope-ai / OpenJudge

OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards

AISBench / benchmark

AISBench Benchmark is a model evaluation tool built on OpenCompass, compatible with OpenCompass’s configuration system, dataset structure, and model backend implementation, while extending support for service-based models.

Purewhiter / mobilegym

[EMNLP 2026] MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research · 浏览器里运行的安卓模拟器 · Browser-hosted Android Simulator · Verifiable Evaluation · Scalable Online RL Training

overmind-core / overmind

The platform for continuously improving AI agents.

allenai / vla-evaluation-harness

One framework to evaluate any VLA model on any robot simulation benchmark.

NVIDIA / SkillEvaluator

Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.

rockmatrix-ai / EdgeRock

EdgeRock: Strong Reasoning at Constant Memory. Multimodal models with O(N) generation and O(1) persistent history. Inference and evaluation code.

whichmen / dxl-commerce-agent

OpenClaw + 大模型驱动的完整电商客服自动化系统：覆盖淘宝、拼多多、抖音、快手、小红书、视频号，包含浏览器/手机 Worker、订单物流与售后工具、记忆、回执、恢复和 Eval。

whitecircle / rise

RISE: Red-teaming via Iterative Strategy Evolution (NeurIPS 2026)

xiaojunlan / care

CARE: corrective data generation and FSR-Bench evaluation for robotic manipulation

AlexJ1ang / sse-agent-bench

Configuration-driven evaluation, regression testing, replay, and load testing for SSE-streaming LLM agents

oumi-ai / oumi

Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!

open-compass / opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.

EvolvingLMMs-Lab / lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

open-compass / VLMEvalKit

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

MichaelGrupp / evo

Python package for the evaluation of odometry and SLAM

truera / trulens

Evaluation and Tracking for LLM Experiments and AI Agents

embeddings-benchmark / mteb

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations