ifixai-ai / iFixAi
Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
Tencent / WeKnora
Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.
trycua / cua
Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
promptfoo / promptfoo
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
langfuse / langfuse
🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
zgcagi / Harness-RSI
Harness self-evolution: update targets, theory, evaluation, and reported evidence. Bilingual reading list and project website.
Folkbench / folkbench
AI API 中转站评测排行与切换工具：实测可用率、延迟与价格 · Open-source AI API relay benchmark, rankings & switcher for Claude Code / Codex
AgentSafeLabs / safelabs-eval
Red-teaming and evaluation framework for AI agents, built around an OWASP-inspired ASI01–ASI10 taxonomy
confident-ai / deepeval
The LLM Evaluation Framework
OpenDCAI / DataFlex
[NeurIPS 2026] DataFlex: A Unified Benchmark and Evaluation Platform for Data-Centric Training of Large Language Models
Sanssssssssssssssss / ERP-agent
A Windows workbench and native Python Pi harness for Odoo 19: approved business actions, state readback, traceable runs, and ERP-Bench evaluation.
JEen-wang / Adaptive-rag
按问题复杂度自动分流的电商客服 Agent：简单问题直接回答；复杂问题经过多跳检索并自我纠错；查单、退改通过工具调用完成，且高风险写操作必须先经人工确认。基于 FastAPI 和 LangGraph，混合检索并附带引用，内置离线评测与 Prometheus 指标。
mll-lab-nu / embodied-first-exam
The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
comet-ml / opik
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
scaleapi / agentenv-framework
Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments tools, dynamism of the environment, reproducibility, and more. There is no open source framework for building these environments effectively. Until now.
groundingpi / GroundingPI
A 4B grounding foundation model for physical intelligence. Unifies detection, referring, pointing, OCR, GUI, and layout grounding, achieving SOTA overall across 34 benchmarks and improving robot generalization and data efficiency in controlled evaluations.
superwesleyhys-ux / factcircuit
FactCircuit: an auditable claim-and-evidence verification loop with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.
mlflow / mlflow
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
dataelement / bisheng
BISHENG is an open LLM devops platform for next generation Enterprise AI applications. Powerful and comprehensive features include: GenAI workflow, RAG, Agent, Unified model management, Evaluation, SFT, Dataset Management, Enterprise-level System Management, Observability and more.
Tencent / AI-Infra-Guard
A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.
jwuthri-cloud / Tracely-ai
Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable.
liguozheng-ac / j-gamelqa
Hybrid rule-based and context-aware LLM evaluation system for Chinese-to-Japanese game localization QA.
Mohamad-Hachem / LangChain_Agent_Evaluation_System
—
Tongyi-MAI / MAI-Gym
An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments
Arize-ai / phoenix
AI Observability & Evaluation
Kiln-AI / Kiln
Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
TIGER-AI-Lab / ClawBench
Open-source benchmark for browser AI agents on daily tasks.
xinxuxin / keystone-bench
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
DocAILab / XRAG
XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation
fstandhartinger / jevbench
JevBench by Benchmark Heaven: benchmark and leaderboard for AI decision models, measuring typed decision accuracy, calibration, latency and cost.
phuryn / bug-hunt-bench
Bug Hunt Bench: 105 real bugs in two production repos, frontier coding models (GPT-6, Claude, Grok, Gemini, DeepSeek...) find and fix them in their own CLI, graded blind. Live leaderboard + every receipt.
wanghui2323 / specialist-model-studio
Conversation-native studio for evidence-backed specialist-model research, training, evaluation, and delivery.
appletea233 / OneSearch-VL
A unified multimodal deep research agent for images and videos, with open models, datasets, and training and evaluation code.
microsoft / scope
Scope | Agentic Experience evaluation platform (Research preview)
FirmwareVersion / choravox
Offline multi-speaker meeting-speech toolkit: planted-ground-truth synthetic meeting factory, VAD, speaker embeddings, diarization (AHC/spectral/VB-HMM), overlap detection, CTC speaker-attributed ASR, cpWER/DER/JER evaluation, online streaming mode, deterministic reports. NumPy core, CPU-only torch extra.
Q00 / ouroboros
Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI, Gemini CLI, OpenCode, Copilot, Kiro and more.
coze-dev / coze-loop
Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring.
modelscope / evalscope
A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.
bespokelabsai / nimble
Local typed decisions, contrastive data curation, and model evaluation.
thetahealth / mirobody-env
HAEnv — synthetic longitudinal patients with code-derived gold, for evaluating health agents over time
NVIDIA / garak
the LLM vulnerability scanner
langwatch / langwatch
Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.
lmnr-ai / lmnr
Laminar - open-source observability platform purpose-built for AI agents. YC S24.
MCPJam / inspector
Testing and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.
Barca0412 / Introduction-to-Quantitative-Finance
AI+金融（量化）：1.多因子股票量化框架开源教程 2.学界和业界的经典资料收录 3.AI + 金融的相关工作，包括LLM, Agent, benchmark(evaluation), etc.
microsoft / prompty
Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developers.
alibaba / skill-up
An evaluation and evolution tool for Agent Skills.
keli-wen / agy-staff
Hire Google's Antigravity CLI (agy) as a fast Gemini staffer for Claude Code and OpenAI Codex.
robocurve / inspect-robots
Open source evals for physical AI. Run any LLM/VLA on any arm/humanoid against any real/sim benchmark.
paper-instruments / feather-train
Feather agent harness, training integrations, and evaluation tools for knowledge work
Jaasssoooonnnnn / FirstLight_CR
Offline Clash Royale AI agent research: native simulation, imitation learning, PPO, and interactive evaluation. 离线对局模拟、模仿学习、强化学习与交互评测。
Danielxu0208 / rsi_eval
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.
evaldock / EvalDock
An end-to-end evaluation framework for heterogeneous agents and plugins.
Helicone / helicone
🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓
Giskard-AI / giskard-oss
🐢 Open-Source Evaluation & Testing library for LLM Agents
JerBouma / FinanceToolkit
Transparent and Efficient Financial Analysis
Jia-Ethan / codex-keysmith
Versioned Codex instruction deployment with preview, ownership manifests, hook isolation, scenario evaluation, and recovery.
cel-expr / cel-go
Fast, portable, non-Turing complete expression evaluation with gradual typing (Go)
genieincodebottle / generative-ai
Comprehensive resources on Generative AI, including a detailed roadmap, projects, use cases, interview preparation, and coding preparation.
future-agi / future-agi
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.