promptfoo / promptfoo

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

confident-ai / deepeval

The LLM Evaluation Framework

Kiln-AI / Kiln

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

huggingface / lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

future-agi / future-agi

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

Cloud-CV / EvalAI

:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI

msoedov / agentic_security

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

ServiceNow / AgentLab

AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

frontier-harness-eval / eval

Public results and task definitions for FrontierHarness Eval

ktwu01 / benchmark-radar

Track 33,707+ AI benchmark, eval, dataset, and data-quality records from 40 public sources, with linked evidence and daily updates.

dreamers-laboratory / ai-evaluation-framework

Accuracy, latency p95, and cost benchmarking for model-based solutions, with per-field ground-truth scoring.