huggingface / lighteval
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
huggingface / lighteval
huggingface / evaluate
🤗 Evaluate: A library for easily evaluating machine learning models and datasets.
Sahir619 / fable-method
The Fable Workflow: how Claude Fable 5 worked, distilled into skills any model can run, with the eval that keeps it honest. Think / act / prove.
galilai-group / stable-worldmodel
A platform for reproducible world model research and evaluation
Cloud-CV / EvalAI
:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI
msoedov / agentic_security
Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪
rentruewang / aioway
AI on the way. An auto deep learning pipe dream. An RDBMS approach to deep learning. Declarative, explainable, scalable, optimizable, easy to deploy, all that good stuff.
InterDigitalInc / CompressAI
A PyTorch library and evaluation platform for end-to-end compression research
sepandhaghighi / pycm
Multi-class confusion matrix library in Python
NVIDIA-NeMo / Gym
Evaluate and improve models and agents using environments
chunxiaoxx / nautilus-compass
Open memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform
cvs-health / uqlm
[JMLR 2026] UQLM: A Python package for LLM hallucination detection using uncertainty quantification
JudgmentLabs / judgeval
The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.
microsoft / AIOpsLab
A holistic framework to enable the design, development, and evaluation of autonomous AIOps agents.
Raudaschl / rag-fusion
RAG-Fusion: multi-query generation + Reciprocal Rank Fusion for better retrieval-augmented generation. Includes evaluation harness with NFCorpus/BEIR.
certsocietegenerale / fame
FAME Automates Malware Evaluation
Kautenja / gym-super-mario-bros
Gymnasium environments for NES Mario games, with task metadata, reward instrumentation, and reproducible evaluation interfaces.
nolabs-ai / deepfabric
Generate High-Quality Synthetics, Train, Measure, and Evaluate in a Single Pipeline
groq / openbench
Provider-agnostic, open-source evaluation infrastructure for language models
chirpz-ai / pandaprobe
open source agent engineering platform: traces, evals, and metrics to debug and improve your AI agents. Integrates with LangGraph, CrewAI, Claude Agent SDK, and more.
claw-eval / claw-eval
Claw-Eval is an evaluation harness for evaluating LLM as agents. All tasks verified by humans.
MigoXLab / dingo
Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool
hailo-ai / hailo_model_zoo
The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment
xmarre / ComfyUI-Spectrum-MiniMax-H3
Training-free Spectrum acceleration for ComfyUI’s native MiniMax H3 audio-video model. Uses Chebyshev ridge feature forecasting to skip selected H3 transformer evaluations, with adaptive scheduling, sampler-aware support for Euler, ER-SDE, RES, SEEDS and SA-Solver, CPU/VRAM history storage, and fail-closed native fallbacks.
EverMind-AI / SkillCorpus
Open-source infrastructure that turns scattered SKILL.md files into curated, retrieval-ready agent-skill corpora—with retrieval and evaluation tooling included.
probabl-ai / skore
Track your Data Science. Skore's open-source Python library accelerates ML model development with automated evaluation reports, smart methodological guidance, and comprehensive cross-validation analysis.
ServiceNow / AgentLab
AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.
ggozad / haiku.rag
Agentic RAG for local and self-hosted document search: hybrid retrieval, reranking and multimodal RAG on embedded LanceDB, with Docling parsing and an MCP server
HowieHwong / TrustLLM
[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.
KRLabsOrg / LettuceDetect
Span-level grounding verification for RAG, code, and tool-grounded AI outputs.
locuslab / open-unlearning
[NeurIPS D&B '25] The one-stop repository for LLM unlearning
danthedeckie / simpleeval
Simple Safe Sandboxed Extensible Expression Evaluator for Python
run-llama / ParseBench
ParseBench - A Document Parsing Benchmark for AI Agents
google / seqio
Task-based datasets, preprocessing, and evaluation for sequence models.
facebookresearch / meta-agents-research-environments
Meta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges.
Mic92 / nix-fast-build
Combine the power of nix-eval-jobs with nix-output-monitor to speed-up your evaluation and building process.
skygazer42 / MimirQ
中文优先的企业 RAG 知识库：可控解析、治理、切块、混合检索、重排、引用、图谱、评测与 Dify 接入。
OmniJev / awesome-jev-gallery
🔥🔥 Awesome Jev: open-source models, projects, benchmarks, and independent evaluations for Jev and System One models.
uoftcprg / pokerkit
An open-source Python library for poker game simulations, hand evaluations, and statistical analysis
official-stockfish / nnue-pytorch
Stockfish NNUE (Chess evaluation) trainer in Pytorch
open-daimax / daimax-appbench
An automated benchmark platform for AI-generated apps — built-in evaluation rules, standard sample sets, and layered test cases across a multi-dimensional metric system. One pipeline runs build, install, E2E tests, and scoring into a shareable report.
aaron-for-value / VeriRun
Evidence-first infrastructure for reproducible, isolated executable evaluation and online rewards.
MirroS-Lab / HarnessEval-W
HarnessEval-W: Agentifying the Evaluation of Visual Worlds
ktwu01 / benchmark-radar
Track 33,707+ AI benchmark, eval, dataset, and data-quality records from 40 public sources, with linked evidence and daily updates.
jinzijian / EvoTrace
Compile real-world Claude Code and Codex trajectories into verified, tradable post-training assets.
xiaobright / modeltest
Personal LLM engineering-maintenance evaluation harness (V4.1b, frozen). Not a public benchmark.
i3T4AN / KADATH
Evolutionary multi-agent runtime that breeds, evaluates, and improves autonomous agents across reproducible epochs to converge on optimization of a goal.
waiterve / wai-play
WAI Play - AI web game testing and quality evaluation platform
roy-tong / AgentMeasure
Inspect Codex / Claude logs, verify metering samples and re-count supported billing exports. Free local tools; bounded $10 diagnosis and $20 QA.
jsdhwfmax / EvalForge
Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.
EastForce / worker-ai-spark
开源劳动者AI研究与评测项目：建设劳动知识、公开评测基准与共同治理机制｜Open worker-centered AI research and evaluation project.
solis-team / XRepoTest
[EMNLP 2026] A multilingual benchmark for repository-level unit test generation, evaluating LLMs across 5 programming languages and real-world code under reproducible, Docker-based execution.
nssmd / RoboRSI
Robot-agent harness with a CLI and local Web console for LIBERO short evaluation
AMAP-ML / LoopArena
Benchmarking models as runtime Controllers for Loop Engineering
Blackwellboy / model-serving-minefield
Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each.
MrZoyo / deslop-GPT
Deletion-first Agent Skill for removing test bloat, verification theater, and speculative fallbacks while preserving behavior.
salikhussain71-code / PAKGOV-RAG-project
Research project on bilingual Urdu-English retrieval-augmented generation for Pakistani government and legal documents, with OCR, hybrid retrieval, citation grounding, and evaluation.
fengzhizi715 / YoloWebAgentStarter
Local-first YOLO workbench for annotation, training, evaluation, and AI assistance.
ConstantineChenn / SakikoMind
Enterprise AI customer-service Agent platform featuring hybrid intent recognition, multi-agent orchestration, MCP-powered RAG, dynamic Skills, memory, human handoff, observability and evaluation. 小祥智能客服系统，支持可追溯、可观测与云端部署。
To3akaRin / VLMForge
Evaluate visual models on your own images, JSON Schema, and production constraints with LangGraph, Pareto analysis, and a no-key Replay demo.