harbor-framework / terminal-bench-science

Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains

hailo-ai / hailo_model_zoo

The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment

michaelthwan / searchGPT

Grounded search engine (i.e. with source reference) based on LLM / ChatGPT / OpenAI API. It supports web search, file content search etc.

UKGovernmentBEIS / inspect_evals

Collection of evals for Inspect AI

xmarre / ComfyUI-Spectrum-MiniMax-H3

Training-free Spectrum acceleration for ComfyUI’s native MiniMax H3 audio-video model. Uses Chebyshev ridge feature forecasting to skip selected H3 transformer evaluations, with adaptive scheduling, sampler-aware support for Euler, ER-SDE, RES, SEEDS and SA-Solver, CPU/VRAM history storage, and fail-closed native fallbacks.

TIGER-AI-Lab / VLM2Vec

This repo contains the code for "VLM2Vec / MMEB" [ICLR 2025], "VLM2Vec-V2 / MMEB-V2" [TMLR 2026], and "MMEB-V3" [COLM 2026]

layumi / University1652-Baseline

ACM Multimedia2020 University-1652: A Multi-view Multi-source Benchmark for Drone-based Geo-localization :helicopter: annotates 1652 buildings in 72 universities around the world.

EverMind-AI / SkillCorpus

Open-source infrastructure that turns scattered SKILL.md files into curated, retrieval-ready agent-skill corpora—with retrieval and evaluation tooling included.

OWASP / OFFAT

The OWASP OFFAT tool autonomously assesses your API for prevalent vulnerabilities, though full compatibility with OAS v3 is pending. The project remains a work in progress, continuously evolving towards completion.

probabl-ai / skore

Track your Data Science. Skore's open-source Python library accelerates ML model development with automated evaluation reports, smart methodological guidance, and comprehensive cross-validation analysis.

Bessouat40 / RAGLight

RAGLight is a modular framework for Retrieval-Augmented Generation (RAG). It makes it easy to plug in different LLMs, embeddings, and vector stores, and now includes seamless MCP integration to connect external tools and data sources.

MIC-DKFZ / nnDetection

nnDetection is a self-configuring framework for 3D (volumetric) medical object detection which can be applied to new data sets without manual intervention. It includes guides for 12 data sets that were used to develop and evaluate the performance of the proposed method.

castorini / rank_llm

RankLLM is a Python toolkit for reproducible information retrieval research using rerankers, with a focus on listwise reranking.

ServiceNow / AgentLab

AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

ggozad / haiku.rag

Agentic RAG for local and self-hosted document search: hybrid retrieval, reranking and multimodal RAG on embedded LanceDB, with Docling parsing and an MCP server

HowieHwong / TrustLLM

[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.

KRLabsOrg / LettuceDetect

Span-level grounding verification for RAG, code, and tool-grounded AI outputs.

locuslab / open-unlearning

[NeurIPS D&B '25] The one-stop repository for LLM unlearning

danthedeckie / simpleeval

Simple Safe Sandboxed Extensible Expression Evaluator for Python

run-llama / ParseBench

ParseBench - A Document Parsing Benchmark for AI Agents

SHITIANYU-hue / SheetasToken

Implementation and resources for Sheet as Token, a graph-enhanced framework for multi-sheet spreadsheet understanding and retrieval.

google / seqio

Task-based datasets, preprocessing, and evaluation for sequence models.

MaxFreedomPollard / Compartment

Encrypted, fully offline agentic memory. One click install, GUI w/ memory map, all OS and agents. Superior memory creation, storage and retrieval.

dbt-labs / dbt-project-evaluator

This package contains native dbt checks to report and fix issues in your dbt project

aws-samples / service-screener-v2

A tool for customers to evaluate their AWS service configurations based on AWS and community best practices and receive recommendations on potential improvements.

facebookresearch / meta-agents-research-environments

Meta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges.

Mic92 / nix-fast-build

Combine the power of nix-eval-jobs with nix-output-monitor to speed-up your evaluation and building process.

skygazer42 / MimirQ

中文优先的企业 RAG 知识库：可控解析、治理、切块、混合检索、重排、引用、图谱、评测与 Dify 接入。

keyuchen21 / agentic-engineering-handbook

The definitive OpenAI, Claude, MCP, Harness, Evals, and Production Agent Systems learning roadmap.

naver / kapture

kapture is a file format as well as a set of tools for manipulating datasets, and in particular Visual Localization and Structure from Motion data.

HKUDS / CatchMe

"CatchMe: Make Your AI Agents Truly Personal"

OmniJev / awesome-jev-gallery

🔥🔥 Awesome Jev: open-source models, projects, benchmarks, and independent evaluations for Jev and System One models.

uoftcprg / pokerkit

An open-source Python library for poker game simulations, hand evaluations, and statistical analysis

official-stockfish / nnue-pytorch

Stockfish NNUE (Chess evaluation) trainer in Pytorch

Pringled / pyversity

Fast Diversification for Search & Retrieval

caura-ai / caura

Caura (formerly MemClaw) — governed shared memory for AI agent fleets. Multi-agent, multi-tenant, MCP-native. Trust tiers, keystone policies, audit trails, knowledge graph, self-improving retrieval. Apache 2.0.

phoenix-zhou / agent-sweep-engine

Built on **Agentic RAG** (Agent-driven Retrieval-Augmented Generation) technology, it not only accurately answers pre-sales and after-sales questions but also generates personalized usage reports and optimization suggestions by deeply analyzing device data.

OpenMOSS / EasyWAM

A unified framework for training, fine-tuning, and evaluating World Action Models

ChidiebereNjoku / ModelBrief

An open-source Python library that automates the analysis and reporting of machine learning experiments, generating structured and comprehensive reports that make model performance, results, and key insights easier to understand and evaluate by data professionals, with optional Groq-powered AI explanations and recommendations for deeper insights.

open-daimax / daimax-appbench

An automated benchmark platform for AI-generated apps — built-in evaluation rules, standard sample sets, and layered test cases across a multi-dimensional metric system. One pipeline runs build, install, E2E tests, and scoring into a shareable report.

aaron-for-value / VeriRun

Evidence-first infrastructure for reproducible, isolated executable evaluation and online rewards.

FengZhenfei / carrel

Ontology-Augmented Generation for AI agents.

MirroS-Lab / HarnessEval-W

HarnessEval-W: Agentifying the Evaluation of Visual Worlds

ktwu01 / benchmark-radar

Track 33,707+ AI benchmark, eval, dataset, and data-quality records from 40 public sources, with linked evidence and daily updates.

jinzijian / EvoTrace

Compile real-world Claude Code and Codex trajectories into verified, tradable post-training assets.

xiaobright / modeltest

Personal LLM engineering-maintenance evaluation harness (V4.1b, frozen). Not a public benchmark.

i3T4AN / KADATH

Evolutionary multi-agent runtime that breeds, evaluates, and improves autonomous agents across reproducible epochs to converge on optimization of a goal.

waiterve / wai-play

WAI Play - AI web game testing and quality evaluation platform

TriWorldBench / TriWorldBench

A Benchmark Evaluating Triple-View Embodied World Models

SodaMem / SodaMem

Agentic memory infra for AI agents

roy-tong / AgentMeasure

Inspect Codex / Claude logs, verify metering samples and re-count supported billing exports. Free local tools; bounded $10 diagnosis and $20 QA.

MoonshotAI / PerceptionBench

PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

jsdhwfmax / EvalForge

Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.

MoMoM101 / RAG-ReActAgent

RAG ReAct Agent - A Retrieval-Augmented Generation system with ReAct (Reasoning+Acting) agent loop for intelligent question answering with multi-hop reasoning

cdcailf / cjkfold

Fold look-alike CJK glyphs (Kangxi radicals, CJK radical supplements) so text extracted from PDFs matches again.

EastForce / worker-ai-spark

开源劳动者AI研究与评测项目：建设劳动知识、公开评测基准与共同治理机制｜Open worker-centered AI research and evaluation project.

KhanCold / merchantbench

MerchantBench is a 365-day, order-level benchmark for evaluating the long-term coherence of LLM agents in seller-side e-commerce operations.

solis-team / XRepoTest

[EMNLP 2026] A multilingual benchmark for repository-level unit test generation, evaluating LLMs across 5 programming languages and real-world code under reproducible, Docker-based execution.

nssmd / RoboRSI

Robot-agent harness with a CLI and local Web console for LIBERO short evaluation

AMAP-ML / LoopArena

Benchmarking models as runtime Controllers for Loop Engineering