rafaelpadilla / Object-Detection-Metrics

Most popular metrics used to evaluate object detection algorithms.

pydantic / logfire

AI observability platform for production LLM and agent systems.

EvolvingLMMs-Lab / lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

layumi / Person_reID_baseline_pytorch

:bouncing_ball_person: Pytorch ReID: A tiny, friendly, strong pytorch implement of person re-id / vehicle re-id baseline. Tutorial 👉https://github.com/layumi/Person_reID_baseline_pytorch/tree/master/tutorial

open-compass / VLMEvalKit

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

MichaelGrupp / evo

Python package for the evaluation of odometry and SLAM

TarrySingh / Artificial-Intelligence-Deep-Learning-Machine-Learning-Tutorials

Synapsa Commons: free, hands-on AI courses that run anywhere (Colab, Kaggle, Binder, Codespaces, Jupyter). EU AI Act conformity evidence, model validation, predictive maintenance, document extraction, retrieval and RAG, AI agents and MCP, a simulated humanoid. Every lesson autograded. From the team building Synapsa.

morphik-org / morphik-core

Open-source multimodal retrieval engine (Morphik Core)

truera / trulens

Evaluation and Tracking for LLM Experiments and AI Agents

RUC-NLPIR / FlashRAG

⚡FlashRAG: A Python Toolkit for Efficient RAG Research (WWW2025 Resource)

embeddings-benchmark / mteb

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

superlinked / sie

Open-source inference server and production cluster for all the models your agent needs.

MemMachine / MemMachine

Universal memory layer for AI Agents. It provides scalable, extensible, and interoperable memory storage and retrieval to streamline AI agent state management for next-generation autonomous systems.

UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations

swirlai / swirl-search

AI Search & RAG Without Moving Your Data. Get instant answers from your company's knowledge across 100+ apps while keeping data secure. Deploy in minutes, not months.

illuin-tech / colpali

The code used to train and run inference with the ColVision models, e.g. ColPali, ColQwen2, and ColSmol.

huggingface / lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

pydata / numexpr

Fast numerical array expression evaluator for Python, NumPy, Pandas, PyTables and more

huggingface / evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

Sahir619 / fable-method

The Fable Workflow: how Claude Fable 5 worked, distilled into skills any model can run, with the eval that keeps it honest. Think / act / prove.

galilai-group / stable-worldmodel

A platform for reproducible world model research and evaluation

castorini / pyserini

Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations.

Cloud-CV / EvalAI

:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI

pykeen / pykeen

🤖 A Python library for learning and evaluating knowledge graph embeddings

youngfish42 / Awesome-FL

Comprehensive and timely academic information on federated learning (papers, frameworks, datasets, tutorials, workshops)

msoedov / agentic_security

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

tsale / EDR-Telemetry

This project aims to compare and evaluate the telemetry of various EDR products.

SaiAkhil066 / CORTEX-AI-SUPER-RAG

CORTEX RAG is an enterprise retrieval and knowledge assistant that helps teams find accurate answers from company data with citations, permission-aware retrieval, and fast deployment

NVIDIA-AI-Blueprints / video-search-and-summarization

NVIDIA AI Blueprint for video search and summarization (VSS) is a GPU-accelerated reference architecture for building video analytics agents with real-time verified alerts, visual Q&A, and automated reporting. The VSS Blueprint uses vision language models (VLMs) such as NVIDIA Cosmos, LLMs such as NVIDIA Nemotron, RAG, and NVIDIA NIMs.

rentruewang / aioway

AI on the way. An auto deep learning pipe dream. An RDBMS approach to deep learning. Declarative, explainable, scalable, optimizable, easy to deploy, all that good stuff.

xhluca / bm25s

Fast BM25 search in Python, powered by Numpy and Numba

Tencent / WeMM-Embedding

WeMM-Embedding is a family of universal multimodal embedding models by the WeChat Vision Team at Tencent, supporting multimodal understanding and retrieval.

InterDigitalInc / CompressAI

A PyTorch library and evaluation platform for end-to-end compression research

ray-r-ren / agent-apprenticeship

The living ecosystem where AI agents complete tasks through workflow loops, improve through iterative execution, are evaluated by mentor agents or humans in the loop, and turn completed work into reusable work experience and data to improve future agents.

KruxAI / ragbuilder

A toolkit to create optimal Production-readyRetrieval Augmented Generation(RAG) setup for your data

sepandhaghighi / pycm

Multi-class confusion matrix library in Python

NVIDIA-NeMo / Gym

Evaluate and improve models and agents using environments

chunxiaoxx / nautilus-compass

Open memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform

Mathics3 / mathics-core

An open-source Mathematica Kernel. This repository contains the Python modules for WL Built-in functions, variables, core primitives, e.g. Symbol, a parser to create Expressions, and an evaluator to execute them.

cvs-health / uqlm

[JMLR 2026] UQLM: A Python package for LLM hallucination detection using uncertainty quantification

superlinear-ai / raglite

🥤 RAGLite is a Python toolkit for Retrieval-Augmented Generation (RAG) with DuckDB or PostgreSQL

Azure-Samples / chat-with-your-data-solution-accelerator

A Solution Accelerator for the RAG pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences. This includes most common requirements and best practices.

haowang02 / codex-candy-eval

Codex 降智测试

EmbeddedLLM / JamAIBase

The collaborative spreadsheet for AI. Chain cells into powerful pipelines, experiment with prompts and models, and evaluate LLM responses in real-time. Work together seamlessly to build and iterate on AI applications.

JudgmentLabs / judgeval

The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.

braintrustdata / autoevals

AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices.

pytorch / benchmark

TorchBench is a collection of open source benchmarks used to evaluate PyTorch performance.

microsoft / AIOpsLab

A holistic framework to enable the design, development, and evaluation of autonomous AIOps agents.

Raudaschl / rag-fusion

RAG-Fusion: multi-query generation + Reciprocal Rank Fusion for better retrieval-augmented generation. Includes evaluation harness with NFCorpus/BEIR.

certsocietegenerale / fame

FAME Automates Malware Evaluation

Kautenja / gym-super-mario-bros

Gymnasium environments for NES Mario games, with task metadata, reward instrumentation, and reproducible evaluation interfaces.

Natooz / MidiTok

MIDI / symbolic music tokenizers for Deep Learning models 🎶

nolabs-ai / deepfabric

Generate High-Quality Synthetics, Train, Measure, and Evaluate in a Single Pipeline

groq / openbench

Provider-agnostic, open-source evaluation infrastructure for language models

usemoss / moss

The retrieval layer for production AI systems. Lightning-fast (<10ms) search without vector databases. Built for browser, edge, on-device, and cloud.

chirpz-ai / pandaprobe

open source agent engineering platform: traces, evals, and metrics to debug and improve your AI agents. Integrates with LangGraph, CrewAI, Claude Agent SDK, and more.

claw-eval / claw-eval

Claw-Eval is an evaluation harness for evaluating LLM as agents. All tasks verified by humans.

huggingface / upskill

Generate and evaluate agent skills for code agents like Claude Code, Open Code, OpenAI Codex

jonfairbanks / local-rag

Ingest files for retrieval augmented generation (RAG) with open-source Large Language Models (LLMs), all without 3rd parties or sensitive data leaving your network.

MigoXLab / dingo

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool