MadsLorentzen / ai-job-search

The job search that runs on your machine. AI job application framework built on Claude Code: evaluate postings, tailor CVs, write cover letters, prep interviews. Fork it and own it.

ifixai-ai / iFixAi

Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

Gen-Verse / DiscoveryRSI

DiscoveryRSI: Recursively Learning to Discover Valuable Hypotheses

Aoisop / mawoagent

An auditable, reversible, evaluable local agent runtime that turns feedback into skills — and ships them only when evidence and tests say so.

FirmwareVersion / choravox

Offline multi-speaker meeting-speech toolkit: planted-ground-truth synthetic meeting factory, VAD, speaker embeddings, diarization (AHC/spectral/VB-HMM), overlap detection, CTC speaker-attributed ASR, cpWER/DER/JER evaluation, online streaming mode, deterministic reports. NumPy core, CPU-only torch extra.

mlflow / mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

comet-ml / opik

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

google / adk-python

An open-source, code-first Python toolkit for building, evaluating, and deploying sophisticated AI agents with flexibility and control.

confident-ai / deepeval

The LLM Evaluation Framework

ShishirPatil / gorilla

Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)

dataelement / bisheng

BISHENG is an open LLM devops platform for next generation Enterprise AI applications. Powerful and comprehensive features include: GenAI workflow, RAG, Agent, Unified model management, Evaluation, SFT, Dataset Management, Enterprise-level System Management, Observability and more.

Arize-ai / phoenix

AI Observability & Evaluation

NVIDIA / garak

the LLM vulnerability scanner

oumi-ai / oumi

Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!

open-compass / opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.

interviewstreet / hiring-agent

AI agent to evaluate and score resumes.

Tencent / AI-Infra-Guard

A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.

harbor-framework / harbor

Framework for evaluating and improving agents

Q00 / ouroboros

Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI, Gemini CLI, OpenCode, Copilot, Kiro and more.

google / agents-cli

The CLI and skills that turn any coding assistant into an expert at creating, evaluating, and deploying AI agents on Google Cloud.

Giskard-AI / giskard-oss

🐢 Open-Source Evaluation & Testing library for LLM Agents

JerBouma / FinanceToolkit

Transparent and Efficient Financial Analysis

Kiln-AI / Kiln

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

transformerlab / transformerlab-app

The open source research environment for AI researchers to seamlessly train, evaluate, and scale models from local hardware to GPU clusters.

rafaelpadilla / Object-Detection-Metrics

Most popular metrics used to evaluate object detection algorithms.

pytorch / ignite

High-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.

Jia-Ethan / codex-keysmith

Versioned Codex instruction deployment with preview, ownership manifests, hook isolation, scenario evaluation, and recovery.

EvolvingLMMs-Lab / lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

open-compass / VLMEvalKit

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

MichaelGrupp / evo

Python package for the evaluation of odometry and SLAM

evalstate / fast-agent

Code, Build and Evaluate agents - excellent Model and Skills/MCP/ACP/A2A Support

truera / trulens

Evaluation and Tracking for LLM Experiments and AI Agents

modelscope / evalscope

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

embeddings-benchmark / mteb

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

openai / human-eval

Code for the paper "Evaluating Large Language Models Trained on Code"

domokane / FinancePy

A Python Finance Library that focuses on the pricing and risk-management of Financial Derivatives, including fixed-income, equity, FX and credit derivatives.

OpenDCAI / DataFlex

[NeurIPS 2026] DataFlex: A Unified Benchmark and Evaluation Platform for Data-Centric Training of Large Language Models

UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations

huggingface / lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

pydata / numexpr

Fast numerical array expression evaluator for Python, NumPy, Pandas, PyTables and more

huggingface / evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

IBM / AssetOpsBench

AssetOpsBench - Industry 4.0: A unified benchmark and framework for building, orchestrating, and evaluating domain-specific AI agents for Industry 4.0 asset operations and maintenance, with 460+ scenarios, 5 specialist agents (IoT, FMSR, TSFM, Work Order,...), and multi-agent orchestration blueprints (MetaAgent, AgentHive) over MCP.

Sahir619 / fable-method

The Fable Workflow: how Claude Fable 5 worked, distilled into skills any model can run, with the eval that keeps it honest. Think / act / prove.

galilai-group / stable-worldmodel

A platform for reproducible world model research and evaluation

bespokelabsai / nimble

Local typed decisions, contrastive data curation, and model evaluation.

future-agi / future-agi

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

Cloud-CV / EvalAI

:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI

pykeen / pykeen

🤖 A Python library for learning and evaluating knowledge graph embeddings

msoedov / agentic_security

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

tsale / EDR-Telemetry

This project aims to compare and evaluate the telemetry of various EDR products.

Barca0412 / Introduction-to-Quantitative-Finance

AI+金融（量化）：1.多因子股票量化框架开源教程 2.学界和业界的经典资料收录 3.AI + 金融的相关工作，包括LLM, Agent, benchmark(evaluation), etc.

rentruewang / aioway

AI on the way. An auto deep learning pipe dream. An RDBMS approach to deep learning. Declarative, explainable, scalable, optimizable, easy to deploy, all that good stuff.

vllm-project / guidellm

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

InterDigitalInc / CompressAI

A PyTorch library and evaluation platform for end-to-end compression research

ray-r-ren / agent-apprenticeship

The living ecosystem where AI agents complete tasks through workflow loops, improve through iterative execution, are evaluated by mentor agents or humans in the loop, and turn completed work into reusable work experience and data to improve future agents.

sepandhaghighi / pycm

Multi-class confusion matrix library in Python

harveyai / harvey-labs

A benchmark built to evaluate and improve agent capabilities for supporting legal work.

Topdu / OpenOCR

OpenOCR: An Open-Source Toolkit for General-OCR Research and Applications, integrates a unified training and evaluation benchmark, commercial-grade OCR and Document Parsing systems, and faithful reproductions of the core implementations from a wide range of academic papers.

toluaina / pgsync

Postgres, MySQL, or MariaDB to Elasticsearch/OpenSearch sync

zjunlp / SkillNet

Create, Evaluate, and Connect AI Skills