NVlabs / RoboLab
A simulation benchmarking platform for evaluating generalist robot policies
NVlabs / RoboLab
DocAILab / XRAG
XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation
modelscope / evalscope
A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.
xlang-ai / OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
SemiAnalysisAI / InferenceX
Open Source AI Accelerator Research Platform Standard / εΌζΊζ¨ηη η©ΆεΉ³ε°
AetherLabsAI / Video2World
Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos - Rethinking Embodied Real2Sim from a Software Engineering Perspective
Danielxu0208 / rsi_eval
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.
locustio / locust
Write scalable load tests in plain Python ππ¨
uber / ADR
ADR secures enterprise AI agents through observability, security benchmarking, and threat detection. Deployed at Uber.
Farama-Foundation / Metaworld
Collections of robotics environments geared towards benchmarking multi-task and meta reinforcement learning
ai-dynamo / aiperf
AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution.
morluto / flameox
Runtime evidence that helps agents trace, profile, and burn down hotspots in application and native code, GPU kernels, and inference stacks.
4DCodeBench / 4DCodeBench
4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes
facebookresearch / swe-sweep
How many bugs can LMs find & fix in large codebases?
ZeamoxWang / pdmd
Official Training and Benchmarking Implementation of PDMD: Projected Distribution Matching Distillation for Video Diffusion Models
imitator-game / The-Imitator-Game
Official repo for The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction
VAIL-UCLA / NavSafe-infty
NavSafe-β: Benchmarking Closed-Loop Driving Safety in Photorealistic Environments
Gen-Verse / Skill-Entropy-RL
[NeurIPS 2026] Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning
intel / gpu-ai-skills
Agent Skills for running, benchmarking, and profiling Hugging Face models on Intel GPUs β PyTorch, vLLM-XPU, and SGLang-XPU
Junjue-Wang / DORA
[NeurIPS 2026 Oral] DORA: Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations
elastic / rally
Macrobenchmarking framework for Elasticsearch
sintel-dev / Orion
Unsupervised time series anomaly detection library
redis / memtier_benchmark
NoSQL Redis and Memcache traffic generation and benchmarking tool.
airspeed-velocity / asv
Airspeed Velocity: A simple Python benchmarking tool with web-based reporting
psf / pyperf
Toolkit to run Python benchmarks
ServiceNow / AgentLab
AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.
mlcommons / ck
Collective Knowledge (CK), Collective Mind (CM/CMX) and MLPerf automations: community-driven projects to learn how to run AI, ML, and other emerging workloads more efficiently and cost-effectively across diverse models, datasets, software, and hardware using MLPerf methodology and benchmarks.
IPNL-POLYU / UrbanNavDataset
UrbanNav:An Open-sourced Multisensory Dataset for Benchmarking Positioning Algorithms Designed for Urban Areas
SalesforceAIResearch / MCP-Universe
MCP-Universe is a comprehensive framework designed for RL training, benchmarking, and developing AI agents for general tool-use.
itbench-hub / ITBench
An open source benchmarking framework for IT automation
ktwu01 / benchmark-radar
Track 33,707+ AI benchmark, eval, dataset, and data-quality records from 40 public sources, with linked evidence and daily updates.
HarlandZZC / OmniVChat
Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue
AMAP-ML / LoopArena
Benchmarking models as runtime Controllers for Loop Engineering
Blackwellboy / model-serving-minefield
Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each.
YuyaoGe / doc2harness
Official Code for the paper "Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents "
yasir-mo / AI-watermark-remover-GUI
Open-source research tool for inspecting, benchmarking, and scrubbing AI document metadata and invisible tracking artifacts.
PriorLabs / relarena
Open and Reproducible Benchmarking for Relational Learning
zzl0720-2025 / cxxpilot-local
Local C++ performance testing and source-to-source optimization with deterministic rules, semantic checks, and paired benchmarking.
swirl-uk / BiGym2
BiGym 2.0: Benchmarking Learned and Agent-Developed Policies for Humanoid Household Manipulation (arXiv:2610.07594)
sauravsingla / agentweave
Pre-inference tool and agent routing for MCP, A2A and tool-rich LLM systems. Reduce candidate tools before model inference with policy-aware routing and recovery.
AlibabaResearch / cut-craft-bench
Beyond Coherence: Benchmarking Professional Editing-Technique Execution in Multi-Shot Audio-Video Generation
wieslawsoltes / Performance-Skill
Cross-platform .NET performance engineering skill for coding agents, covering CPU, memory, GC, benchmarking, concurrency, startup, native profiling, GPU rendering, and production diagnostics on macOS, Windows, and Linux.
Gen-Verse / PAST-Bench
PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents
ExVRAM / exvram-lab
Compute-for-memory research lab for running larger LLMs on memory-constrained consumer GPUs. Official website exvram.pro
SharkSpicy-NLP / Beyond-Factual-Knowledge
[EMNLP 2026] Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models
lexmount / Lexbench-Headless-Browser
Benchmarking headless browsers as agent runtimes
SharkSpicy-NLP / RuleWeaver
[EMNLP 2026] RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models
zsxm1998 / awesome-perception-aware-rlvr
Unified reproductions, benchmarking & evaluation for perception-aware RLVR and OPD in VLMs. 19 methods Β· 30+ benchmarks Β· Official code of CGPO (ACM MM 2026 Oral).
AbdelStark / jev-benchmarks
Probability-aware evaluation for typed decision models: calibration, selective risk, latency, and reproducible benchmarks.
rsasaki0109 / lidar_slam_ros2
ROS 2 LiDAR SLAM for pointcloud-map authoring, benchmarking, and Autoware-compatible map workflows.
Simreal-AI / Simreal-MLBench
[Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.