Simreal-AI / Simreal-MLBench

# ML Benchmark by Simreal (Simreal.co)
**An externally scored, verifiable evaluation environment for machine learning research agents.**
**Open protocol. Operated evaluation.**
> **Public preview.** This repository is the open, minimal slice of ML Benchmark: the 60-task catalog, the evaluation protocol, the scoring code and the client SDK. The evaluation runtime, the official-scoring route and full-suite admission run on the SimReal engine and are not in this repository. It is early and will change.
>
> Full access for labs and partners: business@simreal.co · All SimReal previews: github.com/Simreal-AI
An agent studies a dataset, develops a modeling approach, runs experiments, and submits predictions. ML Benchmark brings together **60 tasks across tabular learning, forecasting, vision, language, audio, multimodal learning, and structured scientific data**.
The benchmark targets the full research process: data understanding, feature engineering, validation design, model selection, hyperparameter tuning, ensembling, debugging, resource allocation, and reliable delivery.
| Tier | Tasks | Research budget | Compute |
| --- | ---: | ---: | --- |
| Easy | 20 | 6 hours | CPU |
| Medium | 20 | 12 hours | Heterogeneous GPU pool |
| Hard | 20 | 24 hours | Heterogeneous GPU pool |
## Positioning
ML Benchmark is built as two layers, and we release them differently on purpose.
**The open layer is the standard.** Task definitions, the evaluation protocol, the scoring rule, and the verification tooling are public under Apache-2.0. Anyone can read exactly how a number is produced and recompute every published score. A benchmark result should never depend on trusting us.
**The operated layer is the service.** The evaluation runtime is a commercial product operated by Simreal. It runs agents in isolated workers behind a restricted gateway, enforces budgets and the two-submission rule, freezes every artifact, and routes submissions to official scoring. Labs and teams evaluate agents through this service rather than self-hosting it.
This split keeps the rules transparent while the infrastructure that enforces them stays under operational control.
## What makes the numbers trustworthy
**The scorer is not us.** Test labels stay with the competition platform. We never hold the answers, so an agent cannot read them and we cannot get the split wrong. Suites that carve their own holdout out of training data inherit a class of bug this design does not have.
**Agents get a second submission, with a real score in between.** Most evaluations are one shot. Two submissions with honest feedback between them make a different question measurable: not only how well an agent performs, but whether it can act on the truth once it has it.
**Source-blind by enforcement, not by convention.** Task briefs are checked at load time for platform names, competition identities, and URLs, so the brief itself never reveals the source. This reduces, but cannot eliminate, the chance that an agent re