Simreal-AI / Xitadel-QuantBench

# Xitadel-QuantBench
**The first public trading benchmark scored against what human competitors
actually made on the same day.**
> **Public preview.** This repository is the open, minimal slice of Xitadel, SimReal's trading environment: the published tasks with their training data, the strategy SDK, the scoring specification and the public report. The matching engine, the counterparty model, the held-out scored days, the session controller and the reserve tasks run on the SimReal engine and are not in this repository. It is early and will change.
>
> Full access for labs and partners: business@simreal.co · All SimReal previews: github.com/Simreal-AI
Seven research tasks across two market suites. An agent receives historical
order books, product rules and position limits; it researches, writes a Python
strategy, and ships it. We replay that strategy on a market day it has never
seen and score the result against the best human competition strategy on that
same day.
Human reference scores 80. Twice that combined performance scores 100.
| Rank | Model | Score / 100 | Tasks at or above human |
| --- | --- | ---: | ---: |
| 1 | GPT 6 | 77.28 | 5 / 7 |
| 2 | GLM 5.3 | 30.12 | 1 / 7 |
| 3 | Kimi K3 | 27.68 | 0 / 7 |
| 4 | DeepSeek V4 Pro | 24.58 | 0 / 7 |
**No model has cleared the human bar overall.**
Published results use a 2h/4h pilot allocation; the standard protocol is 6h/12h.
The allocation is identical for every model in a comparison and is reported with
the result.
## Why the score means something
**The verifier is a simulator, not an opinion.** No judge model, no rubric, no
LLM grading another LLM. A submitted strategy executes against a historical
order book and is scored on realized P&L, maximum drawdown and Sharpe. The same
code yields the same number every time.
**The reference is a real result, not an estimate.** Human strategies are
replayed from published competition submissions under the same isolation, the
same test episode and the same metric convention as every model run.
**The scored day is never seen.** Training episodes are published; the scored
episode is not, and neither are the raw metrics measured on it — those would
characterise the episode well enough to tune against. `COMMITMENT.json`
publishes their hash, so the full evaluation can be audited without being
disclosed and released once those episodes retire.
## Designed as a trading problem, not a prediction task
**Execution is modelled, not assumed.** Orders join a queue, cross the touch or
rest behind it. Every fill is attributed against the book as it stood when the
order was submitted, so passive and aggressive fills are separable and expired
orders are counted.
**Risk is priced into the score, not reported beside it.** Combined performance
is `max(0, P - 0.1 x D)` scaled by a Sharpe quality factor. This is not
cosmetic: in our own reference set, repairing three logic defects in one human
strategy raised its P&L by 63% while its Sharpe fell from 7.27 to 3.03. Ranking
on P