ruyianry / JevGym

# 🏟️ JevGym
**A timestamped benchmark and trading arena for real-world probabilistic forecasting.**
*Can a decision model forecast — and profitably trade — real-world events it has never seen?*
JevGym replays **real, already-resolved prediction markets** as a decision task: a model sees only
the information available at a past moment, commits to a probability, and is then scored against
what actually happened. Because a prediction market quotes a **direct probability** for every
event, the crowd becomes a strong, continuously-updated baseline — directly comparable to a
model's output (probability in, probability out). **Realized P&L in the trading arena is the
headline metric**; calibration (Brier, log-loss, ECE) is the check.
It measures the **Jev-series** ("System One") decision models, but the arena is open to any
model — official Jev, open Jev-wire servers, a fine-tuned Qwen, or a plain LLM — over the same
markets, the same frozen prompt, and the same scoring.
## ✨ Why it's different
- **No leakage, by construction.** Every piece of evidence carries a public-availability
 timestamp and `available_at ≤ snapshot.timestamp` is enforced; agents that search the web are
 bounded to the same moment via the Internet Archive. Models forecast the past *as if it were
 the present*.
- **Reward, not just accuracy.** A sequential arena turns each forecast into a trade against the
 contemporaneous market price and scores realized P&L — the thing that actually matters.
- **The crowd is a baseline, never the oracle.** The oracle is the resolved outcome. The market
 price is just a strong reference to beat.
- **One dataset, many uses.** The same leakage-safe snapshots power the benchmark, the arena,
 and an interactive site.
## 📊 What the numbers say (Kalshi, live)
On real resolved Kalshi markets, official **Jev is profitable in the arena — +11.8% average
return per contract** on the clearer markets, across **weather · crypto · the Fed**, and it
carries real signal *before a contract is even priced*. Near resolution the crowd is hard to
out-calibrate — which is exactly where the headroom is. Full method,
per-difficulty tables, and honest caveats: **KALSHIJEV.md**.
🎮 **Can you beat Jev?** — a slider game on
resolved questions, one per category. 🧪 **Evaluate Jev yourself**
— bring your own key, ask any question, grade the answer.
## 🧩 What's inside
| Component | Status | Explore |
| --- | --- | --- |
| **Kalshi** — the `KalshiJev` subset | ✅ **live** | **→ Kalshi report** |
| Polymarket | 🔜 soon | normalizes into the same schema |
| More sources & categories | 🔜 expanding | added continuously |
Coverage spans weather, crypto, and economics/the Fed today, with more categories flowing in.
Harder markets are **enriched with extra context** a model can leverage — macro-economic trends
and the series' own prior decisions — all timestamped and leakage-safe.
## ⚡ Quickstart (runs offline, no keys)
The CLI is **`jevgym`** — benchmark, arena, and dataset build in one too