Simreal-AI / future-prediction-bench

# Future Prediction Bench
A probability forecasting benchmark and verifiable real-world agent environment for research on delayed feedback, evidence, and rollout collection. Forecasting agents research unresolved events and submit probabilities; coding agents change an isolated workspace and receive independently checked outcomes.
**Status: alpha research prototype.** The forecasting protocol, offline demos, live source adapters, and scripted environment integrations are implemented. There are no formal model forecasting results, hosted leaderboard, or complete LLM training pipeline. Text trajectory exports remain `trainer_ready=false`.
| Track | Agent task | Implemented output |
| --- | --- | --- |
| Research Benchmark | Research a future event and submit a probability for every option | Frozen forecasts, evidence traces, and outcome-based scores |
| Live RL Environment | Collect repeated train-only forecasting rollouts | Delayed rewards, policy-revision checks, and grouped text samples with advantages |
| Real-World Task RL | Inspect and change an isolated software workspace | Hidden verification, auditable task rewards, and train-only text trajectories |
The intended forecasting scope is multi-domain events targeted within seven days after the prediction deadline. Current automated coverage is **MLB baseball and USGS earthquakes**. The software track currently provides bounded coding tasks through Docker and experimental QEMU backends.
## Five-minute offline demo
Use **Python 3.10 or newer**. The core runtime has no third-party Python dependencies and uses the standard library. From a repository checkout:
 [code] 
Choose a new or empty output directory. The demo requires no API keys, network calls, model, or Docker. It creates **three synthetic questions, five submissions, and four exported train trajectories**: two train questions receive two scripted rollouts each, and one test question receives one benchmark submission. All five submissions first await outcomes; simulated resolution then scores them. The test submission never enters training export.
Inspect `report.json`, `bench.sqlite`, `questions.jsonl`, `pending_receipts.json`, and `training_trajectories.jsonl` in the output directory. The fixture clock and database mode keep this synthetic exercise distinct from live forecasting. Local `runs/` artifacts are ignored by Git.
## Forecasting contract
Binary and categorical questions use the same distribution format:
 [code] 
Every option must appear exactly once. Probabilities must be finite, lie in `[0, 1]`, and sum to one within `1e-6`; validation never silently normalizes. Categorical options must be mutually exclusive and collectively exhaustive. Question payloads, deadlines, resolution rules, and submissions are frozen.
The primary score is normalized Brier: `0.5 * sum((p_i - y_i)^2)`, where `y_i` marks the observed outcome. Valid forecasts remain pending until an authoritative outcome arrives. Missing, delayed, canceled, and void even