StartLuxLabs / DecisionPivot

# DecisionPivot
**Evaluating decision models at the control points of LLM systems.**
Paper (arXiv:2610.28522) · Dataset (Hugging Face) · Citation
*Figure 1 of the paper.* (a) A contrast pair from `state_goal`: the two traces differ only in the order of steps 3 and 5, which flips
whether the goal is reached. Among the models shown, only the two SL-27B variants answer both members right. (b) Six open 27B decision
models and the hosted Jev on the public test sets and on DecisionPivot: models with similar public scores differ widely in pair accuracy
and in how much of the work they can decide at a 5% error rate.
DecisionPivot was called DecideBench until 14 October 2026. The items and their hashes did not change.
A *decision model* reads a case, the *state*, and a typed question with a fixed set of options, and returns a probability for every option
in one forward pass. LLM systems use such models to route requests, guard tool calls, check outputs and decide when a case goes to a
person. DecisionPivot measures whether these decisions are right, whether they stay right when one decisive detail of the case changes, and
how much of the work a model can take over at a fixed error rate.
This repository holds the code of the benchmark: the generators and independent solvers of the core track, the audits, the scorer, the
runners, rebuild scripts for 13 of the reference tasks, the task cards and hashes of every release, and tests.
The items themselves are distributed through the Hugging Face dataset (see Get the items).
> **Evaluation only.** The items must never be used as training data or put into the prompts of a model that is being trained. Some
> reference tasks of layer B contain harmful text by design, as data to classify.
## The benchmark at a glance
| | core track | reference track |
|---|---|---|
| what | 13 program-generated families | 35 public test sets converted to the same request format |
| items | 7,800 (600 per family, 200 on each of three levels) | 34,104 in release 1.1, 32,233 in release 1.3 |
| answers | computed by the generator, re-derived by an independent solver | the labels of the source |
| structure | contrast groups: the same case two to four times, with one planted detail that changes the answer | as in the source |
| main scores | accuracy, pair accuracy, level-3 pair accuracy, automation at 5% error, accuracy at 0.90, calibration error | the metric of the source, chance-corrected, averaged per layer |
The 48 tasks follow the seven places where a decision sits in an LLM system:
| layer | decision point | core families | reference tasks |
|---|---|---|---:|
| A | before generation: can the request be answered, will a named model answer it | `answerable_facts` | 6 |
| B | guards: is the request allowed under written rules, is it unsafe | `perm_inherit`, `policy_case` | 7 |
| C | after generation: is the claim or the response right | `log_count_check` | 8 |
| D | agent loop: is the tool call valid, is the goal reached | `sched_con