PostHog / jeeves

# Jeeves – Reasoning improves Jev-like decision models
A Jev-style classifier that reasons when it is unsure, with a diffusion drafter, trained on public data.
## Acknowledgements
Inspired by Kev.
## Highlights
- A 9B Jev-like model (Qwen3.5-9B, LoRA, pointer head) that reasons before it decides when it is unsure, with a block-4 diffusion drafter and the full training code and train/dev/test data.
- Reasoning only when unsure, it beats Kev-9B, Jev and JevK5 on test data it was never trained on (0.905 vs 0.822, 0.857 and 0.802) and on JevBench's public tiers (0.896 vs 0.866 for Jev and 0.861 for JevK5).
- Supports yes/no (`noul`), multiple-choice (`choice`), and rating (`score`) questions in the same request, through a Jev-compatible API.
- About 20 ms per request without thinking and a 1.2 s median with it on one H100 with `--precision fp8`. Can be sped up by truncating chain length.
- Runs on CUDA in bf16, or with FP8 linear layers (`--precision fp8`) on GPUs with compute capability 8.9 or higher. Inference also runs on Apple Silicon (MPS) in bf16 or FP8.
## Problem
Jev-like models give calibrated decision probabilities, but at low accuracy. A lot of pipelines therefore rely on a reasoning model as a fallback. Jeeves trains a Jev-like Qwen3.5-9B (LoRA and a pointer head) to reason before it decides when it is unsure.
This results in better performance on out of domain tasks, and outperforms Jev in JevBench hard (public).
## Results
Jeeves v2 and v1 answer directly when their calibrated confidence is at least 0.9 and otherwise reason first (`nothink_threshold` 0.9, greedy, 2,560-token cap); the percentage is the share of questions that reasoned. JevK5 (v0.3) does not reason; we ran it ourselves on the same items. The Kev-9B and Jev columns are the numbers Kev publishes.
| bench | Kev-9B | Jev | JevK5 | Jeeves v1 | Jeeves v2 |
| ------------------------------------------------------------ | ----------------------- | ----------------------- | ----------------------- | ------------------------------ | ------------------------------- |
| **Test overall** (out-of-domain and held-out, item-weighted) | 0.822 | 0.857 | 0.802 | 0.884 (41%) | 0.905 (39%) |
| **Transfer overall** (MMLU-Pro and buried state) | 0.579 | 0.800 | 0.561 | 0.728 (86%) | 0.780 (76%) |
| **JevBench overall** (231 public items) | 0.715\* | 0.866 | 0.861 | 0.913 (44%) | 0.896 (29%) |
| QNLI | 0.925 | 0.925 | 0.900 | 0.912 | 0.950 |
| SciQ | 0.963 | 0.988 | 0.948 | 0.991 | 0.983 |
| TweetEval offensive | 0.775 | 0.813 | 0.850 | 0.825 | 0.838 |
| PAWS | 0.763 | 0.788 | 0.838 | 0.875 | 0.875 |
| MMLU | 0.738 | 0.900 | 0.690 | 0.810 | 0.845 |
| Emotion | 0.600 | 0.588 | 0.664 | 0.638 | 0.707 |
| Held-out rule structures | 0.896 | 0.885 | 0.729 | 0.979 | 1.000 |
| Contrastive policies | 0.900 | 0.963 | 0.827 | 0.980 | 0.985 |
| MMLU-Pro (10-way) | 0.515 | 0.840 | 0.504 | 0.726 | 0.770 |
| Buried state | 0.740 | 0.700 | 0.672 | 0.733 | 0.802 |
| Unknowable answered at p ≥ 0.9 (lower is bett