openlayer-ai / jevals

# jevals
Evals and guardrails for agents, using Jev-style decision models instead of an LLM judge. All the evals for a trace go out as one request that costs a few thousandths of a cent and comes back in a few hundred milliseconds, so you can run them on every trace and inside the agent loop.
Works with Jev through the TypeSafe or Vercel APIs, with any decision model on OpenRouter (Jev, Cloudflare's Clef, Liquid's d1 and more), with Kev or Laya running locally on a Mac, with Eikos on your own GPU, or with a regular chat LLM if that's all you have (slower, costs more).
 [code] 
 [code] 
That's one HTTP request for all eight, and those are the numbers Jev returned for the trace in `examples/quickstart.py`. `messages` is the OpenAI chat format (user, assistant with `tool_calls`, tool). If you have Anthropic content blocks or LangChain message objects you can pass those directly.
## The problem
Most teams eval a small sample of their traffic, if they eval at all, and the reason is usually cost. The judge is a frontier LLM and it is the expensive part of the pipeline.
Ragas is where most of us got our metric names, and its source shows where an LLM judge spends its money. Faithfulness is two LLM calls. Answer relevancy is three, plus embeddings. Context precision is one call per retrieved chunk. Every call carries few-shot examples, generates JSON token by token, and retries when the JSON doesn't parse. Running four metrics on one sample works out to six to eleven round trips depending on how many chunks you retrieved (measured below), and several seconds. At that price you sample 1%, run it nightly, and the results never get anywhere near the request path.
Agents make this harder. The traces are long, there's more to check (did it pick the right tool, did it use the result, did it stay in scope, did a tool result contain instructions it shouldn't have followed), and the judge is non-deterministic on top of being slow. LangChain ran the comparison: on identical traces, GPT and Claude judges had 92x to 913x the score variance of Jev. A judge that changes its mind between runs makes a poor test suite.
## What changed
Jev doesn't generate text. You send it some state and a set of typed questions (yes/no, pick one of these, score on this rubric) and it returns a calibrated probability for each one in a single forward pass. The questions are evaluated independently and in parallel, so asking 40 costs about the same latency as asking one. Pricing is $0.042 per million input tokens with no output tokens. Through Vercel's gateway we measured p50 244ms, p95 371ms per request.
There are already open-weight models speaking the same API: Kev (Qwen3, runs on a Mac), Laya (ModernBERT, about 10ms on Apple Silicon) and Eikos (Qwen3.5, 4B and 27B, MIT, served by vLLM on one GPU or by PyTorch on a Mac). The request shape, `state + {id: {type, instructions, criteria}}`, looks like it's going to stick.
Most of what an LLM judge is asked to decide fits those three questio