mode-io / vllm-jev

Fast structured decisions on Linux and Apple Silicon 
 Documentation ·
 Demos ·
 Getting Started ·
 Supported Models ·
 Example ·
 Updates 
---
## Demos
### 🎉 Celebrating RadixFlow! (experimental)
https://github.com/user-attachments/assets/d99f5a8d-d715-4a79-8870-e56a6e48c073
Our first PR, which explored shared-prefix reuse and cache-on/off behavior, inspired us to look more closely at what happens after a cache hit. Reuse matters especially for prefill: Jev scores candidate branches through prefill, so reusing their common prefix avoids repeated computation.
We found a scheduling gap: a matching prefix can still have hybrid state being produced in the current round. Stopping the scan at that pending branch leaves room that independent ready branches could use. **RadixFlow defers the pending branch, continues selecting ready work, and rechecks it after completion.** The engineering integration recovered executable capacity in the tested hybrid models; the pressure comparisons report the resulting worker rounds, speed and token reuse.
RadixFlow is **experimental and disabled by default**. Enable it with `--radixflow true` on supported configurations.
We welcome bug reports, feedback, and discussions—please share your experience with RadixFlow!
### Video understanding: live comparison
The same Valen checkpoint answers **24 structured questions about one video**, using eight sampled frames and one A800 per runtime. In this live run, Valen's PyTorch reference path took **5.31 s**, while vLLM Jev took **1.22 s** (**4.3× faster**); all 24 selected answers agreed. Model loading is excluded. Watch the MP4.
### Apple Silicon: live image decisions
Valen 2B answers six GameQA Maze image questions locally on an Apple M5 using MLX. The display follows live HTTP responses. Watch the MP4.
 More demos 
### Multimodal: live Sokoban
Both runtimes played the same five levels from Valen's Sokoban evaluation set with the same checkpoint and one A800 each.
### Text: 40 concurrent decisions
The Open-Jev-2B author HTTP server and vLLM Jev received the same 40 text-only Choice requests at once, on one A800 each.
## About
vLLM Jev serves compatible Jev-style checkpoints through vLLM. Give it a question and candidate answers; it returns a label and a probability for each candidate.
- **Native vLLM serving on Linux:** scheduling, batching, compilation, KV cache, and metrics.
- **Apple Silicon preview:** run supported text models through MLX or PyTorch MPS, and Valen multimodal decisions through MLX.
- **Structured decisions:** Choice, Noul (yes/no), and Score (ordered levels) over HTTP.
- **Multimodal inference:** Valen, vjev, RSI-Jev v4.0-VL, JevAny Pointer, and Clef accept text and images. Valen and Clef also accept short videos; platform availability and request formats are model-specific.
- **RadixFlow (experimental, opt-in):** bounded admission and state-ready traversal fill shared-prefix execution rounds. See configuration and pressure-test results.
- **Automatic setu