kyle-pena-nlp / jevchat

# jevchat
We know Jev.
`jevchat` turns that into a chat model. At every step it asks Jev one question:
> Given the user's question and the reply written so far, which symbol comes next?
The options are an alphabet plus an option to stop emitting. Jev returns a probability for each
one, and the sampler draws the next symbol from that normalised distribution.
Append, repeat, and stop when STOP is drawn.
There are several alphabets (including truncated token lists) and sampling strategies available. 
The idea is for fun, the cost is somewhat impractical, and the results are hilarious.
This was a Claude accelerated experiment. I described the sampling algorithms, strategies, and so on, and it implemented them.
## Setup
 [code] 
Put your Jev key in `.env` next to `pyproject.toml` (git-ignored):
 [code] 
`JEV_API_KEY` and `TYPESAFE_API_KEY` are also accepted. Values in `.env` win over
exported ones, so editing the file is enough to switch keys.
## Use
 [code] 
The reply appears as it is sampled, in a panel with a live readout of the
generation rate — symbols/s, characters/s, milliseconds per API call, elapsed
time — and the top few symbols Jev scored at the last step, so you can watch the
distribution the sampler is drawing from.
**Ctrl-C cancels.** The first press stops generation once the in-flight request
returns and keeps the partial reply; a second press aborts immediately. In chat,
the partial reply stays in the conversation history. `ask` exits 130 when cancelled.
Chat commands: `/help`, `/alphabet [name]`, `/temp `, `/stop-bias `,
`/reset`, `/stats`, `/exit`.
## Modes
Two things are swappable: **how** the distribution over the next symbol is
obtained (`-s/--strategy`), and **what** it is over (`-a/--alphabet`). Every
combination below is a runnable command.
### Strategies
`choice` asks one question over the whole alphabet. `bisect` sorts the alphabet
and asks earlier/later yes-no questions until the group is small, then asks one
choice question inside it.
 [code] 
### Presentations
 [code] 
### Beam search
 [code] 
Costs one score per live beam per step. Above width 1, `temperature`, `top_p` and
`top_k` stop applying — beams are ranked by probability, not drawn from.
### Alphabets
 [code] 
### Combining them
 [code] 
## Hypothesis options
There are two ways to ask Jev the same question. Under `--presentation symbol` the
options are the symbols themselves — `'a'`, `'i'`, `' the'` — and Jev has to append
the option to the reply in its head before judging it. The instructions used to say
exactly that: *"judge grammar and spelling on the concatenation, not on the option
on its own."*
Under `--presentation hypothesis` the options are the **resulting texts**:
 [code] 
The append is already done, so Jev only ranks finished strings — which is what a
decision model is built for. It is the single largest improvement in the project:
on character alphabets it roughly triples top-1 and doubles the probability mass
landing on the right symbol, for fewer inpu