actual-computer / toks

# toks
**The tokenizer that keeps up with your memory bus.** (ﾉ◕ヮ◕)ﾉ*:・ﾟ✧
toks is Actual Computer's tokenizer. Give it the `tokenizer.json` your model ships with and it returns exactly the
ids Hugging Face tokenizers would, 13-151x faster. It's a small C
library with hand-written asm kernels for arm64 (NEON) and x86-64 (AVX2), no runtime and no dependencies, plus a
Python package with an hf-style `Tokenizer` API.
- **Exact.** Same ids as hf tokenizers 0.23.2, checked over hundreds of thousands of cases per model on every
 CPU tier. A tokenizer toks can't reproduce exactly gets refused at load, with the missing feature named. You
 never get quietly different ids.
- **Fast.** On one core with fresh text, toks is 13-151x faster than hf tokenizers in every cell of the speed
 table, 4-23x faster than tiktoken in every cell tiktoken can run, and ahead of gigatoken, the fastest tokenizer
 we know of, in 252 of 255 cold cells (receipts below).
- **Tiny and embeddable.** A plain C ABI (`include/toks.h`) with caller-owned buffers. Nothing
 is allocated after load, the core has no threads and no callbacks, and the input is read where it sits without
 being copied. It's built to live inside inference engines.
**Try it in your browser at toks.actual.inc.** The page runs toks's portable C core,
compiled to WebAssembly, right in the tab: the same ids as the native library, timed on the scalar tier instead of
the asm kernels (How fast has those).
> **toks 0.3.2 is released** (tag `v0.3.2`): C bundles for linux arm64, linux x86-64 and macOS arm64, and Python
> wheels for CPython 3.10-3.14. 0.3.2 is the first release under the Apache License, Version 2.0 (0.3.0
> and 0.3.1 shipped under the Business Source License 1.1), and it carries the work since 0.3.1: ABI 0.4's hf and
> tiktoken primitives (`toks_template`, `toks_added`, `TOKS_NO_TRUNCATE` / `TOKS_NO_PAD`, `TOKS_DECODE_RAW`) with
> hf's and tiktoken's `Encoding` on the Python `Tokenizer`, a SentencePiece exactness fix (a pending unk across
> byte-fallback chars), `make test-guard` (every table and scratch region on its own pages) and the stall screen. Its
> exactness receipts are the release commit's CI (both tiers on linux x86-64 and arm64, macOS, Windows) and the
> nightly full-parity run on it, named in the release notes. The 0.3.0 release report,
> `docs/release/0.3.md`, carries every gate of the 0.3 goal with its receipt and an UNMET
> table for the ones still open, the fuzzing budget and the proof package among them. The speed numbers below were
> measured at `245cc5c`, the 0.3.0 release candidate, and are re-measured at a later cut. Commit ids and PR numbers
> quoted in this README and under `docs/` from before the first public commit belong to the private history this
> tree was cut from; the receipts they name are in the tree.
## How fast
On one core with fresh text, toks runs 13-151x faster than hf tokenizers in every cell of the speed table on the
asm tiers: 88 cells per machine on gb10c, tr9970x a