1amageek / swift-gigatoken
git clone https://github.com/1amageek/swift-gigatoken.git
670 MB/s Swift tokenizer inspired by Marcel Roed’s gigatoken. Zero-copy, NEON/CRC32, WebAssembly, and Embedded Swift.
—
stars, 24h
—
stars, 7 days
—
stars, 30 days
29
★ total
0
forks · 0 open issues
Star history starts 2026-10-20; the chart appears after the second daily snapshot.
LanguageSwift
LicenseMIT
Created2026-07-23 (2mo ago)
Last push2mo ago
OwnerUser
GitHub topics
arm64bpebyte-pair-encodingcore-aiembedded-swiftgigatokenhigh-performancelarge-language-modelsllmmlx-swiftneonsimdswiftswift-package-managertiktokentokenizationtokenizerwasmwebassemblyzero-copy