huggingface / tokenizers
git clone https://github.com/huggingface/tokenizers.git
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
—
stars, 24h
—
stars, 7 days
—
stars, 30 days
11k
★ total
1.2k
forks · 179 open issues
Star history starts 2026-10-20; the chart appears after the second daily snapshot.
LanguageRust
LicenseApache-2.0
Created2019-11-01 (6y ago)
Last push1d ago
OwnerOrganization