alienflm / MacTok
[CVPR 2026 Highlight] Official PyTorch implementation of MacTok, a masked 1D continuous image tokenizer that compresses images into just 64 or 128 tokens.
alienflm / MacTok
OpenBMB / VoxCPM
VoxCPM2: Tokenizer-Free TTS for Multilingual Speech Generation, Creative Voice Design, and True-to-Life Cloning
natasha / natasha
Solves basic Russian NLP tasks, API for lower level Natasha projects
lovit / soynlp
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
Natooz / MidiTok
MIDI / symbolic music tokenizers for Deep Learning models 🎶
sanderland / ctok
Claude's Tokenizer, Offline, Kinda
vincenzo-afk / Zen-2
A modern 124M-parameter decoder-only LLM built from scratch. RoPE · RMSNorm · SwiGLU · GQA · KV Cache · BPE Tokenizer · Full training pipeline. Research-grade, production-quality.
VizuaraAI / mini-kimi-k3
The pretraining scaffold behind 'Pretraining a Mini Kimi K3' (free book): six-source HF data mix, benchmark decontamination, tokenizer, WSD training loop, run monitoring. 5B tokens for $252.
majiayu000 / stealthprint
Python toolkit for fingerprinting anonymous LLMs on OpenAI-compatible endpoints using tokenizer, context, error, and vision probes
onurtirpan / agentic-llm-token-and-performance-tests
The same HTTP API in 10 languages at 3 sizes, 30 implementations. Token cost for LLM agents, plus cold start, latency, throughput, CPU and memory. All measured, all verified over live HTTP.