facebookresearch / audiocraft
Audiocraft is a library for audio processing and generation with deep learning. It features the state-of-the-art EnCodec audio compressor / tokenizer, along with MusicGen, a simple and controllable music generation LM with textual and melodic conditioning.
alienflm / MacTok
[CVPR 2026 Highlight] Official PyTorch implementation of MacTok, a masked 1D continuous image tokenizer that compresses images into just 64 or 128 tokens.
rasbt / LLMs-from-scratch
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
OpenBMB / VoxCPM
VoxCPM2: Tokenizer-Free TTS for Multilingual Speech Generation, Creative Voice Design, and True-to-Life Cloning
google / sentencepiece
Unsupervised text tokenizer for Neural Network-based text generation.
huggingface / tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Chevrotain / chevrotain
Parser Building Toolkit for JavaScript
datawhalechina / diy-llm
Covers pre-training data, Tokenizer, Transformer, MoE,distributed training, Scaling Laws, inference & alignment .6 progressive code assignments for full-stack LLM learning | 涵盖预训练数据、分词器、Transformer、MoE、分布式训练、缩放定律、推理与对齐，6 项渐进代码作业，掌握 LLM 全栈知识
natasha / natasha
Solves basic Russian NLP tasks, API for lower level Natasha projects
uhop / stream-json
A micro-library of stream components for building custom JSON and JSONC processing pipelines with a minimal memory footprint — parse, filter, and transform JSON far larger than available memory with a SAX-inspired token API, on Node.js or Web Streams.
lovit / soynlp
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
ikawaha / kagome
Self-contained Japanese Morphological Analyzer written in pure Go
Natooz / MidiTok
MIDI / symbolic music tokenizers for Deep Learning models 🎶
niieani / gpt-tokenizer
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
bab2min / Kiwi
Kiwi(지능형 한국어 형태소 분석기)
knuddelsgmbh / jtokkit
JTokkit is a Java tokenizer library designed for use with OpenAI models.
lindera / lindera
A multilingual morphological analysis library.
glayzzle / php-parser
NodeJS PHP Parser - extract AST or tokens
actual-computer / toks
exact tokenizers ids, as fast as the hardware allows
mattn / tensai
A tiny neural-network framework in pure Go with AVX2 SIMD kernels (GOEXPERIMENT=simd)
sanderland / ctok
Claude's Tokenizer, Offline, Kinda
gameplayerroblox9-cmyk / RoNet-BPE-Tensor-Forge
RoNet 2026: Build Transformer Models and Byte-Level BPE Tokenizers in Pure Tensors
gillverd / accelerando-8192
Accelerando-8192 — an 8,192-token alien glyph alphabet, interactive encoder, and complete visual atlas.
OpenLlamas / model-xray
Model authenticity / dilution detection CLI: one command, one report. Detect fake or watered-down LLM endpoints via identity, context, tokenizer, capability and latency checks.
SergeevDmitry / llm-kit
Small, independent TypeScript packages for LLM plumbing — token budgets, streaming JSON repair, cost accounting, retries, embedding caches. No provider SDKs.
Greninja9257 / LabLLM
A native macOS lab for teaching tiny language models to think — build the architecture, train the weights, and watch a small LLM emerge from scratch, locally on Apple Silicon with custom data, tokenizers, checkpoints, and MLX acceleration.
vincenzo-afk / Zen-2
A modern 124M-parameter decoder-only LLM built from scratch. RoPE · RMSNorm · SwiGLU · GQA · KV Cache · BPE Tokenizer · Full training pipeline. Research-grade, production-quality.
VizuaraAI / mini-kimi-k3
The pretraining scaffold behind 'Pretraining a Mini Kimi K3' (free book): six-source HF data mix, benchmark decontamination, tokenizer, WSD training loop, run monitoring. 5B tokens for $252.
majiayu000 / stealthprint
Python toolkit for fingerprinting anonymous LLMs on OpenAI-compatible endpoints using tokenizer, context, error, and vision probes
1amageek / swift-gigatoken
670 MB/s Swift tokenizer inspired by Marcel Roed’s gigatoken. Zero-copy, NEON/CRC32, WebAssembly, and Embedded Swift.
onurtirpan / agentic-llm-token-and-performance-tests
The same HTTP API in 10 languages at 3 sizes, 30 implementations. Token cost for LLM agents, plus cold start, latency, throughput, CPU and memory. All measured, all verified over live HTTP.