google / sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.

PyThaiNLP / pythainlp

Thai natural language processing in Python

messense / jieba-rs

The Jieba Chinese Word Segmentation Implemented in Rust

bab2min / Kiwi

Kiwi(지능형 한국어 형태소 분석기)