google / sentencepiece

git clone https://github.com/google/sentencepiece.git

Unsupervised text tokenizer for Neural Network-based text generation.