mandeal303 / tiktokenizer

git clone https://github.com/mandeal303/tiktokenizer.git

An interactive playground for comparing how different language models tokenize the same text.