huggingface / datatrove

git clone https://github.com/huggingface/datatrove.git

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.