internetarchive / heritrix3

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

apache / nutch

Apache Nutch is an extensible and scalable web crawler