apache / nutch

Apache Nutch is an extensible and scalable web crawler

MarginaliaSearch / MarginaliaSearch

Internet search engine for text-oriented websites. Indexing the small, old and weird web.

apache / stormcrawler

A scalable, mature and versatile web crawler based on Apache Storm