projectdiscovery / katana

A next-generation crawling and spidering framework.

gocolly / colly

Elegant Scraper and Crawler Framework for Golang

StJudeWasHere / seonaut

Open source SEO audit tool.

MontFerret / ferret

Declarative data automation language and Go runtime for structured extraction workflows.

edoardottt / cariddi

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Zulwatha / content-parity

Measure what a website serves to machine readers. Fetches any page as several honest identities, including a cryptographically signed agent, and reports what each one received.