Autumn-27 / ScopeSentry

ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes

TecharoHQ / anubis

Weighs the soul of incoming HTTP requests to stop AI crawlers

projectdiscovery / katana

A next-generation crawling and spidering framework.

StJudeWasHere / seonaut

Open source SEO audit tool.

gocolly / colly

Elegant Scraper and Crawler Framework for Golang

edoardottt / cariddi

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

chushuai / wscan

Wscan is a web security scanner that focuses on web security, dedicated to making web security accessible to everyone.

monperrus / crawler-user-agents

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star: