D4Vinci / Scrapling
🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
unclecode / crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
scrapy / scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
apify / crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
gocolly / colly
Elegant Scraper and Crawler Framework for Golang
lexmount / moli
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
getmaxun / maxun
Turn any website into a structured API. Extract, automate, search and monitor the web.
projectdiscovery / katana
A next-generation crawling and spidering framework.
codelucas / newspaper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
apify / crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
adbar / trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
wzdnzd / aggregator
One-stop Proxies Crawling and Aggregation Platform
MontFerret / ferret
Declarative data automation language and Go runtime for structured extraction workflows.
exa-labs / exa-mcp-server
Exa MCP for web search and web crawling!
ai-robots-txt / ai.robots.txt
A list of AI agents and robots to block.
hardkoded / puppeteer-sharp
Headless Chrome .NET API
edoardottt / cariddi
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
internetarchive / heritrix3
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
apache / nutch
Apache Nutch is an extensible and scalable web crawler
symfony / panther
A browser testing and web crawling library for PHP and Symfony
brightdata / brightdata-mcp
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
NateScarlet / holiday-cn
📅🇨🇳中国法定节假日数据 自动每日抓取国务院公告
zorlan / skycaiji
蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统
howie6879 / ruia
Async Python 3.6+ web scraping micro-framework based on asyncio
rebrowser / rebrowser-patches
Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.
webrecorder / browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
scrapfly / scrapfly-scrapers
Scalable Python web scraping scripts for +40 popular domains
janreges / siteone-crawler
SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
dondai44423 / donsetch
Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
scrapinghub / scrapyrt
HTTP API for Scrapy spiders
StJudeWasHere / seonaut
Open source SEO audit tool.
BlessedRebuS / Krawl
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
tiliondev / fortress
Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.
rubycdp / vessel
Fast high-level web crawling Ruby framework
scrapinghub / spidermon
Scrapy Extension for monitoring spiders execution.
nuxt-modules / robots
Tame the robots crawling and indexing your Nuxt site.
beyondtahir / beyondseo
Your complete SEO house: native crawling, SEO/AEO/GEO, content, competitors and reputation. Includes 206 publishing sources with DR context and tailored backlink plans. No API keys required by the native engine.
bunoriapp / bunori
Read, Manage, Download & Export Light Novels, all in one Android app.
RankSpotAI / awesome-seo-mcp
A curated list of MCP servers for SEO. Search Console, keywords, backlinks, crawling, SERPs and AI search visibility, verified with install details and auth requirements.
tomaszboloz / SEOmi
Native macOS and Windows workspace for technical SEO audits, site crawling, semantic maps, keyword research, rank tracking, DataForSEO, Search Console, and AI visibility.
Zulwatha / content-parity
Measure what a website serves to machine readers. Fetches any page as several honest identities, including a cryptographically signed agent, and reports what each one received.