D4Vinci / Scrapling
🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
unclecode / crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
scrapy / scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
codelucas / newspaper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
apify / crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
adbar / trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
wzdnzd / aggregator
One-stop Proxies Crawling and Aggregation Platform
ai-robots-txt / ai.robots.txt
A list of AI agents and robots to block.
NateScarlet / holiday-cn
📅🇨🇳中国法定节假日数据 自动每日抓取国务院公告
howie6879 / ruia
Async Python 3.6+ web scraping micro-framework based on asyncio
scrapfly / scrapfly-scrapers
Scalable Python web scraping scripts for +40 popular domains
scrapinghub / scrapyrt
HTTP API for Scrapy spiders
BlessedRebuS / Krawl
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
tiliondev / fortress
Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.
scrapinghub / spidermon
Scrapy Extension for monitoring spiders execution.
beyondtahir / beyondseo
Your complete SEO house: native crawling, SEO/AEO/GEO, content, competitors and reputation. Includes 206 publishing sources with DR context and tailored backlink plans. No API keys required by the native engine.
RankSpotAI / awesome-seo-mcp
A curated list of MCP servers for SEO. Search Console, keywords, backlinks, crawling, SERPs and AI search visibility, verified with install details and auth requirements.