D4Vinci / Scrapling
🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
unclecode / crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
NanmiCoder / MediaCrawler
小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 ｜ 评论爬虫、微博帖子 ｜ 评论爬虫、百度贴吧帖子 ｜ 百度贴吧评论回复爬虫 | 知乎问答文章｜评论爬虫
scrapy / scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
ScrapeGraphAI / Scrapegraph-ai
Python scraper based on AI
Evil0ctal / Douyin_TikTok_Download_API
🚀 自托管的抖音、TikTok 数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
codelucas / newspaper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
s0md3v / Photon
Incredibly fast crawler designed for OSINT.
zubair-trabzada / geo-seo-claude
GEO-first SEO skill for Claude Code. Comprehensive AI search optimization for any website — citability scoring, AI crawler analysis, brand authority, schema markup, platform-specific optimization, and PDF reports. If you want learn how to sell this to real businesses, check out the skool community
apify / crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
TeamWiseFlow / xiaobei
为OPC/中小微企业量身打造的自媒体获客智能体
hect0x7 / JMComic-Crawler-Python
Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀
adbar / trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
subzeroid / instagrapi
🔥 The fastest and powerful Python library for Instagram Private API 2026 with HikerAPI SaaS
wzdnzd / aggregator
One-stop Proxies Crawling and Aggregation Platform
DedSecInside / TorBot
Dark Web OSINT Tool
ai-robots-txt / ai.robots.txt
A list of AI agents and robots to block.
CharlesPikachu / videodl
Videodl: A lightweight video downloader written in pure python. (轻量级视频下载器，优先高清无水印，支持抖音，快手，小红书，B站，TikTok，YouTube，FIFA+，优酷，腾讯，爱奇艺，1905电影网，乐视，芒果，咪咕，PPTV，搜狐，Facebook，Twitter，新浪微博，今日头条，网易公开课，全民K歌，CCTV央视频，酷狗音乐MV，新片场，知乎，百度贴吧，TED等海量流媒体平台)
howie6879 / owllook
owllook-小说搜索引擎
vladkens / twscrape
Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.
lncrawl / lightnovel-crawler
Generate and download e-books from online sources.
3441293738 / creatorhub
多平台内容监控·采集·搬运 一个 Web 面板管起抖音 / 小红书 / 快手
NateScarlet / holiday-cn
📅🇨🇳中国法定节假日数据 自动每日抓取国务院公告
howie6879 / ruia
Async Python 3.6+ web scraping micro-framework based on asyncio
rushter / selectolax
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
eatmoreduck / boss-zhipin-scraper
Boss直聘爬虫 / BOSS直聘职位数据抓取工具，基于 Chrome CDP 协议复用真实登录态，绕过字体反爬，输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.
0x676e67 / wreq-python
An ergonomic, privacy-aware Python HTTP Client
sethblack / python-seo-analyzer
An SEO tool that analyzes the structure of a site, crawls the site, count words in the body of the site and warns of any technical SEO issues.
openwpm / OpenWPM
A web privacy measurement framework
tavily-ai / tavily-python
The Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. Easily integrate smart search, content extraction, and research capabilities into your applications, harnessing Tavily's powerful features.
scrapfly / scrapfly-scrapers
Scalable Python web scraping scripts for +40 popular domains
PhialsBasement / LibreCrawl
Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!
xiaoxiunique / x-kit
Pure-Python protocol tools and research notes for X.com, with VibeLoft Twitter account datasets
0xAllenChen / spider_reverse
爬虫逆向案例，已完成：TLS指纹｜瑞数｜震坤行 | 网易易盾 | 微信小程序反编译逆向(百达星系) | 同花顺 | rpc解密 | 加速乐 | 极验滑块验证码 | 巨量算数 | Boss直聘 | 企查查 | 中国五矿 | qq音乐 | 产业政策大数据平台 | 企知道 | 雪球网(acw_sc__v2) | 1688 | 七麦数据 | whggzy | 企名科技 | mohurd | 艺恩数据 | 欧科云链
TikHub / TikHub-API-Python-SDK
High-performance asynchronous Douyin(抖音) TikTok Xiaohongshu(小红书) Kuaishou(快手) Weibo(微博) Instagram YouTube(油管) Twitter(X) Captcha Solver(验证码解决器) Temp Mail(临时邮箱) API(接口).
scrapinghub / scrapyrt
HTTP API for Scrapy spiders
rndinfosecguy / Scavenger
Crawler (Bot) searching for credential leaks on paste sites.
tigerless-labs / seo-ops
SEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.
internetarchive / brozzler
brozzler - distributed browser-based web crawler
BlessedRebuS / Krawl
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
chadi0x / TheBigBrother
The Big Brother is a weaponized OSINT platform featuring username enumeration (473+ platforms), quad-vector visual intelligence, Sky Radar tracking, crypto wallet analysis, SSL intelligence, digital footprint reconstruction, EXIF extraction, advanced dorking, and network reconnaissance.
tiliondev / fortress
Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.
MigoXLab / dingo
Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool
polyrabbit / hacker-news-digest
:newspaper: Let LLM Summarize Hacker News for You
martin-majlis / Wikipedia-API
Python wrapper for Wikipedia
PyPtt / PyPtt
The best PTT library
Madi-S / Lead-Generation
Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation.
oooldtoy / SSTAP_ip_crawl_tool
一个自动获取游戏远程ip，并自动写成SSTAP/NETCH规则文件的脚本
CharlesPikachu / freeproxy
FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理，支持爬取数十个免费代理分享源，支持自定义规则代理筛选，爬虫与数据分析必备，每日更新海量免费代理。)
scrapinghub / spidermon
Scrapy Extension for monitoring spiders execution.
beyondtahir / beyondseo
Your complete SEO house: native crawling, SEO/AEO/GEO, content, competitors and reputation. Includes 206 publishing sources with DR context and tailored backlink plans. No API keys required by the native engine.
matthewprice / JobFinderOS
An agentic job search that runs on Claude Code: a recruiter, a crawler, and a market analyst writing to an Obsidian vault.
zhansan379 / boss-crawler-skill
一套完整的 BOSS 直聘求职自动化工具：爬取岗位 → 解析简历 → 智能匹配 → 可视化报告 → 自动生成优化简历图片 → 自动投递。帮助你更好精投简历。支持上传专精特精公司名单爬取数据
ripmilla / netwalk
Read-only network survey toolkit for AI coding agents: crawl a site from one device, diagnose it, draw it, and hand over a report — without ever changing a device or seeing a credential.
RayenMlayeh / jevcrawl
MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.
Alexxxxxxxxxxxxy / schinza-wechat-certificate
最新微信公众号文章抓取工具，修复7月29号微信公众号接口下架后不能抓取的问题，支持windows、MacOS系统
aldemaroc / forage
Lightweight self-hosted web search & extract service for Hermes Agent, a drop-in Firecrawl alternative
vsmutok / ytscrape
Free, open-source YouTube scraper library for Python. Search YouTube and extract video, channel, and playlist data without an API key or quota limits.
simonlin1212 / FactReach
面向 AI Agent 的互联网搜索 Skill：覆盖 23 个渠道（全网、X、小红书、抖音、微博、B站、Reddit、YouTube、GitHub、微信公众号等），零鉴权优先。Internet search skill for AI agents across 23 channels.
rockbenben / free-llm-intel
AI 厂商动态别挨家追：每天巡检 OpenAI、Claude、Gemini、DeepSeek、通义、Kimi 等的官方博客与更新页，生成可订阅的 RSS，英文原文逐段精译为中文，并附 70 家免费额度 / 限时活动总表（实时抓自官方页、带官方链接可复核）