rebrowser / rebrowser-patches
Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.
rebrowser / rebrowser-patches
fastcrw / crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
TikHub / TikHub-API-Python-SDK
High-performance asynchronous Douyin(抖音) TikTok Xiaohongshu(小红书) Kuaishou(快手) Weibo(微博) Instagram YouTube(油管) Twitter(X) Captcha Solver(验证码解决器) Temp Mail(临时邮箱) API(接口).
StJudeWasHere / seonaut
Open source SEO audit tool.
crawlseo / crawlseo
Open-source SEO monitoring. GSC + Site Crawler + Core Web Vitals in one dashboard.
vakra-dev / reader
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
beyondtahir / beyondseo
Your complete SEO house: native crawling, SEO/AEO/GEO, content, competitors and reputation. Includes 206 publishing sources with DR context and tailored backlink plans. No API keys required by the native engine.
dengmeiluan / ticket-monitoring
✈️ 自托管的多渠道机票价格监控：五平台明细采集 → 钉钉图文推送 → 达标四层强提醒。国内航线与国际航线（五洲目的地）同一套采集-判定-推送链路；数据留在本机，配置全在网页，开源免费。
s0md3v / Photon
Incredibly fast crawler designed for OSINT.
faisalman / ua-parser-js
UAParser.js: The Essential Web Development Tool for User-Agent Detection. Detect Browsers, OS, Devices, Bots, Apps, AI Crawlers, and more. Run in Browser or Node
wzdnzd / aggregator
One-stop Proxies Crawling and Aggregation Platform
MontFerret / ferret
Declarative data automation language and Go runtime for structured extraction workflows.
symfony / dom-crawler
Eases DOM navigation for HTML and XML documents
hardkoded / puppeteer-sharp
Headless Chrome .NET API
edoardottt / cariddi
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
any4ai / AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
internetarchive / heritrix3
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
symfony / panther
A browser testing and web crawling library for PHP and Symfony
crawl / crawl
Dungeon Crawl: Stone Soup official repository
facundoolano / google-play-scraper
Node.js scraper to get data from Google Play
howie6879 / owllook
owllook-小说搜索引擎
jae-jae / QueryList
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
brightdata / brightdata-mcp
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
xtuhcy / gecco
Easy to use lightweight web crawler（易用的轻量化网络爬虫）
JayBizzle / Crawler-Detect
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
AAndyProgram / SCrawler
🏳️🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, BlueSky, TikTok, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.
TeamNewPipe / NewPipeExtractor
NewPipe's core library for extracting data from streaming sites
zhegexiaohuozi / SeimiCrawler
一个简单、敏捷、分布式的支持SpringBoot的Java爬虫框架;An agile, distributed crawler framework.
coder-hxl / x-crawl
Flexible Node.js crawler library
schemacrawler / SchemaCrawler
Free database schema discovery and comprehension tool
howie6879 / ruia
Async Python 3.6+ web scraping micro-framework based on asyncio
rushter / selectolax
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
gildas-lormeau / single-file-cli
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
orangecoding / fredy
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)
0x676e67 / wreq-python
An ergonomic, privacy-aware Python HTTP Client
sethblack / python-seo-analyzer
An SEO tool that analyzes the structure of a site, crawls the site, count words in the body of the site and warns of any technical SEO issues.
ShilongLee / Crawler
抖音爬虫(a_bogus最新版)、快手、哔哩哔哩、小红书、淘宝、京东、微博等平台爬虫开源api接口服务器。docker一键快速部署。
dadoonet / fscrawler
Elasticsearch File System Crawler (FS Crawler)
openwpm / OpenWPM
A web privacy measurement framework
tavily-ai / tavily-python
The Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. Easily integrate smart search, content extraction, and research capabilities into your applications, harnessing Tavily's powerful features.
monperrus / crawler-user-agents
Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:
JustinBeckwith / linkinator
Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.
webrecorder / browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
omrilotan / isbot
🤖/👨🦰 Detect bots/crawlers/spiders using the user agent string
hisorange / browser-detect
Browser Detection for Laravel by hisorange!
codelibs / fess
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
scrapfly / scrapfly-scrapers
Scalable Python web scraping scripts for +40 popular domains
0x676e67 / wreq
An ergonomic, privacy-aware Rust HTTP Client
PhialsBasement / LibreCrawl
Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!
apache / stormcrawler
A scalable, mature and versatile web crawler based on Apache Storm
darbra / awesome-ai-reverse
ai reverse 逆向一把梭
fredwu / crawler
A high performance web crawler / scraper in Elixir.
xiaoxiunique / x-kit
Pure-Python protocol tools and research notes for X.com, with VibeLoft Twitter account datasets
radiantearth / stac-spec
SpatioTemporal Asset Catalog specification - making geospatial assets openly searchable and crawlable
0xAllenChen / spider_reverse
爬虫逆向案例，已完成：TLS指纹｜瑞数｜震坤行 | 网易易盾 | 微信小程序反编译逆向(百达星系) | 同花顺 | rpc解密 | 加速乐 | 极验滑块验证码 | 巨量算数 | Boss直聘 | 企查查 | 中国五矿 | qq音乐 | 产业政策大数据平台 | 企知道 | 雪球网(acw_sc__v2) | 1688 | 七麦数据 | whggzy | 企名科技 | mohurd | 艺恩数据 | 欧科云链
scrapinghub / scrapyrt
HTTP API for Scrapy spiders
skrapeit / skrape.it
A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.
rndinfosecguy / Scavenger
Crawler (Bot) searching for credential leaks on paste sites.
openclaw / discrawl
cli for Discord with sqlite backend
internetarchive / brozzler
brozzler - distributed browser-based web crawler