D4Vinci / Scrapling

🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

unclecode / crawl4ai

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf

NanmiCoder / MediaCrawler

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 ｜ 评论爬虫、微博帖子 ｜ 评论爬虫、百度贴吧帖子 ｜ 百度贴吧评论回复爬虫 | 知乎问答文章｜评论爬虫

scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

ScrapeGraphAI / Scrapegraph-ai

Python scraper based on AI

Evil0ctal / Douyin_TikTok_Download_API

🚀 自托管的抖音、TikTok 数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

codelucas / newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

s0md3v / Photon

Incredibly fast crawler designed for OSINT.

zubair-trabzada / geo-seo-claude

GEO-first SEO skill for Claude Code. Comprehensive AI search optimization for any website — citability scoring, AI crawler analysis, brand authority, schema markup, platform-specific optimization, and PDF reports. If you want learn how to sell this to real businesses, check out the skool community

apify / crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

TeamWiseFlow / xiaobei

为OPC/中小微企业量身打造的自媒体获客智能体

hect0x7 / JMComic-Crawler-Python

Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀

adbar / trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

subzeroid / instagrapi

🔥 The fastest and powerful Python library for Instagram Private API 2026 with HikerAPI SaaS

DedSecInside / TorBot

Dark Web OSINT Tool

ai-robots-txt / ai.robots.txt

A list of AI agents and robots to block.

CharlesPikachu / videodl

Videodl: A lightweight video downloader written in pure python. (轻量级视频下载器，优先高清无水印，支持抖音，快手，小红书，B站，TikTok，YouTube，FIFA+，优酷，腾讯，爱奇艺，1905电影网，乐视，芒果，咪咕，PPTV，搜狐，Facebook，Twitter，新浪微博，今日头条，网易公开课，全民K歌，CCTV央视频，酷狗音乐MV，新片场，知乎，百度贴吧，TED等海量流媒体平台)

howie6879 / owllook

owllook-小说搜索引擎

vladkens / twscrape

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

lncrawl / lightnovel-crawler

Generate and download e-books from online sources.

3441293738 / creatorhub

多平台内容监控·采集·搬运 一个 Web 面板管起抖音 / 小红书 / 快手

howie6879 / ruia

Async Python 3.6+ web scraping micro-framework based on asyncio

rushter / selectolax

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

eatmoreduck / boss-zhipin-scraper

Boss直聘爬虫 / BOSS直聘职位数据抓取工具，基于 Chrome CDP 协议复用真实登录态，绕过字体反爬，输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.

0x676e67 / wreq-python

An ergonomic, privacy-aware Python HTTP Client

openwpm / OpenWPM

A web privacy measurement framework

scrapfly / scrapfly-scrapers

Scalable Python web scraping scripts for +40 popular domains

PhialsBasement / LibreCrawl

Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!

xiaoxiunique / x-kit

Pure-Python protocol tools and research notes for X.com, with VibeLoft Twitter account datasets

0xAllenChen / spider_reverse

爬虫逆向案例，已完成：TLS指纹｜瑞数｜震坤行 | 网易易盾 | 微信小程序反编译逆向(百达星系) | 同花顺 | rpc解密 | 加速乐 | 极验滑块验证码 | 巨量算数 | Boss直聘 | 企查查 | 中国五矿 | qq音乐 | 产业政策大数据平台 | 企知道 | 雪球网(acw_sc__v2) | 1688 | 七麦数据 | whggzy | 企名科技 | mohurd | 艺恩数据 | 欧科云链

TikHub / TikHub-API-Python-SDK

High-performance asynchronous Douyin(抖音) TikTok Xiaohongshu(小红书) Kuaishou(快手) Weibo(微博) Instagram YouTube(油管) Twitter(X) Captcha Solver(验证码解决器) Temp Mail(临时邮箱) API(接口).

scrapinghub / scrapyrt

HTTP API for Scrapy spiders

rndinfosecguy / Scavenger

Crawler (Bot) searching for credential leaks on paste sites.

tigerless-labs / seo-ops

SEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.

internetarchive / brozzler

brozzler - distributed browser-based web crawler

BlessedRebuS / Krawl

Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.

chadi0x / TheBigBrother

The Big Brother is a weaponized OSINT platform featuring username enumeration (473+ platforms), quad-vector visual intelligence, Sky Radar tracking, crypto wallet analysis, SSL intelligence, digital footprint reconstruction, EXIF extraction, advanced dorking, and network reconnaissance.

tiliondev / fortress

Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.

polyrabbit / hacker-news-digest

:newspaper: Let LLM Summarize Hacker News for You

martin-majlis / Wikipedia-API

Python wrapper for Wikipedia

PyPtt / PyPtt

The best PTT library

Madi-S / Lead-Generation

Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation.

CharlesPikachu / freeproxy

FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理，支持爬取数十个免费代理分享源，支持自定义规则代理筛选，爬虫与数据分析必备，每日更新海量免费代理。)

beyondtahir / beyondseo

Your complete SEO house: native crawling, SEO/AEO/GEO, content, competitors and reputation. Includes 206 publishing sources with DR context and tailored backlink plans. No API keys required by the native engine.

matthewprice / JobFinderOS

An agentic job search that runs on Claude Code: a recruiter, a crawler, and a market analyst writing to an Obsidian vault.

zhansan379 / boss-crawler-skill

一套完整的 BOSS 直聘求职自动化工具：爬取岗位 → 解析简历 → 智能匹配 → 可视化报告 → 自动生成优化简历图片 → 自动投递。帮助你更好精投简历。支持上传专精特精公司名单爬取数据

RayenMlayeh / jevcrawl

MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.

Alexxxxxxxxxxxxy / schinza-wechat-certificate

最新微信公众号文章抓取工具，修复7月29号微信公众号接口下架后不能抓取的问题，支持windows、MacOS系统

vsmutok / ytscrape

Free, open-source YouTube scraper library for Python. Search YouTube and extract video, channel, and playlist data without an API key or quota limits.

rockbenben / free-llm-intel

AI 厂商动态别挨家追：每天巡检 OpenAI、Claude、Gemini、DeepSeek、通义、Kimi 等的官方博客与更新页，生成可订阅的 RSS，英文原文逐段精译为中文，并附 70 家免费额度 / 限时活动总表（实时抓自官方页、带官方链接可复核）

RedNote / Xiaohongshu-Shield-Algorithm

小红书Shield算法 小红书签名算法 小红书x-mini-mua 小红书APP签名 小红书逆向 小红书纯算 小红书白盒AES 小红书MD5 小红书RC4 小红书libxyass Xiaohongshu Shield Xiaohongshu signature RedNote Shield algorithm x-mini-mua x-mini-sig x-mini-gid x-mini-s1 xy-common-params xy-platform-info 小红书爬虫 小红书采集 小红书数据 小红书API 小红书接口 小红书APP逆向 小红书SO逆向 小红书native签名 小红书加密算法 App Shield 算法还原 纯算还原 Python逆向 Android逆向 SO逆向

makralabs / makra

Makra is a memory layer between the web and your AI agents. It serves real-time structured data from the open web at the cost of a vector search.

Janusilver / multiplatform-downloader

多平台无水印下载器：抖音 / 小红书 / 快手 / B站 / X (Twitter) / Instagram 一键下载（Python + tkinter GUI）