D4Vinci / Scrapling

🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

unclecode / crawl4ai

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf

Mahanaicoach / google-maps-scraper-kit

Free Google Maps scraper for lead generation. Exports phones, emails, websites and socials to CSV. Runs locally in Docker, no API keys. Claude Code skill included.

stickerdaniel / linkedin-mcp-server

Open-source MCP server for LinkedIn. Give Claude and any MCP-compatible AI agent access to profiles, companies, jobs, and messages.

subzeroid / instagrapi

🔥 The fastest and powerful Python library for Instagram Private API 2026 with HikerAPI SaaS

instaloader / instaloader

Download pictures (or videos) along with their captions and other metadata from Instagram.

apify / crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

vladkens / twscrape

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

KatrielMoses / MailAccess

Free email OSINT tool, 5000+ platforms, identity clustering, breach detection. No API keys required. pip install mailaccess

Coding-Doctor-Omar / reddrix

Reddrix is an open-source, blazing-fast Reddit Scraper for Python (No Login or API Key)

yusufkaraaslan / Skill_Seekers

Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection

XD-MHLOO / facebook-gql-scraper

Scrape Facebook GraphQL data without an API key — profiles, posts, comments, reactions, groups, marketplace, search_engine.

Evil0ctal / Douyin_TikTok_Download_API

🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

codelucas / newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

speedyapply / JobSpy

Jobs scraper library for LinkedIn, Indeed, Glassdoor, ZipRecruiter, Bayt, Naukri, BDJobs & HelloWork

lncrawl / lightnovel-crawler

Generate and download e-books from online sources.

hhursev / recipe-scrapers

Python package for scraping recipes data

Altimis / Scweet

Scrape tweets, profiles, followers and following from Twitter/X, no API key needed. Python library with smart multi-account pooling, proxy support and async.

eatmoreduck / boss-zhipin-scraper

Boss直聘爬虫 / BOSS直聘职位数据抓取工具，基于 Chrome CDP 协议复用真实登录态，绕过字体反爬，输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.

alltheplaces / alltheplaces

A set of spiders and scrapers to extract location information from places that post their location on the internet.

fa0311 / TwitterInternalAPIDocument

Twitter Internal API Document

justoneapi-labs / douyin-api

抖音API、抖音接口、抖音爬虫示例。

itsOwen / CyberScraper-2077

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

sunscrapers / djoser

REST implementation of Django authentication system.

AWeirdDev / flights

Fast, robust Google Flights scraper (API) for Python. (Probably)

rushter / selectolax

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

emcf / thepipe

Get clean data from tricky documents, powered by vision-language models ⚡

0x676e67 / wreq-python

An ergonomic, privacy-aware Python HTTP Client

egbertbouman / youtube-comment-downloader

Simple script for downloading Youtube comments without using the Youtube API

scrapfly / scrapfly-scrapers

Scalable Python web scraping scripts for +40 popular domains

ranahaani / GNews

A Happy and lightweight Python Package that Provides an API to search for articles on Google News and returns a JSON response.

ythx-101 / x-tweet-fetcher

Fetch X/Twitter tweets, replies, timelines, and articles without login or API keys — field tool for AI agents.

openstates / openstates-scrapers

source for Open States scrapers

scrapinghub / scrapyrt

HTTP API for Scrapy spiders

tducret / amazon-scraper-python

Non-official client to get some info about products sold on Amazon

sudoguy / tiktokpy

Tool for automated TikTok interactions

openzim / zimit

Make a ZIM file from any Web site and surf offline!

opensanctions / opensanctions

An open database of international sanctions data, persons of interest and politically exposed persons

vil / H4X-Tools

A modular, terminal-based toolkit for OSINT, reconnaissance, and scraping - built in Python, runs on Linux and Windows.

flosell / trailscraper

A command-line tool to get valuable information out of AWS CloudTrail

tiliondev / fortress

Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.

martin-majlis / Wikipedia-API

Python wrapper for Wikipedia

saifalisew1508 / Telegram-Members-Adder

Telegram Members Adding Software/Script Using Termux.

Madi-S / Lead-Generation

Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation.

BullsEye0 / ghost_eye

Ghost Eye Informationgathering Footprinting Scanner and Recon Tool Release. Ghost Eye is an Information Gathering Tool I made in python 3. To run Ghost Eye, it only needs a domain or ip. Ghost Eye can work with any Linux distros if they support Python 3. Author: Jolanda de Koff

gunkonchi / xscrape

Async Python client for collecting public data from X (Twitter) - search, profiles, threads, replies, and media. Built-in account rotation, adaptive rate limiting, and export to JSON/CSV/SQLite.

ksanjeev284 / reddit-universal-scraper

Universal Reddit Scraper - Works on any Subreddit or User

jannisborn / paperscraper

A bibliometrics tool for publication metadata (pubmed, arxiv, medrxiv, biorxiv, chemrxiv) and citation analyses

Xewdy444 / CF-Clearance-Scraper

A simple program for scraping Cloudflare clearance (cf_clearance) cookies from websites issuing Cloudflare challenges to visitors

JXZ666 / slgking

黄油之王：自由检索挑选你喜欢的黄油，机器学习你的xp，方便快捷地管理本地黄油

dengji85 / avm

Scrape. Organize. Play. Analyze. — Your local JAV hub.一款现代化的本地 JAV 管理平台。不只是管理文件，更能通过播放记录、标签频次、演员分布等数据，自动生成你的个人观影报告。集刮削、整理、播放、分析于一体，是你的私人影库，更是你的观影喜好洞察工具。无需安装，下载即用。

vedantdhande04 / tweetytweets

Autonomous X/Twitter posting pipeline for your AI agent. No API key, no paid scrapers: researches your niche, writes in your voice, publishes through a real browser, and verifies its own posts.

vsmutok / ytscrape

Free, open-source YouTube scraper library for Python. Search YouTube and extract video, channel, and playlist data without an API key or quota limits.

nodemaven / google-browser-scraper

Self-hosted Google search scraper for Python. Scrape Google SERP results with a real Chrome browser and your own residential proxy. Self-hosted SERP API alternative with JSON output, an HTTP API and an MCP server for AI agents.

sami-mag07 / ultimate-scraper-agent

Research skill and researcher subagent for Claude Code: multi-engine search behind one interface, parallel agent waves, and a verify gate that blocks any citation without a verbatim quote. No personal data; configured via .env.

rockbenben / ai-job-search-cn

一条命令跑完国内求职：猎聘 BOSS 智联前程一起搜，学历、户口、外包这些硬性条件先筛掉陪跑岗，打招呼话术和中文简历都备好，你只管点发送。数据不出本机，Claude Code、Codex、Qoder、Antigravity、Qwen、MiMo 都照样跑。

ismailsaoulaj / reddit-mcp-server

A production-grade Model Context Protocol (MCP) server. Empowers AI assistants (Claude, Cursor) to search, fetch, and analyze Reddit discussions with resilient rate-limiting and smart LLM noise filtering.

noahneancy-byte / netreach

No-login web access layer for AI agents — 7 zero-config platforms

RedNote / Xiaohongshu-Shield-Algorithm

小红书Shield算法 小红书签名算法 小红书x-mini-mua 小红书APP签名 小红书逆向 小红书纯算 小红书白盒AES 小红书MD5 小红书RC4 小红书libxyass Xiaohongshu Shield Xiaohongshu signature RedNote Shield algorithm x-mini-mua x-mini-sig x-mini-gid x-mini-s1 xy-common-params xy-platform-info 小红书爬虫 小红书采集 小红书数据 小红书API 小红书接口 小红书APP逆向 小红书SO逆向 小红书native签名 小红书加密算法 App Shield 算法还原 纯算还原 Python逆向 Android逆向 SO逆向

makralabs / makra

Makra is a memory layer between the web and your AI agents. It serves real-time structured data from the open web at the cost of a vector search.