Panniantong / Agent-Reach
Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Panniantong / Agent-Reach
D4Vinci / Scrapling
🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
unclecode / crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
soimort / you-get
:arrow_double_down: Dumb downloader that scrapes the web
ScrapeGraphAI / Scrapegraph-ai
Python scraper based on AI
Evil0ctal / Douyin_TikTok_Download_API
🚀 自托管的抖音、TikTok 数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
yusufkaraaslan / Skill_Seekers
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
codelucas / newspaper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
instaloader / instaloader
Download pictures (or videos) along with their captions and other metadata from Instagram.
apify / crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
subzeroid / instagrapi
🔥 The fastest and powerful Python library for Instagram Private API 2026 with HikerAPI SaaS
speedyapply / JobSpy
Jobs scraper library for LinkedIn, Indeed, Glassdoor, ZipRecruiter, Bayt, Naukri, BDJobs & HelloWork
stickerdaniel / linkedin-mcp-server
Open-source MCP server for LinkedIn. Give Claude and any MCP-compatible AI agent access to profiles, companies, jobs, and messages.
itsOwen / CyberScraper-2077
A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama
vladkens / twscrape
Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.
sunscrapers / djoser
REST implementation of Django authentication system.
lncrawl / lightnovel-crawler
Generate and download e-books from online sources.
simonw / shot-scraper
A CLI utility for taking screenshots of websites, recording video demos and scraping sites using JavaScript
vasu-devs / JustHireMe
Local-first AI job intelligence workbench for scraping roles, ranking fit, and generating tailored application materials.
hhursev / recipe-scrapers
Python package for scraping recipes data
probberechts / soccerdata
⛏⚽ Scrape soccer data from Club Elo, ESPN, FBref, Football-Data.co.uk, Sofascore, SoFIFA, Understat and WhoScored.
AWeirdDev / flights
Fast, robust Google Flights scraper (API) for Python. (Probably)
rushter / selectolax
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
KatrielMoses / MailAccess
Free email OSINT tool, 5000+ platforms, identity clustering, breach detection. No API keys required. pip install mailaccess
Altimis / Scweet
Scrape tweets, profiles, followers and following from Twitter/X, no API key needed. Python library with smart multi-account pooling, proxy support and async.
eatmoreduck / boss-zhipin-scraper
Boss直聘爬虫 / BOSS直聘职位数据抓取工具，基于 Chrome CDP 协议复用真实登录态，绕过字体反爬，输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.
Mahanaicoach / google-maps-scraper-kit
Free Google Maps scraper for lead generation. Exports phones, emails, websites and socials to CSV. Runs locally in Docker, no API keys. Claude Code skill included.
emcf / thepipe
Get clean data from tricky documents, powered by vision-language models ⚡
0x676e67 / wreq-python
An ergonomic, privacy-aware Python HTTP Client
egbertbouman / youtube-comment-downloader
Simple script for downloading Youtube comments without using the Youtube API
scrapfly / scrapfly-scrapers
Scalable Python web scraping scripts for +40 popular domains
ranahaani / GNews
A Happy and lightweight Python Package that Provides an API to search for articles on Google News and returns a JSON response.
ythx-101 / x-tweet-fetcher
Fetch X/Twitter tweets, replies, timelines, and articles without login or API keys — field tool for AI agents.
openstates / openstates-scrapers
source for Open States scrapers
scrapinghub / scrapyrt
HTTP API for Scrapy spiders
tducret / amazon-scraper-python
Non-official client to get some info about products sold on Amazon
sudoguy / tiktokpy
Tool for automated TikTok interactions
openzim / zimit
Make a ZIM file from any Web site and surf offline!
opensanctions / opensanctions
An open database of international sanctions data, persons of interest and politically exposed persons
vil / H4X-Tools
A modular, terminal-based toolkit for OSINT, reconnaissance, and scraping - built in Python, runs on Linux and Windows.
alltheplaces / alltheplaces
A set of spiders and scrapers to extract location information from places that post their location on the internet.
flosell / trailscraper
A command-line tool to get valuable information out of AWS CloudTrail
tiliondev / fortress
Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.
martin-majlis / Wikipedia-API
Python wrapper for Wikipedia
fa0311 / TwitterInternalAPIDocument
Twitter Internal API Document
saifalisew1508 / Telegram-Members-Adder
Telegram Members Adding Software/Script Using Termux.
Madi-S / Lead-Generation
Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation.
BullsEye0 / ghost_eye
Ghost Eye Informationgathering Footprinting Scanner and Recon Tool Release. Ghost Eye is an Information Gathering Tool I made in python 3. To run Ghost Eye, it only needs a domain or ip. Ghost Eye can work with any Linux distros if they support Python 3. Author: Jolanda de Koff
gunkonchi / xscrape
Async Python client for collecting public data from X (Twitter) - search, profiles, threads, replies, and media. Built-in account rotation, adaptive rate limiting, and export to JSON/CSV/SQLite.
ksanjeev284 / reddit-universal-scraper
Universal Reddit Scraper - Works on any Subreddit or User
jannisborn / paperscraper
A bibliometrics tool for publication metadata (pubmed, arxiv, medrxiv, biorxiv, chemrxiv) and citation analyses
Xewdy444 / CF-Clearance-Scraper
A simple program for scraping Cloudflare clearance (cf_clearance) cookies from websites issuing Cloudflare challenges to visitors
JXZ666 / slgking
黄油之王：自由检索挑选你喜欢的黄油，机器学习你的xp，方便快捷地管理本地黄油
dengji85 / avm
Scrape. Organize. Play. Analyze. — Your local JAV hub.一款现代化的本地 JAV 管理平台。不只是管理文件，更能通过播放记录、标签频次、演员分布等数据，自动生成你的个人观影报告。集刮削、整理、播放、分析于一体，是你的私人影库，更是你的观影喜好洞察工具。无需安装，下载即用。
Coding-Doctor-Omar / reddrix
Reddrix is an open-source, blazing-fast Reddit Scraper for Python (No Login or API Key)
Conradqh / scrapemychats
Export a ChatGPT account (incl. business/Team) to a local searchable offline archive - conversations, files, and a single-file HTML viewer
justoneapi-labs / douyin-api
抖音API、抖音接口、抖音爬虫示例。
vedantdhande04 / tweetytweets
Autonomous X/Twitter posting pipeline for your AI agent. No API key, no paid scrapers: researches your niche, writes in your voice, publishes through a real browser, and verifies its own posts.
Maas6696 / cloudflare-challenge-clearance
Bypass the Cloudflare challenge (Just a moment / 5s) in Python and get a valid cf_clearance cookie via the Peak API - headless-friendly, proxy-aware.
maximilianfeix / proxy-scraper
Free HTTP/SOCKS proxies with every check on the record: passed, failed or couldn't run, and when. Hourly list, --pick --verify, --diagnose, rotating proxy, Python API, MCP server.