browser-use / browser-use
Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.
browser-use / browser-use
D4Vinci / Scrapling
🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
unclecode / crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
scrapy / scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
soxoj / maigret
🕵️♂️ Collect a dossier on a person by username from 6K websites
dgtlmoon / changedetection.io
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!
CloakHQ / CloakBrowser
Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.
ScrapeGraphAI / Scrapegraph-ai
Python scraper based on AI
assafelovic / gpt-researcher
An autonomous agent that conducts deep research on any data using any LLM providers
Evil0ctal / Douyin_TikTok_Download_API
🚀 自托管的抖音、TikTok 数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
MODSetter / SurfSense
Air-gapped, privacy-focused, open-source NotebookLM alternative. AI agent that researches, transforms, and edits your documents. Join our Discord: https://discord.gg/ejRNvftDp9
yusufkaraaslan / Skill_Seekers
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
seleniumbase / SeleniumBase
📊 Python APIs for Web Automation, E2E Testing, and Data Scraping. CDP Mode bypasses bot-detection and handles CAPTCHAs. Includes a Stealth mode for Playwright.
daijro / camoufox
🦊 Anti-detect browser
apify / crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
adbar / trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
lexiforest / curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
ultrafunkamsterdam / nodriver
Successor of Undetected-Chromedriver. Providing a blazing fast framework for web automation, webscraping, bots and any other creative ideas which are normally hindered by annoying anti bot systems like Captcha / CloudFlare / Imperva / hCaptcha
snooppr / snoop
Snoop — инструмент разведки на основе открытых данных (OSINT world)
itsOwen / CyberScraper-2077
A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama
vibheksoni / stealth-browser-mcp
Stealth browser automation for AI agents over MCP and the Chrome DevTools Protocol: navigation, network hooks, DOM extraction, and pixel-accurate UI cloning.
vladkens / twscrape
Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.
scrapinghub / dateparser
python parser for human readable dates
feder-cr / invisible_playwright
Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
simonw / shot-scraper
A CLI utility for taking screenshots of websites, recording video demos and scraping sites using JavaScript
3441293738 / creatorhub
多平台内容监控·采集·搬运 一个 Web 面板管起抖音 / 小红书 / 快手
vasu-devs / JustHireMe
Local-first AI job intelligence workbench for scraping roles, ranking fit, and generating tailored application materials.
hhursev / recipe-scrapers
Python package for scraping recipes data
pmh1314520 / WebRPA
A powerful no-code automation tool. Build workflows via drag-and-drop modules for web scraping, form filling and automated testing. | 一款功能强大的自动化工具，通过拖拽模块的方式快速构建自动化工作流，无需编写任何代码即可实现网页数据采集、表单填写、自动化测试等任务。【QQ交流群：115069513】
howie6879 / ruia
Async Python 3.6+ web scraping micro-framework based on asyncio
rushter / selectolax
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
Altimis / Scweet
Scrape tweets, profiles, followers and following from Twitter/X, no API key needed. Python library with smart multi-account pooling, proxy support and async.
Mahanaicoach / google-maps-scraper-kit
Free Google Maps scraper for lead generation. Exports phones, emails, websites and socials to CSV. Runs locally in Docker, no API keys. Claude Code skill included.
Kaliiiiiiiiii-Vinyzu / patchright-python
Undetected Python version of the Playwright testing and automation library.
emcf / thepipe
Get clean data from tricky documents, powered by vision-language models ⚡
requests-cache / requests-cache
Persistent HTTP cache for python requests
0x676e67 / wreq-python
An ergonomic, privacy-aware Python HTTP Client
tinyfish-io / agentql
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.
kalil0321 / reverse-api-engineer
The agent that turns websites into APIs!
cdpdriver / zendriver
A blazing fast, async-first, undetectable webscraping/web automation framework based on ultrafunkamsterdam/nodriver. Now with Docker support!
okfn-brasil / querido-diario
📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.
scrapy / parsel
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
feder-cr / invisible_playwright_mcp
Playwright MCP server undetected by anti-bots and captchas: AI agent browses the web on anti-detect stealth Firefox, Python, undetected browser automation, scraping, computer use.
johnwmillr / LyricsGenius
Download song lyrics and metadata from Genius.com 🎶🎤
soxoj / socid-extractor
⛏️ The extraction engine behind Maigret: turn any profile URL into a structured OSINT record across 150+ sites
scrapfly / scrapfly-scrapers
Scalable Python web scraping scripts for +40 popular domains
Anil-matcha / awesome-agent-apis
660+ muapi-hosted generative-media models plus community-submitted third-party API tools (SEO, enrichment, social, scraping) — one YAML file per entry, browsable by capability.
ail-project / ail-framework
AIL framework - Analysis Information Leak framework
scrapinghub / extruct
Extract embedded metadata from HTML markup
runesleo / x-reader
Universal content reader MCP Server for 10+ platforms
BullsEye0 / dorks-eye
Dorks Eye Google Hacking Dork Scraping and Searching Script. Dorks Eye is a script I made in python 3. With this tool, you can easily find Google Dorks. Dork Eye collects potentially vulnerable web pages and applications on the Internet or other awesome info that is picked up by Google's search bots. Author: Jolanda de Koff
scrapinghub / scrapyrt
HTTP API for Scrapy spiders
ttlns / Selenium-Driverless
a stealthy browser automation framework
openzim / zimit
Make a ZIM file from any Web site and surf offline!
vil / H4X-Tools
A modular, terminal-based toolkit for OSINT, reconnaissance, and scraping - built in Python, runs on Linux and Windows.
feder-cr / invisible_selenium
Selenium that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
Lookyloo / lookyloo
Lookyloo is a web interface that allows users to capture a website page and then display a tree of domains that call each other.
tiliondev / fortress
Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.
carlmontanari / scrapli
Fast, flexible, sync/async, Python 3.10+ screen scraping client specifically for network devices
ScriptSmith / socialreaper
Social media scraping / data collection library for Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs