firecrawl / cli
Firecrawl CLI and agent skills: search, scrape, and interact with the web from your terminal or any coding agent, plus parse, crawl, map, Agent and Monitor. Firecrawl is the context API for AI agents.
firecrawl / cli
toki-plus / video-mover
全自动短视频搬运工具，支持自动下载、去重、AI生成标题+标签、上传，可二开扩展至多平台，例如：TikTok->视频号/抖音/小红书、抖音->TikTok/视频号/小红书......video-processing, automation, tiktok, selenium, pyqt5, ffmpeg, bot, data-scraping, video-deduplication.
osintshifu / awesome-osint-repos
Continuously updated catalogue of open-source OSINT tools, skills, plugins, MCP servers, and agentic integrations.
XD-MHLOO / facebook-gql-scraper
Scrape Facebook GraphQL data without an API key — profiles, posts, comments, reactions, groups, marketplace, search_engine.
gocolly / colly
Elegant Scraper and Crawler Framework for Golang
Evil0ctal / Douyin_TikTok_Download_API
🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
ultrafunkamsterdam / nodriver
Successor of Undetected-Chromedriver. Providing a blazing fast framework for web automation, webscraping, bots and any other creative ideas which are normally hindered by annoying anti bot systems like Captcha / CloudFlare / Imperva / hCaptcha
spider-rs / spider
Foundational low latency web data collecting in Rust
0xMassi / webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
hhursev / recipe-scrapers
Python package for scraping recipes data
TheGP / untidetect-tools
List of anti-detect and humanizing tools and browsers, including captcha solvers and sms-activation.
Altimis / Scweet
Scrape tweets, profiles, followers and following from Twitter/X, no API key needed. Python library with smart multi-account pooling, proxy support and async.
karust / openserp
Self-hosted SERP API for AI, SEO & automation. Browser-rendered Google, Bing, Yandex, Baidu, DuckDuckGo and Ecosia search with page extraction 🎉
tinyfish-io / agentql
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.
cdpdriver / zendriver
A blazing fast, async-first, undetectable webscraping/web automation framework based on ultrafunkamsterdam/nodriver. Now with Docker support!
rebrowser / rebrowser-patches
Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.
sardanioss / httpcloak
Go HTTP client with browser-identical TLS/HTTP2 fingerprinting. Bypass bot detection by perfectly mimicking Chrome, Firefox, and Safari at the cryptographic level (JA3/JA4, Akamai fingerprint, header order). Supports HTTP/1.1, HTTP/2, HTTP/3, sessions, cookies, and proxies.
gooseworks-ai / goose-skills
Library of Growth & GTM skills + data APIs for Claude Code, Codex, Cursor to run ads, social, content, lead gen, seo and data scraping
plabayo / rama
modular service framework to move and transform network packets
neon-mmd / websurfx
:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine
fastcrw / crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
soxoj / socid-extractor
⛏️ The extraction engine behind Maigret: turn any profile URL into a structured OSINT record across 150+ sites
feder-cr / invisible_playwright_mcp
Playwright MCP server undetected by anti-bots and captchas: AI agent browses the web on anti-detect stealth Firefox, Python, undetected browser automation, scraping, computer use.
alpkeskin / rota
A high-performance proxy rotation engine with automated IP management and real-time health monitoring
Yuvi9587 / Kemono-Downloader
Kemono pawchive Downloader is a fast, PyQt5 app for archiving content from a wide array of sites, including Kemono, Coomer, Bunkr, Erome, Saint2.su, nhentai, and Discord. It features a Creator Browser, update checker, and supports multi-threaded, multi-part downloads. Sessions can pause, resume, or recover. filters
AnonCatalyst / Ominis-OSINT
This Python application is an OSINT (Open Source Intelligence) tool called "Ominis OSINT - Web Hunter." It performs online information gathering by querying Google for search results related to a user-inputted query. The tool extracts relevant information such as titles, URLs, and potential mentions of the query in the results.
vakra-dev / reader
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
programminghistorian / jekyll
Jekyll-based static site for The Programming Historian
chienbm98 / foxprofile
Free, open-source anti-detect browser profile manager: per-profile fingerprint, cookies and proxy, Camoufox and Chrome engines, desktop app, REST API and MCP server for AI agents. Self-hosted alternative to GoLogin / MoreLogin.
cporter202 / job-data-apis-and-scrapers
A curated directory of job data APIs and scrapers for listings, hiring signals, salaries, and recruiting.
jhy / jsoup
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.
MontFerret / ferret
Declarative data automation language and Go runtime for structured extraction workflows.
IonicaBizau / scrape-it
🔮 A Node.js scraper for humans.
snooppr / snoop
Snoop — инструмент разведки на основе открытых данных (OSINT world)
jstrieb / github-stats
Better GitHub statistics images for your profile, with stats from private repos too
any4ai / AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
php-curl-class / php-curl-class
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
itsOwen / CyberScraper-2077
A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama
symfony / panther
A browser testing and web crawling library for PHP and Symfony
x4nth055 / pythoncode-tutorials
The Python Code Tutorials
scrapinghub / dateparser
python parser for human readable dates
brightdata / brightdata-mcp
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
botswin / BotBrowser
Advanced Privacy Browser Core with Unified Fingerprint Defense: Cloudflare, Akamai, Kasada, Shape, DataDome, PerimeterX, hCaptcha, FunCaptcha, Imperva, reCAPTCHA, ThreatMetrix, Adscore
apify / fingerprint-suite
Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.
saifyxpro / HeadlessX
The undetected self-hosted browser automation platform. Powered by Camoufox (Firefox) for 0% detection rates. Built for speed, privacy, and scalability.
Tanu-N-Prabhu / Python
This repository helps you learn Python and Machine Learning from scratch.
anaskhan96 / soup
Web Scraper in Go, similar to BeautifulSoup
php-embed / Embed
Get info from any web service or page
iiab / iiab
Internet-in-a-Box - Build your own LIBRARY OF ALEXANDRIA with a Raspberry Pi !
pmh1314520 / WebRPA
A powerful no-code automation tool. Build workflows via drag-and-drop modules for web scraping, form filling and automated testing. | 一款功能强大的自动化工具，通过拖拽模块的方式快速构建自动化工作流，无需编写任何代码即可实现网页数据采集、表单填写、自动化测试等任务。【QQ交流群：115069513】
microlinkhq / browserless
The headless Chrome/Chromium driver on top of Puppeteer. Take screenshots, generate PDFs, extract text and HTML with a production-ready API.
howie6879 / ruia
Async Python 3.6+ web scraping micro-framework based on asyncio
rushter / selectolax
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
gildas-lormeau / single-file-cli
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
orangecoding / fredy
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)
emcf / thepipe
Get clean data from tricky documents, powered by vision-language models ⚡
tidyverse / rvest
Simple web scraping for R
requests-cache / requests-cache
Persistent HTTP cache for python requests
0x676e67 / wreq-python
An ergonomic, privacy-aware Python HTTP Client
okfn-brasil / querido-diario
📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.