firecrawl / firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
firecrawl / firecrawl
apify / crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
getmaxun / maxun
Turn any website into a structured API. Extract, automate, search and monitor the web.
apify / apify-mcp-server
The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.
KnockOutEZ / wigolo
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
any4ai / AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
coder-hxl / x-crawl
Flexible Node.js crawler library
quemsah / awesome-claude-plugins
Automated index of Claude Code plugins and adoption metrics across GitHub repositories, powered by a Node.js/TypeScript crawler
JustinBeckwith / linkinator
Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.
webrecorder / browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
omrilotan / isbot
🤖/👨🦰 Detect bots/crawlers/spiders using the user agent string
crawlseo / crawlseo
Open-source SEO monitoring. GSC + Site Crawler + Core Web Vitals in one dashboard.
vakra-dev / reader
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
cv-cat / catbus
一个命令操作小红书、抖音、TikTok、B 站、快手、微博、闲鱼、淘宝、京东、X：统一的命令与 JSON 输出，给人和 AI Agent 用
h100envy / crawl-hive
CrawlHive: open-source memecoin safety swarm. Bundles, snipers, linked operators, bubble map, LP locks and dev history on Solana, BNB, Base and Robinhood Chain, plus a Larpcheck second opinion. Free scans, Telegram bot, Chrome extension.
GoldenLoaf24h / browserpaw
BrowserClaw - Control your everyday Chrome browser from AI agents, without losing logins or focus
tomaszboloz / SEOmi
Native macOS and Windows workspace for technical SEO audits, site crawling, semantic maps, keyword research, rank tracking, DataForSEO, Search Console, and AI visibility.
vincenzo-afk / Intelis-Agent
A self-running research agent that monitors the web on your schedule, verifies findings with LLM analysis, and delivers intelligence reports by email and in-app.
Adityavardhanjain / WikiCrawl
Interactive Wikipedia knowledge graph - crawl topics, visualize connections, and explore unexpected paths.
Aaqibhafeezkhan / OpenNewsGrid
Real-time global news aggregation platform built with Next.js 15, TypeScript, and Tailwind CSS. It aggregates news from 300+ international sources through RSS feeds, NewsAPI, GNews, and other APIs, with smart deduplication to prevent duplicate stories.