outcrawlai / scout
Scout: the open-source browser agent, by Outcrawl. Mirrored automatically from Outcrawl's own repository.
outcrawlai / scout
firecrawl / firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
firecrawl / open-lovable
🔥 Clone and recreate any website as a modern React app in seconds
apify / crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
getmaxun / maxun
Turn any website into a structured API. Extract, automate, search and monitor the web.
apify / apify-mcp-server
The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.
KnockOutEZ / wigolo
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
exa-labs / exa-mcp-server
Exa MCP for web search and web crawling!
any4ai / AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
tavily-ai / tavily-mcp
Production ready MCP server with real-time search, extract, map & crawl.
coder-hxl / x-crawl
Flexible Node.js crawler library
quemsah / awesome-claude-plugins
Automated index of Claude Code plugins and adoption metrics across GitHub repositories, powered by a Node.js/TypeScript crawler
JustinBeckwith / linkinator
Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.
webrecorder / browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
omrilotan / isbot
🤖/👨🦰 Detect bots/crawlers/spiders using the user agent string
firecrawl / cli
Firecrawl CLI and agent skills: search, scrape, and interact with the web from your terminal or any coding agent, plus parse, crawl, map, Agent and Monitor. Firecrawl is the context API for AI agents.
crawlseo / crawlseo
Open-source SEO monitoring. GSC + Site Crawler + Core Web Vitals in one dashboard.
iannuttall / seo
SEO for agent-native teams. Your agent gets your real Search Console, Google Analytics, Bing and crawl data through an MCP server and CLI, so it stops guessing.
vakra-dev / reader
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
nuxt-modules / robots
Tame the robots crawling and indexing your Nuxt site.
cv-cat / catbus
一个命令操作小红书、抖音、TikTok、B 站、快手、微博、闲鱼、淘宝、京东、X：统一的命令与 JSON 输出，给人和 AI Agent 用
SouravBeraAkaSpeed / xiryn
Xiryn — the open-source personal agent. Does everything Muse, Grok Bot and Dots do, on your own computer: each agent gets its own Linux desktop in Docker, any model with your keys, an encrypted vault, and control from Telegram.
h100envy / crawl-hive
CrawlHive: open-source memecoin safety swarm. Bundles, snipers, linked operators, bubble map, LP locks and dev history on Solana, BNB, Base and Robinhood Chain, plus a Larpcheck second opinion. Free scans, Telegram bot, Chrome extension.
GoldenLoaf24h / browserpaw
BrowserClaw - Control your everyday Chrome browser from AI agents, without losing logins or focus
tomaszboloz / SEOmi
Native macOS and Windows workspace for technical SEO audits, site crawling, semantic maps, keyword research, rank tracking, DataForSEO, Search Console, and AI visibility.
vincenzo-afk / Intelis-Agent
A self-running research agent that monitors the web on your schedule, verifies findings with LLM analysis, and delivers intelligence reports by email and in-app.
Adityavardhanjain / WikiCrawl
Interactive Wikipedia knowledge graph - crawl topics, visualize connections, and explore unexpected paths.
Aaqibhafeezkhan / OpenNewsGrid
Real-time global news aggregation platform built with Next.js 15, TypeScript, and Tailwind CSS. It aggregates news from 300+ international sources through RSS feeds, NewsAPI, GNews, and other APIs, with smart deduplication to prevent duplicate stories.