apify / crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

getmaxun / maxun

Turn any website into a structured API. Extract, automate, search and monitor the web.

exa-labs / exa-mcp-server

Exa MCP for web search and web crawling!

webrecorder / browsertrix-crawler

Run a high-fidelity browser-based web archiving crawler in a single Docker container

nuxt-modules / robots

Tame the robots crawling and indexing your Nuxt site.

tomaszboloz / SEOmi

Native macOS and Windows workspace for technical SEO audits, site crawling, semantic maps, keyword research, rank tracking, DataForSEO, Search Console, and AI visibility.