apify / apify-mcp-server

The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.

ScrapeGraphAI / Scrapegraph-ai

Python library that uses LLMs and graph-based pipelines to extract structured data from websites and local documents from a natural-language prompt.

TecharoHQ / anubis

Weighs the soul of incoming HTTP requests to stop AI crawlers

apify / crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify / crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

ai-robots-txt / ai.robots.txt

A list of AI agents and robots to block.

JayBizzle / Crawler-Detect

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

monperrus / crawler-user-agents

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:

omrilotan / isbot

🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string

LambdaHack / LambdaHack

Haskell game engine library for roguelike dungeon crawlers; please offer feedback, e.g., after trying out the sample game with the web frontend at

cyohei9907 / web-crawlers

A procedural spider that walks on the DOM — its feet plant on real words, every touch glitches the page. Plain JS × CSS.

faisalman / ua-parser-js

UAParser.js: The Essential Web Development Tool for User-Agent Detection. Detect Browsers, OS, Devices, Bots, Apps, AI Crawlers, and more. Run in Browser or Node

scrapfly / Antibot-Detector

Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.

ZensInk / awesome-open-source-seo

Open-source SEO tools only — crawlers, SERP, GEO/AI-search, CLI toolkits. No SaaS, no affiliate links.