firecrawl / firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
firecrawl / firecrawl
lexmount / moli
A headless browser runtime in Rust that AI agents use to fetch, extract, search, and automate web pages.
unclecode / crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
ScrapeGraphAI / Scrapegraph-ai
Python library that uses LLMs and graph-based pipelines to extract structured data from websites and local documents from a natural-language prompt.
firecrawl / firecrawl-mcp-server
Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
apify / crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
apify / crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
KnockOutEZ / wigolo
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
DedSecInside / TorBot
Dark Web OSINT Tool
apache / nutch
Apache Nutch is an extensible and scalable web crawler
spider-rs / spider
Foundational low latency web data collecting in Rust
0xMassi / webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
MarginaliaSearch / MarginaliaSearch
Internet search engine for text-oriented websites. Indexing the small, old and weird web.
fastcrw / crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
vakra-dev / reader
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
beyondtahir / beyondseo
Your complete SEO house: native crawling, SEO/AEO/GEO, content, competitors and reputation. Includes 206 publishing sources with DR context and tailored backlink plans. No API keys required by the native engine.
gildas-lormeau / single-file-cli
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
JustinBeckwith / linkinator
Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.
webrecorder / browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
omrilotan / isbot
🤖/👨🦰 Detect bots/crawlers/spiders using the user agent string
scrapfly / scrapfly-scrapers
Scalable Python web scraping scripts for +40 popular domains
PhialsBasement / LibreCrawl
Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!
apache / stormcrawler
A scalable, mature and versatile web crawler based on Apache Storm
martin-majlis / Wikipedia-API
Python wrapper for Wikipedia
Madi-S / Lead-Generation
Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation.
RayenMlayeh / jevcrawl
MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.
cyohei9907 / web-crawlers
A procedural spider that walks on the DOM — its feet plant on real words, every touch glitches the page. Plain JS × CSS.
vincenzo-afk / NEXUS-ENGINE
A full-featured search engine in Rust. Local filesystem + web crawl, BM25+PageRank ranking, BPE query language, bangs, Tor mode, HTTP/WebSocket API, and a privacy-first frontend.
makralabs / makra
Makra is a memory layer between the web and your AI agents. It serves real-time structured data from the open web at the cost of a vector search.
vincenzo-afk / Intelis-Agent
A self-running research agent that monitors the web on your schedule, verifies findings with LLM analysis, and delivers intelligence reports by email and in-app.
Adityavardhanjain / WikiCrawl
Interactive Wikipedia knowledge graph - crawl topics, visualize connections, and explore unexpected paths.
Aaqibhafeezkhan / OpenNewsGrid
Real-time global news aggregation platform built with Next.js 15, TypeScript, and Tailwind CSS. It aggregates news from 300+ international sources through RSS feeds, NewsAPI, GNews, and other APIs, with smart deduplication to prevent duplicate stories.