lexmount / moli
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
lexmount / moli
spider-rs / spider
Foundational low latency web data collecting in Rust
0xMassi / webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
fastcrw / crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
0x676e67 / wreq
An ergonomic, privacy-aware Rust HTTP Client
janreges / siteone-crawler
SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
vincenzo-afk / NEXUS-ENGINE
A full-featured search engine in Rust. Local filesystem + web crawl, BM25+PageRank ranking, BPE query language, bangs, Tor mode, HTTP/WebSocket API, and a privacy-first frontend.