RayenMlayeh / jevcrawl

### The web crawler for AI agents that finds the answer, not just pages.
A small decision model picks every link and every passage, so your agent's LLM reads
**only what answers its question** at **a few cents per crawl**.
Quick start · Benchmarks · Tools ·
How it works · Configuration
## Why jevcrawl
Agents that browse a site have two bad options. They can **read everything** a crawler fetches,
which buries the answer in navigation and boilerplate and burns tokens. Or they can **let the LLM
pick each link**, which is slow and costs dollars per question. jevcrawl does neither:
- 🎯 **Finds the answer page more often.** 54% of tasks vs 33% when Claude picks the links, and
 2x as often as Crawl4AI's goal-directed crawl.
- 💸 **About 8x cheaper per task, 10x per correct answer.** The crawl decisions cost about a cent.
- ⚡ **2x faster than an LLM-driven crawl.** Four pages at a time, one decision request per page.
- 📜 **Verbatim, with sources.** Every passage is copied from the page with its URL, heading path
 and character offsets. The decision model only scores, so it cannot invent text.
- 🛡️ **Safe against prompt injection by design.** Page text is only ever scored, never followed.
- 🔌 **Works everywhere MCP does:** Claude Code, Claude Desktop, Cursor, or the CLI.
## Quick start
You need a Jev API key from console.typesafe.ai and
uv.
**Claude Code**
 [code] 
**Claude Desktop, Cursor and other MCP clients** (`mcpServers` in the client's config file)
 [code] 
Then ask your agent something like *"Use jevcrawl to find which Python version added the walrus
operator, starting from docs.python.org"*. Check the setup with
`uvx jevcrawl doctor`. Agents that load skills
can also use `skills/jevcrawl/SKILL.md`.
**Pages that need JavaScript** (optional). jevcrawl reads plain HTML. To also render pages that
are empty without JavaScript, add the `js` extra and download Chromium once:
 [code] 
(In `mcpServers`, use `"args": ["--from", "jevcrawl[js]", "jevcrawl"]`.) Only pages detected as
JavaScript shells are opened in the browser, at most five per call, and they are listed in
`stats.js_rendered_urls`. Everything else is still fetched as plain HTML.
## Benchmarks
Every system starts on a site's home page with the same 30-page budget and the same question.
What each one collects goes to the **same answering LLM**, and the **same judge** grades the
answer, so the crawler is the only thing that changes.
### jevcrawl vs Crawl4AI
Crawl4AI is one of the most popular open-source crawlers
for LLMs. Its `BestFirstCrawlingStrategy` with a keyword scorer is its way to crawl toward a question
without an LLM, so we compared the two on whether they fetch the page that holds the answer.
### Where the numbers come from
| | |
|---|---|
| **Tasks** | WebWalkerQA from *WebWalker: Benchmarking LLMs in Web Traversal* (Wu et al., Alibaba Tongyi Lab, ACL 2025): questions whose answers sit several clicks deep in real websites (conferences, universities, organizations, games), in English