unclecode / crawl4ai

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf

ScrapeGraphAI / Scrapegraph-ai

Python library that uses LLMs and graph-based pipelines to extract structured data from websites and local documents from a natural-language prompt.

apify / crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

DedSecInside / TorBot

Dark Web OSINT Tool

beyondtahir / beyondseo

Your complete SEO house: native crawling, SEO/AEO/GEO, content, competitors and reputation. Includes 206 publishing sources with DR context and tailored backlink plans. No API keys required by the native engine.

scrapfly / scrapfly-scrapers

Scalable Python web scraping scripts for +40 popular domains

PhialsBasement / LibreCrawl

Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!

martin-majlis / Wikipedia-API

Python wrapper for Wikipedia

Madi-S / Lead-Generation

Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation.

RayenMlayeh / jevcrawl

MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.

makralabs / makra

Makra is a memory layer between the web and your AI agents. It serves real-time structured data from the open web at the cost of a vector search.