gunkonchi / xscrape

# xscrape 
> Asynchronous Python client for collecting public data from X (Twitter).
**xscrape** is a lightweight library for asynchronously collecting public data from X (Twitter): posts, profiles, threads, replies, and media. It supports account rotation, a built-in rate limiter, and export to JSON / CSV / SQLite.
---
## Table of Contents
- Features
- Installation
- Quick Start
- How It Works
- Configuration
- CLI
- Examples
- Docker
- Testing
- Project Layout
- Roadmap
- Contributing
- Disclaimer
- License
---
## Features
- 🔍 **Post search** — by keywords, hashtags, and operators (`from:`, `since:`, `until:`)
- 👤 **User profiles** — metadata, followers, activity counters
- 🧵 **Threads and replies** — reconstruction of conversation chains
- 🖼 **Media** — extraction of image and video links
- 🔄 **Account rotation** — session pool with automatic failover
- ⏱ **Rate limiting** — adaptive control of request frequency
- 💾 **Storage backends** — JSON, CSV, SQLite out of the box
- 🧩 **Plugins** — custom handlers and exporters
- 🖥 **CLI** — ready-to-use command line interface
- 🐳 **Docker-ready** — single command deployment
- 📊 **Structured logging** — JSON logs with request tracing
---
## 📦 Installation
### From source
 [code] 
### Requirements
- Python 3.10+
- `aiohttp`, `pydantic`, `tenacity`, `orjson`
### Optional
 [code] 
---
## Quick Start
 [code] 
Run `xscrape --help` for the full command reference.
## How It Works
`xscrape` talks to public GraphQL endpoints of X using session cookies. The pipeline looks like this:
 [code] 
1. **Client** — public interface (`search`, `user`, `thread`, `replies`).
2. **AuthPool** — session pool; picks a free account, handles 429/401.
3. **Fetcher** — low-level HTTP requests with retries and exponential backoff.
4. **Parser** — normalizes raw responses into typed models (`Tweet`, `User`).
5. **RateLimiter** — per-account token bucket plus a global cap.
6. **Storage** — serialization of results into the chosen format.
See docs/ARCHITECTURE.md for details.
---
## Configuration
All options are read from environment variables (see `.env.example`):
| Variable | Description | Default |
|---|---|---|
| `XSCRAPE_COOKIES` | Cookie string (`auth_token`, `ct0`) | — |
| `XSCRAPE_POOL` | Path to JSON with account pool | `None` |
| `XSCRAPE_CONCURRENCY` | Max parallel requests | `4` |
| `XSCRAPE_TIMEOUT` | Request timeout (seconds) | `20` |
| `XSCRAPE_RETRIES` | Number of retries on error | `3` |
| `XSCRAPE_USER_AGENT` | Custom User-Agent | built-in |
| `XSCRAPE_PROXY` | Proxy (`http://user:pass@host:port`) | `None` |
| `XSCRAPE_LOG_LEVEL` | Logging level | `INFO` |
| `XSCRAPE_LOG_FORMAT` | `text` or `json` | `text` |
Full reference: docs/CONFIGURATION.md.
---
## Examples
The `examples/` directory contains ready-to-run scripts:
- `search_tweets.py` — search with pagination and filters
- `user_timeline.py` — collect a user's timeline
- `export_to_csv.py` — dump results to CSV
- `export_to_sqlite.py` — persist results into SQLite