huggingface / datasets

🤗 Datasets is a lightweight library providing **two** main features:
- **one-line dataloaders for many public datasets**: one-liners to download and pre-process any of the major public datasets (image datasets, audio datasets, text datasets in 467 languages and dialects, 3D medical images, video datasets, agent traces, etc.) provided on the HuggingFace Datasets Hub. With a simple command like `squad_dataset = load_dataset("rajpurkar/squad")`, get any of these datasets ready to use in a dataloader for training/evaluating a ML model (Numpy/Pandas/PyTorch/TensorFlow/JAX/Polars),
- **efficient data pre-processing**: simple, fast and reproducible data pre-processing for the public datasets as well as your own local datasets in CSV, JSON, JSONL, Parquet, HDF5, XML, text, PNG, JPEG, WAV, MP3, PDF, NIfTI, and more. With simple commands like `processed_dataset = dataset.map(process_example)`, efficiently prepare the dataset for inspection and ML model evaluation and training.
🎓 **Documentation** 🔎 **Find a dataset in the Hub** 🌟 **Share a dataset on the Hub**
# 🚀 Key Features
🤗 Datasets is designed to let the community easily add and share new datasets, and provides powerful capabilities for data manipulation:
| Feature | Description |
|---------|-------------|
| 📦 **One-line dataset loading** | Load AI-ready datasets from the Hugging Face Hub or local files with `load_dataset()` |
| 🔍 **Multiple formats** | Native support for CSV, JSON, JSONL, Parquet, Arrow, XML, Text, Webdataset, and more |
| 🖼️ **Multi-modal data** | Built-in support for text, audio, image, video, PDF, and NIfTI (3D medical) data |
| 🚀 **Streaming mode** | Stream datasets without downloading — iterate over data on-the-fly with `streaming=True` (now up to **100x faster** with Xet backend) |
| 💾 **HF Storage Buckets** | Read and write directly from/to Hugging Face Storage Buckets for mutable, large-scale raw data |
| 🧠 **AI Agent Traces** | Load and process AI agent traces (prompts, tool calls, responses) from the Hub |
| ⚡ **Apache Arrow backend** | Zero-copy memory-mapped storage — datasets naturally free you from RAM limitations |
| 🔄 **Smart caching** | Never wait for your data to process twice — cached results are automatically reused |
| 📊 **Multi-framework interoperability** | Native conversion to/from NumPy, Pandas, Polars, Arrow, PyTorch, TensorFlow, JAX, and Spark |
| 🏎️ **Multi-processing** | Fast parallel data processing with `map(num_proc=N)` |
| 🔎 **Search & index** | Built-in FAISS and Elasticsearch index support for similarity search |
| 📦 **JSON type** | Flexible JSON/structured data support with `Json()` feature type |
# Installation
## With pip
🤗 Datasets can be installed from PyPi and should be installed in a virtual environment (venv or conda for instance):
 [code] 
For the latest development version:
 [code] 
## With conda
 [code] 
## Optional dependencies
🤗 Datasets supports various optional features via extras:
 [code] 
For more details on installation, check the