pathwaycom / pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

rocketride-org / rocketride-server

High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.

OpenDCAI / DataFlow

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

datajuicer / data-juicer

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

unionai-oss / pandera

A light-weight, flexible, and expressive statistical data testing library

datachain-ai / datachain

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

bytewax / bytewax

Python Stream Processing

NVIDIA-NeMo / Curator

Scalable data pre processing and curation toolkit for LLMs

lotus-data / lotus

Optimized Agentic and LLM Bulk Processing Over Your Data

quixio / quix-streams

Python Streaming DataFrames for Kafka

TangleML / tangle

Tangle is a web app that allows the users to build and run Machine Learning pipelines without having to set up development environment.