pathwaycom / pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
pathwaycom / pathway
rocketride-org / rocketride-server
High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.
cocoindex-io / cocoindex
Incremental engine for long horizon agents 🌟 Star if you like it!
onceupon / Bash-Oneliner
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
johnkerl / miller
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
OpenDCAI / DataFlow
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
TomWright / dasel
Unified querying, transformation, and modification of JSON, TOML, YAML, XML, INI, HCL, KDL and CSV.
datajuicer / data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
NVIDIA / DALI
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
fluvio-community / fluvio
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
unionai-oss / pandera
A light-weight, flexible, and expressive statistical data testing library
numaproj / numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
datachain-ai / datachain
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
bytewax / bytewax
Python Stream Processing
NVIDIA-NeMo / Curator
Scalable data pre processing and curation toolkit for LLMs
lotus-data / lotus
Optimized Agentic and LLM Bulk Processing Over Your Data
siddhi-io / siddhi
Stream Processing and Complex Event Processing Engine
quixio / quix-streams
Python Streaming DataFrames for Kafka
data-prep-kit / data-prep-kit
Open source project for data preparation for GenAI applications
Th0rgal / open_oura
A Rust toolkit for the Oura Ring (Gen 3/4/5): reverse-engineered BLE protocol, event decoders, and reimplemented data-processing algorithms. Sync, store, and analyze your data locally.
TangleML / tangle
Tangle is a web app that allows the users to build and run Machine Learning pipelines without having to set up development environment.
Puchaczov / Musoq
SQL Runtime without any database