dagucloud / dagu
Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler.
arielshakaramiro / assignment-data-pipeline
A Python/Pandas ETL pipeline that turns a deliberately messy automotive dataset into production-ready data, with 15 pytest tests and GitHub Actions CI.
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
NVIDIA / DALI
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
pathwaycom / pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
airbytehq / airbyte
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
debezium / debezium
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
datajuicer / data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
bruin-data / ingestr
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
apache / shardingsphere
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Unstructured-IO / unstructured
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
OpenDCAI / DataFlow
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
elementary-data / elementary
The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
datazip-inc / olake
OLake - Fastest Databases, Kafka & S3 Replication to Apache Iceberg with Table optimization (Called OLake Fusion). ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Supported sources : Postgres, MongoDB, MySQL, Oracle, MSSql, DB2, Kafka, S3.
apache / dolphinscheduler
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
kedro-org / kedro
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
snowplow / snowplow
The leader in Customer Data Infrastructure
apache / flink-cdc
Flink CDC is a streaming data integration tool
fluvio-community / fluvio
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
clidey / whodb
Where data access meets operational intelligence
rudderlabs / rudder-server
Privacy and Security focused Segment-alternative, in Golang and React
ucbepic / docetl
A system for agentic LLM-powered data processing and ETL
adilkhash / Data-Engineering-HowTo
A list of useful resources to learn Data Engineering from scratch
Netflix / maestro
Maestro: Netflix’s Workflow Orchestrator
superlinked / sie
Open-source inference server and production cluster for all the models your agent needs.
meltano / meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
feldera / feldera
The Feldera Incremental Computation Engine
pydoit / doit
CLI task management & automation tool
bruin-data / bruin
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Multiwoven / multiwoven
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
quixio / quix-streams
Python Streaming DataFrames for Kafka
combust / mleap
MLeap: Deploy ML Pipelines to Production
slothflowlabs / duckle
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
opendatadiscovery / odd-platform
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
estuary / flow
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
ConduitIO / conduit
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
backblaze-labs / genblaze
Genblaze is an open source Python SDK for orchestrating generative AI media pipelines across video, audio, and image providers with built in provenance for every output.
elementary-data / dbt-data-reliability
This dbt package captures metadata, artifacts, and test results so you can detect anomalies, monitor data quality, and build metadata tables. It powers Elementary OSS and feeds the wider context layer used by Elementary Cloud’s full Data & AI Control Plane.
Hebbian-Robotics / hflow
SDK for robotics teams to verify the quality of their data used for AI model training.
parz0val0 / scan-to-practice
Scan-to-Practice: a field-tested AI skill and methodology for turning scanned learning materials into structured practice products.
mrcanlaco / PeakPulse-AI
PeakPulse AI — Real-Time Financial Market Anomaly Detection & Time-Series Machine Learning Engine
MahdiKordian / DataForge
Automated data preparation for machine learning. Load a dataset, clean, transform, encode, scale and split it, explore it with built-in analysis, and export model-ready data with a full report. Built with Python and Streamlit.
Unknown-333 / awesome-data-engineering-skills
36 data-engineering skills for AI coding agents — dbt, Airflow, Dagster, Prefect, Spark, Snowflake, BigQuery, Databricks, Kafka, Iceberg. Portable SKILL.md Agent Skills (the format Anthropic launched as Claude Skills), with idempotency, backfill, and data-quality know-how. Runs in Claude Code, Cursor, Codex, Copilot, Gemini CLI and more.
rocketride-org / rocketride-server
High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.
authorjapps / zerocode
zerocode-tdd is a community-developed, free, open-source, outcome-driven automated testing framework for Data Pipelines, ETL, REST API, Kafka(Data Streams), Databases and Load scenarios. all defined in simple JSON or YAML — with zero coding.