Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler.
Open-source developer platform to power your entire infra and turn scripts into webhooks, workflows and UIs. Fastest workflow engine (13x vs Airflow). Open-source alternative to Retool and Temporal.
End-to-end streaming data platform on AWS: Kafka β Iceberg lake β Kimball model β Athena, orchestrated with Airflow 3. Terraform, Glue PySpark, dbt.
Workflow Engine for Kubernetes
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
Run your dbt (Core, Fusion, OSS) projects as Apache Airflow DAGs and Task Groups with a few lines of code
Curated list of resources about Apache Airflow
Construct Apache Airflow DAGs Declaratively via YAML configuration files
Elyra extends JupyterLab with an AI centric approach.
Open-source, extensible platform for your data stack. Connect tools, install plugins, and build custom capabilities.
36 data-engineering skills for AI coding agents β dbt, Airflow, Dagster, Prefect, Spark, Snowflake, BigQuery, Databricks, Kafka, Iceberg. Portable SKILL.md Agent Skills (the format Anthropic launched as Claude Skills), with idempotency, backfill, and data-quality know-how. Runs in Claude Code, Cursor, Codex, Copilot, Gemini CLI and more.
SDK for robotics teams to verify the quality of their data used for AI model training.