semantica-agi / semantica
Graph-Native Infrastructure for Context and Accountable AI Systems
semantica-agi / semantica
schumannx / full-stack-data-engineering
End-to-end streaming data platform on AWS: Kafka → Iceberg lake → Kimball model → Athena, orchestrated with Airflow 3. Terraform, Glue PySpark, dbt.
apache / superset
Apache Superset is a Data Visualization and Data Exploration Platform
Zleap-AI / SAG
A new SOTA for RAG — an original retrieval architecture and an open-source knowledge base for humans and agents.
arielshakaramiro / assignment-data-pipeline
A Python/Pandas ETL pipeline that turns a deliberately messy automotive dataset into production-ready data, with 15 pytest tests and GitHub Actions CI.
PrefectHQ / prefect
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
airbytehq / airbyte
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
andkret / Cookbook
The Data Engineering Cookbook
xonsh / xonsh
🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.
feast-dev / feast
The Open Source Feature Store for AI/ML
ruc-datalab / DeepAnalyze
DeepAnalyze is the first agentic LLM for autonomous data science. 🎈你的AI数据分析师，自动分析大量数据，一键生成专业分析报告！
sodadata / soda-core
Data Contracts engine for the modern data stack. https://www.soda.io
zoltanctoth / complete-dbt-bootcamp-zero-to-hero
Supplementary Materials for the The Complete dbt (Data Build Tool) Bootcamp Udemy course
fivetran / great_expectations
Always know what to expect from your data.
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Zipstack / unstract
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
aws / aws-sdk-pandas
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
meltano / meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
bytewax / bytewax
Python Stream Processing
mlrun / mlrun
MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.
quixio / quix-streams
Python Streaming DataFrames for Kafka
pyjanitor-devs / pyjanitor
Clean APIs for data cleaning. Python implementation of R package Janitor
Nike-Inc / koheesio
Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
dbt-labs / dbt-mcp
A MCP (Model Context Protocol) server for interacting with dbt.
kimtth / software-architect-mindmap
🧠Mindmap of 🗺️Software Architecture, Software engineering: An Overview of Software Terminologies and Concepts.
Hebbian-Robotics / hflow
SDK for robotics teams to verify the quality of their data used for AI model training.
jddevtime / open-lakehouse-compute-fabric
Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.
ashishsinha1602 / schemagate
Text-to-SQL access control: schema selection that gives the LLM only the tables the caller is allowed to read. Row-level-security-aware table pruning for LangChain, MCP and any SQL agent. Postgres, Oracle, MySQL, SQL Server. 75% fewer prompt tokens.
Snowflake-Labs / data-eng-bench
Data-engineering benchmark for coding agents (DuckDB + Snowflake dbt tasks).
darshilparmar / zomato-ai-data-engineering-end-to-end-project
zomato-ai-data-engineering-end-to-end-project
Unknown-333 / awesome-data-engineering-skills
36 data-engineering skills for AI coding agents — dbt, Airflow, Dagster, Prefect, Spark, Snowflake, BigQuery, Databricks, Kafka, Iceberg. Portable SKILL.md Agent Skills (the format Anthropic launched as Claude Skills), with idempotency, backfill, and data-quality know-how. Runs in Claude Code, Cursor, Codex, Copilot, Gemini CLI and more.
dlt-hub / dlt
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️