semantica-agi / semantica
Graph-Native Infrastructure for Context and Accountable AI Systems
semantica-agi / semantica
schumannx / full-stack-data-engineering
End-to-end streaming data platform on AWS: Kafka → Iceberg lake → Kimball model → Athena, orchestrated with Airflow 3. Terraform, Glue PySpark, dbt.
apache / superset
Apache Superset is a Data Visualization and Data Exploration Platform
DataTalksClub / data-engineering-zoomcamp
Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼
cocoindex-io / cocoindex
Incremental engine for long horizon agents 🌟 Star if you like it!
kestra-io / kestra
Open-source event-driven orchestration platform for scheduling and running data, AI, and infrastructure workflows defined in YAML.
Zleap-AI / SAG
A new SOTA for RAG — an original retrieval architecture and an open-source knowledge base for humans and agents.
arielshakaramiro / assignment-data-pipeline
A Python/Pandas ETL pipeline that turns a deliberately messy automotive dataset into production-ready data, with 15 pytest tests and GitHub Actions CI.
PrefectHQ / prefect
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
airbytehq / airbyte
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
andkret / Cookbook
The Data Engineering Cookbook
risingwavelabs / risingwave
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
getnao / nao
👾 nao is an open source analytics agent. (1) Create context with nao-core cli, (2) deploy nao chat interface for everyone
argoproj / argo-workflows
Workflow Engine for Kubernetes
xonsh / xonsh
🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.
growthbook / growthbook
Open Source Feature Flags, Experimentation, and Product Analytics
feast-dev / feast
The Open Source Feature Store for AI/ML
ruc-datalab / DeepAnalyze
DeepAnalyze is the first agentic LLM for autonomous data science. 🎈你的AI数据分析师，自动分析大量数据，一键生成专业分析报告！
dathere / qsv
Blazing-fast, Neuro-symbolic Data-Wrangling toolkit
sodadata / soda-core
Data Contracts engine for the modern data stack. https://www.soda.io
Kaelio / ktx
ktx is an executable context layer for data and analytics agents 🐙 Allow Claude Code, Codex, or other AI agents to query analytical databases accurately and with full context of your company
DataWithBaraa / sql-ultimate-course
The most comprehensive SQL guide from a real-world expert! Learn everything from basics to advanced queries, optimizations, and real-world SQL
datacontract / datacontract-cli
Open-source data contract testing for Snowflake, Databricks, BigQuery, Kafka & 15 more. Built on the Open Data Contract Standard (ODCS). MIT.
DataWithBaraa / sql-data-warehouse-project
A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.
zoltanctoth / complete-dbt-bootcamp-zero-to-hero
Supplementary Materials for the The Complete dbt (Data Build Tool) Bootcamp Udemy course
fivetran / great_expectations
Always know what to expect from your data.
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
redpanda-data / connect
Fancy stream processing made operationally mundane
Zipstack / unstract
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
evidence-dev / evidence
Business intelligence as code: build fast, interactive data visualizations in SQL and markdown
cloudquery / cloudquery
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Eventual-Inc / Daft
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
treeverse / lakeFS
lakeFS - Data version control for your data lake | Git for data
rudderlabs / rudder-server
Privacy and Security focused Segment-alternative, in Golang and React
aws / aws-sdk-pandas
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
adilkhash / Data-Engineering-HowTo
A list of useful resources to learn Data Engineering from scratch
Netflix / maestro
Maestro: Netflix’s Workflow Orchestrator
lakehq / sail
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
apache / devlake
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
meltano / meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
apache / hamilton
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
metarank / metarank
A low code Machine Learning personalized ranking service for articles, listings, search results, recommendations that boosts user engagement. A friendly Learn-to-Rank engine
bytewax / bytewax
Python Stream Processing
munich-data-engineering / reflect-cpp
A C++20 library for fast serialization, deserialization and validation using reflection. Supports JSON, Avro, Boost Serialization, BSON, Cap'n Proto, CBOR, Cereal, Command line interfaces, CSV, Environment variables, flexbuffers, msgpack, parquet, TOML, UBJSON, XML, YAML, yas / msgpack.org[C++20]
mlrun / mlrun
MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.
Multiwoven / multiwoven
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
quixio / quix-streams
Python Streaming DataFrames for Kafka
slothflowlabs / duckle
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
pyjanitor-devs / pyjanitor
Clean APIs for data cleaning. Python implementation of R package Janitor
opendatadiscovery / odd-platform
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
quiltdata / quilt
Quilt is a Scientific Data Management Platform on AWS that helps teams and AI find, trust, and reuse data through deeply versioned, context-rich data packages.
estuary / flow
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
odpi / egeria
Egeria core
bacalhau-project / bacalhau
Community-driven, simple, yet powerful framework for fast, cost-effective distributed Compute over Data.
Nike-Inc / koheesio
Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
AltimateAI / altimate-code
Open-source agentic data engineering harness for dbt, SQL, and cloud warehouses. 100+ tools, 10 warehouses, AI-powered.
risesoft-y9 / DataFlow-Engine
数据流引擎是一款面向数据集成、数据同步、数据交换、数据共享、任务配置、任务调度的底层数据驱动引擎。数据流引擎采用管执分离、多流层、插件库等体系应对大规模数据任务、数据高频上报、数据高频采集、异构数据兼容的实际数据问题。
rpbouman / huey
Browser-based data analytics & BI with DuckDB-WASM. Privacy-first: your data stays local; no server required.