schumannx / full-stack-data-engineering

End-to-end streaming data platform on AWS: Kafka → Iceberg lake → Kimball model → Athena, orchestrated with Airflow 3. Terraform, Glue PySpark, dbt.

ibis-project / ibis

the portable Python dataframe library

microsoft / SynapseML

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

JohnSnowLabs / spark-nlp

State of the Art Natural Language Processing

lakehq / sail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

narwhals-dev / narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

palantir / pyspark-style-guide

This is a guide to PySpark code style presenting common situations and the associated best practices based on the most frequent recurring topics across the PySpark repos we've encountered.

graphframes / graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

Nike-Inc / koheesio

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

MrPowers / chispa

PySpark test helper methods with beautiful error messages

capitalone / datacompy

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

eakmanrq / sqlframe

Turning PySpark Into a Universal DataFrame API

databrickslabs / dbldatagen

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

zorost / AI-Engineering-Lab

A free, self-paced 24-week AI engineering course: Python, machine learning, LLMs, RAG, fine-tuning, agents and MCP, Azure and Vertex and Bedrock, and Databricks. 43 runnable notebooks, one continuous case study. MIT licensed, no signup. By Zorost Intelligence AI Lab.

DataFabric-Academy / ms-fabric-data-science

Hands-on Microsoft Fabric Data Science labs (FreshMart churn) with Thai study notes, PySpark/MLflow, and DP-604-aligned learning path.

Jiarui-0410 / real-time-recommendation-analytics

End-to-end retail analytics and recommendation platform using PySpark, Kafka, PostgreSQL, Power BI, PyTorch Two-Tower, Qdrant, FastAPI, Docker and Azure.

maurinruneacco / luau-diagrams-and-defects

Production Data Engineering Cloud Architecture Workflow Automation Portfolio 2026

dmux / minilake

Minilake is a free, local Databricks API emulator — a single-developer tool for testing databricks-sdk/Terraform code against real SQL, real Delta Lake, and real Job execution, without paying for cloud compute.