mlflow / mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

treeverse / lakeFS

lakeFS - Data version control for your data lake | Git for data

microsoft / SynapseML

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

lakehq / sail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

kubeflow / spark-operator

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

dotnet / spark

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

japila-books / apache-spark-internals

The Internals of Apache Spark

graphframes / graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

sparklyr / sparklyr

R interface for Apache Spark

LucaCanali / sparkMeasure

This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.

jddevtime / open-lakehouse-compute-fabric

Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.

EmbrasureAI / spark-observability-skills

Open-source agent skills for debugging and optimizing Apache Spark workloads