apache / spark

Apache Spark - A unified analytics engine for large-scale data processing

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator

apache / gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.

awslabs / deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

apache / carbondata

High performance data store solution

microsoft / SynapseML

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

JohnSnowLabs / spark-nlp

State of the Art Natural Language Processing

zio / zio-quill

Compile-time Language Integrated Queries for Scala

almond-sh / almond

A Scala kernel for Jupyter

combust / mleap

MLeap: Deploy ML Pipelines to Production

graphframes / graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

NVIDIA / cudf-spark

NVIDIA cuDF for Apache Spark plugin - accelerate Apache Spark with GPUs

apache / livy

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

delta-io / delta-sharing

An open protocol for secure data sharing

RedisLabs / spark-redis

A connector for Spark that allows reading and writing to/from Redis cluster

typelevel / frameless

Expressive types for Spark.

LucaCanali / sparkMeasure

This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.

mrpowers-io / spark-daria

Essential Spark extensions and helper methods ✨😲

AbsaOSS / spline

Data Lineage Tracking And Visualization Solution

nightscape / spark-excel

A Spark plugin for reading and writing Excel files