apache / spark

Apache Spark - A unified analytics engine for large-scale data processing

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator

apache / gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.

awslabs / deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

apache / carbondata

High performance data store solution

NVIDIA / cudf-spark

NVIDIA cuDF for Apache Spark plugin - accelerate Apache Spark with GPUs

LucaCanali / sparkMeasure

This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.

apache / openwhisk

Apache OpenWhisk is an open source serverless cloud platform

microsoft / SynapseML

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

spotify / scio

A Scala API for Apache Beam and Google Cloud Dataflow.

apache / pekko

Build highly concurrent, distributed, and resilient message-driven applications using Java/Scala

sbt / sbt-native-packager

sbt Native Packager

graphframes / graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

apache / livy

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

gearpump / gearpump

Lightweight real-time big data streaming engine over Apache Pekko

apache / incubator-toree

Mirror of Apache Toree (Incubating)

TouK / nussknacker

Low-code tool for automating actions on real time data | Stream processing for the users.

wvlet / airframe

Essential Building Blocks for Scala

VirtusLab / scala-cli

Scala CLI is a command-line tool to interact with the Scala language. It lets you compile, run, test, and package your Scala code (and more!)

xerial / sbt-pack

A sbt plugin for creating distributable Scala packages.

gvolpe / trading

💱 Trading application written in Scala 3 that showcases an Event-Driven Architecture (EDA) and Functional Programming (FP)

yahoo / CMAK

CMAK is a tool for managing Apache Kafka clusters