apache / spark
Apache Spark - A unified analytics engine for large-scale data processing
apache / spark
apache / datafusion-comet
Apache DataFusion Comet Spark Accelerator
apache / gluten
Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
awslabs / deequ
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
apache / kyuubi
Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
apache / carbondata
High performance data store solution
NVIDIA / cudf-spark
NVIDIA cuDF for Apache Spark plugin - accelerate Apache Spark with GPUs
LucaCanali / sparkMeasure
This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.
apache / openwhisk
Apache OpenWhisk is an open source serverless cloud platform
microsoft / SynapseML
Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark
spotify / scio
A Scala API for Apache Beam and Google Cloud Dataflow.
apache / pekko
Build highly concurrent, distributed, and resilient message-driven applications using Java/Scala
graphframes / graphframes
GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs
apache / livy
Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.
gearpump / gearpump
Lightweight real-time big data streaming engine over Apache Pekko
apache / incubator-toree
Mirror of Apache Toree (Incubating)
TouK / nussknacker
Low-code tool for automating actions on real time data | Stream processing for the users.
gvolpe / trading
💱 Trading application written in Scala 3 that showcases an Event-Driven Architecture (EDA) and Functional Programming (FP)
yahoo / CMAK
CMAK is a tool for managing Apache Kafka clusters