rustfs / rustfs

RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.

juicedata / juicefs

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

databendlabs / databend

Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. — rebuilt from scratch. Unified architecture on your S3.

apache / shardingsphere

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

volcano-sh / volcano

A Cloud Native Batch System (Project under CNCF)

fluid-cloudnative / fluid

Fluid, elastic data abstraction and acceleration for BigData/AI applications in cloud. (Project under CNCF)

apache / celeborn

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.

NationalSecurityAgency / datawave

DataWave is an ingest/query framework that leverages Apache Accumulo to provide fast, secure data access.

apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.

apache / avro

Apache Avro is a data serialization system.

apconw / Aix-DB

Aix-DB 基于 LangChain/LangGraph 框架，结合 MCP Skills 多智能体协作架构，实现自然语言到数据洞察的端到端转换。

Mdz-Bigdata / lattice

数据平台

DTStack / chunjun

A data integration framework

dotnet / spark

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

Netflix / genie

Distributed Big Data Orchestration Service

opendatadiscovery / odd-platform

First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.

apache / livy

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

gearpump / gearpump

Lightweight real-time big data streaming engine over Apache Pekko

AbsaOSS / spline

Data Lineage Tracking And Visualization Solution

marmotdata / marmot

The open-source context layer for your AI. Catalog your tables, topics, queues and APIs then expose real metadata to your AI agents.

cas-bigdatalab / piflow

πflow is a big data flow engine with spark support

taosdata / TDengine

High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios

oxnr / awesome-bigdata

A curated list of awesome big data frameworks, ressources and other awesomeness.