seaweedfs / seaweedfs
SeaweedFS is a distributed storage system for object storage (S3), file systems, and Iceberg tables, designed to handle billions of files with O(1) disk access and effortless horizontal scaling.
seaweedfs / seaweedfs
apache / doris
Apache Doris is a real-time analytics and hybrid search database for AI agents.
trinodb / trino
Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
StarRocks / starrocks
The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
risingwavelabs / risingwave
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
apache / iceberg
Apache Iceberg
Eventual-Inc / Daft
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
lakehq / sail
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
timeplus-io / proton
The Fastest Unified Streaming SQL Engine in a Single C++ Binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute real-time context from streams, logs, metrics, events, and CDC.
apache / polaris
Apache Polaris, the interoperable, open source catalog for Apache Iceberg
nisshi-io / nisshi
Apache Kafka® compatible broker with S3, PostgreSQL, SQLite, Apache Iceberg and Delta Lake
Snowflake-Labs / pg_lake
pg_lake: Postgres with Iceberg and data lake access
datazip-inc / olake
OLake - Fastest Databases, Kafka & S3 Replication to Apache Iceberg with Table optimization (Called OLake Fusion). ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Supported sources : Postgres, MongoDB, MySQL, Oracle, MSSql, DB2, Kafka, S3.
projectnessie / nessie
Nessie: Transactional Catalog for Data Lakes with Git-like semantics
lakekeeper / lakekeeper
Apache Iceberg REST Catalog in Rust — access control, credential vending and audit for every engine and AI agent. Apache 2.0.
apache / iceberg-rust
Apache Iceberg
supabase / storage
S3 compatible object storage service that stores metadata in Postgres
apache / incubator-xtable
Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.
apache / amoro
Apache Amoro(incubating) is a Lakehouse management system built on open data lake formats.
apache / iceberg-python
PyIceberg
ClickHouse / ClickBench
ClickBench: a Benchmark For Analytical Databases
jddevtime / open-lakehouse-compute-fabric
Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.
schumannx / full-stack-data-engineering
End-to-end streaming data platform on AWS: Kafka → Iceberg lake → Kimball model → Athena, orchestrated with Airflow 3. Terraform, Glue PySpark, dbt.
parqdb-io / parqdb
Billion-scale embedded vector database built entirely on Parquet and Arrow.
EmbrasureAI / flow
Stream PostgreSQL changes into standard Apache Iceberg tables, built in Rust.
dataglotai / dataglot
Rust-native federated SQL query engine with governance enforced in the query plan. One PostgreSQL endpoint for all your databases.
apache / datafusion-iceberg
Apache DataFusion Iceberg Integration
Unknown-333 / awesome-data-engineering-skills
36 data-engineering skills for AI coding agents — dbt, Airflow, Dagster, Prefect, Spark, Snowflake, BigQuery, Databricks, Kafka, Iceberg. Portable SKILL.md Agent Skills (the format Anthropic launched as Claude Skills), with idempotency, backfill, and data-quality know-how. Runs in Claude Code, Cursor, Codex, Copilot, Gemini CLI and more.