pathwaycom / pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
pathwaycom / pathway
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
airbytehq / airbyte
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
abetlen / llama-cpp-python
Python bindings for llama.cpp
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Zipstack / unstract
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
ucbepic / docetl
A system for agentic LLM-powered data processing and ETL
aws / aws-sdk-pandas
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
SQLMesh / sqlmesh
Scalable and efficient data transformation framework - backwards compatible with dbt.
xingpingcn / enhanced-FaaS-in-China
提升部署在cloudflare、vercel或netlify的网页在中国的访问速度和稳定性 cf优选域名 | cf优选ip | cloudflare | vercel | netlify | 加速 | 国内 | 中国 | 境内 | 大陆
nerevu / riko
A Python stream processing engine modeled after Yahoo! Pipes
Open-Source-Legal / OpenContracts
The open document intelligence platform for builders and hackers - DMS for the agentic world
getlogbook / logbook
A cool logging replacement for Python.
kjtsune / embyToLocalPlayer
etlp - Emby/Jellyfin 调用外部本地播放器，并回传播放记录。适配 Plex。
toluaina / pgsync
Postgres, MySQL, or MariaDB to Elasticsearch/OpenSearch sync
zane-ops / zane-ops
A beautiful and fast self-hosted PaaS for deploying and managing web apps, databases, static websites and more.
petl-developers / petl
Python Extract Transform and Load Tables of Data
VLSIDA / OpenRAM
An open-source static random access memory (SRAM) compiler.
svinota / pyroute2
Python Netlink and PF_ROUTE library — network configuration and monitoring
getludic / ludic
🌳 A type-safe HTML template engine for Python. Build dynamic web pages using Python components with a React-like approach.
ipspace / netlab
Making virtual networking labs suck less
awslabs / aws-glue-libs
AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
catalyst-cooperative / pudl
The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
Tencent / wave-mcp
Open-source, license-free MCP server for RTL waveform debug: reads FST waveforms (VCD/FSDB auto-convert) plus a SystemVerilog netlist, with 34 tools covering driver analysis, value/X tracing, pass-fail waveform diff, a browser wave viewer the agent drives, and pre-simulation static analysis.
ibrews / unreal-custodian
Reclaim the Intermediate, Binaries and cache folders your Unreal Engine projects and engines quietly hoard — without touching anything you authored.
arielshakaramiro / assignment-data-pipeline
A Python/Pandas ETL pipeline that turns a deliberately messy automotive dataset into production-ready data, with 15 pytest tests and GitHub Actions CI.
nathanrs / detllm
Fully deterministic cross-device LLMs inference
zealxz / quietloop
Evidence-driven, privacy-first reflective automation for personal agents.
Unknown-333 / awesome-data-engineering-skills
36 data-engineering skills for AI coding agents — dbt, Airflow, Dagster, Prefect, Spark, Snowflake, BigQuery, Databricks, Kafka, Iceberg. Portable SKILL.md Agent Skills (the format Anthropic launched as Claude Skills), with idempotency, backfill, and data-quality know-how. Runs in Claude Code, Cursor, Codex, Copilot, Gemini CLI and more.