Zipstack / unstract
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Zipstack / unstract
datajuicer / data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
snowplow / snowplow
The leader in Customer Data Infrastructure
clearml / clearml
ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
kubernetes-sigs / metrics-server
Scalable and efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines.
pditommaso / awesome-pipeline
A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin
gonglei007 / GameDevMind
最全面的游戏开发技术图谱(Game Development Map)。帮助游戏开发者们在已知问题上节省时间，省出更多的精力投入到更有创造性的工作中去。
cloudquery / cloudquery
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
nteract / papermill
📚 Parameterize, execute, and analyze notebooks
apache / flink-cdc
Flink CDC is a streaming data integration tool
dramaclaw / dramaclaw
A general-purpose AIGC video engine: script to finished film in one pipeline — dramas, ads, product videos, otome games, and more. | 通用 AIGC 视频引擎 —— 从剧本到成片一条流水线，漫剧、广告、电商、乙游皆可
PaddlePaddle / PaddleX
All-in-One Development Tool based on PaddlePaddle
Eventual-Inc / Daft
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
NVIDIA / DALI
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
OpenBMB / UltraRAG
A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines
zenml-io / zenml
ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.
devtron-labs / devtron
The only Kubernetes dashboard you need
fluvio-community / fluvio
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
Marker-Inc-Korea / AutoRAG
Your Personal Librarian Agent. Ask anything on your computer, organize files, remember important info and files.
jitsucom / jitsu
Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
tencentmusic / cube-studio
cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库，AI模型市场，支持国产cpu/gpu/npu 昇腾生态，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano等分布式
diggerhq / digger
Digger is an open source IaC orchestration tool. Digger allows you to run IaC in your existing CI pipeline ⚡️
clidey / whodb
Where data access meets operational intelligence
feyninc / chonkie
🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines
jenkins-x / jx
Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
rudderlabs / rudder-server
Privacy and Security focused Segment-alternative, in Golang and React
dagucloud / dagu
Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler.
GiovanniPasq / agentic-rag-for-dummies
A modular Agentic RAG built with LangGraph — learn Retrieval-Augmented Generation Agents in minutes.
kubeflow / pipelines
Machine Learning Pipelines for Kubeflow
ucbepic / docetl
A system for agentic LLM-powered data processing and ETL
firecow / gitlab-ci-local
Tired of pushing to test your .gitlab-ci.yml?
DaKheera47 / job-ops
job-ops: DevOps principles applied to job hunting. A self-hosted pipeline to track, analyze, and assist your application process
adilkhash / Data-Engineering-HowTo
A list of useful resources to learn Data Engineering from scratch
dora-rs / dora
DORA (Dataflow-Oriented Robotic Architecture) is middleware designed to streamline and simplify the creation of AI-based robotic applications. It offers low latency, composable, and distributed dataflow capabilities. Applications are modeled as directed graphs, also referred to as pipelines.
bruin-data / ingestr
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
seandavi / awesome-single-cell
Community-curated list of software packages and data resources for single-cell, including RNA-seq, ATAC-seq, etc.
Netflix / maestro
Maestro: Netflix’s Workflow Orchestrator
google / deepvariant
DeepVariant is an analysis pipeline that uses a deep neural network to call genetic variants from next-generation DNA sequencing data.
mapillary / OpenSfM
Open source Structure-from-Motion pipeline
Ontos-AI / knowhere
Knowhere extracts, parses, and outputs structured chunks ready for AI Agents and RAG.
polyaxon / polyaxon
AI Infra / AI Orchestration / AI Control Plane
ericosiu / ai-marketing-skills
Open-source AI marketing skills — growth experiments, sales pipeline, content ops, outbound, SEO, and finance automation
microsoft / azure-pipelines-tasks
Tasks for Azure Pipelines
terramate-io / terramate
Open-source Infrastructure as Code (IaC) orchestration platform: GitOps workflows, orchestration, code generation, observability, drift detection, asset management, policies, Slack notifications, and more. Integrates with Terraform, OpenTofu, Terragrunt, Kubernetes, GitHub Actions, GitLab CI/CD, BitBucket Pipelines, and any other CI/CD platform.
grafana / alloy
OpenTelemetry Collector distribution with programmable pipelines
nextflow-io / nextflow
A workflow language for data-driven computational pipelines
lynote-ai / humanize-text
Open-source pipeline for rewriting AI-generated text into natural human prose. Need it in your app? Try the Lynote Humanizer API → lynote.ai/ai-humanizer-api
luigifreda / pyslam
pySLAM is a hybrid Python/C++ Visual SLAM pipeline supporting monocular, stereo, and RGB-D cameras. It provides a broad set of modern local and global feature extractors, multiple loop-closure strategies, a volumetric reconstruction module, integrated depth-prediction models, and semantic segmentation capabilities for enhanced scene understanding.
argilla-io / distilabel
Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.
superlinked / sie
Open-source inference server and production cluster for all the models your agent needs.
huggingface / datatrove
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
apptension / saas-boilerplate
SaaS Boilerplate - Open Source and free SaaS stack that lets you build SaaS products faster in React, Django and AWS. Focus on essential business logic instead of coding repeatable features!
Unity-Technologies / Graphics
Unity Graphics - Including Scriptable Render Pipeline
Flangvik / SharpCollection
Nightly builds of common C# offensive tools, fresh from their respective master branches built and released in a CDI fashion using Azure DevOps release pipelines.
aldegad / sprite-gen
Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.
microsoft / pg_durable
PostgreSQL in-database durable execution
numaproj / numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
snapotter-hq / SnapOtter
Open-source, self-hosted file-processing tool. Convert, compress, OCR, transcribe & run local AI across image, video, audio, PDF & documents, via UI, REST API & pipelines. Your files never leave your network.
argoproj / argo-events
Event-driven Automation Framework for Kubernetes
meltano / meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.