langgenius / dify
Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.
langgenius / dify
calesthio / OpenMontage
World's first open-source, agentic video production system. 12 production pipelines, 100+ tools, 700+ agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.
pathwaycom / pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
alibaba / open-code-review
Secure, fast, efficient, battle-tested at Alibaba's scale. Hybrid architecture code review tool: deterministic pipelines + LLM Agent, precise line-level comments, built-in multi-language ruleset (NPE, thread-safety, XSS, SQL injection), OpenAI & Anthropic compatible.
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
DataTalksClub / data-engineering-zoomcamp
Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼
harness / harness
Harness Open Source is an end-to-end developer platform with Source Control Management, CI/CD Pipelines, Hosted Developer Environments, and Artifact Registries.
deepset-ai / haystack
Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.
jenkinsci / jenkins
Jenkins automation server
PrefectHQ / prefect
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
vectordotdev / vector
A high-performance observability data pipeline.
openobserve / openobserve
Open source observability platform for logs, metrics, traces, RUM (web, android, ios), Session replay, pipelines, SLO and LLM observability. A sophisticated, simple and highly performant alternative to Datadog, Splunk, and Elasticsearch with 140x lower storage costs and single binary deployment.
airbytehq / airbyte
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
modelscope / FunASR
Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
spotify / luigi
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
argoproj / argo-workflows
Workflow Engine for Kubernetes
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
Unstructured-IO / unstructured
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
llmware-ai / llmware
Unified framework for building enterprise RAG pipelines with small, specialized models
apache / dolphinscheduler
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
kubescape / kubescape
Kubescape is an open-source Kubernetes security platform for your IDE, CI/CD pipelines, and clusters. It includes risk analysis, security, compliance, and misconfiguration scanning, saving Kubernetes users and administrators precious time, effort, and resources.
data-privacy-stack / presidio
An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.
openspug / spug
Spug is a lightweight agent-free automatic operation and maintenance platform designed for small and medium-sized enterprises. It integrates host management, host batch execution, host online terminal, file management, application release and deployment, pipelines, online task planning, configuration center, monitoring, alarm and so on.
kedro-org / kedro
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
bentoml / BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
OpenDCAI / DataFlow
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
concourse / concourse
Concourse is a container-based automation system written in Go. It's mostly used for CI/CD.
flyteorg / flyte
Type-safe, distributed orchestration of agents, ML pipelines, and real-time inference on your k8s — in pure Python with async/await, also other languages (rust, go and ts)
kubernetes-sigs / metrics-server
Scalable and efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines.
cloudquery / cloudquery
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
PaddlePaddle / PaddleX
All-in-One Development Tool based on PaddlePaddle
OpenBMB / UltraRAG
A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines
zenml-io / zenml
ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.
fluvio-community / fluvio
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
feyninc / chonkie
🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines
jenkins-x / jx
Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
kubeflow / pipelines
Machine Learning Pipelines for Kubeflow
ucbepic / docetl
A system for agentic LLM-powered data processing and ETL
dora-rs / dora
DORA (Dataflow-Oriented Robotic Architecture) is middleware designed to streamline and simplify the creation of AI-based robotic applications. It offers low latency, composable, and distributed dataflow capabilities. Applications are modeled as directed graphs, also referred to as pipelines.
Netflix / maestro
Maestro: Netflix’s Workflow Orchestrator
polyaxon / polyaxon
AI Infra / AI Orchestration / AI Control Plane
microsoft / azure-pipelines-tasks
Tasks for Azure Pipelines
terramate-io / terramate
Open-source Infrastructure as Code (IaC) orchestration platform: GitOps workflows, orchestration, code generation, observability, drift detection, asset management, policies, Slack notifications, and more. Integrates with Terraform, OpenTofu, Terragrunt, Kubernetes, GitHub Actions, GitLab CI/CD, BitBucket Pipelines, and any other CI/CD platform.
grafana / alloy
OpenTelemetry Collector distribution with programmable pipelines
nextflow-io / nextflow
A workflow language for data-driven computational pipelines
argilla-io / distilabel
Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.
Flangvik / SharpCollection
Nightly builds of common C# offensive tools, fresh from their respective master branches built and released in a CDI fashion using Azure DevOps release pipelines.
microsoft / pg_durable
PostgreSQL in-database durable execution
snapotter-hq / SnapOtter
Open-source, self-hosted file-processing tool. Convert, compress, OCR, transcribe & run local AI across image, video, audio, PDF & documents, via UI, REST API & pipelines. Your files never leave your network.
argoproj / argo-events
Event-driven Automation Framework for Kubernetes
meltano / meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
oxsecurity / megalinter
🦙 MegaLinter analyzes 50 languages, 22 formats, 21 tooling formats, excessive copy-pastes, spelling mistakes and security issues in your repository sources with a GitHub Action, other CI tools or locally.
ArnasDon / wacrm
Self-hostable CRM template for WhatsApp — shared inbox, contacts, sales pipelines, broadcasts, and no-code automations. Fork it, brand it, host it.
elementary-data / elementary
The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
pypa / cibuildwheel
🎡 Build Python wheels for all the platforms with minimal configuration.
tensorflow / tfx
TFX is an end-to-end platform for deploying production ML pipelines
feldera / feldera
The Feldera Incremental Computation Engine
zereight / gitlab-mcp
MCP server for GitLab: 100+ tools for projects, MRs, issues, pipelines, wikis and more via the GitLab REST/GraphQL API
doobidoo / mcp-memory-service
Open-source persistent memory for AI agent pipelines (LangGraph, CrewAI, AutoGen) and Claude. REST API + knowledge graph + autonomous consolidation.