langgenius / dify
Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.
langgenius / dify
calesthio / OpenMontage
World's first open-source, agentic video production system. 12 production pipelines, 100+ tools, 700+ agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.
pathwaycom / pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
deepset-ai / haystack
Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.
PrefectHQ / prefect
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
airbytehq / airbyte
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
modelscope / FunASR
Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
spotify / luigi
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
llmware-ai / llmware
Unified framework for building enterprise RAG pipelines with small, specialized models
data-privacy-stack / presidio
An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.
kedro-org / kedro
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
bentoml / BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
OpenDCAI / DataFlow
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
PaddlePaddle / PaddleX
All-in-One Development Tool based on PaddlePaddle
OpenBMB / UltraRAG
A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines
zenml-io / zenml
ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.
feyninc / chonkie
🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines
ucbepic / docetl
A system for agentic LLM-powered data processing and ETL
argilla-io / distilabel
Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.
meltano / meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
pypa / cibuildwheel
🎡 Build Python wheels for all the platforms with minimal configuration.
tensorflow / tfx
TFX is an end-to-end platform for deploying production ML pipelines
doobidoo / mcp-memory-service
Open-source persistent memory for AI agent pipelines (LangGraph, CrewAI, AutoGen) and Claude. REST API + knowledge graph + autonomous consolidation.
elyra-ai / elyra
Elyra extends JupyterLab with an AI centric approach.
NVIDIA-NeMo / Curator
Scalable data pre processing and curation toolkit for LLMs
mlrun / mlrun
MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.
quixio / quix-streams
Python Streaming DataFrames for Kafka
koaning / scikit-lego
Extra blocks for scikit-learn pipelines.
explosion / spacy-llm
🦙 Integrating LLMs into structured NLP pipelines
EmbeddedLLM / JamAIBase
The collaborative spreadsheet for AI. Chain cells into powerful pipelines, experiment with prompts and models, and evaluate LLM responses in real-time. Work together seamlessly to build and iterate on AI applications.
mosecorg / mosec
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
SpikeInterface / spikeinterface
A Python-based module for creating flexible and robust spike sorting pipelines.
Haervwe / open-webui-tools
Open‑WebUI Tools is a modular toolkit designed to extend and enrich your Open WebUI instance, turning it into a powerful AI workstation. With a suite of over 15 specialized tools, function pipelines, and filters, this project supports academic research, agentic autonomy, multimodal creativity, workflows, and more
Nike-Inc / koheesio
Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
JulianKemmerer / PipelineC
PypelineC: A hardware description language (HDL) adding high level synthesis(HLS)-like automatic pipelining as a language construct/compiler feature.
GuanYixuan / pyCapCut
A lightweight, flexible, and easy-to-use Python tool for generating CapCut drafts to build automated video editing/remix pipelines! Another similar project: https://github.com/GuanYixuan/pyJianYingDraft
kubeflow / kale
Kubeflow’s superfood for Data Scientists
Azure / azure-devops-cli-extension
Azure DevOps Extension for Azure CLI
onnx / sklearn-onnx
Convert scikit-learn models and pipelines to ONNX
evelyyyyynnnnn / 4.0-Decision-Intelligence-Framework
A research-oriented framework for building optimization-driven decision intelligence systems, integrating causal inference, risk modeling, and AI pipelines for large-scale socio-technical systems
backblaze-labs / genblaze
Genblaze is an open source Python SDK for orchestrating generative AI media pipelines across video, audio, and image providers with built in provenance for every output.
elementary-data / dbt-data-reliability
This dbt package captures metadata, artifacts, and test results so you can detect anomalies, monitor data quality, and build metadata tables. It powers Elementary OSS and feeds the wider context layer used by Elementary Cloud’s full Data & AI Control Plane.
TangleML / tangle
Tangle is a web app that allows the users to build and run Machine Learning pipelines without having to set up development environment.
WILLOSCAR / research-units-pipeline-skills
Research pipelines as semantic execution units: each skill declares inputs/outputs, acceptance criteria, and guardrails. Evidence-first methodology prevents hollow writing through structured intermediate artifacts.
databrickslabs / dbldatagen
Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines
sharziki / semdecide
Typed semantic decisions for Unix pipelines and CI, powered by TypeSafe AI Jev.
xiaoyeuzixi / revlab
REVLab - Windows PE reverse engineering workflow platform (static analysis, unpacking, disassembly, decompile, dynamic sandbox, network capture, custom pipelines, MCP + AI agent integration). For lab/legal compliance testing only.
AhmadJeddi / Plant-Disease-Classification
A ResNet50-DCBAM deep learning framework for plant disease classification using the PlantVillage dataset with reproducible training and evaluation pipelines.
EvolvingLMMs-Lab / GenAI-Caption-Pipeline
Reconstructing Big Tech T2I/T2V captioning pipelines for production and research
00200200 / repro-lens
Static reproducibility checks, experiment replay, and verification for ML pipelines.
neural-maze / substack-brain-course
A 6-week open-source course on harness engineering: durable pipelines, temporal knowledge graphs, error analysis and MCP.
fraunhoferhhi / RIPEpp
[LIMIT@ECCV 2026] RIPE, minus the negatives, plus a trainable matcher - full sparse matching pipelines learned from positive image pairs alone.