dharmendradevops11 / the-self-healing-pipeline
Companion code for the book The Self-Healing Pipeline: How I Taught My CI/CD Pipeline to Fix Itself at 2 AM
dharmendradevops11 / the-self-healing-pipeline
langgenius / dify
Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.
calesthio / OpenMontage
World's first open-source, agentic video production system. 12 production pipelines, 100+ tools, 700+ agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.
pathwaycom / pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Imbad0202 / academic-research-skills
Academic Research Skills for Claude Code: research → write → review → revise → finalize
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
deepspeedai / DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
hpcaitech / ColossalAI
Making large AI models cheaper, faster and more accessible
deepset-ai / haystack
Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.
PrefectHQ / prefect
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
marimo-team / marimo
A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.
airbytehq / airbyte
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
modelscope / FunASR
Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
spotify / luigi
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
rocketride-org / rocketride-server
High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
llmware-ai / llmware
Unified framework for building enterprise RAG pipelines with small, specialized models
Imbad0202 / academic-research-skills-codex
Codex-native Academic Research Skills suite for human-in-the-loop academic research workflows
fivetran / great_expectations
Always know what to expect from your data.
data-privacy-stack / presidio
An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.
kedro-org / kedro
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
bentoml / BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
google-deepmind / alphafold3
AlphaFold 3 inference pipeline.
OpenDCAI / DataFlow
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
Emily2040 / seedance-2.0
Comprehensive production pipeline for quad-modal AI filmmaking with Seedance 2.0
Zipstack / unstract
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
datajuicer / data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
clearml / clearml
ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
gonglei007 / GameDevMind
最全面的游戏开发技术图谱(Game Development Map)。帮助游戏开发者们在已知问题上节省时间，省出更多的精力投入到更有创造性的工作中去。
nteract / papermill
📚 Parameterize, execute, and analyze notebooks
dramaclaw / dramaclaw
A general-purpose AIGC video engine: script to finished film in one pipeline — dramas, ads, product videos, otome games, and more. | 通用 AIGC 视频引擎 —— 从剧本到成片一条流水线，漫剧、广告、电商、乙游皆可
PaddlePaddle / PaddleX
All-in-One Development Tool based on PaddlePaddle
OpenBMB / UltraRAG
A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines
zenml-io / zenml
ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.
feyninc / chonkie
🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines
ucbepic / docetl
A system for agentic LLM-powered data processing and ETL
google / deepvariant
DeepVariant is an analysis pipeline that uses a deep neural network to call genetic variants from next-generation DNA sequencing data.
mapillary / OpenSfM
Open source Structure-from-Motion pipeline
Ontos-AI / knowhere
Knowhere extracts, parses, and outputs structured chunks ready for AI Agents and RAG.
ericosiu / ai-marketing-skills
Open-source AI marketing skills — growth experiments, sales pipeline, content ops, outbound, SEO, and finance automation
lynote-ai / humanize-text
Open-source pipeline for rewriting AI-generated text into natural human prose. Need it in your app? Try the Lynote Humanizer API → lynote.ai/ai-humanizer-api
luigifreda / pyslam
pySLAM is a hybrid Python/C++ Visual SLAM pipeline supporting monocular, stereo, and RGB-D cameras. It provides a broad set of modern local and global feature extractors, multiple loop-closure strategies, a volumetric reconstruction module, integrated depth-prediction models, and semantic segmentation capabilities for enhanced scene understanding.
argilla-io / distilabel
Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.
superlinked / sie
Open-source inference server and production cluster for all the models your agent needs.
huggingface / datatrove
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
aldegad / sprite-gen
Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.
meltano / meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
TianyuCodings / NanoJev
A nano replica of Jev: parallel decisions, dynamic candidates, and an end-to-end training pipeline.
data-infra / cube-studio
cubestudio开源云原生一站式机器学习/深度学习/大模型AI平台/MaaS/mlops/人工智能平台/训推平台，算法全链路流程，多租户，算力租赁平台，token中转，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务，VGPU虚拟化，云边端协同，边缘计算，自动化标注平台，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库llmops智能体，AI模型市场，支持国产异构算力调度,昇腾/寒武纪/海光/摩尔/沐曦等，支持ib/roce/RDMA，信创支持
sodadata / soda-core
Data Contracts engine for the modern data stack. https://www.soda.io
pypa / cibuildwheel
🎡 Build Python wheels for all the platforms with minimal configuration.
tensorflow / tfx
TFX is an end-to-end platform for deploying production ML pipelines
syv-ai / HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
pydoit / doit
CLI task management & automation tool
tdrussell / diffusion-pipe
A pipeline parallel training script for diffusion models.
doobidoo / mcp-memory-service
Open-source persistent memory for AI agent pipelines (LangGraph, CrewAI, AutoGen) and Claude. REST API + knowledge graph + autonomous consolidation.
elyra-ai / elyra
Elyra extends JupyterLab with an AI centric approach.
galaxyproject / galaxy
Data intensive science for everyone.
NVIDIA-NeMo / Curator
Scalable data pre processing and curation toolkit for LLMs