CyberDefenderAI / AI-Data-Security-Governance
In the era of Generative AI, the data is the primary attack surface. This module focuses on securing the ML data pipeline from ingestion to vector storage, ensuring that the foundation of the AI model remains uncompromised and privacy-compliant.
coderamp-labs / gitingest
Replace 'hub' with 'ingest' in any GitHub URL to get a prompt-friendly extract of a codebase
The-Vibe-Company / quivr
An open-source engine that turns continuous content streams into search and monitoring. Durable ingestion, hybrid search, alerts, and plugins for formats, models and business rules.
getlago / lago
Open Source Metering and Usage Based Billing API ⭐️ Consumption tracking, Subscription management, Pricing iterations, Payment orchestration & Revenue analytics
bruin-data / ingestr
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
jitsucom / jitsu
Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
feyninc / chonkie
🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines
datazip-inc / olake
OLake - Fastest Databases, Kafka & S3 Replication to Apache Iceberg with Table optimization (Called OLake Fusion). ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Supported sources : Postgres, MongoDB, MySQL, Oracle, MSSql, DB2, Kafka, S3.
apache / seatunnel
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
The-Vibe-Company / megaparse
File Parser optimised for LLM Ingestion with no loss 🧠 Parse PDFs, Docx, PPTx in a format that is ideal for LLMs.
morphik-org / morphik-core
Open-source multimodal retrieval engine (Morphik Core)
apache / paimon
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
lakesoul-io / LakeSoul
LakeSoul is an end-to-end, realtime cloud-native Lakehouse framework for fast data ingestion, concurrent updates, incremental analytics, multimodal data processing and vector search — powering next-generation BI and AI workloads.
rsyslog / rsyslog
High-performance log ingestion and ETL engine
apache / gobblin
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
Micke-K / IntuneManagement
Copy, export, import, delete, document and compare policies and profiles in Intune and Azure with PowerShell script and WPF UI. Import ADMX files and registry settings with ADMX ingestion. View and edit PowerShell script.
bruin-data / bruin
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
tilo / smarter_csv
Fastest end-to-end CSV ingestion for Ruby (with C acceleration). SmarterCSV auto-detects formats, applies smart defaults, and returns Rails-ready hashes for seamless use with ActiveRecord, Sidekiq, parallel jobs, and S3 pipelines — even for messy user-uploaded real-world data.
nvk / llm-wiki
LLM-compiled knowledge bases for any AI agent. Parallel multi-agent research, thesis-driven investigation, source ingestion, wiki compilation, querying, and artifact generation.
StarlightSearch / EmbedAnything
Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust 🦀
CrunchyData / pg_parquet
Copy to/from Parquet in S3, Azure Blob Storage, Google Cloud Storage, http(s) stores, local files or standard inout stream from within PostgreSQL
Basekick-Labs / arc
Open, SQL-native time-series database for telemetry you need to keep. 38M+ records/sec ingestion, 25M+ rows/sec queries. InfluxDB Line Protocol and Telegraf compatible. Open Parquet on your storage. Single binary. S3/Azure native. Air-gap ready. AGPL-3.0.
ggozad / haiku.rag
Agentic RAG for local and self-hosted document search: hybrid retrieval, reranking and multimodal RAG on embedded LanceDB, with Docling parsing and an MCP server
mozilla-services / socorro
Socorro is the Mozilla crash ingestion pipeline. It accepts and processes Breakpad-style crash reports. It provides analysis tools.
Ali-Marandi / reportflow-automation
Reliable multi-source financial data ingestion with provenance, retries, normalization, and standard HTML/CSV/JSON reports.
sushiterminal / RagRank
RagRank utilizes real-time data ingestion and high-performance processing to modeling and anomaly detection on a scalable engine platform.
sfrangulov / minirag-mcp
Local-first RAG MCP server in Python: hybrid search (semantic + BM25 keyword boost) over your own documents. 11 MCP tools + CLI, markitdown ingestion for 12 formats, multilingual embeddings, LanceDB. Nothing leaves your machine.
MnemosyneOS / mnemosyne
Zero-dependency local-first AI memory engine: graph memory, multimodal ingestion, reranking, temporal reasoning, hash-chained ledger, capsule compression, memory formation + runtime layers. 52 MCP tools (41 native + 11 compatible). MIT.
needtobreathe / AuditHook
High-throughput HTTP event ingestion gateway with live SSE telemetry, payload auditing, and localhost dispatch capabilities.
mahialjaber / SOC-Log-Analysis-Datasets
Curated, simulated dataset of security logs (Windows, Web, Firewall, Cloud) with embedded attack vectors for SOC analysts to practice SIEM ingestion, threat hunting, and log analysis.
Victor-Kipruto-Rop / medallion-lakehouse-platform
A production-grade data engineering platform implementing the Bronze → Silver → Gold medallion architecture for scalable data ingestion, transformation, quality, governance, and analytics.