opendatalab / MinerU
Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.
opendatalab / MinerU
docling-project / docling
Get your documents ready for gen AI
HelixDB / helix-foundry
Connect your company's data, let an agent draft an ontology over it, and load it into HelixDB as one graph you can explore and ask questions of. Hosted at foundry.helix-db.com, or run it on your own machine.
datalab-to / indra
1M param OCR model - runs on CPU, 15x faster than tesseract, close to 1B+ param models in accuracy
duckdb / duckdb
DuckDB is a high-performance analytical database system
FengZhenfei / carrel
Ontology-Augmented Generation for AI agents.
DataTalksClub / data-engineering-zoomcamp
Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼
seaweedfs / seaweedfs
SeaweedFS is a distributed storage system for object storage (S3), file systems, Iceberg table buckets, and Lance vector buckets, designed to handle billions of files with O(1) disk access and effortless horizontal scaling.
redis / redis
For developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.
jxwfdpt42t-ship-it / ReskillSQL
A Text-to-SQL agent that diagnoses failed queries and promotes validated, versioned query-writing skill updates instead of changing model weights.
kestra-io / kestra
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
datalab-to / marker
Convert PDF to markdown + JSON quickly with high accuracy
trinodb / trino
Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
elastic / elasticsearch
Free and Open Source, Distributed, RESTful Search Engine
ModernRelay / omnigraph
Object-storage native graph database with branching and typed ontology
YOLOW92 / pagesift
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
opendataloader-project / opendataloader-pdf
PDF Parser for AI-ready data. Automate PDF accessibility. Open-source.
apache / spark
Apache Spark - A unified analytics engine for large-scale data processing
Unstructured-IO / unstructured
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
OpenTenBase / OpenTenBase
OpenTenBase is an enterprise-level distributed HTAP open source database.
apache / kafka
Apache Kafka - A distributed event streaming platform
freerouting / freerouting
Advanced PCB auto-router
apache / doris
Apache Doris is a real-time analytics and hybrid search database for AI agents.
teslamate-org / teslamate
A self-hosted data logger for your Tesla 🚘 [main maintainer=@JakobLichterfeld]
StarRocks / starrocks
The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
CAPCOM-TD-OSS / REDox
High-performance, token-based structured data engine for .NET. A core component of REX, the technology behind CAPCOM's next-generation game engine.
colinhacks / motherload
Import data files from TypeScript, fully typed. Powered by TypeScript 7.1 content mappers.
apache / nifi
Apache NiFi
dmux / minilake
Minilake is a free, local Databricks API emulator — a single-developer tool for testing databricks-sdk/Terraform code against real SQL, real Delta Lake, and real Job execution, without paying for cloud compute.
apache / flink
Apache Flink
argoproj / argo-workflows
Workflow Engine for Kubernetes
DaveGamble / cJSON
Ultralightweight JSON parser in ANSI C
hydra-db / hydradb
HydraDB - fast graph database on object storage
datalab-to / lift
Extract structured data from documents quickly and accurately.
apache / beam
Apache Beam is a unified programming model for Batch and Streaming data processing.
apache / cassandra
Open source transactional distributed database. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure without compromising performance.
apache / dolphinscheduler
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
tarantool / tarantool
Get your data in RAM. Get compute close to data. Enjoy the performance.
OpenRefine / OpenRefine
OpenRefine is a free, open source power tool for working with messy data and improving it
apache / camel
Apache Camel is an open source integration framework with 350+ connectors. Write routes in Java, YAML, or XML. Run on Spring Boot, Quarkus, or standalone. Apache License 2.0.
elstongun / leviathan
Deep memory for agents over large datasets. Leviathan is a single static binary that turns your records (JSONL, JSON, CSV/TSV, SQLite, or anything a database CLI can export) into a ranked full-text index.
apache / kyuubi
Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
xtdb / xtdb
An immutable SQL database for application development, time-travel reporting and data compliance. Developed by @juxt
apache / gluten
Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
gpsaggese / gpsaggese.github.io
Course materials, tutorials, and projects for two University of Maryland graduate machine learning and big data classes.
mathesar-foundation / mathesar
An intuitive spreadsheet-like interface that lets users of all technical skill levels view, edit, query, and collaborate on Postgres data directly. 100% open source and self hosted, with native Postgres access control.
apache / carbondata
High performance data store solution
CMSgov / price-transparency-guide
The technical implementation guide for the tri-departmental price transparency rule.
Swirrl / grafter
Linked Data & RDF Manufacturing Tools in Clojure
zoltanctoth / dbt-student-repo
Starter Repository for the Complete dbt Bootcamp Zero to Hero Course
secretGeek / AwesomeCSV
🕶️A curated list of awesome tools for dealing with CSV.