Ephemeral-AI-Lab / layerfs
Ephemeral Workspaces, durable shared history. LayerFS gives every agent a complete, writable filesystem Workspace over shared committed history, without copying the base. Agents work in parallel, commit what changed as deduplicated checkpoints, then branch, rewind or merge. Try it with LayerFS × DSH.
surya-koritala / Glyd
Lossless AI compression: 33% less GPU memory, bit for bit.
restic / restic
Fast, secure, efficient backup program
jiji262 / douyin-downloader
A practical Douyin downloader for both single-item and profile batch downloads, with progress display, retries, SQLite deduplication, and browser fallback support. 抖音批量下载工具，去水印，支持视频、图集、合集、音乐(原声)。
kopia / kopia
Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
toki-plus / video-mover
全自动短视频搬运工具，支持自动下载、去重、AI生成标题+标签、上传，可二开扩展至多平台，例如：TikTok->视频号/抖音/小红书、抖音->TikTok/视频号/小红书......video-processing, automation, tiktok, selenium, pyqt5, ffmpeg, bot, data-scraping, video-deduplication.
borgbackup / borg
Deduplicating archiver with compression and authenticated encryption.
windirstat / windirstat
Windows disk usage analyzer and cleanup tool with interactive treemaps, large-file discovery, duplicate detection, and file-system monitoring.
moj-analytical-services / splink
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
PlakarKorp / plakar
plakar is a backup solution powered by Kloset and ptar
prometheus / alertmanager
Prometheus Alertmanager
bup / bup
Very efficient backup system based on the git packfile format, providing fast incremental saves and global deduplication (among and within files, including virtual machine images). Please post problems or patches to the mailing list for discussion (see the end of the README below).
mhx / dwarfs
A fast high-compression read-only file system for Linux, FreeBSD, macOS and Windows
sahib / rmlint
Extremely fast tool to remove duplicates and other lint from your filesystem
borgmatic-collective / borgmatic
Mirror of borgmatic: Simple, configuration-driven backup software for servers and workstations
NVIDIA-NeMo / Curator
Scalable data pre processing and curation toolkit for LLMs
karanhudia / borg-ui
Replace complex Borg Backup terminal commands with a beautiful web UI. Create, schedule, and restore backups with just a few clicks.
zinggAI / zingg
Scalable master data management, identity resolution, entity resolution, and deduplication using ML
MinishLab / semhash
Fast Multimodal Semantic Deduplication & Filtering
data-prep-kit / data-prep-kit
Open source project for data preparation for GenAI applications
rin-nas / postgresql-patterns-library
Коллекция готовых SQL запросов для PostgreSQL по часто возникающим задачам (получение и модификация данных, ускорение запросов, обслуживание БД)
RazgrizHsu / immich-deduper
duplicate photo finder for Immich - find and remove similar images
CarlosZ16420 / hamster-archiver
Windows local-first file organizer and media catalog with MCP tools for AI assistants. Verified batch archiving. 本地文件整理、媒体建库与可校验批量归档，支持 AI 通过 MCP 调用。
lincwang123-bot / humanized-social-publisher
Agent-friendly local publisher for Xiaohongshu, Douyin and WeChat Channels with multi-account safety, scheduling and deduplication.
Aaqibhafeezkhan / OpenNewsGrid
Real-time global news aggregation platform built with Next.js 15, TypeScript, and Tailwind CSS. It aggregates news from 300+ international sources through RSS feeds, NewsAPI, GNews, and other APIs, with smart deduplication to prevent duplicate stories.