Ephemeral-AI-Lab / layerfs
Ephemeral Workspaces, durable shared history. LayerFS gives every agent a complete, writable filesystem Workspace over shared committed history, without copying the base. Agents work in parallel, commit what changed as deduplicated checkpoints, then branch, rewind or merge. Try it with LayerFS × DSH.
restic / restic
Fast, secure, efficient backup program
kopia / kopia
Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
windirstat / windirstat
Windows disk usage analyzer and cleanup tool with interactive treemaps, large-file discovery, duplicate detection, and file-system monitoring.
jiji262 / douyin-downloader
A practical Douyin downloader for both single-item and profile batch downloads, with progress display, retries, SQLite deduplication, and browser fallback support. 抖音批量下载工具，去水印，支持视频、图集、合集、音乐(原声)。
kucherenko / jscpd
Copy/paste detector for source code. 220+ languages, Rust engine, SARIF/HTML/badge reporters, GitHub Action, MCP server for AI agents.
GalacticBehindvalor / ua-free-telegram-rpocket-team-autopilot
Portable Windows autopilot for collecting tech news, AI rewriting with factual QA, deduplication and automatic Telegram publishing.
fallow-rs / fallow
Codebase intelligence for TypeScript and JavaScript. Health, complexity hotspots, duplication, architecture boundaries, circular dependencies, design-system drift, and unused code, from one graph. CLI, GitHub Action, LSP, MCP, and VS Code. Rust, MIT licensed.
rustic-rs / rustic
rustic - fast, encrypted, and deduplicated backups powered by Rust
borgbackup / borg
Deduplicating archiver with compression and authenticated encryption.
wu529778790 / panhub.shenzjd.com
PanHub 网盘聚合搜索 · Netdisk Search Aggregator。一个搜索框聚合夸克/阿里云盘/百度/115/迅雷等网盘资源，多源聚合去重、智能排序、插件熔断隔离。零数据库轻量部署，支持 Docker / Vercel / Cloudflare。Aggregate netdisk links into one search.
moj-analytical-services / splink
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
duplicati / duplicati
Store securely encrypted backups in the cloud!
scanaislop / aislop
Catch and fix the code-quality issues AI coding agents leave behind - dead code, unsafe casts, swallowed errors, duplication, security risks, and more. 50+ deterministic rules across 10 language targets, with CLI, CI, and GitHub Actions. No LLM at runtime. MIT.
NVIDIA-NeMo / Curator
Scalable data pre processing and curation toolkit for LLMs
sahib / rmlint
Extremely fast tool to remove duplicates and other lint from your filesystem
OpenGene / fastp
An ultra-fast all-in-one FASTQ preprocessor (QC/adapters/trimming/filtering/splitting/merging...)
RazgrizHsu / immich-deduper
duplicate photo finder for Immich - find and remove similar images
toki-plus / video-mover
全自动短视频搬运工具，支持自动下载、去重、AI生成标题+标签、上传，可二开扩展至多平台，例如：TikTok->视频号/抖音/小红书、抖音->TikTok/视频号/小红书......video-processing, automation, tiktok, selenium, pyqt5, ffmpeg, bot, data-scraping, video-deduplication.
prometheus / alertmanager
Prometheus Alertmanager
karanhudia / borg-ui
Replace complex Borg Backup terminal commands with a beautiful web UI. Create, schedule, and restore backups with just a few clicks.
ayushjain1144 / trackeverything
Code repository for "TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations"
PlakarKorp / plakar
plakar is a backup solution powered by Kloset and ptar
darula-hpp / shimmer-from-structure
A structure-aware skeleton loader that mirrors your rendered UI at runtime. Zero layout duplication. Built for modern frameworks.
fcorbelli / zpaqfranz
Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
bup / bup
Very efficient backup system based on the git packfile format, providing fast incremental saves and global deduplication (among and within files, including virtual machine images). Please post problems or patches to the mailing list for discussion (see the end of the README below).
mhx / dwarfs
A fast high-compression read-only file system for Linux, FreeBSD, macOS and Windows
borgmatic-collective / borgmatic
Mirror of borgmatic: Simple, configuration-driven backup software for servers and workstations
CarlosZ16420 / hamster-archiver
Windows local-first file organizer and media catalog with MCP tools for AI assistants. Verified batch archiving. 本地文件整理、媒体建库与可校验批量归档，支持 AI 通过 MCP 调用。
agentic-box / memora
Give your AI agents persistent, collective memory — with deduplicating absorb, supersession lineage, semantic search, and a graph UI. Speaks MCP.
tool-bunker / code-buster
Code Buster reports dependency wiring, dead code, cycles, duplication, complexity, architecture-policy violations, feature flags, security and style heuristics, quality scores, and remediation plans.
zinggAI / zingg
Scalable master data management, identity resolution, entity resolution, and deduplication using ML
MinishLab / semhash
Fast Multimodal Semantic Deduplication & Filtering
data-prep-kit / data-prep-kit
Open source project for data preparation for GenAI applications
rin-nas / postgresql-patterns-library
Коллекция готовых SQL запросов для PostgreSQL по часто возникающим задачам (получение и модификация данных, ускорение запросов, обслуживание БД)
Yomguithereal / talisman
Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.
alt8cio-source / tarkov-stash-replication-utility
Tarkov Item Duplication Exploit 2026: Dupe Any Item in 30 Seconds with FIR Status
elliribeiro1-spec / tarkov-fir-stash-mirror
Escape from Tarkov Item Duplication Guide 2026: Stash Dupe Method for All Items
tools4imps / exhale-ruby
The contraction gate for Rails: no PR merges while the codebase holds duplication the Contract doesn't keep
lincwang123-bot / humanized-social-publisher
Agent-friendly local publisher for Xiaohongshu, Douyin and WeChat Channels with multi-account safety, scheduling and deduplication.
Cavernzecool / Duplicati-Complete-Download
duplicati download — Duplicati for Windows 11 and 10. Installer in README, setup walkthrough, and configuration notes.
researchradar / research-radar
Self-hosted research intelligence for collecting, deduplicating, ranking, searching, and browsing the researchers, topics, and sources you care about.
poncippg-spec / liquidDX11
Real-time desktop glass for Dear ImGui using Win32, DirectX 11, DXGI Desktop Duplication, and HLSL.
Aaqibhafeezkhan / OpenNewsGrid
Real-time global news aggregation platform built with Next.js 15, TypeScript, and Tailwind CSS. It aggregates news from 300+ international sources through RSS feeds, NewsAPI, GNews, and other APIs, with smart deduplication to prevent duplicate stories.
NoteNerdOfficial / ai-skills-manager
AI Skills Manager brings your Claude, Codex, Pi, and other tool-specific skills, agents, commands, and rules into one searchable Obsidian library. Browse, tag, favorite, and group items across global and project folders without duplicating files. Enable or disable items in place with symlink-aware handling that keeps project links intact.
OpenDataBox / awesome-data-llm
Official Repository of "LLM × DATA" Survey Paper
ra1nty / DXcam
A Python high-performance screen capture library for Windows using Desktop Duplication API - Updated 2026
ivandilelio / kybes
Know Your Backend is a Plugin that help your AI to generate a structured Model/View/Controller application, witouth surprise, code duplication and different modality in a different project
surya-koritala / Glyd
Lossless AI compression: 33% less GPU memory, bit for bit.