moj-analytical-services / splink

> [!IMPORTANT]
> 🎉 Splink 5 has been released! Examples of new syntax are here and a release announcement is here.
# Fast, accurate and scalable data linkage and deduplication
Splink is a Python package for probabilistic record linkage (entity resolution) that allows you to deduplicate and link records from datasets that lack unique identifiers.
It is used widely by within government, academia and the private sector - see use cases.
## Key Features
⚡ **Speed:** Capable of linking a million records on a laptop in around a minute. 
🎯 **Accuracy:** Support for term frequency adjustments and user-defined fuzzy matching logic. 
🌐 **Scalability:** Execute linkage in Python (using DuckDB) or big-data backends like Spark for 100+ million records. 
🎓 **Unsupervised Learning:** No training data is required for model training. 
📊 **Interactive Outputs:** A suite of interactive visualisations help users understand their model and diagnose problems. 
Splink's linkage algorithm is based on Fellegi-Sunter's model of record linkage, with various customisations to improve accuracy.
## What does Splink do?
Consider the following records that lack a unique person identifier:
Splink predicts which rows link together:
and clusters these links to produce an estimated person ID:
## What data does Splink work best with?
Splink performs best with input data containing **multiple** columns that are **not highly correlated**. For instance, if the entity type is persons, you may have columns for full name, date of birth, and city. If the entity type is companies, you could have columns for name, turnover, sector, and telephone number.
High correlation occurs when one column is highly predictable from another - for instance, city can be predicted from postcode. Correlation is particularly problematic if **all** of your input columns are highly correlated.
Splink is not designed for linking a single column containing a 'bag of words'. For example, a table with a single 'company name' column, and no other details.
## Documentation
The homepage for the Splink documentation can be found here, including a tutorial and examples that can be run in the browser.
The specification of the Fellegi Sunter statistical model behind `splink` is similar as that used in the R fastLink package. Accompanying the fastLink package is an academic paper that describes this model. The Splink documentation site and a series of interactive articles also explores the theory behind Splink.
The Office for National Statistics have written a case study about using Splink to link 2021 Census data to itself.
## Installation
Splink supports python 3.9+. To obtain the latest released version of splink you can install from PyPI using pip:
 [code] 
or, if you prefer, you can instead install splink using conda:
 [code] 
### Installing Splink for Specific Backends
For projects requiring specific backends, Splink offers optional installations for **Spark** and **PostgreSQL**. These can be installed by appending the