uk_address_matcher 1.0.0.dev24
A package for matching UK addresses using a pretrained Splink model
License: MIT
Requires Python: >=3.9

# High performance UK addresses matcher (geocoder)

Extremely fast address matching using a pre-trained [Splink](https://github.com/moj-analytical-services/splink) model.

```
Full time taken: 11.05 seconds
to match 176,640 messy addresses to 273,832 canonical addresses
at a rate of 15,008 addresses per second

(On Macbook M4 Max)
```

## Installation

```bash
pip install --pre uk_address_matcher
```

## Usage

High performance address matching using a pre-trained [Splink](https://github.com/moj-analytical-services/splink) model.

Will match two datasets provided in this format:

| unique_id | address_concat                          |
|-----------|-----------------------------------------|
| 1         | 123 Fake Street, Faketown, FA1 2KE      |
| 2         | 456 Other Road, Otherville, NO1 3WY     |
| ...       | ...                                     |


- You may also provide a separate column called `postcode`, which, if provided will trump any postcode information provided in `address_concat`.
- If you have labelled data (you know the ground truth), you may provide a column called `ukam_label`, if provided, this will propagate through your results for accuracy analysis.

Postcode handling rules:
- If you provide a separate `postcode` column, `address_concat` should ideally not include the postcode.
- If you do not provide `postcode`, the matcher will attempt to extract it during cleaning.


Generally one dataset will be a dataset of 'messy addresses' which need matching, and the second will be a 'canonical dataset' of addresses to match to.

## Preparing AddressBase for use in `uk_address_matcher`

`uk_address_matcher` can be used to any canonical list of addresses provided in the format above.

Many users will wish to link to Ordnance Survey address products.


### Simplest route (lower accuracy)

The simplest Ordnance Survey product to use for this purpose is [NGD Built Address](https://docs.os.uk/osngd/data-structure/address/gb-address/built-address).

You can use this 'out of the box' as your canonical list of addresses by selecting data from BuiltAddress as follows:

```
select uprn as unique_id, fulladdress as address_concat
from builtaddress
where {your_filter_here}
```

And providing the result output to `uk_address_matcher`.  You will generally improve accuracy if you filter the data down to the geographical region of interest, and filter the addresses down as much as possible to include only those of interest (e.g. residential only, if you're matching residential addresses)

### Full prep (higher accuracy)

Higher accuracy can be achieve by processing Ornance Survey data in a more sophisticated way.

For instance, Ordnance Survey provides multiple representations of a single address in Addressbase Premium and also in [NGD Address](https://docs.os.uk/osngd/data-structure/address/related-components/alternate-address).

By providing multiple addresses representations of each canonical address to `uk_adress_matcher`, you will have a better chance of higher precisison matching.

We provide a recommendation for automated build scripts for how to build such a file from Addressbase Premium and the NGD datasets here:
- [AddressBase Premium build script](https://github.com/moj-analytical-services/prepare_addressbase_for_address_matching)
- [NGD build script](https://github.com/moj-analytical-services/prepare_ngd_for_address_matching)


### Basic Matching

> [!NOTE]
> Two runnable examples with live sample data are included for experimentation:
> - [`examples/example_matching.py`](./examples/example_matching.py): End-to-end matching example, including loading data, running the matcher, and previewing results.
> - [`examples/example_prepare_canonical.py`](./examples/example_prepare_canonical.py): Example of preparing a canonical dataset for repeated use, demonstrating how to persist prepared data to disk and load it for matching.
>
> Both use parquet files in [`example_data/`](./example_data/) so you can run and adapt them immediately. You will need to download the example data from the releases page to run them, or you can adapt the code to use your own data.

```python
import duckdb

from uk_address_matcher import AddressMatcher, ExactMatchStage, SplinkStage

con = duckdb.connect()

df_canonical = con.read_parquet("your_canonical_addresses.parquet")
df_messy = con.read_parquet("your_messy_addresses.parquet")

matcher = AddressMatcher(
    canonical_addresses=df_canonical,
    addresses_to_match=df_messy,
    con=con,
)

result = matcher.match()  # returns a DuckDBPyRelation
result.limit(10).show(max_width=500)
```

The default stages are `ExactMatchStage` followed by `SplinkStage`. You can
customise them by passing your own `stages` list:

```python
from uk_address_matcher import (
    AddressMatcher,
    ExactMatchStage,
    SplinkStage,
    UniqueTrigramStage,
)

matcher = AddressMatcher(
    canonical_addresses=df_canonical,
    addresses_to_match=df_messy,
    con=con,
    stages=[
        ExactMatchStage(),
        UniqueTrigramStage(),
        SplinkStage(
            final_match_weight_threshold=20.0,
            final_distinguishability_threshold=5.0,
        ),
    ],
)

result = matcher.match()
```

### Pre-preparing canonical data

Cleaning a large canonical dataset (e.g. AddressBase) is expensive. Use
`prepare_canonical_folder` to do it once and write the artefacts to disk.
Subsequent runs load the prepared folder directly, skipping cleaning entirely.

```python
from uk_address_matcher import AddressMatcher, prepare_canonical_folder

# One-time preparation
prepare_canonical_folder(
    df_canonical,
    output_folder="./ukam_prepared_canonical",
    con=con,
    overwrite=True,
)

print("Prepared canonical data written to ./ukam_prepared_canonical/")

# Fast matching — pass the folder path instead of a relation
matcher = AddressMatcher(
    canonical_addresses="./ukam_prepared_canonical",
    addresses_to_match=df_messy,
    con=con,
)

result = matcher.match()
```

### Matching one or more AddressRecord entries

If you want to match a small number of addresses, or you have them in-memory as Python dictionaries, you can pass them directly as `addresses_to_match` without needing to create a DuckDB relation first.

You can pass a list of `AddressRecord` entries directly as
`addresses_to_match`. The matcher also accepts a list of dicts with
`address_concat`, `postcode`, and `unique_id`, or a DuckDB relation.

```python
import duckdb

from uk_address_matcher import AddressMatcher, AddressRecord

con = duckdb.connect()

df_canonical = con.read_parquet("your_canonical_addresses.parquet")

records = [
    AddressRecord(
        unique_id="m_1",
        address_concat="10 downing street westminster london",
        postcode="SW1A 2AA",
    ),
    AddressRecord(
        unique_id="m_2",
        address_concat="11 downing street westminster london",
        postcode="SW1A 2AB",
    ),
]

matcher = AddressMatcher(
    canonical_addresses=df_canonical,
    addresses_to_match=records,
    con=con,
)

result = matcher.match()
```


### Two-Pass Matching Approach

The Splink phase uses a two-pass approach to achieve high accuracy matching:

1. **First Pass**: A standard probabilistic linkage model using Splink generates candidate matches for each input address.

2. **Second Pass**: Within each candidate group, the model analyzes distinguishing tokens to refine matches:
   - Identifies tokens that uniquely distinguish addresses within a candidate group
   - Detects "punishment tokens" (tokens in the messy address that don't match the current candidate but do match other candidates)
   - Uses this contextual information to improve match scores

This approach is particularly effective when matching to a canonical (deduplicated) address list, as it can identify subtle differences between very similar addresses.



## Development

The scripts and tests will run better if you create .vscode/settings.json with the following:

```json
{
    "jupyter.notebookFileRoot": "${workspaceFolder}",
    "python.analysis.extraPaths": [
        "${workspaceFolder}"
    ],
    "python.testing.pytestEnabled": true,
    "python.testing.unittestEnabled": false,
    "python.testing.pytestArgs": [
        "-v",
        "--capture=tee-sys"
    ]
}
```