BPID : A Benchmark for Personal Identity Deduplication (2024)

Abstract: Data deduplication is a critical task in data management and mining, focused on consolidating duplicate records that refer to the same entity. Personally Identifiable Information (PII) is a critical class of data for deduplication across various industries. Consumer data, stored and generated through various engagement channels, is crucial for marketers, agencies, and publishers. However, a major challenge to PII data deduplication is the lack of open-source benchmark datasets due to stringent privacy concerns, which hinders the research, development, and evaluation of robust solutions.This paper addresses this critical lack of PII deduplication benchmarks by introducing the first open-source, high-quality dataset for this task. We provide two datasets: one with 1,000,000 unlabeled synthetic PII profiles and a subset of 10,000 pairs curated and labeled by trained annotators as matches or non-matches. Our datasets contain synthetic profiles built from publicly available sources that do not represent any real individuals, thus ensuring privacy and ethical compliance. We provide several challenging data variations to evaluate the effectiveness of various deduplication techniques, including traditional supervised methods, deep-learning approaches, and large language models (LLMs). Our work aims to set a new standard for PII deduplication, paving the way for more accurate and secure solutions. We share our data publicly at this link - https://zenodo.org/records/13932202.

## introduction data deduplication is a field of study dedicated to removing duplicate records that belong to the same entity, and is an essential problem in natural language processing (nlp) and data mining @xcite @xcite . for instance, grammarly's plagiarism checker detects plagiarism from billions of web pages and academic databases; google news identifies all versions of the same news article from different sources for comprehensive coverage; and amazon web services (aws) has an identity resolution service for linking customer identifiers from various sources into a unified customer profile. personal identifiable information (pii) encompasses a wide range of data, including names, addresses, email addresses, social security numbers, and more, which can uniquely identify individuals. handling pii presents unique challenges in data deduplication. similar-looking strings in names, addresses, and other personal details sometimes represent different individuals and often require deep semantic understanding to accurately determine whether two records refer to the same individual. simple string matching techniques are insufficient, as variations in spelling, abbreviations, and typographical errors can lead to incorrect conclusions. accurate pii data deduplication requires sophisticated algorithms to understand and interpret these nuances. the significance of accurate personally identifiable information (pii) data deduplication is evident in its influence across government agencies and industries. consumer records, as well as financial, criminal or property records are generated in various applications and engagement channels at a rapid pace @xcite . unifying and mapping this data enables agencies to identify individuals across different channels and personalize advertising and marketing campaigns. traditionally, consumer records were unified using thirdparty cookies and device ids. however, with the increasing deprecation of third-party cookies and device ids to enhance consumer privacy, marketers and publishers must develop new consumer identity resolution capabilities. organizations often invest considerable time creating customized solutions that link consumer identifiers, such as names, emails, and phone numbers. these solutions are not only expensive to develop, but they also require continuous maintenance due to the diversity of consumer data. in the absence of a diverse benchmark dataset for testing, both rule-based and machine learning-based methods are likely to commit errors on real-world consumer data. despite the significant progress in entity resolution (er) technologies, a major challenge persists in the lack of open-source benchmark datasets for pii entity resolution. while numerous existing datasets are available for non-pii entity resolution, such as products, academic papers, and music @xcite , no equivalent datasets exist for pii due to privacy concerns. industry providers have been reluctant to release testing datasets, creating a barrier to the development and benchmarking of robust er solutions. this gap restricts the ability to perform reliable evaluations and comparisons, which are crucial to advance the state-of-the-art in this field. to address this lack of benchmarking data, we leverage publicly available data sources (which are free to share and distribute) to construct the first open-source, high-quality benchmark dataset for the personal identity resolution problem. we build upon these sources to generate diverse and challenging testing examples that include both matching and non-matching pairs of profiles. our dataset comprises synthetic data that is curated by trained annotators to capture diverse potential pii data variations, and does not represent any real individual. this design ensures that our benchmark dataset challenges identity resolution services that rely solely on pre-existing consumer databases or simplistic heuristics. in our benchmark release, we provide two datasets for evaluation. the first blocking dataset includes 1,000,000 synthetic personal profiles without labels connecting duplicate identities, offering a broad testing ground for various er techniques. the second matching dataset is a subset of the first, containing 10,000 pairs of identities, each labeled as either a match or a no-match, to facilitate more detailed and supervised testing. we aim to set a new standard for pii entity resolution by introducing this benchmark dataset. this initiative is crucial for advancing er technologies and ensuring they can meet the growing demands for accuracy, privacy, and security in handling personal identity information. the rest of the paper is structured as follows: in section 2, we will discuss related work in pii deduplication. section 3 describes how we generate the datasets, including the data sources and the creation process for synthetic personal profiles. in section 4, we evaluate various algorithms on this dataset, including traditional methods, deep learning approaches, and llms, and find that our dataset proves challenging even for cutting-edge methods. in this work, we are particularly interested in evaluating deduplication for pii, broadly referred as entity resolution (er) , the process of consolidating records that represent the same real-world entity @xcite . er typically consists of two primary phases: blocking and matching. the blocking phase generates candidate pairs of entities, and the matching phase provides a final match/no-match decision for each candidate pair. our benchmark provides two datasets, one to evaluate each of these two phases. a considerable body of work has proposed deeplearning techniques for the matching phase @xcite @xcite @xcite . recent work proposes contrastive learning methods and/or labeled data for bertbased models in er tasks @xcite @xcite , as well as cutting edge large language models (llms) @xcite . we evaluate a representative set of methods in our experiments (section 4), and show that our dataset proves challenging even to state-of-the-art methods. however, we note that the blocking phase is critical to reduce the computational load of the matching system, since the number of candidate pairs potentially grows as the square of the dataset size. more recently, @xcite and @xcite have benchmarked blocking workflows. the challenge of the blocking phase is to achieve a minimal candidate set to reduce computation while maximizing the identification of true matches. to better evaluate blocking methods, we also provide a large blocking dataset of one million records. an ideal blocking method can correctly identify all matching pairs in our matching dataset, while reducing the overall candidate count. ## bpid dataset construction in this section, we describe our generation approach for our benchmark dataset. we include five universal personally identifiable attributes that are common to industry consumer records @xcite as well as governmental recordsthe name, any physical addresses, email addresses, phone numbers associated with the individual, and their date of birth, to match or not-match a pair of personal profiles. we first collect raw values for each of these five attributes from the below sources: real-world personal profile data is often incomplete. typically, 20% or more attribute values are unavailable @xcite , which significantly impacts pii data deduplication efforts. to make our dataset representative of real-world usecases, we randomly set 20% of the attribute values to empty strings. ## evaluations in this section, we provide results for both the blocking dataset (which contains one million unannotated profiles) and the matching dataset, which contains ten-thousand profile pairs with match/nomatch human annotations. note that the set of profiles in the matching set is a subset of the blocking set. we evaluate state-of-the-art entity matching and blocking methods over our benchmark datasets. our matching methods include a traditional supervised random forest @xcite , pre-trained language model based methods, ditto @xcite and sudowoodo @xcite , and llm-based methods @xcite . we designed a llm prompt (provided in appendix a) to determine if two profiles represent the same individual, and applied the prompt to various cuttingedge llms including anthropic model claude3-sonnet foot_3 , meta ai llama3-70b-instruct (meta, 2024), mistral ai mistral large foot_4 , and openai gpt4 turbo (achiam et al., 2023) 7 . we use the f1 score on the annotated matching dataset as our evaluation metric. we selected sudowoodo @xcite , nlshblock @xcite , sparkly @xcite , and contriever @xcite for the blocking task. we measure the recall (the percentage of the matched pairs in the matching dataset that be retrieved from the blocking dataset) and the candidate set size generated by each method. note that a higher recall and lower candidate size are preferred for blocking methods. these methods cover pre-trained language model based solutions, traditional tf-idf based solution, and dense information retrieval solutions. we use an aws ec2 p4d instance in our experiments for blocking and matching. ## conclusion in this paper, we introduce the first fully public benchmark dataset to facilitate the evaluation of data deduplication methods for personally identifiable information (pii). our dataset is meticulously designed to provide a rigorous and challenging testbed, surpassing the limitations of simplistic rules or heuristic techniques. even state-of-the-art large language models (llms) exhibit non-trivial error rates on our dataset, underscoring the complexity of the task and setting a high bar for evaluation. through this benchmark, we aim to foster advancements in pii data deduplication, promoting the development of innovative methods that prioritize privacy and data security while also enabling effective data management and analysis.