Pre-trained Language Models for Entity Blocking: A Reproducibility Study (2024)

Abstract: Entity Resolution (ER) is an essential task in data integration and its goal is to find records that represent the same entity in a dataset. Deep learning models, especially large pre-trained language models, have achieved state-of-the-art results on this task. A typical ER pipeline consists of Entity Blocking and Entity Matching: Entity Blocking finds candidate record pairs that potentially match and Entity Matching determines if the pairs match. The goal of the entity blocking step is to include as many matching pairs as possible while including as few non-matching pairs as possible. On the other hand, the blocking task can also be considered as an Information Retrieval (IR) task. However, state-of-the-art neural IR models that are based on large language models have not been evaluated on the ER task. What’s more, the generalization ability of state-of-the-art methods for entity blocking is not well-studied but an import aspect in real-world applications. In this work, we evaluate state-of-the-art models for Entity Blocking along with neural IR models on a wide range of real-world datasets, and also study their in-distribution and out-of-distribution generalization abilities.

## introduction entity resolution (er) aims to find data instances that belong to the same entity, and is an essential problem in data integration @xcite @xcite . over the decades of study, the focus has shifted from rule-based methods @xcite and machine learning-based methods @xcite , to deep learning-based methods, especially language model-based methods @xcite @xcite @xcite . in various real-world applications, data instances representing entities, such as product profiles or news articles, are commonly represented as text items composed of words. a pair of text items is deemed a match if they correspond to the same real-world entity. one may compare every pair of items to solve the problem, but the computational cost of comparing each pair in a dataset to identify matches is prohibitively high, particularly for large datasets where the number of all pairs grows quadratically. to address this challenge, the entity resolution (er) pipeline typically contains two primary stages: blocking and matching. the blocking stage aims to identify potential matches while excluding non-matching pairs, and the matching stage verifies whether candidate pairs, obtained through the blocking stage, indeed match. in the ir literature, this is a classic two-stage filter-and-refine pipeline, where the first stage (entity blocking) is a candidate generation phase. in recent literature of er, significant advances in accuracy have been achieved through the finetuning of large pre-trained language models (plms) @xcite @xcite @xcite . compared to previous methods, these plm-based methods excel at capturing the semantics of text items and correlations of the words in the items. it has been demonstrated that plm-based methods achieve state-of-the-art performance on both the blocking phase and the matching phase of the er pipeline in @xcite @xcite . within the literature, the blocking stage is typically composed of two steps: embedding and similarity search @xcite . in the embedding step, pretrained language models are fine-tuned using either unsupervised or supervised learning approaches to encode entities into embeddings that encapsulate their semantic meaning. subsequently, in the similarity search step, the search for similar item pairs is conducted directly on the embeddings. on the other hand, the entity blocking task can be viewed as an information retrieval (ir) task, which has seen significant advancements in recent ir literature with the emergence of large language models (plms). these plms play a crucial role in both dense and sparse neural ranking models, such as contriever @xcite , colbert (khattab and zaharia, 2020), and splade @xcite . dense models, such as contriever @xcite and colbert @xcite , leverage pre-trained language models and contrastive loss to learn dense representations for text items. although these dense neural ir models are primarily fine-tuned for question answering tasks, they can be naturally applied in the entity blocking task. sparse retrieval models, exemplified by splade @xcite , predict the importance of tokens in text items and combine this importance with token embeddings generated by large language models for ranking purposes. these sparse neural ir models show potential for the entity blocking task. since the performance of both dense and sparse llm-based ir models has yet to be thoroughly investigated within the entity resolution literature, we also include them in our reproducibility study to better understand the advancement of both the ir and the er literature. in real-world applications of large-scale entity resolution (er), labeled data is often only available for a small subset of the entire dataset or not available at all. what's more, it is not always feasible to use a large corpus of diverse real data during the training process, and real customer data is sometimes inaccessible during the training process due to various constraints such as privacy and security. hence, it is crucial to assess the generalization capabilities of plm-based methods, for the blocking task-an aspect largely unexplored in recent literature. besides, commonly used datasets for evaluating entity resolution are not specifically designed for evaluating generalization. in this paper, we examine the reproducibility and generalization ability of deep learning methods based on large language models (plms) for entity blocking. we conduct an evaluation and comparison of state-of-the-art methods from both the entity resolution and information retrieval literature as well as non-plms based methods, prepare new datasets and experimental settings that are specifically designed for evaluating generalization abilities of entity blocking methods. ## problem definition the entity resolution (er) process frequently operates on two distinct tables a and b, each containing item profiles. the primary objective is to identify all pairs (x, y) where @xmath0. when focusing on identifying duplicates within a single dataset, we consider @xmath1. figure 1 provides an example with two tables, each containing product profiles with identical attributes ("product name", "manufacturer", "price"). solid arrows in the figure represent matches, i.e., the second profile in table a matches the first profile in table b, and the third profile in table a matches the third profile in table b. the dashed arrow denotes an unmatched pair: the second item in table a does not match the second item in table b. to use pre-trained language models for processing item profiles, the raw texts are first serialized the same way as in existing methods @xcite @xcite : for each data entry @xmath2, val i ) 1≤@xmath3, we let serialize(e) ::= [col] attr 1 [val] val 1 ... [col] attr k [val] val k , where [col] and [val] are special tokens that indicate the beginning of attribute names and values respectively. for example, the first item in the left table in figure 1 can be serialized as follows: [col] product name [val] instant immersion spanish deluxe 2.0 [col] manufacturer [val] topics entertainment [col] price [val] 39.99 definition 2.1 (embedding in the interest of simplicity, we normalize all output vectors, implying that the l-2 norm ∥m emb (x)∥ 2 = 1 for each data item @xmath4. definition 2.2 (entity blocking). entity blocking, as a critical step in the entity resolution process, involves generating a candidate set of pairs @xmath5, y)|@xmath6, @xmath7. product name manufacturer price instant immersion spanish deluxe 2.0 topics entertainment 39.99 adventure workshop 4th-6th grade 7th edition encore software 29.99 sharp printing calculator sharp el1192b 45.63 product name manufacturer price encore inc adventure workshop 4th-6th grade 7th edition encore 26.49 adventure workshop 4th-6th grade 8th edition null 39.99 new-sharp shr-el1192bl two-color printing calculator 12-digit lcd black red sharp 45.99 if we denote the ground-truth matches as g, the aim of the blocking step is to maximize recall |c ∩ g|/|g|, while minimizing the size of the candidate set |c|. a smaller |c| for the same recall implies fewer non-matching pairs in c and higher precision, leading to reduced computational cost for the subsequent matching phase. for a given blocking model, a larger candidate set generally enhances recall but diminishes precision. therefore, striking a balance between recall and precision is a necessity in real-world applications. post fine-tuning of m emb , we apply the embedding model m emb to each data item to generate high-dimensional vectors. in this work, we use the high dimensional similarity search library faiss @xcite to find the k most similar embeddings for every embedding, forming the candidate set. we note that k is a tunable parameter that controls the candidate set size and recall. ## reproducibility dimensions of entity blocking for each data entry, the model m emb will convert each serialized text into in an embedding. in this study, we focus on utilizing a pre-trained transformer-based language model, such as bert @xcite and its variations like roberta @xcite . transformerbased language models excel in generating highly contextualized embeddings that offer enhanced understanding of textual information @xcite @xcite . ## evaluations we conduct the evaluations on a machine with a 12-core amd ryzen cpu (3.7ghz), 64gb main memory, and a nvidia rtx 4090 gpu (24 gb gpu memory). we limit the length of input tokens of text items to 128 for all methods. for each evaluated method that involves training, we use the same training data to ensure fair comparisons. we followed the best practice of each selected method according to their paper and official github repository, to conduct experiments in the entity blocking task. the main performance metric is recall in the problem of entity blocking @xcite . a higher recall can always be achieved by including more candidate pairs, but will a larger number of candidate pairs will incur higher cost for the entity matching step in entity resolution. to compare the effectiveness of difference methods, we draw the curve that represent the relation between the size of the candidate set (or the number of candidates) and the recall in our evaluation. each figure presented in this section and the following section displays the recall of the corresponding candidate set size on the yaxis, and the number of candidate pairs obtained by knn search is shown on the x-axis. in our evaluation, k ranges from 1 to 20. this means we find k candidates for each profile in the table, and the total number of candidates equals k multiplies the number of rows in the table. in comparing two methods for entity resolution, our primary focus is on the recall metric. this is because the ultimate goal of entity resolution is to accurately identify all data instances that correspond to the same entity. under the assumption of achieving equal recall performance, the method that yields the smallest candidate set size is considered as the best. this is because a smaller candidate set size reduces the computational overhead of downstream processing and facilitates more efficient data handling. we note that bm25 is not trainable and we use bm25 directly on the datasets in our evaluations, so the curves of bm25 are not related to "out-ofdistribution" or "in-distribution". we include the performance of bm25 for easy comparisons with other methods. ## related work the field of entity resolution (er) -the process of determining data items that represent the same realworld entity -has been a significant area of research over the past few decades @xcite . this process typically consists of two primary phases: blocking and matching. while a considerable body of work has proposed deep learning techniques for the matching phase (kasai et al., 2019; peeters et al., 2020; li 6 https://github.com/facebookresearch/faiss/wiki/faissindexes @xcite @xcite , the literature on techniques for the blocking phase remains comparatively sparse. most recently, @xcite and @xcite perform experimental studies on traditional blocking workflows and embedding-based nearest-neighbor search methods. they analyze the relative performance of the main representatives per category over numerous established datasets. however, these works do not study the generalization capabilities of state of the art methods in the literature, and they overlook latest advancements in the information retrieval (ir) literature which is important because the entity blocking problem can also be solved using ir methods. our work focuses on the generalization capabilities of latest advancements in both the entity resolution and information retrieval literature. blocking is pivotal as it minimizes the number of pairs to be compared in the matching phase. this is particularly vital considering the number of potential pairs can potentially reach the square of the dataset size, making a naive approach computation-ally prohibitive, especially when employing intricate models such as deep neural networks @xcite . these networks may need to process millions or even billions of items. the goal of the blocking phase is to identify as many genuine matches as feasible while maintaining a minimal candidate set. the research community has proposed a variety of blocking techniques, including rule-based blocking @xcite @xcite , schema-agnostic blocking @xcite , meta-blocking @xcite , deep learning techniques @xcite , and locality-sensitive hashing (lsh)-based blocking techniques @xcite . these methods can manage billions of items for entity matching. most recently, sparkly @xcite is a tf-idf based method and achieves state-of-theart performance on various datasets. however, its success relies on sophisticated tuning of the parameters for each specific dataset, which cannot generalize in the out-of-distribution setting. recently, pre-trained language models, such as bert-based models, have been leveraged to encapsulate the semantics of text items @xcite @xcite . these models are refined using contrastive learning methods and/or labeled data to generate item embeddings. similar item pairs can then be discerned by executing a similarity search on the embeddings. for instance, peeters et al. @xcite proposed r-supcon, a supervised contrastive learning model for product matching, utilizing the learned embeddings for blocking. entity blocking can also be framed as an information retrieval (ir) task. recent literature in ir @xcite has seen methods such as dpr @xcite , gtr @xcite , and contriever @xcite , which utilize pretrained language model-based (plm) methods to learn dense document representations. candidate pairs can be identified by conducting a similarity search on these dense representations using faiss @xcite . conversely, splade++ (formal et al., 2022) learns sparse representations and constructs an inverted index for lexical matching. this fundamentally circumvents the necessity for pair-wise comparison and scales appropriately to large datasets. the out-of-distribution generalization capabilities has been studied in the ir liter-ature @xcite , and the evaluations incidate neural retrievers or rankers can poorly generalize to different domains/datasets . ## conclusion in this paper, we study the reproducibility of large language model based methods for entity blocking. our study shows that state-of-the-art plm-based methods from both the entity resolution and information retrieval literature performs generally well in the in-distribution generalization evaluations for the entity blocking task. we also provide detailed break up of the running time for comprehensive understanding of each part of these plm-based methods. the majority of these methods also demonstrate excellent out-of-distribution generalization abilities. we highlight the challenge of achieving good performance when the density of matched profiles is higher on larger mixed datasets.