Learning from Natural Language Explanations for Generalizable Entity Matching (2024)

Abstract: Entity matching is the task of linking records from different sources that refer to the same real-world entity. Past work has primarily treated entity linking as a standard supervised learning problem. However, supervised entity matching models often do not generalize well to new data, and collecting exhaustive labeled training data is often cost prohibitive. Further, recent efforts have adopted LLMs for this task in few/zero-shot settings, exploiting their general knowledge. But LLMs are prohibitively expensive for performing inference at scale for real-world entity matching tasks.As an efficient alternative, we re-cast entity matching as a conditional generation task as opposed to binary classification. This enables us to “distill” LLM reasoning into smaller entity matching models via natural language explanations. This approach achieves strong performance, especially on out-of-domain generalization tests (10.85% F-1) where standalone generative methods struggle. We perform ablations that highlight the importance of explanations, both for performance and model robustness.

## introduction entity matching, also known as record linkage or data deduplication, refers to matching records from different sources which refer to the same underlying entity, in the absence of unique identifiers. this is a practically important task across a diverse set of domains, e.g., database management, healthcare, customer relationship management, and financial services; in such applications, normalizing entities to realize a unified view of data is imperative. most prior work on entity matching has adopted supervised techniques, training a model to link entities within a particular domain. performing pair-wise comparison on all record pairs is computationally prohibitive, especially on large scale * work perfomed during internship at amazon. datasets; typical entity resolution pipelines therefore perform blocking followed by matching @xcite . the former step entails identifying candidate record pairs which may reference the same entity, while in the latter one attempts to infer whether this candidate is indeed a match. ## entity matching via text generation we treat entity matching as a conditional text generation task. for a dataset of n entity pairs x @xmath0, entity_b i ), we model the probability of generating classification label (e.g., "match"/"no match") as a string this is the standard conditional language modeling objective. during training, we use "teacherforcing", i.e., condition production of outputs ("match" or "not") on reference prefixes. ## assessing the usefulness of explanations through ablations for each of the following ablations (a-e), we make targeted changes to the original llm-generated explanations and then retrain the smaller lm to test the corresponding effects. a. junk substituion we start by substituting llm-generated explanations by sentences comprising random 'junk' tokens, which are generated at random foot_4 from the english language vocabulary. we retain the original length of the explanation, e.g., in the example above the llm-generated explanation is substituted with the following text substituted: contour fix nap egregious text nimble perhaps the aim is to assess whether it is the presence of meaningful text (rather than any text) that leads to performance gains under the above settings. aggregate performance under ablation a drops 28.17%, and this is consistent across train-test pairs. b. random token-drop we alter the llmgenerated explanations by reducing their length. we start by removing all stop-words from the explanation, then randomly drop tokens to further reduce its length until we reduce the total length by half (50%). in the running example, the llmgenerated explanation might be replaced by the following text substituted: entities red "hard 3tb sata 3.5" use attached storage nas match. c. tf-idf here we attempt to sample tokens from the llm-generated explanation to assess if the presence of certain key tokens is all that is needed to realize the observed performance gains. we use tf-idf @xcite as a measure of word importance. specifically, we treat entity descriptions and their corresponding labels as documents, and llm-generated explanations as a summary of these. we then sample tokens from the explanation based on the tf-idf scores of individual tokens until we retain 50% of the original length of the explanation. in the running example, the llm-generated explanation might be replaced by the following text: substituted: drive to entity refers while 3tb and are attached both entities for hard iii in match nas network not red refer sata specifically storage perhaps surprisingly, sampling tokens in this way does not help, compared to randomly sampling them like as in (b); the performance degradation is about the same (5.57% vs 5.69%; ## conclusions we proposed a novel model distillation approach to train a small, more-robust model for generalizable entity matching. eliciting target label rationales from llms enables transfer of grounded "reasoning" to the smaller models. our experiments show this translates to strong performance in diverse settings, outperforming existing models designed for domain adaptation that struggle to generalize. ablation studies provide insight into the importance of explanation generation for achieving robust match-ing performance.