L ink T ransformer: A Unified Package for Record Linkage with Transformer Language Models (2024)

Abstract: Many computational analyses require linking information across noisy text datasets. While large language models (LLMs) offer significant promise, approximate string matching packages in popular statistical softwares such as R and Stata remain predominant in academic applications. These packages have simple interfaces and can be easily extended to a diversity of languages and settings, and for academic applications, ease-of-use and extensibility are essential. In contrast, packages for record linkage with LLMs require significant familiarity with deep learning frameworks and often focus on specialized applications of commercial value in English. The open-source package LinkTransformer aims to bridge this gap by providing an end-to-end software for performing record linkage and other data cleaning tasks with transformer LLMs, treating linkage as a text retrieval problem. At its core is an off-the-shelf toolkit for applying transformer models to record linkage. LinkTransformer contains a rich repository of pre-trained models for multiple languages and supports easy integration of any transformer language model from Hugging Face or OpenAI, providing the extensibility required for many scholarly applications. Its APIs also perform common data processing tasks, e.g., aggregation, noisy de-duplication, and translation-free cross-lingual linkage. LinkTransformer contains comprehensive tools for efficient model tuning, allowing for highly customized applications, and users can easily contribute their custom-trained models to its model hub to ensure reproducibility. Using a novel benchmark dataset geared towards academic applications, we show that LinkTransformer - with both custom models and Hugging Face or OpenAI models off-the-shelf - outperforms string matching by a wide margin. By combining transformer LMs with intuitive APIs, LinkTransformer aims to democratize these performance gains for those who lack familiarity with deep learning frameworks.

## introduction linking information across sources is fundamental to a variety of analyses in social science, business, and government. a recent literature, focused on matching across e-commerce datasets, shows the promise of transformer large language models (llms) for improving record linkage (alternatively termed entity resolution or approximate dictionary matching). yet these methods have not yet made widespread inroads in social science applications, with rule-based methods continuing to overwhelmingly predominate (e.g., see reviews by @xcite ; @xcite ). in particular, researchers commonly employ stringbased matching tools available in statistical software packages such as r or stata. in academic applications, extensibility to a diversity of human societies (historic and present) and ease of use for those not familiar with deep learning are essential. string matching algorithms in widely used statistical packages meet these requirements because they require little coding expertise and can easily be applied across different languages and settings. in contrast, existing tools for large language model matching require considerable technical expertise to implement. this makes sense in the context for which these models were developed -classifying and linking products for e-commerce firms, which employ data scientists -but it is a significant impediment for scholarly use. to bridge the gap between the ease-of-use of widely employed string matching packages and the power of modern llms, we developed linktransformer, a general purpose, user friendly package for record linkage with transformer llms. linktransformer treats record linkage as a text retrieval problem (see figure 1 ). the api can be thought of as a drop-in replacement to popular dataframe manipulation frameworks like pandas or tools like r and stata, catering to those who lack extensive coding experience. linktransformer integrates: while transfer learning can facilitate strong off-the-shelf performance, heterogeneity in how out-of-domain applications are from llm training corpora -combined with settings that demand extremely high accuracy -create many scenarios where custom training may be needed. linktransformer makes it straightforward for users to tune their own customized models. linktransformer performs well on challenging record linkage applications. it is equally applicable to tasks with a single field -e.g., linking 1940s mexican tariff product classes across time -and applications that require concatenating an array of noisily measured fields -e.g., linking 1950s japanese firms across different largescale, noisy databases using the firm name, location, products, shareholders, and banks. this type of linkage problem would be highly convoluted with traditional string matching, as there are many noisily measured fields (e.g., products can be described in different ways, different subsets of managers and shareholders are listed, etc). using linktransformer to automatically concatenate the information and feed it to a llm handles these challenges with ease. linktransformer has a gnu general public license and is being actively maintained. a demo is available at https://youtu.be/hfrh8k1puki . more resources are available on our package website https://linktransformer.github.io/ . this study is organized as follows: section 2 provides an overview of related work. the core linktransformer library is described in section 3. section 4 evaluates linktransformer performance on various use cases, and section 5 considers ethics. ## relation to the existing literature the record linkage literature is sprawling -with large literatures in quantitative social science (particularly economics), statistics, computer science, and industry applications. these literatures are highly disjoint, taking very different approaches and even using different terms (record linkage, entity resolution/matching, approximate dictionary matching, etc.) to refer to the same task. a 2022 interdisciplinary science advances review, "(almost) all of entity resolution" @xcite , concludes that deep neural models are unlikely to be applicable to record linkage using structured data. it argues that training datasets are small and there is not much to be gained from llms since text fields are often short. yet there is an active literature on e-commerce applications that underscores the utility of llms for linking structured datasets, even when text fields are short. bench-marks in this literature (e.g., @xcite al. (2010); das et al. (2015); primpeli et al. (2019) ) focus on high resource commercial applications in english, such as matching electronics and software products between amazon-google and walmart-amazon listings, matching itunes and amazon music listings, and matching restaurants between fodors and zagat. recent studies have used masked language models @xcite @xcite , gpt @xcite , or both, significantly outperforming static word embedding and other older linkage methods. the siloed nature of the literature is reflected in softwares. the main existing package for record linkage with llms is ditto @xcite , which implements @xcite . it requires significant programming expertise to deploy. while this is appropriate for an e-commerce target audiencewhere data scientists predominate -the technical expertise required and the lack of pre-trained models targeted to multilingual social science applications has likely hindered further takeup. moreover, most of the literature examining record linkage with llms poses record linkage as a classification task @xcite , which is appropriate for the e-commerce benchmarks. however, this significantly limits extensibility, as in many social science and government applications the number of entities to be linked numbers in the millions, making it computationally infeasible to compute a softmax over all possible classes (entities). in the social sciences, string matching with statistical packages predominates. linktransformer frames record linkage as a knn-retrieval task, in which the nearest neighbor for each entity in a query embedding dataset is retrieved from a key embedding dataset, using cosine similarity implemented with an faiss backend @xcite . linktransformer includes functionality to tune a no-match threshold -since not all entities in the query need to have a match in the key -and allows for retrieving multiple neighbors, to accommodate many-tomany matches between the query and the key. the linktransformer architecture was inspired by biencoder applications with unstructured texts, e.g., passage retrieval @xcite , entity disambiguation @xcite , and entity coreference resolution (hsu and horwood, 2022). the knn retrieval structure of linktransformer also supports noisy de-duplication, a closely re-lated task that finds noisily duplicated observations within a dataset, following the methods developed in @xcite . linktransformer departs from much of the literature in utilizing llms trained for semantic similarity, combined with a supervised contrastive loss @xcite . off-the-shelf llms such as bert have anisotropic geometries @xcite , which makes them unsuitable off-the-shelf for metric learning problems like linktransformer nearest neighbor retrieval. contrastive training for semantic similarity reduces anisotropy, improving alignment between semantically similar pairs to be linked and improving sentence embeddings @xcite . linktransformer builds closely upon the contrastively trained sentence bert @xcite , whose semantic similarity library inspired many of the features in linktransformer. ## applications the llms in the linktransformer model zoo excel at a variety of tasks. as expected, custom-tuned models typically achieve the best performance, with off-the-shelf models still outperforming edit distance matching, typically by a wide margin. the custom-trained models are often plausibly achieving human-level accuracy, as cases that they get wrong are often impossible to resolve from the information provided, e.g., in cases where a firm is referred to by two completely disparate acronyms. second, we examine historical applications, which are central to understanding long-run phenomena like economic growth or social mobility and typically lack unique identifiers for linkage. first, we link two tariff schedules published by the mexican government in the 1940s (secretaria de economía de @xcite . tariffs were applied at an extremely disaggregated product level and each of the many thousands of products in the tariff schedule is identified only by a text description, which can change each time the tariff schedule is updated. around 2,000 products map to different descriptions across the schedules in a rare crosswalk published by the government (typically, homogenized crosswalks do not exist). we link the tariff schedules using an off-the-shelf semantic similarity model, as well as a model tuned on the in-domain historical data and open ai embeddings. all transformer models widely outperform edit distance. while there are considerable debates on the role that trade policies have played in long-run development, empirical evidence is limited largely due to the considerable challenges of homogenizing tariff schedules across time. language model linking offers the opportunitiy to bring novel quantitative evidence to this important question. we also link firms across two different 1950s publications created by different japanese credit bureaus @xcite . one has around 7,000 firms and the other has around 70,000, including many small firms. firm names can be written differently across publications and there are many duplicated or similar firm names. we concatenate information on the firm's name, prefecture, major products, shareholders, and banks. these variables contain ocr noise and the information included varies, e.g. in terms of how a firm's products are described, which shareholders are included, etc. this makes rule-based methods quite brittle, whereas the custom-tuned model links 91% of firms correctly. in the supplemental materials, we examine the various e-commerce and industry benchmarks that prevail in this literature. we use the same training procedure for each benchmark, to avoid overfitting, which is often not the case in the literature. we have generally comparable performance, sometimes outperformed by other models (that could be used with linktransformer if on hugging face) and sometimes outperforming other models. when ocr errors are severe, too much information may have been destroyed to achieve the desired accuracy with the garbled texts. a multimodal matching framework @xcite that uses aligned language and vision transformers to incorporate the original image crops or a matching framework that incorporates character visual similarity @xcite -as ocr confuses visually similar characters -may be required. vision and multimodal linking support will be incorporated into future releases of linktransformer. ## ethics statement linktransformer is ethically sound. it is built using public domain training data. because it is built upon transformer language models, it will not be suitable for lower resource languages that lack pretrained llms. however, it can utilize any hugging face or openai embedding model and hence will be extensible as the low-resource transformer literature expands to lower resource settings, as long as relevant embedding models are posted on hugging face or made available commercially by openai.