Goodwill Hunting: Analyzing and Repurposing Off-the-Shelf Named Entity Linking Systems (2021)

Abstract: Named entity linking (NEL) or mapping “strings” to “things” in a knowledge base is a fundamental preprocessing step in systems that require knowledge of entities such as information extraction and question answering. In this work, we lay out and investigate two challenges faced by individuals or organizations building NEL systems. Can they directly use an off-the-shelf system? If not, how easily can such a system be repurposed for their use case? First, we conduct a study of off-the-shelf commercial and academic NEL systems. We find that most systems struggle to link rare entities, with commercial solutions lagging their academic counterparts by 10%+. Second, for a use case where the NEL model is used in a sports question-answering (QA) system, we investigate how to close the loop in our analysis by repurposing the best off-the-shelf model (Bootleg) to correct sport-related errors. We show how tailoring a simple technique for patching models using weak labeling can provide a 25% absolute improvement in accuracy of sport-related errors.

## introduction named entity linking (nel), the task of mapping from "strings" to "things" in a knowledge base, is a fundamental component of commercial systems such as information extraction and question answering @xcite . given some text, nel systems perform contextualized linking of text phrases, called mentions, to a knowledge base. if a user asks her personal assistant "how long would it take to drive a lincoln to lincoln", the nel system underlying the assistant should link the first mention of "lincoln" to the car company, and the second "lincoln" to lincoln in nebraska, in order to answer correctly. as nel models have direct impact on the success of downstream products @xcite , * e-mail: kgoel@cs.stanford.edu all major technology companies deploy large-scale nel systems; e.g., in google search, apple siri and salesforce einstein. while these companies can afford to build custom nel systems at scale, we consider how a smaller organization or individual could achieve the same objectives. we start with a simple question: how would someone, starting from scratch, build an nel system for their use case? can existing nel systems be used off-the-shelf, and if not, can they be repurposed with minimal engineer effort? our "protagonist" here must navigate two challenging problems, as shown in figure 1: 1. off-the-shelf capabilities. industrial nel systems provide limited transparency into their performance, and the majority of academic nel systems are measured on standard benchmarks biased towards popular entities @xcite . however, prior works suggest that nel systems struggle on so-called "tail" entities that appear infrequently in data @xcite . as the majority of user queries are over the tail @xcite , it is critical to understand the extent to which nel systems struggle on the tail in offthe-shelf academic and commercial systems. in response to these questions, we investigate the limitations of existing systems and the possibility of repurposing them: we believe these principles of understanding fine-grained failure modes in the nel system and correcting them with data engineering apply to large-scale industrial pipelines where the nel model or its embeddings are used in numerous downstream products. ## named entity linking given some text, nel involves two steps: the identification of all entity mentions (mention extraction), and contextualized linking of these mentions to their corresponding knowledge base entries (mention disambiguation). for example, in "what ingredients are in a manhattan", the mention "manhattan" links to manhattan (cocktail), not manhattan (borough) or the manhattan project. internally, most systems have an intermediate step that generates a small set of possible candidates for each mention (candidate generation) for the disambiguation model to choose from. given the goal of building a nel system for a specific use case, we need to answer two questions: ## understanding failure modes we begin by analyzing the fine-grained performance of off-the-shelf academic and commercial systems for nel. ## amazon performs named entity recognition (ner) to we compare to 3 state-of-the-art systems: (i) boot-leg, a self-supervised system, (ii) rel, which combines existing state-of-the-art approaches, (iii) wat an extension of the tagme @xcite linker. we also compare to a simple heuristic (iv) pop, which picks the most popular entity among candidates provided by bootleg. datasets. we compare methods on examples drawn from two datasets: (i) wikipedia, which contains 100, 000 entity mentions mined from gold anchor links across 37, 492 sentences from a november 2019 wikipedia dataset, and (ii) aida, the aida test-b benchmark dataset 2 . metrics. as wikipedia is sparsely labeled (ghaddar and langlais, 2017), we compare performance on recall. for aida, we use macro-f1, since aida provides a more dense labeling of entities. results. our results for wikipedia and aida are reported in figures 3 , 4 respectively. ## related work identifying errors. a key step in assessing offthe-shelf systems is fine-grained evaluation, to determine if a system exhibits undesirable behavior. prior work on fine-grained evaluation in nel @xcite characterizes how to more consistently evaluate nel models, with an analysis that focuses on academic systems. by contrast, we consider both academic and industrial off-the-shelf systems, and describe how to assess them in the context of a downstream use-case. we use robustness gym @xcite , an open-source evaluation toolkit for performing the analysis, although other evaluation toolkits @xcite are possible to use, depending on the objective of the assessment. patching errors. if a system is assessed to have some undesirable behavior, the next step is to correct its errors and repurpose it for use. the key challenge lies in how to correct these errors. although similar to the related fields of domain adaptation @xcite and transfer learning @xcite where the goal is to transfer knowledge from a pretrained, source model to a related task in a potentially different domain, our work focuses on user-guided behavior correction when using a pretrained model on the same task. for industrial nel applications, orr et al. (2020) describe how to use data management techniques such as augmentation @xcite @xcite , weak supervision @xcite , and slice-based learning @xcite to correct underperforming, user-defined sub-populations of data. focusing on image data @xcite use domain translation models to generate synthetic augmentation data that improves underperforming subpopulations. nel. nel has been a long standing problem in industrial and academic systems. standard, predeep-learning approaches to nel have been rulebased @xcite , but in recent years, deep learning systems have become the new standard (see @xcite for an overview of deep learning approaches to nel), often relying on contextual knowledge from language models such as bert @xcite for state-of-the-art performance. despite strong benchmark performance, the long tail of nel @xcite in industrial workloads has remained a challenge. recent papers @xcite ; @xcite have begun to measure and improve performance on unseen entities, but it remains an open problem. ## conclusion we studied the performance of off-the-shelf nel models and how to repurpose them for a downstream use case. in line with prior work, we found that off-the-shelf models struggle to disambiguate rare entities. using a sport qa system as a case study, we showed how to use a data engineering solution to patch a bootleg model from mispredicting countries instead of sports teams. we hope that our study of data engineering to effectuate model behavior inspires future work in this direction.