On the Surprising Effectiveness of Name Matching Alone in Autoregressive Entity Linking (2023)

Abstract: Fifteen years of work on entity linking has established the importance of different information sources in making linking decisions: mention and entity name similarity, contextual relevance, and features of the knowledge base. Modern state-of-the-art systems build on these features, including through neural representations (Wu et al., 2020). In contrast to this trend, the autoregressive language model GENRE (De Cao et al., 2021) generates normalized entity names for mentions and beats many other entity linking systems, despite making no use of knowledge base (KB) information. How is this possible? We analyze the behavior of GENRE on several entity linking datasets and demonstrate that its performance stems from memorization of name patterns. In contrast, it fails in cases that might benefit from using the KB. We experiment with a modification to the model to enable it to utilize KB information, highlighting challenges to incorporating traditional entity linking information sources into autoregressive models.

## introduction early work in entity linking in wikipedia @xcite followed by the formulation of the task at the tac kbp shared task @xcite @xcite has led to more than a decade of research into how to match textual mentions of entities to grounded entities in a knowledge base (kb). this large body of research has led to some clear findings @xcite @xcite @xcite @xcite @xcite . entity linking is commonly modeled as a ranking task, in which a triaged set of kb entities is ranked by comparison to a textual entity mention. these ranking systems rely on different information sources. first, the en-tity mention is compared to the entity name in the kb (name matching), with allowances for aliases, acronyms, etc. second, the context of the mention is compared to entity descriptions in the kb to select the correct entity among a set of similarly named candidates. third, other relevant information from the kb (type information, links to related entities, popularity, etc.) can help disambiguate between candidates. this information is formulated as features (either engineered or learned) into the ranking system. the recent emergence of autoregressive large language models as multi-task learners @xcite has led to numerous new applications of these models. these models have been particularly effective in few-shot learning settings @xcite , but typically fall behind supervised training of traditional systems that can flexibly incorporate a range of features. despite this trend, de @xcite presented genre, an autoregressive language model that uses supervised training to link textual mentions to entities in a kb. given a sentence and a previouslyidentified mention span, the model generates an entity name selected from a set of (triaged) candidates, with the option to generate entities without any constraints (with worse performance). surprisingly, aside from the entity name, genre uses no information from the kb, in contrast to other high-performing entity linking systems that rely on textual entity descriptions @xcite or type information @xcite . we may expect an autoregressive lm to do well, but how can it beat the best available feature-based entity linking systems? we explore the benefits and drawbacks of autoregressive entity linking. first, we ask -why genre performs so well? our answer comes from an analysis of the behavior of genre across several different entity linking datasets. specifically, we measure the generalization ability of the model by looking at performance on new datasets and knowledge bases. we find that genre relies heavily on memorization of name patterns, meaning that it struggles to generalize to new entities and kbs. kb information is often found to be useful in these cases, but its absence from genre means it struggles when name matching fails. therefore, our second question is: can genre make use of information from the kb when available? specifically, we provide descriptive information about an entity from the kb to genre and measure its resulting performance in various settings. we find that while it sometimes can make use of this information, it still struggles to learn generalizable patterns. our analysis shows opportunities for incorporating kb information into an autoregressive entity linker, but also the challenges of doing so given current model architectures. genre formulates entity linking as text generation as follows. given the selected entity mention and its left context within the sentence, the model is trained to predict the next tokens as the normalized entity name. consider the example in figure 1 . the model encodes the context two of the party's european, and is trained to generate the correct normalized entity name european parliament for this context. during training, the model is trained to minimize the smoothed cross-entropy loss between the generated entity name and the correct (normalized) entity name, where the normalized entity name matches the title of the associated node in the kb (wikipedia page title). in this setup, negative sampling is not required. genre starts with a pretrained bart model and continues training on 9 million example entity mentions selected from wikipedia, where the entity name is appended after each entity mention (see section 5). asking genre to freely generate a normalized name is both extremely challenging and unneces-sary. in practice, a pre-filtering (triage) step can be used to automatically select the most likely entity candidates for a textual reference via a name matching algorithm. foot_0 de @xcite evaluated genre under several conditions. first, a free decoding step whereby the model could output any string; this did not do well. second, constraining the model to generate a valid entity name from the kb. third, constraining the model to generate an entity from the small set of triaged candidates. for the constrained generation case, the authors constructed a trie t , where each node of the trie consists of a vocabulary entry, with a specialized token in the root. for each subword @xmath0, its children are allowed subword continuations. in an evaluation on the several entity linking datasets, including wikipedia and msnbc @xcite , genre achieved state-of-theart results compared to traditional entity linking systems. yet the shocking thing about this result is what genre lacks. first, genre uses no information from the kb. typical entity linking systems consider contextual overlap between the mention string and the kb entity description; genre does not. for example, when linking the textual mention america, a system would measure overlap with the kb description the united states of america is a transcontinental country primarily located in north america (united states) or americans are the citizens and nationals of the united states of america. (american). another popular feature is entity type, for example, country (united states) or nationality (american). other feature such as entity popularity, entity type, and related entities, are not available to genre. this information has long been used to disambiguate entities, and recent systems continue to show their ongoing effectiveness. @xcite use type information to help disambiguate entities that do not occur frequently. blink @xcite build contexualized embeddings for each entity using entity descriptions. none of this information is available to genre. furthermore, due to the generation nature of bart, genre only uses the left context of the entity mention. in sentences such as that in figure 1 , a very limited left context is availble to provide any information. while genre can memorize associations between the limited left context and the entity name, it cannot generalize even this limited information to new settings. despite these limitations, genre represents a state-of-the-art entity linker. ## genre and generalization how does genre achieve great entity linking results with such limited information? we explore this through the issue of generalization: how well does the model do on new unseen data? since the model does not have access to the kb, its predictions on new data are based entirely on what it can learn about entities fron training data. de @xcite suggested that genre predicts entities with contextualized name matching by leveraging large amounts of entity linking annotations during training. for example, while the original authors show that the model performs acceptably on rare entities (e.g., approximately 80% accuracy on wikipedia entities seen once in the training data), the accuracy for entities unseen in the training data is only 50%. @xcite show that genre is very data-intensive to train; reducing training to 0.01% of the original size performs 11% worse than blink. constrained decoding is also necessary for accurate predictions. generating without triaged candidates drops the accuracy by 9.2%. however, the importance of training data is clearly central, as triage could be adapted to new settings separately. what is genre learning from the massive training data? one possibility is that it learns how to normalize entity name (bill clinton to william clinton) from annotated data. pretraining on massive amounts of unannotated text followed by a large amount of entity linking annotations may also allow it to learn how to normalize certain informal names (america) to formal ones (the united states). furthermore, pretraining may allow for robust modeling of the context before mentions. fi-nally, as in other nlp tasks, the effect of using the encoding of the context provided by the sentence is likely valuable. if genre exhibits these behaviors, it can generalize certain abilities to new domains. however, if instead it is memorizing the training data, e.g. learning specific entities that appear in training, it cannot generalize. for example, wikipedia titles and mentions follow conventions, which may be learnable by the model, but will not generalize to settings that do not use wikipedia data or kbs. additionally, de @xcite report results on examples where the gold entity is found in the triage step, which biases toward lexical matches. examples that can be lexically matched are likely more likely to be solved by name matching. these links are far more common in wikipedia than other domains. in short, while generalization is a challenge for any machine learning model, it may be especially challenging for the mechanisms used by genre to learn from the training data. our first question is: does genre learn generalizable patterns or does it memorize the entities in the training data? we answer by probing how genre leverages its training data to perform linking. we evaluate genre on new datasets (section 5) more challenging than those reported in the original paper. we begin with datasets linked to wikipedia kbs, the proceed to datasets with different kbs. these new kbs contain entities unobserved in training, especially difficult for genre because it cannot access the kb. ## genre and the knowledge base genre faces challenges in generalization from its lack of access to the kb, which contains information about unseen entities. if genre was able to access the kb, could it better generalize to new data? a long line of entity linking research suggests that the answer should be "yes". in this section, we modify the training data to provide this information to genre. the key idea is to augment the training data with short descriptions of information in the kb. specifically, we add several keywords that summarize an entity's description in the kb to each training instance. genre is then asked (and trained) to generate the entity title followed by these keywords after each entity mention. this approach uses an unchanged genre model architecture to both learn to normalize names and bias the model towards entity descriptions (via keywords) that are most triggered by the (left) context of the mention. we choose to use keywords instead of the full text description for several reasons. first, in many kbs (especially wikipedia) entity descriptions are quite long, often multiple paragraphs. this stretches the context beyond what genre can reasonably model. even selecting a short snippet, e.g. the first sentence, also pushes the model beyond what is reasonable. instead, selecting a few important phrases from the description allows us to easily control the length of the produced string. furthermore, if selected correctly, these keyword can highlight topically related content, signaling a match with the left context of the entity. context enables genre to match the topic of the context with that of the candidate entity. in figure 1 , which entity best matches the the term european is ambigious. although the correct entity european union has a partial lexical match, other entities do as well (e.g., european parliament), and others are close lexical variants (europe). genre's ability to link this mention correctly would likely solely be based on whether it is seen in the training data, given the ambigiuity in the knowledge base. adding additional keywords can signal that european union and european parliament are potentially related, given political-related keywords such as party and council, whereas europe is less related. the same approach may be helpful to other mentions that could be amibigously linked in the knowledge base, such as washington. the keywords for washington d.c., district city congress united states metropolitan area, can help differentiate that entity from washington (state), which is paired with keywords seattle united states british columbia cascade range. this idea is in the same spirit as @xcite , which uses autoregressive language models for search, but de-codes entire spans from a corpus, as opposed to keywords. ## data wikipedia genre was trained on the blinkcreated version of a wikipedia dataset @xcite based on a may 2019 english wikipedia dump with 5.9 million entities. they use a 9 million-sized subset of wikipedia-linked mentions (e.g., links within wikipedia pages to other wikipedia pages). the kb consists of all pages within that snapshot of wikipedia. we exclusively use this dataset to train genre-kb. while we also report evaluation results on this dataset, we primarily target more challenging datasets. for evaluation, we use the provided candidate sets. tac the 2015 tac kbp entity linking dataset @xcite consists of newswire and discussion forum posts linked to an english kb. the discussion forum posts with informal entity mentions are especially challenging. chinese and spanish data are also included, but we only consider english. while this dataset does not directly link to wikipedia, almost all entities linked in the english dataset include a wikipedia title in their metadata. therefore, we convert all entities with wikipedia links to their respective entry in the wikipedia kb and convert all others to nil (no relevant entity). to generate a candidate set at inference time, we use the system of @xcite , which is largely based on work in @xcite . this approach uses wikipedia cross-links to generate a prior probability p prior (e i |m) by estimating counts from those mentions. this prior is used to provide the top k english wikipedia page titles for each mention. wikia to explore how genre and genre-kb work on datasets where wikipedia is not the kb, we include the wikia dataset @xcite . wikia was constructed from the wikia.com website (now fandom), which consists of community-written encyclopedias on a particular subject or theme. this was constructed in the same manner as the wikipedia dataset -mentions were taken from in-page hyperlinks, and each document served as an entity. the authors collect 16 wikias, each with a different topic and kb, thus serving as a challenging adaptation for our wikipedia-trained models. the authors exclude all nil entities and provide candidate sets for each mention of size 64, retrieved via bm25. topics are partitioned across training, validation, and test sets so that each appears in only one set. each mention is categorized by the amount of token overlap between the mention text and the normalized entity title. the categories include high overlap (5% of mentions), which represent exact matches; multiple categories (28% of mentions), where the entity title is the mention text plus a disambiguation phrase (e.g., mention batman, entity title batman (lego) ); and ambiguous substring (8% of mentions), where the mention is a substring of the title. the category other (59% of mentions) includes all remaining mentions. we believe the original label of low overlap is misleading, as many examples in that category have a high degree of lexical similarity. for example, of the other examples that have a candidate identified in the validation set, 28.96% of mention span -entity title pairs have a jaro-winkler lexical similarity (winkler, 1990) of over 0.794. ## experimental setup for genre-kb, we train all models on the wikipedia dataset alone and select the bestperforming model using the wikipedia validation set's loss. in all cases, we do not use the wikia or tac training data for training but only as a validation set. for wikia and tac data, we provide the model with the sentence where the mention occurs. sentence boundaries are identified with spacy @xcite . we adopt the method of reporting results from @xcite , which reports normalized accuracy, which is calculated over the set of examples that are non-nil and have the gold standard entity in their candidate set. as this restricts the types of examples to those that have mentions which are lexically similar to the entity name, we also report oracle results for some datasets, where we add the gold standard entity to all non-nil examples if not already present. ## results our experiments address two questions. first, why does genre perform so well? we answer this through evaluating generalization to new datasets. second, can genre utilize kb information to improve generalization (genre-kb)? ## related work entity linking has been broadly studied @xcite @xcite @xcite @xcite . recent work @xcite highlights the utility of type information in making linking decisions for rarer entities. other work has applied autoregressive models to other information extraction tasks @xcite . @xcite seeks to alleviate some of the performance challenges with genre during inference, although initial experiments found this performed worse in new domains. @xcite proposed a method that allows both the left and right context surrounding an entity mention to be modeled by producing the link at the end of the sequence. ## conclusion autoregressive transformer-based sequence-tosequence models, such as bart, have found increasing success in information extraction tasks. the genre model, which applies autoregressive sequence-to-sequence approaches to entity linking, has high performance on many datasets linked to the wikipedia domain. however, its performance on other domains with different challenges produces mixed results. we suggest that adding previously-explored entity linking features to genre can address some of these pitfalls. specifically, descriptions are a commonly used source of text to make linking decisions. while we see performance decreases in the original wikipedia datasets, we see some improvements in both newswire text and in applying genre-kb to previously unseen knowledge bases for more challenging matches. yet, the ability of genre to work in even challenging settings suggests that it can memorize patterns useful for mention-entity pairs with high lexical similarity. there are several unexplored directions for our model. specifically, we used an off-the-shelf keyword selection method. selecting keywords in a more targeted fashion -perhaps by selecting keywords for an entity that best separates it from another entity -may improve performance. having the computational resources to train a model from scratch would also likely improve performance, as opposed to training from a genre checkpoint. moreover, we focus on integrating descriptive information within the original genre framework. future work may consider an autoregressive entity linker with a novel architecture that can integrate and learn representations of entities would better utilize this information in learning.