Highly Parallel Autoregressive Entity Linking with Discriminative Correction (2021)

Abstract: Generative approaches have been recently shown to be effective for both Entity Disambiguation and Entity Linking (i.e., joint mention detection and disambiguation). However, the previously proposed autoregressive formulation for EL suffers from i) high computational cost due to a complex (deep) decoder, ii) non-parallelizable decoding that scales with the source sequence length, and iii) the need for training on a large amount of data. In this work, we propose a very efficient approach that parallelizes autoregressive linking across all potential mentions and relies on a shallow and efficient decoder. Moreover, we augment the generative objective with an extra discriminative component, i.e., a correction term which lets us directly optimize the generator’s ranking. When taken together, these techniques tackle all the above issues: our model is >70 times faster and more accurate than the previous generative method, outperforming state-of-the-art approaches on the standard English dataset AIDA-CoNLL. Source code available at https://github.com/nicola-decao/efficient-autoregressive-EL

## introduction entity linking (el; @xcite @xcite @xcite is a fundamental task in nlp employed as a building block for text understanding @xcite , question answering @xcite @xcite , dialog modeling @xcite , and information extraction @xcite , to name a few. popular earlier methods address the mention detection (md) and entity disambiguation (ed) stages of el separately @xcite @xcite while modern techniques leverage their mutual dependency @xcite @xcite . a new line of work @xcite departs from linking mentions using a vector space and instead uses large language models fine-tuned with a generative objective (i.e., predicting a textual identifier as the entity identifier). employing autoregressive language models better leverages the implicit knowledge accumulated during pre-training, exploiting a full cross-encoder of entities and their context. for ed, autoregressive generation is remarkably good (even in multilingual settings), while for el, although state-of-theart on multiple datasets, it suffers from several and critical limitations. the generative model of de @xcite outputs a version of the input document which is markup-annotated with mentions linked to their respective entities. this necessitates using an autoregressive decoder, precluding parallelism across mentions. generation also has a high computational cost due to relying on a complex and deep transformer @xcite decoder. transformers are state-less and their memory footprint scales with sequence length, making them memory-consuming when generating long sequences. additionally, transformers-based decoders are notably data-hungry, and their effective training requires large amounts of data. for example, de cao et al. (2021a) had to pre-train their model on wikipedia abstracts. in this work, we revisit the generative approach to el and generate mention-entity pairs conditionally independently given the input. this allows for parallelism across mentions, which we exploit by employing a shallow lstm-based decoder. to optimize more explicitly the generator's ranking, we use a discriminative correction term that pushes the score of the correct predictions to be higher than the rest. moreover, to enable conditioning on long inputs, we employ an efficient transformer encoder @xcite designed to support long sequences. figure 1 outlines our model. contributions we propose a highly parallel model for autoregressive entity linking that retains the advantages of being generative while @xmath0. we optimize for the correctness of the decoder's ranking with a discriminative loss to improve autoregressive el further. the model outperforms state-of-the-art approaches on the standard english aida dataset. related work el is typically decomposed in mention detection (md, i.e., the task of finding mention spans in text) and entity disambiguation (ed, i.e., the task of disambiguating a mention to its respective entity). many methods @xcite @xcite treat these sub-tasks separately, training different modules. more modern approaches -known as end-to-end el -instead use a shared (typically neural) architecture. @xcite use a bidirectional lstm @xcite as an encoder and then local and global scoring functions to link mentions. they exploit pre-computed entity embeddings by @xcite and match the embeddings to contextualized mention representations. martins et al. (2019) also explore joint learning of named entity recognition (ner) and el showing that the two tasks benefit from joint training, while @xcite approach el specifically for questions. in this work, we focus on monolingual el in english while there is a line of work that explores cross-lingual entity linking @xcite , that is linking from any source language to a standard one (e.g. english), and multilingual entity linking @xcite ) that is a generalization of both. autoregressive linking the genre model by de cao et al. (2021a) departs from framing el as matching in vector space, and instead frames it as a sequence-to-sequence problem. genre tackles md and ed for all mention-entity pairs jointly by autoregressively generating a version of the input markup-annotated with the entities' unique identifiers expressed in natural language. although we focus on el, genre was also applied to ed alone as well as to page-level document retrieval for fact-checking, open-domain question answering, slot filling, and dialog @xcite . mgenre @xcite is the multilingual extension of genre. modern techniques @xcite are based on a dense retriever module that uses maximum inner-product search (mips) to match mention vectors to entity embeddings. in contrast with mips for linking, generative models i) exploit knowledge learned during pre-training, ii) are memory-efficient as they do not need to store pre-computed entity representations, and iii) are full cross-encoders of context and entity since decoders can use attention to context. bi-encoders solutions may be sub-optimal and memory inefficient although memory-efficient dense retrieval has recently received attention @xcite @xcite . a caveat of joint modeling all mention-entity pairs with an autoregressive model (i.e., without any independence assumptions) is the lack of parallelism, which makes genre extremely slow for the complete task of el. in addition, generation of open-ended text calls for a deep decoder and thus requires very large corpora for training. ## method our method learns by generating observed mentionentity pairs y given an input document x. to enable, we assume that, given the document x, each mention-entity pair m, @xmath1. moreover, each pair's probability is further factorized as a product of an md and an ed components: p(y|x, θ) mention detection there are different ways to model p(m|x, θ md ) (i.e., the probability that the span m in x contains a mention). one is to score all possible spans which requires a number of evaluations that is quadratic in sequence length. for long documents, that is clearly unfeasible. thus, for maximizing efficiency, we opt for factorizing the probability of a span as the probability of its start m s times the conditional probability of its end m e given the start: the first term is the probability that position m s starts a mention, and the second is the probability that the mention has size m e -m s + 1, to which we give categorical treatment. 3 such factorization allows both for fast training and inference. during training, mentions are known. for inference, we consider only the positions for which the probability of starting a mention exceeds a threshold chosen to maximise micro-f 1 on the validation set. ## conclusion we revisit the generative approach to el exploiting independence assumptions that enable parallelism across mentions with a shallow lstm decoder. despite a simple and scalable design, our model sets a new state-of-the-art on english aida without a large decoder pre-training.