E xt E n D : Extractive Entity Disambiguation (2022)

Abstract: Local models for Entity Disambiguation (ED) have today become extremely powerful, in most part thanks to the advent of large pre-trained language models. However, despite their significant performance achievements, most of these approaches frame ED through classification formulations that have intrinsic limitations, both computationally and from a modeling perspective. In contrast with this trend, here we propose ExtEnD, a novel local formulation for ED where we frame this task as a text extraction problem, and present two Transformer-based architectures that implement it. Based on experiments in and out of domain, and training over two different data regimes, we find our approach surpasses all its competitors in terms of both data efficiency and raw performance. ExtEnD outperforms its alternatives by as few as 6 F1 points on the more constrained of the two data regimes and, when moving to the other higher-resourced regime, sets a new state of the art on 4 out of 4 benchmarks under consideration, with average improvements of 0.7 F1 points overall and 1.1 F1 points out of domain. In addition, to gain better insights from our results, we also perform a fine-grained evaluation of our performances on different classes of label frequency, along with an ablation study of our architectural choices and an error analysis. We release our code and models for research purposes at https://github.com/SapienzaNLP/extend .

## introduction being able to associate entity mentions in a given text with the correct entity they refer to is a crucial task in natural language processing (nlp). formally referred to as entity disambiguation (ed), this task entails, given a mention m occurring in a text c m , identifying the correct entity e * out of a set of candidates e 1 , . . . , e n , coming from a reference knowledge base (kb). first introduced by bunescu and @xcite , ed aims to identify the actors involved in human language and, as such, has shown potential in downstream applications like question answering @xcite , information extraction @xcite , text generation @xcite and semantic parsing @xcite . since the advent of deep learning within the nlp community, this task has mostly been framed as a multi-label classification problem @xcite , especially leveraging the bi-encoder paradigm @xcite . however, although simple and yet powerful enough to push scores past 90% inkb micro f 1 on standard benchmarks, this formulation suffers from a number of downsides. first, the actual disambiguation is only modeled through a dot product between independent mention and entity vectors, which may not capture complex mention-entity interactions. second, from a computational perspective, entities are represented through high-dimensional vectors that are cached in a pre-computed index. thus, classifying against a large kb has a significant memory cost that, in fact, scales linearly with respect to the number of entities. besides this, adding a new entity also requires modifying the index itself. to address these issues, de @xcite have recently proposed an auto-regressive formulation where, given mentions in their context, models are trained to generate, token-by-token, the correct entity identifiers. 1 while this approach has addressed the aforementioned issues effectively, it requires an autoregressive decoding process, which has speed implications, and, what is more, does not let the model see its possible output choices, something that has shown significant potential in other semantic tasks @xcite . in this work, we focus on these shortcomings and, inspired by this latter research trend, propose extractive entity disambiguation (extend), the first entity disambiguator that frames ed as a text extraction task. given as input a context c m in which a mention m occurs, along with a text representation for each of the possible candidates e 1 , . . . , e n , a model has to extract the span associated with the text representation of the entity that best suits m. we implement this formulation through 2 architectures: i) a transformer system @xcite that features an almost identical modeling power to that of previous works, and ii) a variant that relaxes the computational requirements of our approach when using common transformer-based architectures. evaluating our two systems over standard benchmarks, we find our formulation to be particularly suited to ed. in particular, when restricting training resources to the aida-conll dataset @xcite only, extend appears to be significantly more data-efficient than its alternatives, surpassing them by more than 6 inkb micro f 1 points on average across in-domain and out-of-domain datasets. furthermore, when pre-training on external ed data as in de cao et al. (2021b), our system sets a new state of the art on 4 out of 6 benchmarks under consideration, with average improvements of 0.7 overall and 1.1 when moving out of domain. finally, we also perform a thorough investigation of our system performances, providing insights and pinpointing the reasons behind our improvements via a fine-grained evaluation on different label-frequency classes. our contributions are therefore as follows: we release our code and models for research purposes at https://github.com/ sapienzanlp/extend . ## related work entity disambiguation (ed) is the task of identifying, given a mention in context, the most suitable entity among a set of candidates stored in a knowledge base (kb). generally the last step in an entity linking system @xcite , coming immediately after mention detection and candidate generation, this task has been the object of a vast and diverse literature, with approaches typically clustered into two groups, depending on how they model co-occurring mentions in the same document. on the one hand, global models strive to enforce a global coherence across the disambiguations within the same document, leveraging different techniques and heuristics to approximate this objective foot_1 @xcite @xcite @xcite . on the other hand, local models disambiguate each mention independently of the others, conditioning the entity choice only on the mention and its context. thanks to the advent of large pretrained language models, this group has recently witnessed a significant improvement in performances, which are nowadays on par with, or even above, those achieved by state-of-the-art global systems @xcite . these approaches usually frame ed as a multi-label classification problem @xcite and a diverse set of formulations have been proposed. among these, the bi-encoder paradigm @xcite has been particularly successful @xcite @xcite : here, two encoders are trained to learn vector representations in a shared space for mentions in context and entities, respectively. classification of a given mention is then performed by retrieving the entity whose representation is closest according to some metric (e.g. cosine similarity). although remarkably powerful, these formulations present a number of disadvantages, such as their large memory footprint (each entity in the kb needs to be represented by a high-dimensional vector) and the fact that the actual disambiguation process is only expressed via a dot product of independently computed vectors, potentially neglecting mention-entity interactions. while a number of works @xcite attempt to address the latter issue via multi-stage approaches where a cross-encoder is stacked after an initial bi-encoder foot_2 or other retrieval functions, an interesting alternative direction that tackles both problems was recently presented by de cao et al. (2021b): the authors frame ed as a generation problem and, leveraging an auto-regressive formulation, train a sequence-to-sequence model to generate the correct entity identifier for a given mention and its context. nevertheless, while this approach can model more complex interactions, some of these can only occur indirectly inside the backtracking of their beam search. furthermore, the disambiguation involves an auto-regressive decoding that, although mitigated by later efforts @xcite , has intrinsic speed limitations. in contrast, here we propose an extractive formulation, where a model receives as input the mention, its context and the text representation of each candidate, and has to extract the span corresponding to the representation of the entity that best matches the (mention, context) pair under consideration. note that this differs from the aforementioned cross-encoder formulations @xcite where, instead, each entity was encoded together with the (mention, context) pair, but independently from all the other entities. with our schema, complex mention-entity and entity-entity interactions can be explicitly modeled by the neural system, as all the information is provided in input. glancing over other related tasks in the area of semantics, arguably closest to our work is esc @xcite , where the authors propose a new framing of word sense disambiguation (wsd) as an extractive sense comprehension task. yet, differently from their work, we propose here a new framing for ed, i.e. focus on entity descriptions rather than word sense definitions, present a baseline system that implements it and devise an additional architecture that deals with the computational challenges that arise from such implementation. ## model we now introduce extend, our proposed approach for ed. we first present the formulation we adopt (section 3.1) and, then, describe the two architectures that implement it (section 3.2). ## entity disambiguation evaluation we now assess the effectiveness of extend on entity disambiguation. we first introduce the experimental setup we consider (section 4.1). then, we present the results achieved by extend both in terms of raw performances (section 4.2) and via a breakdown of its behavior on different classes of label frequency (section 4.3). for ease of readability, we focus here only on the longformer-based architecture, which we consider as our main model. we defer the comparison with the bart-based system to section 5. ## model ablation while the above-mentioned experiments showed our approach to be rather effective, we only focused on the longformer-based architecture, to which we resorted owing to the computational challenges we mentioned in section 3.2. we now investigate this model choice, evaluating first how the bartbased system fares. then, we ablate the attention pattern we propose for the longformer and, finally, discuss the trade-off between our two proposed architectures. bart strictly speaking, the results we reported in the previous section are not exactly conclusive as to whether or not our formulation is beneficial. indeed, while it is true that we use a new formulation, we also rely upon a transformer model that none of our comparison systems considered. therefore, to better pinpoint the origin of the improvements, we train our bart-based architecture in the aida setting; we refer to this model as bart. note that the underlying transformer is identical to that of de @xcite , except for the final classification heads. 11 as shown in ## error analysis to further investigate the generalization capabilities of extend, we performed a black-box testing (ribeiro et al., 2020) of our system leveraging the available test sets. apart from the problem of label frequencies (e.g. unseen entities), we discovered two additional main classes of errors, namely i) insufficient context, and ii) titles alone might not be enough. insufficient context since the average number of candidates for each mention is roughly 50, the probability of having multiple valid candidates given the input context is far from negligible. for instance, let us consider the following example: "in the last game ronaldo scored two goals despite coming from a bad injury.". in this sentence, the mention ronaldo can refer both to cristiano ronaldo, the portuguese player, and to ronaldo de lima, the brazilian player. while this particular problem holds for several instances in the test sets, the performance drop is, in fact, mitigated by the labels skewness towards the most frequent candidates. indeed, the model appears always to predict the most frequent candidate for this kind of instance, therefore being right in the majority of cases. titles might not be enough for both comparability and performance purposes, the text representation we use for a given entity in this work is simply its wikipedia title. while article titles in wikipedia are rather informative, in several circumstances they do not contain enough information to make them sufficiently distinguishable from other candidates. for example, several pages describing "persons" are entitled just with their respective names and surnames. this kind of identifier is especially ineffective if the mentions taken into consideration were not present in the training dataset, or were rare or unseen during the underlying transformer pre-training. to this end, we strongly believe that future research might benefit from focusing on enriching entities' identifiers by adding a small description of the articles (summary) or at least some keyword representing the domain the entity belongs to. ## conclusion in this work we presented extend, a novel local formulation for ed that frames this task as a text extraction problem: given as input a string containing a marked mention in context and the text representation of each entity in its candidate set, a model has to extract the span corresponding to the text representation of the correct entity. together with this formulation, we also presented two transformer models that implement it and, by evaluating them across several experiments, we found our approach to be particularly suited to ed. first, it is extremely data efficient, surpassing its alternatives by more than 6 f 1 points when considering an aida-only training setting. second, pre-training on blink data enables the model to set a new state of the art on 4 out of 6 benchmarks under consideration and yield average improvements of 0.7 f 1 points overall and 1.1 f 1 points when focusing only on out-of-domain evaluation datasets. as future work, we plan to relax the requirements towards the candidate set and explore adapting this local formulation to a global one, so as to enforce coherence across predictions. for instance, we believe integrating the feedback loop strategy we proposed in @xcite would be an interesting direction to pursue.