Strong Heuristics for Named Entity Linking (2022)

Abstract: Named entity linking (NEL) in news is a challenging endeavour due to the frequency of unseen and emerging entities, which necessitates the use of unsupervised or zero-shot methods. However, such methods tend to come with caveats, such as no integration of suitable knowledge bases (like Wikidata) for emerging entities, a lack of scalability, and poor interpretability. Here, we consider person disambiguation in Quotebank, a massive corpus of speaker-attributed quotations from the news, and investigate the suitability of intuitive, lightweight, and scalable heuristics for NEL in web-scale corpora. Our best performing heuristic disambiguates 94% and 63% of the mentions on Quotebank and the AIDA-CoNLL benchmark, respectively. Additionally, the proposed heuristics compare favourably to the state-of-the-art unsupervised and zero-shot methods, Eigenthemes and mGENRE, respectively, thereby serving as strong baselines for unsupervised and zero-shot entity linking.

## introduction while many of the most famous historic quotes are wise irrespective of their origin, this is less true for the majority of contemporary quotes in the news, which require speaker attribution to be useful in journalism or the social and political sciences. this observation is the motivation behind the construction of quotebank, a corpus of 178 million unique quotations that are attributed to speaker mentions and were extracted from 162 million news articles published between 2008 and 2020 @xcite . however, given the ambiguity of names, attributing quotes to mentions is insufficient for proper attribution, and thus, named entity disambiguation is required, a feature which quotebank lacks. to tackle this shortcoming and investigate the disambiguation of person mentions in quote-bank as a prototypical example of a web-scale corpus, we explore the suitability of scalable named entity linking (nel) heuristics, which map mentions of entity names in the text to a unique identifier in a referent knowledge base (kb) and thus, resolve the ambiguity. nel is an established task and solutions have been used for a variety of applications such as kb population @xcite or information extraction @xcite , yet the frequency of emerging and unseen entities in news data renders the adaptation of supervised nel approaches difficult and tends to require unsupervised or zero-shot methods. while such unsupervised methods @xcite and zero-short methods @xcite have been developed in recent years, scalability is an issue. for example, fully disambiguating quotebank with the state-of-the-art zero-shot nel method, mgenre @xcite , would require approximately 37 years on a single gpu according to our experimental estimates. therefore, we investigate the suitability of heuristic nel methods that rely on signals that are simple to extract from mention contexts or entity entries in a kb. in contrast to mgenre, we find that our best-performing heuristics can solve the same task in 108 days on a single cpu core, i.e., orders of magnitude faster and on cheaper hardware, while achieving comparable performance. ## related work viable learning-based methods for nel in settings without available training data can be classified into zero-shot and unsupervised learning. zero-shot nel was introduced by @xcite with the objective of linking mentions to entities that were unseen during training. later, @xcite proposed a bert-based model for this task. @xcite proposed genre, a supervised nel method that leverages bart to retrieve entities by generating their unique names autoregressively, conditioned on the context by employing beam search. while genre uses wikipedia as its referent kb and is not directly compatible with our setting, we compare our methods to mgenre @xcite , a multilingual adaptation of genre using wikidata. unsupervised nel. @xcite proposed τ mil-nd, a bilstm model trained on noisy labels, which are generated via a heuristic that ranks the candidate entities of a mention based on matching words in a mention and candidate labels. @xcite experiment with distant learning for nel and create training data by merging freebase with wikipedia. recently, @xcite proposed eigenthemes, which is based on the observation that vector representations of gold entities lie in a low-rank subspace of the full embedding space. these low-rank subspaces are used to perform collective entity disambiguation. while powerful, the aforementioned methods are designed for general domains and multiple entity types, and thus, cannot capitalize on domain-and entity-specific signals. in the following, we investigate the suitability of unsupervised nel heuristics for person disambiguation in the domain of news quotes in comparison to these methods. ## problem formalization the input to our nel system are articles @xmath0. in each article a, a set of entity mentions m a is annotated. each such mention @xmath1, which are uniquely identified by their wikidata qid identifier (for further details regarding wikidata, see appendix d). if multiple entity candidates are available for a mention, we refer to this mention as ambiguous. conversely, unambiguous mentions have only a single candidate entity. given an article @xmath2, an ambiguous mention @xmath3, and all candidate entities e m , the task of nel is to identify the entity @xmath4. we assume that nel methods assign a rank r(e, m) to each candidate entity @xmath5, which corresponds to the likelihood that e is the correct entity for m. consequently, we assume that methods cannot identify cases in which the entity does not exist in the kb or is not contained in the list of candidates (i.e., out-of-kb or nil predictions). thus, our focus is on the evaluation of methods in cases where at least one candidate is available. ## scoring methods we consider three main signals for entity candidate ranking methods: entity popularity, entity-content similarity, and entity-entity similarity. implementation details are provided in appendix b. ## data we focus on quotebank data, but also investigate the performance on aida-conll as a benchmark. similar to @xcite @xcite we label the mentions as either 'easy' or 'hard'. in quotebank, we deem a mention easy if it can be correctly disambiguated using ns and hard otherwise, while in aida-conll we use the definition proposed by @xcite in quotebank is a collection of quotes that were extracted from 127 million news articles and attributed to one of 575 million speaker mentions @xcite , out of which 75% are unambiguous. for our evaluation, we use a randomly sampled subset of 300 articles that are manually annotated with 1,866 disambiguated person mentions. 70% of these mentions are unambiguous. out of the ambiguous mentions, it was possible to determine ground truth labels for 310 (57%), which we use in our evaluation. we split the ground truth into 245 mentions (79%) for evaluation and 65 mentions (21%) for parameter tuning. for a more thorough description of the quotebank ground truth, see appendix a. ## evaluation all the resources (code, datasets, etc.) required to reproduce the experiments in this paper are available at https://github.com/ epfl-dlab/nelight . we use micro precision at one (p@1) and mean reciprocal rank (mrr) as the evaluation metrics. the metrics are aggregated over all ambiguous mentions for which ground truth data is available. performance is reported with 95% bootstrapped confidence intervals (cis) over 10, 000 bootstrap samples. to identify optimal weight parameters for the composite metrics, we perform a grid search over the range [0, 1]. for the quotebank data, the best performance is obtained for weights (1, 1, 1) for uiscore and (0.45, 0.9, 0.2) for ucse. for the aida-conll data, we perform the parameter optimization on the official validation set, where the best performance is obtained for weights (0.9, 0, 1) and (0, 1, 1) for uiscore and ucse, respectively. tie breaking. several ranking methods introduce ties, which we break by using popularity heuristics. among the popularity heuristics, only lqid is injective and always outputs distinct scores for different entities. in our experiments, we, therefore, use lqid to break ties if they remain after using other tie-breakers. a full breakdown of the tie-breaking performance for all popularity-based methods can be found in appendix e.3. ## discussion overall, the results highlight the practicality of the proposed heuristics. our simple heuristics outperform those based on word embeddings and are competitive in comparison to mgenre. ## conclusions and future work we tackled the problem of entity linking in quote-bank by employing heuristics that rely on simple signals in the context of mentions and the referent kb. the solid overall performance of the proposed heuristics on quotebank, their low computational complexity, and competitive performance on the aida-conll benchmark suggest that they can be used as strong baselines for unsupervised entity linking in large datasets. future work. we plan to experiment with weighting schemes that account for word importance, utilize additional signals from the kb, and include improved candidate generation methods. finally, we aim to provide a disambiguated version of quote-bank to the community.