Unified Examination of Entity Linking in Absence of Candidate Sets (2024)

Abstract: Despite remarkable strides made in the development of entity linking systems in recent years, a comprehensive comparative analysis of these systems using a unified framework is notably absent. This paper addresses this oversight by introducing a new black-box benchmark and conducting a comprehensive evaluation of all state-of-the-art entity linking methods. We use an ablation study to investigate the impact of candidate sets on the performance of entity linking. Our findings uncover exactly how much such entity linking systems depend on candidate sets, and how much this limits the general applicability of each system. We present an alternative approach to candidate sets, demonstrating that leveraging the entire in-domain candidate set can serve as a viable substitute for certain models. We show the trade-off between less restrictive candidate sets, increased inference time and memory footprint for some models.

## introduction entity linking foot_1 is the task of annotating plain text with references to pre-defined entries -also known as entities -in a knowledge base. the annotation process involves finding specific parts of the textalso known as mentions -that may contain a knowledge base reference, and linking those to the right knowledge base record. for example, in the headline soccer -late goals give japan win over syria, the entity linking system can link japan (underlined) to japan national football team. candidate generation is an integral part of many entity linking systems, in which a small set of candidate knowledge base entities are selected for each potential candidate span. many recent entity linking techniques assume the presence of the preexisting candidate sets for all potential spans. the most popular such set is kb+yago @xcite which is used in a number of recent methods @xcite @xcite @xcite @xcite . another less popular set is pprforned @xcite which assumes the test sets are available to the annotators. due to this assumption, very few recent publications evaluate using pprforned @xcite @xcite . unified evaluation of the different entity linking systems with respect to the application of candidate sets should play a crucial role in a better understanding of the strengths and weaknesses of each system. this will give the research community and commercial deployments better ways to select the most suitable system based on their needs while providing them a platform to identify avenues for enhancement. in this paper, we propose a novel unified blackbox evaluation framework for recent state-of-theart entity linking systems. our contributions in this paper are as follows: ## unified black-box evaluation we benchmark the recent entity linking systems, unchanged and as provided originally by their authors. in these experiments, we intend to examine the suitability of these systems as off-the-shelf systems which can be integrated in future downstream applications. we unify the evaluation environment as ger-bil @xcite plus gerbil_connect @xcite . in the evaluation procedure, gerbil will provide the testing documents to gerbil_connect and receives the entity annotations in the format of (begin char., end char., entity annotation) from gerbil_connect. we implement gerbil_connect tailored to each entity linking system so that it can transform the evaluation documents to readable inputs for each system. specifically, we have (1) utilized nltk's word tokenizer foot_2 to transform raw non-tokenized evaluation sets into their expected conll tokenized format for the models that depend on reading from aida test files @xcite @xcite , (2) simulated long text splitting and result merging strategies for the models with input length constraints @xcite @xcite , (3) implemented a subword token id to character id conversion for the models that output annotations as tokenized subword ids @xcite @xcite , and (4) provided the external data sources such as the prebuilt candidate sets to the model initializers where necessary @xcite . appendix b provides more details on the mentioned alterations. empirically, running the models without adding these techniques significantly hurts performance. removing the tokenization step alone can drop the model performance by up to 20 micro-f1 points. once done, gerbil_connect translates the produced annotations in each system back to the unified annotation format, understandable for ger-bil. we train the models that are not released by the authors @xcite , using their own released source code, and do not consider the models which we were not able to acquire their training source code or were not able to get their training scripts to converge. @xcite @xcite @xcite appendix c briefly discusses each of the evaluated systems. we direct the readers to the original publication of each model for a more in-depth understanding of their contributions. we use conll/aida evaluation sets testa and testb -reported by all entity linking systems tested in different evaluation frameworks -as well as the newly annotated aida/testc evaluation set. the results tables show the gerbil inkb micro-f1 evaluation results. in our experiments, we found that @xcite suffered the most, with an approximate loss of 5% when comparing our results to the originally reported scores. the rest of the models were hit by at most 2%, confirming the reliability of our framework for further analysis. testc is a more challenging evaluation set which contains novel entities that typically hurt model recall. in our experimental results, we found that text generation models (from previous work) performed better on testc. however, the best performing model was not generative, but rather used structured prediction @xcite . ## candidate set ablations candidate sets are an integral part of entity linking systems, many of which assume the presence of good quality sets to perform well. although this assumption holds when linking to english wikipedia, it does not necessarily hold when considering other ontologies (e.g. umls @xcite 2021b ) showed a low variance of 0.08 in the results. thus, the results mirror those reported by gerbil, with the exception of @xcite , which is averaged over two runs. languages 4 . in this section, we ablate the mention-specific candidate sets from the entity linking systems to study their performance in absence of the handcrafted candidate sets. for our experiments, we selected the candidate-set-independent setting of the models in any system that provides such a setting. for the other systems that require a candidate set, and we cannot remove the candidate set dependence, we return the entire in-domain mention vocabulary of aida (the in-domain fixed candidate set of @xcite as the replacement for the required candidate sets (5598 entities including the none entity). where applicable, we add priors such that each candidate has an equal probability. in these systems, models that generate mention representations by combining candidate entity representations see their performance decreased to 20%-35%, while models that generate mention representations by combining the word or token representations within or surrounding the mention perform too poorly to be present in we count the number of annotations in four error categories of over-generated, under-generated, incorrect mention and incorrect entity, and divide each by the total number of gold annotations. figure 1 presents the calculated error analysis ratios. over-generation refers to annotations made by the model and not in the gold set. under-generation refers to annotations in the gold set but not made by the model. incorrect entity refers to annotations where the model linked the wrong entity. incorrect mention refers to annotations where the span's start or end is incorrect. before ablation of candidate sets (figure 1 -left), @xcite ) has the highest rate of over-generation followed by @xcite , while @xcite shows the lowest overgeneration rate. on the other hand @xcite has the highest under-generation ratio establishing itself as a conservative entity linking system. comparing the performance of entity systems w/ and w/o candidate sets, the biggest increase is seen in incorrect entity prediction ratios, confirming the dependence of entity linking systems to candidate sets. @xcite sees the biggest increase in incorrect entity predictions followed by @xcite . while @xcite and @xcite report the smallest rate increase in this category as these methods are less dependent on candidate sets. @xcite on the other hand shows an increase in undergeneration signaling the effect of candidate sets in prediction confidence for this system. figure 2 illustrates the disparities in precision and recall pre-and post-ablation of candidate sets for the models outlined in ## conclusion in this study, we have successfully established a unified black-box evaluation framework for modern entity linking techniques. we provide an indepth ablation analysis to examine the significance of hand-crafted candidate sets for the entity linking task. our findings confirm that modern entity linking systems are excessively dependent on candidate sets. our study shows that we need to be less reliant on hand-crafted candidate sets in order to ensure robust, versatile and accurate entity linking systems.