A Fair and In-Depth Evaluation of Existing End-to-End Entity Linking Systems (2023)

Abstract: Existing evaluations of entity linking systems often say little about how the system is going to perform for a particular application. There are two fundamental reasons for this. One is that many evaluations only use aggregate measures (like precision, recall, and F1 score), without a detailed error analysis or a closer look at the results. The other is that all of the widely used benchmarks have strong biases and artifacts, in particular: a strong focus on named entities, an unclear or missing specification of what else counts as an entity mention, poor handling of ambiguities, and an over- or underrepresentation of certain kinds of entities. We provide a more meaningful and fair in-depth evaluation of a variety of existing end-to-end entity linkers. We characterize their strengths and weaknesses and also report on reproducibility aspects. The detailed results of our evaluation can be inspected under https://elevant.cs.uni-freiburg.de/emnlp2023. Our evaluation is based on several widely used benchmarks, which exhibit the problems mentioned above to various degrees, as well as on two new benchmarks, which address the problems mentioned above. The new benchmarks can be found under https://github.com/ad-freiburg/fair-entity-linking-benchmarks.

## introduction entity linking is a problem of fundamental importance in all kinds of applications dealing with natural language. the input is a text in natural language and a knowledge base of entities, each with a unique identifier, such as wikipedia or wikidata. the task is to identify all sub-sequences in the text that refer to an entity, we call these entity mentions, and for each identified entity mention determine the entity from the knowledge base to which it refers. here is an example sentence, with the entity mentions underlined and the corresponding wikidata id in square brackets (and clickable in the pdf): american [q30] athlete whittington @xcite failed to appear in the 2013-14 season [q16192072] due to a torn acl @xcite . for research purposes, the problem is often split in two parts: entity recognition (er; identifying the entity mentions) and entity disambiguation (ed; identifying the correct entity for a mention). in practical applications, the two problems almost always occur together. in this paper, we consider the combined problem, calling it entity linking (el) foot_0 . ## metrics we report micro precision, recall and f1 scores, both for the overall el task and for the er subtask. details for how these measures are computed are provided in section a.1. additionally, we use the fine-grained error metrics provided by the evaluation tool elevant @xcite to analyze the strengths and weaknesses of the evaluated linkers in detail: er false negatives the following metrics analyze special cases of er false negatives. lowercased: the number of lowercased mentions that are not detected. partially included: the number of mentions where only a part of the mention is linked to some entity. er false positives er false positives are predicted mentions that do not correspond to a ground truth mention or that correspond to a ground truth mention annotated with nil. the following metrics analyze special cases of er false positives. lowercased: the number of falsely predicted mentions written in lower case. ground truth nil: the number of predicted mentions that correspond to a ground truth mention annotated with nil. wrong span: the number of predicted mentions that are part of or overlap with a ground truth mention of the predicted entity, but the predicted span is not correct. ## critical review of existing benchmarks we analyze the performance of the entity linking systems included in our evaluation on three of the most widely used existing benchmarks foot_5 . it turns out that each of them has its own quirks and biases, as discussed in the following sections. statistics on the annotated entity mentions for each benchmark are provided in ## our new fair benchmarks we create two benchmarks to address the shortcomings observed in existing entity linking benchmarks. the benchmarks are publicly available through our github repository foot_6 . the first benchmark, wiki-fair, consists of 80 randomly selected wikipedia articles, the second one, news-fair, of 40 randomly selected news articles from a webnews crawl @xcite . in each of these articles, three random consecutive paragraphs were manually annotated with wikidata entities. the rest of the article remains unannotated. this way, a large variety of topics is covered with an acceptable amount of annotation work while still allowing linkers to use complete articles as context. annotating the benchmarks with wikidata entities instead of wikipedia (or dbpedia) entities decreases the likelihood of punishing a linker for correctly linking an entity that was not contained in the knowledge base during benchmark creation, since the number of entities in wikidata is an order of magnitude larger than in wikipedia. we also annotate non-named entities in our benchmarks. in the few existing benchmarks that contain non-named entities, there is typically no discernible rule for which non-named entities were annotated such that the annotations seem rather arbitrary. to address this issue, we define a type whitelist (given in section a.5) and annotate all entities that have an "instance_of"/"subclass_of" path in wikidata to one of these types foot_7 . as discussed by @xcite , existing entity linking benchmarks differ significantly in which mentions are annotated and with which enti- we also annotate coreference mentions. how11 for example, both linking the entire phrase in "chatham, new jersey" to the entity for chatham and linking just "chatham" to the entity for chatham (while linking "new jersey" to the entity for the state new jersey) are considered correct on our benchmark. if a system predicts the mentions from the latter case, the prediction counts as a single true positive if and only if both mentions were correctly recognized and linked to the correct entities. otherwise it is counted as a single fn. this is to avoid the need for fractional tp or fn. fps are counted as usual. the total number of ground truth mentions is shown in ## evaluation of existing entity linkers in the following we analyze six entity linking systems in detail. our evaluation includes linkers to which code or an api are available and functional such that linking results can easily be produced 12 . furthermore, we restrict the set of linkers to those that either achieve strong results on popular benchmarks or are popular in the entity linking community. ## conclusions our in-depth evaluation sheds light on the strengths and weaknesses of existing entity linking systems, as well as on problems with existing benchmarks (in particular, the widely used aida-conll) and reproducibilty issues. we introduce two new benchmarks with clear annotation guidelines and a fair evaluation as primary goals. in particular, we find that even the best systems still have problems with metonym, partial name and rare mentions. all linkers have troubles with non-named entities. they either ignore non-named entities completely or link too many of them. re-fined performs best on almost all benchmarks including our independently designed and fair benchmarks. several systems have reproducibility issues. the two newest systems, refined and rel, are significantly better in that respect. our evaluation was more extensive than what we could fit into nine pages and we identified several frontiers for going deeper or further: describe more systems in detail, provide even more detailed numbers, include systems which only do disambiguation, evaluate also by entity type, and consider other knowledge bases; see section 9.