ELEVANT : A Fully Automatic Fine-Grained Entity Linking Evaluation and Analysis Tool (2022)

Abstract: We present Elevant, a tool for the fully automatic fine-grained evaluation of a set of entity linkers on a set of benchmarks. Elevant provides an automatic breakdown of the performance by various error categories and by entity type. Elevant also provides a rich and compact, yet very intuitive and self-explanatory visualization of the results of a linker on a benchmark in comparison to the ground truth. A live demo, the link to the complete code base on GitHub and a link to a demo video are provided under https://elevant.cs.uni-freiburg.de .

## introduction entity linking is a fundamental problem, and a first step for or component of many nlp applications. in this paper, we consider end-to-end entity linking systems, which do the following: given a text and a set of entities, identify each mention of one of those entities in the text and say which of these entities it is. due to its fundamental importance and wide applicability, there is vast literature on entity linking, and also a large number of concrete software tools. many publications also come with an evaluation, which compares the entity linker introduced in the publication with existing linkers, usually on a variety of benchmarks. there are also several standard benchmarks, which are found in many evaluations. this is a positive and pleasing development. the typical statistics include overall precision and recall, that is, which percentage of the found mentions were correct and which percentage of the correct mentions were found. it is a frequent experience that the numbers in the evaluation are very good, yet the experience when applying that entity linker in an own application are less convincing. and not so rarely, there is even trouble reproducing the results from the publication. the problem is that overall precision, recall, and f1 tell us little about the particular strengths and weaknesses of a particular entity linker for a particular application. particular benchmarks often require very particular skills from an entity linker, and an entity linker may be deliberately or unknowingly tuned towards these particularities. to find out about the strengths and weaknesses of an entity linker, one needs to look at the results in more detail, which typically has three aspects: doing this on the raw input and output files is tedious, so that often small scripts are written to aid this process. however, these scripts are usually quite basic and imperfect. it also means that researchers do the same work over and over again. it is the purpose of this paper to provide a comprehensive and easy-to-use tool, which every entitylinking researcher can and wants to use to analyze and understand the performance of a particular entity linker in detail. ## basic principles the core of elevant is a web app that helps users analyze and compare results of various entity linking systems over various benchmarks in great detail. to this end, the user can add an experiment and evaluate its results using elevant. we define an experiment as a run of a particular linker with particular settings over a particular benchmark. the pipeline for adding an experiment is as follows: the following subsections explain how each of these steps can be executed using elevant. ## conclusion elevant is a powerful, general-purpose, easy-to-use system for the in-depth evaluation and comparison of a set of entity linkers on a given set of benchmarks. typical evaluations of entity linking systems only provide aggregated figures like precision, recall and the f1 score. elevant goes beyond this by providing a breakdown of the results by entity type and by error category, as well as an intuitive visualization of true positives, false negatives, and false positives on the concrete texts. this can help both practitioners (to understand for which kind of texts a given entity linker is suited) as well as researchers (to help understand in detail the particular weaknesses of their entity linker and try to improve those).