ZELDA : A Comprehensive Benchmark for Supervised Entity Disambiguation (2023)

Abstract: Entity disambiguation (ED) is the task of disambiguating named entity mentions in text to unique entries in a knowledge base. Due to its industrial relevance, as well as current progress in leveraging pre-trained language models, a multitude of ED approaches have been proposed in recent years. However, we observe a severe lack of uniformity across experimental setups in current ED work,rendering a direct comparison of approaches based solely on reported numbers impossible: Current approaches widely differ in the data set used to train, the size of the covered entity vocabulary, and the usage of additional signals such as candidate lists. To address this issue, we present ZELDA , a novel entity disambiguation benchmark that includes a unified training data set, entity vocabulary, candidate lists, as well as challenging evaluation splits covering 8 different domains. We illustrate its design and construction, and present experiments in which we train and compare current state-of-the-art approaches on our benchmark. To encourage greater direct comparability in the entity disambiguation domain, we make our benchmark publicly available to the research community.

## introduction entity disambiguation (ed) is the task of disambiguating textual mentions of entities to a corresponding unique entry in a knowledge base. for instance, the entity mention "nba" might refer to one of several organizations with this abbreviation, such as "national basketball association" or "national boxing association". ed resolves these ambiguities and creates links between a knowledge base of unique entities and the various ways an entity may be referred to in text. it is the core component in the larger task of entity linking (el), which includes the identification of entity mentions in text, often handled by a named entity recognition (ner) system. recent progress in the field is driven by advances in large language models @xcite , pushing the scores on standard evaluation datasets to new heights. these models are typically trained in a supervised manner. unlike many other nlp tasks with relatively few target classes, such as sentiment analysis or part-of-speech tagging, ed may have millions of target classes, since each entity in a knowledge base is modeled as a distinct class. accordingly, most current state-ofthe-art ed approaches are trained over very large amounts of annotated text data that often is automatically derived from wikipedia. lack of uniformity in experimental setup. however, while a number of standard evaluation datasets exist to measure final ed accuracy, such as the aida-b test split of the popular aida dataset for newswire data @xcite , we observe that no such standardization exists for the data used to train ed systems. to illustrate this disparity, refer to as approach training data weak labels? data source additional features entity vocab candidate lists aida-b yamada et al. (2022) ∼10m snippets no wikipedia (dec, 2018) -lukegh 128k gh 92.4 -lukep p r 128k ppr 94.6 -lukegh+aida +aida-train 128k gh 95 -lukep p r+aida +aida-train 128k ppr 97.1 barba et al. (2022) 9m snippets † no wikipedia (may, 2019) -extend +aida-train 1.5m gh 92.6 ayoola et al. (2022) ∼20m snippets yes wikipedia (july, 2021) +kb +descriptions +types -ayoola 6.2m gh 90.4 de cao et al. (2021) 9m snippets † no wikipedia (may, 2019) -genre 1.5m gh 89.3 -genre+aida +aida-train 1.5m gh 93.3 -genre+aida-noc +aida-train 1.5m none 91.2 orr et al. (2021) 5.7m sentences yes wikipedia (nov, 2019) +kb +types -bootleg +aida-train 3.3m ppr+custom 96.7 févry et al. (2020) 17.5m snippets no wikipedia (apr, 2019) +aida-train -fevryhf 5.7m hf+custom 92.5 -fevryp p r 5.7m ppr+custom 96.7 broscheit (2019) yes wikipedia (june, 2017) +aida-train -broscheit 700k 8.8m snippets 700k none 78.8 -broscheit 500k 2.4m snippets 500k none 87.9 ## analysis of training setups ed approaches employ large language models to embed an entity mention, its textual context and additional features. to decode, approaches typically either use variants of softmax classification @xcite @xcite @xcite , generative decoding @xcite or retrieval-based models that compute the pairwise similarity of an embedded mention and a textual description for each target entity @xcite . rather than focus on the algorithmic differences of these approaches, this section analyzes current state-of-the-art approaches from the point of view of their respective training setups. ## the zelda benchmark we create zelda to enable analysis and direct comparison of large ed models. refer to ## related work prior work has addressed aspects of evaluating ed. standardized evaluation. gerbil @xcite standardizes ed evaluation over multiple datasets in a unifying framework, but does not define the training data and thus only focuses on comparing already-trained models. similarly, a range of prior works have sought to refine and standardize ed evaluation @xcite @xcite @xcite . in contrast, zelda defines the full experimental setup, including training data, the entity vocabulary and other training signals. manually labeled training data. a few existing ed datasets not only define a test set, but also a training split. an example discussed in this paper is the aida dataset. other datasets include tac-kbp2010 (ji and grishman, 2011), which is not available anymore, and a zero-shot dataset from @xcite . however, these datasets are too small and cover too few entities for evaluation of large ed approaches. deriving training data from wikipedia. all current state-of-the-art approaches derive their training data from wikipedia, though the exact process is often not thoroughly described and/or provided to the public. one exception is the blink corpus created by @xcite that is used in other works. however, this corpus consists only of single-mention snippets and cannot be used for approaches that train global decoders, like luke. most similar to our sampling method is from @xcite . the authors sample a small wikipedia subset by sampling for the mentions of the kore50 benchmark @xcite . unlike our approach, they sample all occurrences of each mention and sample only single sentences, yielding 520k sentences for only 144 mentions. ## conclusion we presented the zelda benchmark to unify experimental setups across large ed approaches, and conducted an evaluation of various approaches. we find that given the exact same training signal, approaches compare differently than published numbers suggest. we release the datasets, our sampling and preprocessing scripts and our fevry reimplementation to the research community as an open source project available at https://github. com/flairnlp/zelda . additionally, we integrate our benchmark into the open source nlp framework flair @xcite . we hope that this will encourage present and fu-ture ed works to compare algorithmic differences on a more equal setting and thus help generate insights to further advance the field of ed.