Robustness Evaluation of Entity Disambiguation Using Prior Probes: the Case of Entity Overshadowing (2021)

Abstract: Entity disambiguation (ED) is the last step of entity linking (EL), when candidate entities are reranked according to the context they appear in. All datasets for training and evaluating models for EL consist of convenience samples, such as news articles and tweets, that propagate the prior probability bias of the entity distribution towards more frequently occurring entities. It was shown that the performance of the EL systems on such datasets is overestimated since it is possible to obtain higher accuracy scores by merely learning the prior. To provide a more adequate evaluation benchmark, we introduce the ShadowLink dataset, which includes 16K short text snippets annotated with entity mentions. We evaluate and report the performance of popular EL systems on the ShadowLink benchmark. The results show a considerable difference in accuracy between more and less common entities for all of the EL systems under evaluation, demonstrating the effect of prior probability bias and entity overshadowing.

## introduction the task of entity linking (el) refers to finding named entity mentions in unstructured documents and matching them with the corresponding entries in a structured knowledge graph @xcite . this matching is usually done using the surface form of an entity, which is a text label assigned to an entity in the knowledge graph @xcite . some mentions may have several possible matches: for example, "michael jordan" may refer either to a well-known scientist or the basketball player, since they share the same surface form. such mentions are ambiguous and require an additional step of entity disambiguation (ed), which is conditioned on the context in which the mentions appear in the text, to be linked correctly. following van erp and groth (2020) we refer to a set of entities that share the same surface form as an entity space. to decide which of the possible matches is the correct one, an ed algorithm typically relies on: (1) contextual similarity, which is derived from the document in which the mention appears, indicating the relatedness of the candidate entity to the document content, and (2) entity importance, which is the prior probability of encountering the candidate entity irrespective of the document content, indicating its commonness @xcite @xcite . the standard datasets currently used for training and evaluating ed models, such as aida-conll @xcite and wikidis-amb30 @xcite , are collected by randomly sampling from common data sources, such as news articles and tweets. therefore, they are expected to mirror the probability distribution with which the entities occur, thereby favouring more frequent entities (head entities) @xcite . from these considerations, we conjecture that the performance of existing el algorithms on the ed task is overestimated. we set out to explore this effect in more detail by introducing a new dataset for ed evaluation, in which the entity distribution differs from the one typically used for training ed algorithms. we perform a systematic study focusing on a particular phenomenon we refer to as entity overshadowing. specifically, we define an entity e 1 as overshadowing an entity e 2 if two conditions are met: (1) e 1 and e 2 belong to the same entity space s, i.e., share the same surface form and, therefore, can be confused with each other outside of the local context; (2) e 1 is more common than e 2 in some corresponding background corpus (e.g. the web), i.e., it has a higher prior probability p (e 1 ) > p (e 2 ). for example, e 1 = "michael jordan" (basketball player) overshadows e 2 = "michael jordan" (scientist) because p (e 1 ) > p (e 2 ) in a typical dataset sampled from the web. we use an unambiguous text sample that contains this mention to evaluate three popular state-of-the-art el systems, genre @xcite , @xcite , and wat @xcite , and empirically verify that the overshadowing effect that we hypothesized, indeed, takes place (see fig. 1a ). even when more information is added to the local context, including the directly related entities that were correctly recognised by the system ("machine learning"), the ed components still fail to recognise the overshadowed entity (see fig. 1b ). the concept of overshadowed entities introduced in this paper is related to long-tail entities @xcite . however, these two concepts are distinct: a long-tail entity may be unambiguous and therefore not overshadowed, while an overshadowed entity may still be too popular to be considered a long-tail one. to systematically evaluate the phenomenon of entity overshadowing that we have identified, we introduce a new dataset, called shadowlink. shad-owlink contains groups of entities that belong to the same entity space. following van erp and groth (2020), we use wikipedia disambiguation pages to collect entity spaces. disambiguation pages group entities that often share the same surface form and may be confused with each other. we then follow the links in the wikipedia disambiguation pages to the individual (entity) wikipedia pages to extract text snippets in which each of the ambiguous entities occur. note that we do not extract the text from these wikipedia pages directly, since pre-trained language models such as bert (typically used in state-of-the-art ed systems) also use wikipedia as a training corpus, and can learn certain biases as well. instead, we parse external web pages that are often linked at the end of a wikipedia page as references. this data collection approach helps us to minimise the possible overlap between the test and training corpus. thereby, every entity in shadowlink is annotated with a link to at least one web page in which the entity is mentioned. we then proceed to extract all text snippets in which the corresponding entity mention appears on the page. an extracted text snippet typically consists of the sentence in which the mention occurs. next, we use shadowlink to answer the following research questions: rq1: how well can existing ed systems recognise overshadowed entities? rq2: how does performance on overshadowed entities compare to long-tail entities? rq3: are ed predictions biased and how can we measure this bias? our contribution is twofold: (1) a new dataset for evaluating entity disambiguation performance of el systems specifically focused on overshadowed entities, and (2) an evaluation of current state-of-the-art algorithms on this dataset, which empirically demonstrates that we correctly identified the type of samples that remain challenging and provide an important direction for future work. ## the shadowlink dataset this section describes the shadowlink dataset: its construction process, structure, and statistics. ## manual evaluation we perform manual evaluation of a random sample from shadowlink to assess its quality, with the goal of ensuring that the extracted text snippets provide context sufficient for disambiguation. human performance also sets the skyline for automated approaches on this dataset. in the following subsections, we describe the evaluation setup and the results of the manual evaluation. ## benchmark experiments in this section, we describe the benchmark experiments designed to evaluate the baseline systems' performance on the shadowlink dataset. for these experiments, we created a subset of the original dataset by sampling only one of the shadow entities at random to make the number of top and shadow equal. note that in our task setup the model's predictions are not restricted to the top versus shadow entity binary decision. the model can predict any entity from the same or different entity space. we describe the experimental setup in section 4.1, report the benchmarking results and analyse them in more detail in section 4.2. ## conclusion we introduced shadowlink, a new benchmark dataset for evaluating entity disambiguation performance, and used it for an extensive analysis of the state-of-the-art systems' results. our experimental results indicate that all systems under evaluation are prone to rely on their priors, which explains their higher performance on more common entities, and much lower performance on the lexically similar overshadowed entities. our work thereby shows that the ed task is still far from solved for overshadowed entities, and shadowlink paves the way for further research in this direction. the shortcomings of existing disambiguation approaches uncovered by the shadowlink dataset stimulate further research towards developing more robust ed algorithms that are better at exploiting context without overrelying on the prior bias. we would also like to explore ways to account for more context around the entity mentions, and when expanding the context is actually needed.