Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP (2021)

Abstract: Retrieval is a core component for open-domain NLP tasks. In open-domain tasks, multiple entities can share a name, making disambiguation an inherent yet under-explored problem. We propose an evaluation benchmark for assessing the entity disambiguation capabilities of these retrievers, which we call Ambiguous Entity Retrieval (AmbER) sets. We define an AmbER set as a collection of entities that share a name along with queries about those entities. By covering the set of entities for polysemous names, AmbER sets act as a challenging test of entity disambiguation. We create AmbER sets for three popular open-domain tasks: fact checking, slot filling, and question answering, and evaluate a diverse set of retrievers. We find that the retrievers exhibit popularity bias, significantly under-performing on rarer entities that share a name, e.g., they are twice as likely to retrieve erroneous documents on queries for the less popular entity under the same name. These experiments on AmbER sets show their utility as an evaluation tool and highlight the weaknesses of popular retrieval systems.

## introduction substantial progress in nlp has been made on "closed" tasks, where queries are paired with relevant documents @xcite . however, there is growing interest in "opendomain" tasks, where relevant documents need to be retrieved from a knowledge source before an nlp system can perform reasoning and produce an answer @xcite . the open-domain setting better reflects real-world usage for tasks where relevant information is generally not provided (e.g., fact checking). q: which battle did abe lincoln fight in? a: world war ii wikipedia documents ranked by blink: 1. abraham lincoln 2. abraham lincoln in the black hawk war 3. abraham lincoln (captain) 4. benjamin lincoln 5. lincoln nebraska 6. lincoln england q: what musical instrument does abe lincoln play? a: trombone wikipedia documents ranked by blink: 1. abraham lincoln 2. john wilkes booth 3. abe (musical) 4. nebraska 5. lincoln nebraska 6. abe lincoln (musician) figure 1 : queries for two entities @xcite with the name "abe lincoln". retrieving the gold document involves disambiguating which "abe lincoln" each query is asking about. blink performs sub-optimally on the second query, as it ranks the document of the president over the gold document. because success hinges on finding relevant documents, open-domain progress has been closely tied to improvements in retrieval systems 2 @xcite @xcite . a crucial challenge when interacting with a large knowledge source (e.g., wikipedia) is entity ambiguity, the phenomenon where a single name can map to multiple entities. resolving this ambiguity is referred to as entity disambiguation and is an important step for effective retrieval. for example, given the query "what musical instrument does abe lincoln play?", documents about the musician should rank higher than other entities with the same name (figure 1 ). although entity disambiguation has been extensively studied in entity linking @xcite @xcite and search @xcite , in the context of open-domain nlp, it is unclear how good retrieval systems are when faced with queries with ambiguous entities. evaluating entity ambiguity is challenging because the popularity of entities follows a long-tail (figure 2 ) and rare entities are seldom covered in naturally-occurring datasets. in this paper we introduce amber sets, a benchmark for evaluating the entity disambiguation capabilities of retrievers across multiple nlp tasks. each amber set is a collection of wikidata entities that share a name, and their corresponding queries for specific nlp tasks. for each set, we define the head entity as the most popular entity and tail entities as the less popular ones. by creating queries for multiple entities that share a name, amber sets provide an accurate test of entity disambiguation capabilities of retrievers and help assess the role of entity popularity in disambiguation. we show examples of amber sets for the question answering task in we use amber sets to conduct a systematic study of various retrieval systems that operate under different principles, such as token overlap and dense embedding similarity. retrievers perform very differently on amber sets in terms of absolute retrieval numbers, with bootleg @xcite , an entity-linking-based retriever, performing best. despite these differences, all retrievers exhibit a large degree of popularity bias, underperforming on inputs concerning tail entities. tf-idf, a token-based retriever, performs about four times worse on tail entity inputs compared to head entity inputs. even with bootleg, the best performing retriever, performance on tail entities is still 1.5 times lower than on head entities. our results on amber sets demonstrate that there is significant work to be done on making retrievers robust in handling entity disambiguation. ## amber sets retrieving relevant documents from large knowledge sources such as wikipedia is an important first step in the open-domain pipeline. an inherent problem in working with such sources is entity disambiguation: resolving a name (mention) to an entity in the knowledge source. entity disambiguation can be challenging because many entities share a name, and the popularity of entities follows a long-tail distribution (figure 2 ). despite the importance of entity disambiguation, it remains an understudied problem for open-domain nlp. we introduce amber sets for evaluating entity disambiguation capabilities of retrievers and analyze the role of entity popularity in disambiguation. ## creating amber sets while amber sets can be manually created, doing so can be time-consuming, requiring a human to manually scour a knowledge base for polysemous task input instance output fc john mayer plays music. true sf nike [sep] country usa qa whose face is on $100 bill? benjamin franklin ## evaluation setup retrieval systems the primary focus of this work is to evaluate entity ambiguity of retrieval systems. we consider four retrievers based on different retrieval paradigms. the first three are tf-idf, a token-based retriever using sparse embeddings, dpr @xcite ## results in this section, we evaluate existing open-domain nlp pipelines using amber sets. we also conduct figure 4 : popularity gap vs retrieval gap. we bin qa queries of pairs of head and tail entities based on the popularity gap between the entities. for each bin, we calculate the retrieval accuracy@1 difference on the head and tail queries. larger popularity gaps tend to lead to a wider gaps in retrieval performance. the red line is retrievers' performance gaps between head and tail queries on the entire collection. top document retrieval we report retrieval performance in we evaluate systems in an oracle setting, where the gold document is provided, and a retrieval setting, where 20 documents are provided from a retriever. ## conclusion entity ambiguity is an inherent problem in retrieval, as many entities can share a name. for evaluating disambiguation capabilities of retrievers, we introduce amber sets; an amber set is a collection of task-specific queries about entities that share a name, but the queries have sufficient content to resolve the correct entity. we create a broad range of amber sets, covering many entity types, with input queries for three open-domain nlp tasks: fact checking, slot filling, and question answering. our experiments demonstrate the struggles of current retrievers in handling entity ambiguity. in particular, we find that the popularity of an entity in relation to other entities that share a name plays a significant role during disambiguation. for instance, we find that all tested retrievers are about twice as likely to retrieve erroneous documents when dealing with less popular entities than the most popular entity with the same name. future goals include improving entity disambiguation capabilities of retrievers, perhaps more directly incorporating ideas from entity linking and coreference resolution. the amber sets and the code for the generation pipeline is available at https: //github.com/anthonywchen/amber-sets.