W iki GUM : Exhaustive Entity Linking for Wikification in 12 Genres (2021)

Abstract: Previous work on Entity Linking has focused on resources targeting non-nested proper named entity mentions, often in data from Wikipedia, i.e. Wikification. In this paper, we present and evaluate WikiGUM, a fully wikified dataset, covering all mentions of named entities, including their non-named and pronominal mentions, as well as mentions nested within other mentions. The dataset covers a broad range of 12 written and spoken genres, most of which have not been included in Entity Linking efforts to date, leading to poor performance by a pretrained SOTA system in our evaluation. The availability of a variety of other annotations for the same data also enables further research on entities in context.

## introduction entity linking (el) involves identifying entities within a text and subsequently linking their mentions to a knowledge base or table of authorities. the former step is often referred to as named entity recognition (ner) and the latter may also be referred to as entity disambiguation. in this study, we will focus on the latter task by following the popular approach of mapping named entities to wikipedia entities @xcite , i.e. wikification. wikification is the task of adding links to wikipedia pages to mentions of named entities in a written or spoken text. this task supports natural language understanding in downstream tasks such as question answering, summarization, and relation extraction. however the scope and structure of el depends heavily on datasets which are either automatically derived from hyperlinked text and thus suffer some limitations, or are created via human annotation, a time-consuming and expensive task. despite numerous existing el datasets @xcite @xcite @xcite , few have attempted to capture nested entity structure, as in figure 1 , which never occurs in hyperlinks, which cannot be nested. tulsa transit provides bus service w: tulsa,_oklahoma w: metropolitan_tulsa_transit_authority org. instead, annotations have focused on flat mention structure from popular online sources, leaving out important information in nested entities that can be useful for downstream tasks. closest to the resource presented here is the nested named entities (nne) dataset @xcite , which is a large, manually-annotated, nested named entity dataset over english newswire, however, it does not include entity linking. although nne includes finegrained semantic information in nested entity types, it is not linked to any identifiers (e.g. a wikipedia page). furthermore, even if used for mention recognition, the data is not ideal for testing on diverse genres, as nne solely covers news text. we also note other datasets capturing some nested entity structure, such as the abstract meaning representation (amr) corpus @xcite , which includes compositional nesting e.g. in possessives such as toronto's international airport, composed of a city and an airport. however, since amr is not word-aligned to text, even those nested entities that are covered are not aligned to their textual position. in this paper, we present and evaluate a gold standard wikified dataset, called wikigum, in which named and non-named entities have been annotated manually. wikigum is based on the existing gum dataset (georgetown university multilayer corpus, zeldes 2017), and goes beyond other el corpora, in covering all mentions of named entities (nes), including non-named and pronominal mentions, as well as nested mentions, for 12 genres of english text. wikigum also enables assessment of el annotations by highlighting challenges that are common in our dataset, and reveals the relatively poor coverage of state-of-the-art nlp systems for el in diverse genres (section 4). taken together, we aim to facilitate new research on nested ner and el, to promote recognition of all ne mentions and a deeper understanding of the hierarchical structure of entities in text. ## wikigum the underying gum @xcite corpus is a manually annotated dataset with multiple layers, including pos tagging (penn tags, claws5, universal pos), sentence types (e.g. declarative, imperative, yes/no question), ud dependency trees @xcite , coreference resolution (including bridging anaphora and split antecedents), and rst discourse parses @xcite . data covers 12 genres: academic, biographies, conversation, fiction, forums, how-to, interviews, news, speeches, textbooks, travel and vlogs. wikigum adds a layer of wikipedia identifiers to all nes in gum, which are identified automatically by having the gold ptb pos tag nnp(s) for their syntactic head, based on gold syntax trees (for some resulting issues, see below), as well as non-named mentions coreferring to them based on coreference annotations. since gum is expanded by students in classroom annotation every year, and we plan to continue adding wikification in the future, no closed or pre-prepared ontology is applied to the wiki identifiers, making the task simpler for student annotators who only need to find a corresponding wikipedia article. that said, the existing 10 entity types in gum foot_0 mean that our el benefits from the same categorization scheme as a rough ontology, and the availability of semantic information from wikidata means that many relationships between entities can be explored. all referential nps, including pronouns and even clauses (if they co-refer with a named entity based on gum's coreference annotations, for example movie titles), were selected as markables for annotation. note that nested markables are always included, for example: in this example, the span this movie should also be linked to the wikipedia page that refers to the terminator (the movie). statistics on wikigum, which is freely available under the same creative commons license as gum, are shown in although the basic wikification task is fairly straightforward, some ambiguous/tricky cases during annotation included: in other cases, we relied on the coreference annotations to establish equivalence: for example england's city of festivals was labeled as coreferring with york, and was therefore considered equivalent to york for el purposes. most current el datasets are based on newswire text, overlooking the impact of genre on el -for example, @xcite notes that the biomedical domain involves complex and unique entity mentions. as el datasets are developed for evaluation of el systems, out-of-domain data could create challenges for conventional tools. furthermore, most previous work @xcite @xcite @xcite has focused on identifying and classifying atomic, flat mention structures, leaving out the semantic information available in nested mentions. as shown in ## evaluation in this section we evaluate inter-annotator agreement, as well as the extent to which existing wikification technology already captures the information in wikigum. inter-annotator agreement measuring agreement for wikification involves two main complementary aspects: span detection and wikification (including the decision whether to link an entity and to what). since gum already contains mention boundaries and named/non-named status, we focus on the latter task, measuring linking agreement. to calculate agreement, we carried out an inter-annotator agreement experiment by double annotating 3,103 tokens of corpus data containing 237 entities after adjudication, about 3% of the data. we compute both cohen's kappa and simple percent agreement (percentage of exact match), shown in results in ## conclusion this paper presented wikigum, the first exhaustive wikification dataset for named entity linking, including nested and pronominal mentions in 12 genres of english text. our evaluation suggests a high level of agreement, as well as coverage for a significant amount of entities not retrieved by a sota neural linking system. we hope that this dataset will enable further research on entity linking and increase coverage for all types of linkable named entities across a broad spectrum of genres, both spoken and written.