In Media Res: A Corpus for Evaluating Named Entity Linking with Creative Works (2020)

Abstract: Annotation styles express guidelines that direct human annotators in what rules to follow when creating gold standard annotations of text corpora. These guidelines not only shape the gold standards they help create, but also influence the training and evaluation of Named Entity Linking (NEL) tools, since different annotation styles correspond to divergent views on the entities present in the same texts. Such divergence is particularly present in texts from the media domain that contain references to creative works. In this work we present a corpus of 1000 annotated documents selected from the media domain. Each document is presented with multiple gold standard annotations representing various annotation styles. This corpus is used to evaluate a series of Named Entity Linking tools in order to understand the impact of the differences in annotation styles on the reported accuracy when processing highly ambiguous entities such as names of creative works. Relaxed annotation guidelines that include overlap styles lead to better results across all tools.

## introduction identifying all entities from texts and linking them to modern knowledge graphs (kgs) like dbpedia and wikidata is the core task of named entity linking (nel) @xcite . a particular challenge in nel is disambiguating the entity references in the surface form (original text). for example, mentions such as nyt or ny times or even times could all be surface forms referencing the entity the new york times and they can then be linked to the associated entries from wellknown public kgs like dbpedia and wikidata (i.e., dbr:the_new_york_times or wd:q9684 foot_0 ). in ad-dition to entities, the relations between them, as defined in the texts themselves or extracted from a kg, can also help enlighten the narratives. all of the nel related tasks are supported by large communities which have a vested interest in advancing their knowledge extraction (ke) capabilities. the various good results for general nel benchmarks (e.g., f1 scores of 0.80 @xcite ) are encouraging and suggest that the tasks tested for in these benchmarks may be solved soon. however, the main focus of most reported evaluations to date has been on the correct annotation of several specific types of entities like persons (per), locations (loc) or organizations (org). sometimes fine-grained typing was also allowed for locations, therefore allowing us to use types like natural locations (loc) to depict naturally occurring locations like mountains or rivers, facility (fac) to describe buildings or infrastructure like bridges or airports and geo-political entities (gpe) to describe cities, region or countries @xcite . only in the last half-decade the focus has slowly shifted towards expanding the typing to encompass a wider array of types, from dates and products to games, sports, books or movies. a large subset of the fine-grained types of entities found naturally in texts, especially in the media domain, are focused on what can generically be called works or sometimes creative works and which would encompass all creative works (manifestations of creative effort by a creator or group of creators), from books or songs, to movies, television episodes and video games, for example. collective works with unknown authors (e.g., religious works, folklore), festivals, concerts or sports events can also be considered creative works, even though they may be labelled differently (e.g., a concert might be labelled as an event, whereas its audio or video recording could be labelled as a creative work). the current generation of automated annotators provide a rather mixed set of results for mediarelated annotations due to inconsistent annotation guidelines related to this domain in the past. for example, a tv show like star trek: deep space nine will either get no result or two different results, one marking the franchise (star trek) and the second the titular space station (deep space nine), but will probably not be annotated with the full correct entity (star trek: deep space nine) by many automated annotator tools. such an annotation could alternatively be considered correct in our view, if instead of the space station, the second annotation would point to the tv show itself. of course, if we examine the characters from the respective tv shows, we can run into similar problems, as some characters might be described by multiple resources, each of them highlighting a particular performance (e.g., james bond is described both as literary characterdbr:james_bond_(literary_character) -and person dbr:james_bond). in order to counteract such issues, we have developed a corpus focused around annotating such media-related entities. we share this corpus, together with the corresponding annotation guidelines, as well as a set of alternative annotations, especially for nested entities foot_1 . since we plan to continuously update this corpus with new partitions every few months, one of its core concepts is the fact that for each ten partitions created there will be a summary partition, a partition that will combine annotations similar to those from the previous nine partitions. if users aim at performing a quick evaluation, they will be able to use this summary partition which we call 'core set' in order to have a first set of results and decide upon their preferred fine-tuning procedure. the expression media annotations is typically used to refer to video or audio annotations, therefore in the context of this paper we have selected the expression media entity annotations to signify the annotation of texts created and/or published by various media conglomerates which contain entities of type creative work. we consider our work as an important step in the road towards building culturally-aware ais since a significant part of human culture is represented by the creative works humanity has produced, ais are needed that can correctly annotate and interpret information about the referenced creative works. the rest of the paper is organized as follows: section 2 showcases somewhat similar work and articles that discuss some of the ideas that we followed during our design process; section 3 describes the design of the corpora; section 4 evaluates several tools on some of the partitions included in our corpora; whereas the final section provides a reflection upon the insights gained from designing a gold standard that focuses on creative works. ## related work since this dataset uses several concepts that are not widely used in nel evaluation datasets (e.g., lenses, core sets, multiple annotation styles), this section also includes explanations for those concepts. after examining a rich set of media annotation papers we have discovered that very few papers actually discuss how to correctly annotate media assets with creative work entities. early results were focused on the correct semantic annotation of text extracted from the multimedia content (e.g., transcription or subtitles), as presented in a set of papers that originated from the linkedtv project like @xcite , but using the classical entity types well supported by nel tools (person, organization, location). more recent media annotation work discussed cross-modal annotations @xcite and story identification @xcite . none of these projects or papers focused on the correct identification of creative works or the various naming variants involved in creative works referencing. an episode name like the trouble with tribbles can be an episode of star trek, but also an instance of a chapter, comic book or book in the same franchise. even though the existing research would help us annotate the respective episode's video, it will not really help us contextualize it, if mentioned in a press release, for example. due to the shortcomings of current approaches, we have also examined several other avenues, including corpora that contain creative works or historical entities that might otherwise be difficult to identify by the general public and were created for different purposes but in somewhat similar situations (e.g., no corpora available, specific domain, etc). the digital humanities domain, and in particular historical documents and literary criticism, are perhaps the closest scenarios to our use case that come to mind. several works in this domain like the historical documents from the impresso collection @xcite , the multilingual news corpora meantime @xcite and dekker's work on extracting small snippets of literary criticism from social media @xcite have served as a starting point in our journey, helping us to define our annotation guidelines. some early ideas about naming variations (in particular @xcite , @xcite and @xcite ), and nested annotations ( @xcite and ( ji et al., 2017) ) have also shaped our understanding of the difficulties of correctly capturing names referencing entities, regardless of the domain, and have led us to multiple works on annotation styles and lenses. the idea of data lenses comes from multiple places, but it was generally inspired by photography where different lenses are used in order to get different views on an object. in nel, and in nlp in general, the idea is to enable different views onto the same dataset. during the last few years, lenses have increasingly been used to transform data between different views @xcite . in the nlp domain, lenses have traditionally been implemented as annotation sets that reflect the view of a human annotator upon the specific data, being an important feature of annotation packages like gate @xcite , as well as for implementation of large parallel corpora typically used for multilingual settings (e.g., @xcite . in the semantic web (sw), ontologies have traditionally been used to act as lenses over data eventually leading to an entire field of study: ontology-based data access @xcite . more recently, lenses have been used to create multiple views over chemistry data @xcite , help manage large data sets @xcite , or understand big data and ai workloads @xcite . with respect to entity linking, an early example of lenses was the concept of approximate matching implemented in tools like neleval @xcite and gerbil @xcite . the computation of partial matches was done based on the number of overlapping characters or entity types with respect to the measures that can be optimized (e.g., precision or recall). many optimization strategies will fail to correctly identify either connections between entities (e.g., parent) or nested entities. what is missing and why, can generally be discovered during the error analysis phase of the evaluations, as shown in @xcite and @xcite . understanding error classification schemes like the ones described in these publications can be the key towards moving the field forward. more recently, lenses have been used to understand entity annotations @xcite and to study the effect of relaxed annotation schemas that include more types than usual @xcite . a theoretical treatment of automatic procedures for building lenses is presented in @xcite . the paper showcases how to automatically build lenses by following simple rules (e.g., expand the surface form to the maximum length -the longest mention -or reduce the surface form to the minimal possible length -the shortest mention). using the longest mention leads to an annotation schema with less entities, whereas using the shortest mention will lead to more entities being annotated. intermediary schemas are also possible, if we take into account the possibilities of combining the previous schemas or if we add additional rules. we test some of the lenses discussed in this paper, as it can be seen in section 3. our in media res corpora contains partitions from multiple domains. due to this aspect, it was decided to introduce core sets partitions at every nine partitions. by adding these core sets, evaluators can later build a summary dataset for the whole corpus, regardless of the number of partitions contained in it. the concept of core sets was borrowed from computational geometry and robotics. a core set is a small engineered subset of a very large dataset that retains its properties as accurately as possible (e.g., a set of points that approximate the shape of a figure). its size typically depends on the desired accuracy, not necessarily on the size of the original data set. core sets have been successfully applied to a variety of problems, from dimensionality reduction of massive data sets @xcite , to vector summarization @xcite or compression of neural networks @xcite . we, therefore, started our corpus design by testing various annotation styles and creating an annotation guideline for creative work entities. after we identified a solution, we created multiple partitions from various domains. the name of the corpus comes from the latin expression in media res that refers to narratives (e.g., books or movies) that start in the middle of the story. similarly, the created corpus has been designed with the aim of understanding different annotation styles, and rather than starting from zero builds upon prior research in this area. also, the fact that most of the annotated documents are related to media (e.g., franchises, books, tv shows) has been one of the reasons why we have selected this name. ## evaluation this section describes the tools used during the evaluation, its design and a discussion around results. ## conclusion and future work the road towards designing culturally-aware ais has started with the expansion of fields like digital humanities during the last decade. while some steps towards this goal were made, the selected topics still depend on funding and on the researcher's own goals, as this is a relatively new field. even so, we find it surprising that there was a lack of guidance in the nel community related to name variation and nested entities for creative works. the corpus and concepts introduced in this paper are a first attempt to address this issue. the results of the various annotators on the in media res corpus are understandably lower than on corpora from traditional domains such as news articles and social media. while applying different annotation lenses improves the results for some annotators, it is clear that there is a need for more progress in this area. we plan to continue maintaining the corpus and provide updates, new partitions, or lenses. we hope that these efforts will contribute to increased accuracy in the annotation of creative work entities and, therefore, aid annotation systems in taking a further step towards culturally-aware ais.