Enriching the Metadata of Community-Generated Digital Content through Entity Linking: An Evaluative Comparison of State-of-the-Art Models (2024)

Abstract: Digital archive collections that have been contributed by communities, known as community-generated digital content (CGDC), are important sources of historical and cultural knowledge. However, CGDC items are not easily searchable due to semantic information being obscured within their textual metadata. In this paper, we investigate the extent to which state-of-the-art, general-domain entity linking (EL) models (i.e., BLINK, EPGEL and mGENRE) can map named entities mentioned in CGDC textual metadata, to Wikidata entities. We evaluate and compare their performance on an annotated dataset of CGDC textual metadata and provide some error analysis, in the way of informing future studies aimed at enriching CGDC metadata using entity linking methods.

## introduction community-generated digital content (cgdc) pertains to digital-born archive collections that have been developed by communities. in the uk, for instance, libraries and museums such as the morrab library in cornwall foot_0 and the sherborne museum in dorset foot_1 employ volunteers to catalogue their archive collections, consisting of historic photographs and papers, respectively. since 1994, the uk national lottery heritage fund has awarded grants to around 5000 community history projects, foot_2 leading to the proliferation of cgdc. encapsulating the collective experiences and narratives of communities over time, such collections serve as indispensable sources of knowledge, offering a window into the past that deepens our understanding of human history and culture @xcite . however, despite their important role in enhancing people's appreciation of their museums-libraries-and-archives heritage, cgdc items remain hard to find @xcite . this can be attributed to the fact that semantic information on cgdc items tends to be buried within their textual metadata, e.g., titles and descriptions, making it difficult to search for and link items related to a given query. such a challenge can be potentially overcome by enriching cgdc metadata using natural language processing (nlp) methods. as a first step, for example, named entity recognition (ner) can be employed to automatically label the names of any entities mentioned within a piece of text @xcite . this is often followed by entity linking (el), a task concerned with normalising name variants (e.g., the canonical and vernacular names of a place) to the same real-world entity @xcite ; typically, this is implemented as a disambiguation task where a unique identifier (denoting an entity) used within a knowledge base, is assigned to a given named entity. in this paper, we focus on assessing the performance of state-of-the-art el models on cgdc textual metadata. these models have demonstrated impressive el performance in the general domain, e.g., on the task of wikification which involves linking entities within text to wikipedia @xcite . cgdc metadata, however, are not as well-formed as general-domain texts such as news articles, mainly due to the fact that there are no established standards that require communities to write their textual metadata in a consistent way. for instance, many cgdc descriptions are short, consisting only of phrases rather than full sentences; misspellings and obsolete names are also commonplace. we thus aim to evaluate how well existing state-of-the-art models perform on cgdc textual metadata and analyse cases on which these models tend to fail. this will help researchers working in the areas of digital humanities and cultural analytics in identifying ways on how existing el approaches can be adapted or optimised for cgdc. to the best of our knowledge, ours is the first work to explore el for cgdc. the handful of efforts that employed el on archive collections focussed mostly on historical newspapers @xcite @xcite , specific centuries @xcite or events such as the second world war @xcite , but not on cgdc. ## dataset to support our evaluation of state-of-the-art entity linking models, we set out to develop our own annotated dataset of cgdc textual metadata. ## problem formulation and models we first provide a formal definition of the el task: given a target knowledge base containing a set of entities e and a textual document in which a set of named entities n have been identified, an el model maps each @xmath0. if the entity that n corresponds to does not exist in e, then n is considered to be unlinkable and is thus linked to a nil entity. in our work, the context in which n appears is also provided as input (together with n itself) and the target knowledge base is wikidata. three state-of-the-art el models were investigated in this study. for a given named entity (ne), each of the models predicts the best matching entity in the knowledge base that it should be linked to by specifying its identifier (id) together with a similarity score, if it is linkable; otherwise, it is linked to nil. blink. this model @xcite employs bert-based architectures @xcite for two subtasks: retrieving candidate entities by encoding the context containing an ne and the definitions of candidate entities in wikipedia, and ranking the candidates. its predicted wikipedia ids are mapped to wikidata ids. ## results and discussion the models were applied to the gold standard person, organisation, location and miscellaneous nes in our cgdc test set, i.e., the morrab and pcw subsets. it is worth noting that we utilised each of the chosen el models out-ofthe-box, i.e., without extending their functionality. all models make use of the text span of a given ne in their analysis, but none of them consider the ne type as a feature, although it is available as part of the input to el. person nes that contain only one token; such nes were automatically given nil as their id, as our preliminary experimentation with the held-out dataset showed that the three el models are unlikely to be able to correctly disambiguate them. ## conclusion and future work in this paper, we present the results of evaluating blink, epgel and mgenre-three of the stateof-the-art, general-domain entity linking modelson an annotated dataset of cgdc textual metadata. our evaluation shows that mgenre obtains superior performance overall and on unlinkable (nil) named entities more specifically, which tend to be prevalent in cgdc. our future work will focus on handling cases that make cgdc textual metadata particularly challenging, e.g., lesser-known named entities and obsolete names, and on combining the strengths of epgel and mgenre in predicting ids for linkable and unlinkable named entities, respectively. furthermore, we will investigate how the performance of these models can be enhanced by making them leverage the ne type of any given named entity as a semantic feature that can inform the el process.