Comparative evaluation of boundary-relaxed annotation for Entity Linking performance (2023)

Abstract: Entity Linking performance has a strong reliance on having a large quantity of high-quality annotated training data available. Yet, manual annotation of named entities, especially their boundaries, is ambiguous, error-prone, and raises many inconsistencies between annotators. While imprecise boundary annotation can degrade a model’s performance, there are applications where accurate extraction of entities’ surface form is not necessary. For those cases, a lenient annotation guideline could relieve the annotators’ workload and speed up the process. This paper presents a case study designed to verify the feasibility of such annotation process and evaluate the impact of boundary-relaxed annotation in an Entity Linking pipeline. We first generate a set of noisy versions of the widely used AIDA CoNLL-YAGO dataset by expanding the boundaries subsets of annotated entity mentions and then train three Entity Linking models on this data and evaluate the relative impact of imprecise annotation on entity recognition and disambiguation performances. We demonstrate that the magnitude of effects caused by noise in the Named Entity Recognition phase is dependent on both model complexity and noise ratio, while Entity Disambiguation components are susceptible to entity boundary imprecision due to strong vocabulary dependency.

## introduction out of many tasks under the natural language processing (nlp) umbrella, entity linking (el) is one of the critical steps for information extraction, allowing the retrieval and understanding of information from unstructured textual sources. by definition, el is the task of associating a named entity (ne) mention (a concept, person, location, etc.) to a unique identifier that describes what that mention refers to in a knowledge base (kb) @xcite . this is especially significant in the biomedical domain, where it can be used to asso-ciate mentions with different surface forms to a single concept in an ontology, such as the unified medical language system (umls) foot_0 , allowing for better indexing and relation extraction @xcite . in this paper's definition, el comprises two phases, as presented in figure 1 . named entity recognition (ner) searches the sentence for ne mentions, and then entity disambiguation (ed) assigns a unique identifier for each one. @xcite . ner is also commonly related to classifying a detected mention into a set of categories. however, for this study, we consider it as only the location of a ne in the text. although some methodologies have presented state-of-the-art (sota) results for this task @xcite , large amounts of high-quality labeled data are still crucial to achieve good performance models. yet, text annotation is no trivial task. manual annotation of vast data is costly due to the excessive workforce and long hours required. besides, it is arduous to guarantee good quality and standardized annotation. notably, when annotating nes, annotators can face trouble deciding on the boundaries of a mention, which can be ambiguous and raises many inconsistencies @xcite . for example, in the sentence presented in figure 1 , arthritis can be referred to as "joint inflammation". however, a longer span "joint inflammation pain" is also applicable when referring to the same manifestation, even though it packs some peripheral information. deciding where the boundary line of entity or non-entity should be drawn is not straightforward, and rigorous annotation guidelines can add even more bureaucracy to an already slow process. @xcite . although imprecise boundary annotation can be considered a type of noise that degrades a model's performance, some applications, especially those in the biomedical field, are not dependent on the exact span of an entity @xcite . due to the existence of normalized vocabularies, most extracted data can be refined to helpful information by linking to such kbs. in this scenario, the burden on annotators could be relieved by loosening the strictness of entity boundary annotation and, instead, focusing on the complete annotation of the conceptual information. we hypothesize that as long as the detected mention is longer than the target mention, in other words, it contains "more information than necessary", the concept matching to a kb should not be greatly degraded. this approach can potentially minimize the effort during corpora annotation process and improve its speed. to validate the hypothesis, we present a case study to evaluate the impact of boundary-relaxed annotation on ner and ed performance. we generated multiple variants of a popular el dataset (aida conll-yago) by expanding subsets of annotated entities' boundaries. we then train two sota models and our baseline model on the modified dataset and compare it with the original dataset performance on each step of the el pipeline. ## related work entity boundary is a common theme in nerrelated research. some evaluation criteria that relax the strictness of boundary matching have been proposed and refined through the years. @xcite compares a few of those strategies, such as matching only the left or right border, any tag overlap, per-token measurements, and semanticallydriven matching. the authors also present potential use cases for each evaluation technique and compare their characteristics. while several studies evaluate the impact of an-notation quality in el @xcite @xcite , others tackle the issue of noisy-labeled data @xcite , which may include but it is not limited to imprecise entity boundary. most of the works that address the ne boundary problems attempt to improve the detection precision @xcite @xcite , proving that better entity boundary detection dramatically improves both entity classification and linking. however, there are a handful of studies that specifically address the impact of relaxed entity boundary annotation on linking performance. @xcite compared the impact of two different annotation guidelines for disease names on model performance, noting that the set of rules that provide low lexical variability, short entity length, and high regularity usually impact performance positively. as a sideline experiment to the main study, @xcite analyzed the impact of expanded entity boundaries and the use of the additional information as context around the annotated expression. ghiasvand and kate (2018) developed an unsupervised method for training clinical ner systems that automatically generate ne annotations. due to the high surface form variability of clinical concepts, a ne boundary determination component was developed. with the ablation of this component, the authors perceived around a 5% decrease in ner precision and recall performance. @xcite propose a boundary regularization technique that reallocates part of the probability to be an entity from an annotated span to its neighbor words. this effectively creates a smooth transition between an entity annotation and its nonentity surroundings to mitigate annotation boundary inconsistencies. ## method we performed a case study to assess the effect of relaxed annotation on the performance of each phase of the el pipeline. figure 2 depicts a representation of our workflow. the designed method aims to simulate model training using a dataset compiled by employing a hypothetical lenient and inclusive entity annotation guideline, where an annotator would roughly select the correct span of the entity mentions, possibly introducing a few unnecessary extra tokens into the annotation when in uncertainty. in particular, we focused on observing the impact of expanded entity annotations on evaluation metrics commonly used in such tasks. ## experimental setup dataset as our dataset, we use aida conll-yago @xcite . this dataset is an extension of the well-known conll-2003 dataset @xcite , and it is vastly used for el benchmarks. it comprises newswire articles from reuters, containing 1,393 articles with 27,820 ne mentions linkable to its corresponding yago2 id and wikipedia article webpage. training and validation sets contain 946 and 216 articles, respectively. test set has 231 articles. models we executed our experimental procedure in three el pipelines. we selected two sota approaches and also developed our own method: an end-to-end model for autoregressive entity linking available under mit license. it comprises a longformer encoder, a mention detection module including feed-forward neural networks, and an ed component based on a long short-term memory network for candidate generation and ranking. the whole model has a total of 202m parameters. as reported by the authors, this model achieves an f-score of 0.85 for entity linking on the aida conll-yago dataset. even though the authors present it as an end-to-end model, the ner and ed modules are separated components. therefore, we could compute metrics for each step individually. while available in multiple variants, in this study we use luke (base), which has a total of 253m parameters. in the ner task it achieves f-score of 0.94, and 0.95 for ed on the aida conll-yago dataset. since this framework is composed by modules that accomplish different tasks, we can independently create an el pipeline using the available ner and ed components. as reported in the original paper @xcite , one of luke's ed model limitations is not being able to handle out-of-vocabulary entities, we replace expanded entities in the model's vocabulary file before training. in case of an entity being expanded differently in two or more instances, we use the shorter of all variants. our developed ner model is intended to be used as a baseline for comparison. this model aims to simulate a simple and low-effort approach. we start from a pre-trained bert-base-cased model @xcite , which has around 110m parameters, and fine-tune it using our dataset instances. the base model is publicly available in the hugging-face foot_4 platform, under apache license 2.0. as we only developed the ner component, we feed the output of our model to luke's ed model, trained on the same dn (l,i) instance, to compute disambiguation scores. training parameters we used a server comprised of two nvidia quadro rtx 8000 gpus for model training. we report all used parameters and computational time needed in appendix a.3. for both sota models (1 and 2), we use the implementation released by the authors and also apply the default training parameters disclosed in the referring papers. ## results and discussion in this section, we present the experimental results of our case study. appendix a.4 presents a summary of the computed evaluation metrics after the processing of the test set by the trained models in ner and ed phases, respectively. we purposefully omit some noise levels for easier visualization, presenting some checkpoints to allow comparison. ## conclusion in this study, we investigated the impact of relaxed entity boundary annotation on model performance by conducting a case study with three different el approaches. we introduced noise into a popular dataset by expanding the boundaries of some annotated entity mentions, trained models on the produced noisy data, and evaluated their output. for ner components, we verified that model complexity plays a significant role in how its influenced by noise. while complex models may achieve higher performance, they had reduced flexibility to adapt to the noisy environment, thus being outperformed by our simpler approach in this scenario. we also noted that those models are resilient against the boundary-expanded environment until more than half of the training annotations are "noisy". we also observed that ed components are sensitive to imprecise entity mentions and strongly reliant on their training vocabulary. we show that both ed approaches evaluated can still correctly predict identifiers even when trained in noisy environments, given that the detected mention boundaries are precise. on the other hand, when boundary imprecision produces out-of-vocabulary mentions, it hinders performance heavily.