Assessing the Limits of Straightforward Models for Nested Named Entity Recognition in S panish Clinical Narratives (2022)

Abstract: Nested Named Entity Recognition (NER) is an information extraction task that aims to identify entities that may be nested within other entity mentions. Despite the availability of several corpora with nested entities in the Spanish clinical domain, most previous work has overlooked them due to the lack of models and a clear annotation scheme for dealing with the task. To fill this gap, this paper provides an empirical study of straightforward methods for tackling the nested NER task on two Spanish clinical datasets, Clinical Trials, and the Chilean Waiting List. We assess the advantages and limitations of two sequence labeling approaches; one based on Multiple LSTM-CRF architectures and another on Joint labeling models. To better understand the differences between these models, we compute task-specific metrics that adequately measure the ability of models to detect nested entities and perform a fine-grained comparison across models. Our experimental results show that employing domain-specific language models trained from scratch significantly improves the performance obtained with strong domain-specific and general-domain baselines, achieving state-of-the-art results in both datasets. Specifically, we obtained F1 scores of 89.21 and 83.16 in Clinical Trials and the Chilean Waiting List, respectively. Interestingly enough, we observe that the task-specific metrics and analysis properly reflect the limitations of the models when recognizing nested entities. Finally, we perform a case study on an aggregated NER dataset created from several clinical corpora in Spanish. We highlight how entity length and the simultaneous recognition of inner and outer entities are the most critical variables for the nested NER task.

## introduction named entity recognition (ner) is a widely studied task that seeks to identify text spans associated with predefined categories. nested named entity recognition is a particular case of ner, where entities can be nested within each other (finkel and manning, 2009), such as the example in figure 1 . traditional ner models simplify nested entities through predetermined rules, such as keeping the most external entity and ignoring inner ones. this simplified problem is better known as flat ner and allows solving the task using traditional sequence labeling architectures such as the bilstm-crf @xcite approach or fine-tuning transformer-based models @xcite . regarding the spanish language, there are several biomedical and clinical datasets containing nested entities, such as the spanish radiology corpus @xcite , nubes @xcite , the chilean waiting list @xcite , clinical trials @xcite . however, most previous works transformed the task into a flat ner. as mentioned in @xcite , this simplification is due to technological rather than ideological reasons, mainly explained by the difficulty of representing nested entities with the traditional annotation scheme, for example, with the iob2 sequence labeling format. we argue that treating the nested ner task as flat ner is not optimal since removing part of the entities could result in a loss of information previously annotated by humans, wasting time and resources, and harming the model's performance. this paper explores simple neural network-based models as a proxy to address the challenging nested ner task. specifically, we revisited the multiple lstm-crf (mlc) and the joint labeling architectures and performed experiments on two spanish clinical corpora. the former consists of training a flat ner model for each entity type following the iob2 format, while the latter transforms the nested ner task into a flat ner using an annotation scheme that allows preserving the nested entities. we analyze the impact of using pre-trained language models trained on specific domains compared to general-domain ones. to evaluate the performance of our models, we provide a detailed analysis of task-specific evaluation metrics that adequately measure the effectiveness of the models in recognizing nested entities, considering variables such as entity length, the nesting depth level, and the different types of nested entities. in addition, to better understand the limitations of these models, we created an aggregated corpus formed from several spanish clinical ner corpora. in summary, the main contributions of our work are the following: ## related work the nested nature of named entities has recently gained special attention from the nlp research community. several models have been proposed to handle the nesting problem, which can be mainly divided into three categories: region-based, hypergraph-based, and sequence labeling-based models. region-based models list potential span candidates and then classify them into predefined categories. in sohrab and miwa (2018), they used an exhaustive neural model enumerating all possible spans within a limited length and then predicted the entity types of those regions using boundary and average internal token representation. @xcite used a sequence labeling layer to identify candidate spans and then classified the selected regions into their entity category labels. another region-based model was proposed by @xcite , who used contextual representations models to encode sentences and two separate mlps to create start and end token representations. they then ranked all possible start-end regions in the sentence using nested constraints to predict the labels. recently, @xcite used a two-stage identifier, using a filter and a regressor to identify high-quality candidate spans and then classifying them into their entity types. hypergraph-based models learn the nested structure of entities in the sentence through hypergraphs. the aim is to capture the relations between inner and outer entities to leverage the extraction of nested entities. in @xcite , they proposed a mention hypergraph representation for both extracting entity boundaries and predicting entity labels. similarly, @xcite designed a directed hypergraph using lstm features to learn the nesting structure. in luo and zhao (2020), they used a flat ner module for recognizing the most external entities and a graph module for inner entities. sequence labeling-based models formulate the nested ner task as several flat ner models. early work from @xcite introduced three crfbased methods to reduce the nested ner as several iob2 tagging problems. @xcite took advantage of inner entity information to improve outer entity recognition. they dynamically stacked lstm-crf layers predicting entities in an insideto-outside manner. in contrast, shibuya and hovy (2020) recognized entities from outermost to inner ones using a recursive method based on separate crfs. this method was improved in @xcite , demonstrating that inner to outermost recognition is best for modeling this task. finally, @xcite recursively introduced the embedding of tokens and regions into flat ner layers simulating the shape of a pyramid and extracting nested entities from the innermost to the outermost entities. the models used in our experiments fall into this category. ## nested ner models in recent years, contextual representational models have improved the performance of many neural network-based models, making it possible to achieve state-of-the-art in several nlp tasks. unlike traditional word embeddings, language models can represent words according to the sentence-level context. regarding the ner task, using contextual word embeddings or fine-tuning a pre-trained language model to a specific domain has boosted the performance of models in datasets from several domains. previous work in clinical ner showed that using domain-specific language models improves results considerably compared to general-domain language models. however, no studies show this behavior occurs when there is a nested structure in the entities, especially in low-resource languages such as spanish. in this work, we study whether this trend is confirmed in nested ner datasets using two sequence labeling-based architectures, the joint labeling, and the multiple lstm-crf models. ## experiments in this section, we present the datasets, settings, and evaluation metrics used in our experiments. ## overall results although the mlc and joint labeling architectures appear to be simple approaches for solving nested ner, we observe that their results are pretty high. specifically, the best setting for the chilean waiting list corpus is the mlc model with embeddings retrieved from the clinical flair model. using the same data splits, we obtained state-ofthe-art results with an improvement of almost three micro f 1 points over the best system to date, as reported in @xcite , where they achieved a micro f 1 score of 80.27. this excellent performance could be explained since clinical flair is a character-level language model, particularly beneficial in datasets with many misspelled and out-ofvocabulary words, such as diagnoses from public hospitals. on the other hand, the best setting in clinical trials is the joint labeling approach with the clinical version of roberta. to date, the only result reported on campillos-llanos et al. (2021) achieved a micro f 1 score of 86.74 without considering the nested entities. in contrast, we obtained a micro f 1 score of 89.21, achieving state-of-the-art in the corpus and demonstrating the importance of considering nested entities. another point to analyze is the multilabel entities. these entities correspond to text spans associated with more than one entity type, as in the case of the medical term htn, which is both an abbreviation and disease. in the chilean waiting list corpus, 1, 030 entities participate in this type of nesting. considering only the f 1 score of both models on these types of entities, the mlc approach with clinical flair obtained 85.1, while joint labeling with clinical roberta obtained 84.21. therefore, the difference in the standard metric cannot be explained by the performance of these types of nestings. in the following sections, we perform a detailed analysis of the model predictions, looking for information that explains the difference in performance between the joint labeling and mlc approaches beyond the domain in which they were trained. ## case study in order corroborate the conclusions presented above, we have created a synthetic nested ner corto generate the aggregated dataset, some important factors have been considered. first, codiesp is not a ner task but a clinical coding task. however, the authors annotated not only the icd-10 codes but also the textual evidence that supports the assigned codes. for this experiment, we used the textual evidence from codiesp as if they were named entities. secondly, we have found that some textual evidences are either discontinuous or partially contained within other evidences, better known as crossing entities. both cases are beyond the scope of this research, so we decided to discard them. thirdly, distemist is an ongoing task, and we do not have access to the test set annotations. for this reason, we only have annotations for 750 clinical cases of the spaccc corpus. finally, once the three datasets were aggregated, we found that there were discontinuous annotations between codiesp and distemist in 17 of the documents. we removed these documents from the corpus, leaving us with 733 documents. the dataset was divided into 80% for training, 10% for validation, and 10% for testing. the statistics of the corpus are shown in according to the standard ner metric, the results for the mlc and joint labeling approaches are 78.53 and 78.25, respectively. although the performance was comparable between both models, analyzing figure 4 , we note that the behavior in the two previous datasets is repeated. the mlc curve is higher than the joint labeling curve for the smallest entities, but as the number of tokens increases, the joint labeling model obtains slightly better results than mlc. ## conclusions and future work since most previous works on nested ner have focused on solving the task in english, this paper contributes to the exploration of diverse models for solving the task on two spanish clinical datasets, resulting in the state-of-the-art in both corpora. specifically, we explore the advantages and limitations of the multiple lstm-crf approach, which consists of training one model for each entity type, and the joint labeling approach, which through an appropriate annotation scheme, allows solving the task by fine-tuning transformer-based models. to assess the limitations, we studied taskspecific metrics for the nested ner task, which consider variables such as the entity position in the nesting, the impact of nesting depth, and entity length. although our approaches achieve high results according to the standard metric, we found limitations concerning the recognition of nested entities. the main drawbacks of these architectures are the low performance when recognizing complete nestings and the outermost entities of a nesting. in addition, the mlc approach combined with a character-level language model performs less when recognizing entities with many tokens. we believe this work can contribute to the nlp community to re-think how the nested ner task is being evaluated, considering task-specific metrics beyond the traditional micro f 1 score. furthermore, our case study on the spaccc aggregated dataset points out many of the challenges of the nested ner task, especially when complex annotations are allowed due to the aggregation pro-cess. therefore, future work will analyze the performance of other existing architectures beyond the sequence labeling-based approach and compare their performance against our models. we also plan to propose new methods to treat the cases of discontinuous entities and crossing entities, which are entities that overlap others but are not fully contained, to address the nested ner task fully.