Sentence-Level Resampling for Named Entity Recognition (2022)

Abstract: As a fundamental task in natural language processing, named entity recognition (NER) aims to locate and classify named entities in unstructured text. However, named entities are always the minority among all tokens in the text. This data imbalance problem presents a challenge to machine learning models as their learning objective is usually dominated by the majority of non-entity tokens. To alleviate data imbalance, we propose a set of sentence-level resampling methods where the importance of each training sentence is computed based on its tokens and entities. We study the generalizability of these resampling methods on a wide variety of NER models (CRF, Bi-LSTM, and BERT) across corpora from diverse domains (general, social, and medical texts). Extensive experiments show that the proposed methods improve span-level macro F1-scores of the evaluated NER models on multiple corpora, frequently outperforming sub-sentence-level resampling, data augmentation, and special loss functions such as focal and Dice loss.

## introduction in natural language processing, named entity recognition (ner) is an important task both on its own and for numerous downstream tasks such as entity linking and question answering. ner has an inherent data imbalance problem: named entities of interest are almost always the minority among irrelevant (other type) tokens in a text corpus. data imbalance is even more severe in real-world bespoke ner tasks, which directly motivated this work. for example, given full-text articles from a medical subfield, domain experts may wish to recognize only those concepts related to specific aspects of the subfield (e.g., symptoms and medicine related to a specific disease). compared to all tokens in the full text, extremely few tokens are annotated with any entity type. because domain experts have limited availability, annotated corpus are usually small in such tasks. as a result, some rare entity types may have less than 10 tokens across the corpus. such severe data imbalance and scarcity makes many ner models suffer. data imbalance in ner challenges machine learning-based models because their learning objective is dominated by entities of the majority type (other), causing the model to be reluctant to predict the types of interest. various techniques have been studied to tackle this challenge. active learning was applied to collect a more balanced dataset at annotation time @xcite . special loss functions including focal loss @xcite and dice loss @xcite are proposed to deal with data imbalance. data augmentation was shown to be effective by enriching entity-bearing sentences through methods like segment shuffling and mention replacement @xcite @xcite . the classical method for alleviating data imbalance is resampling (upsampling the minority class or downsampling the majority class) and its close relative, cost-sensitive learning (assigning larger weight to the minority class or smaller weight to the majority class in the learning objective) @xcite . a natural question is: can we apply resampling to address the data imbalance problem in ner? it turns out that unlike classification tasks, applying resampling to sequence @xcite ; gmb subset @xcite ; anem @xcite ; cadec @xcite ; conll @xcite ; n2c2 ade @xcite ; ontonotes @xcite . 'sent.' = sentences; 'freq.' = frequent. in this paper, we propose sentence-level resampling methods for ner, an underexplored problem in this area. as sentences are the natural units of data in ner, sentence-level resampling leaves the contextual information intact in a natural sentence needed by deep models like transformers. since a sentence may contain a mixture of entities whose types have different levels of rareness, traditional resampling method for imbalanced classification (e.g., inverse probability resampling) cannot be applied. instead, we develop a set of methods for computing integer-valued importance score for a sentence based on its entity composition, and resample the sentence accordingly. experiments show that our methods can improve performance of a variety of ner models and are especially effective on tasks with small annotated corpora, which is often seen in real-world bespoke ner tasks. ## resampling strategy design for a sequence tagging task like ner, resampling cannot be as simple as what it is in classification and regression tasks, in which data points can be individually replicated, discarded, or synthesized. in ner, named entities cannot be resampled out of context. the surrounding context of named entities -albeit tokens from the irrelevant other typeshould be considered as well. resampling named entities with context is a double-edged sword: preserving context will help ner models, but too much context increases the amount of non-entity tokens and aggravates the data imbalance problem. the goal of sentence-level resampling is to find the balance between too little and too much context accompanying named entities in complete sentences. ## experimental evaluation resampling should be a domain-and modelagnostic strategy in tackling data imbalance. therefore, the goal of our experiments is to evaluate if the proposed resampling methods are effective in an extensive array of ner corpora and base models. towards this goal, we apply the four resampling methods (together with baseline methods) on three representative ner models (each has two variants), 2154 and evaluate the resulting models on four corpora from diverse domains. ## conclusion and future work our proposed sentence resampling methods generalize well across diverse ner corpora and models. they enjoy the following advantages: model-agnostic: since resampling only manipulates datasets and not models, the proposed methods can be directly applied to any ner model, requiring no knowledge of its functioning or any change to it. resampling is also more convenient than cost-sensitive learning as the latter still requires changing the model training process.