Continual Named Entity Recognition without Catastrophic Forgetting (2023)

Abstract: Continual Named Entity Recognition (CNER) is a burgeoning area, which involves updating an existing model by incorporating new entity types sequentially. Nevertheless, continual learning approaches are often severely afflicted by catastrophic forgetting. This issue is intensified in CNER due to the consolidation of old entity types from previous steps into the non-entity type at each step, leading to what is known as the semantic shift problem of the non-entity type. In this paper, we introduce a pooled feature distillation loss that skillfully navigates the trade-off between retaining knowledge of old entity types and acquiring new ones, thereby more effectively mitigating the problem of catastrophic forgetting. Additionally, we develop a confidence-based pseudo-labeling for the non-entity type, i.e., predicting entity types using the old model to handle the semantic shift of the non-entity type. Following the pseudo-labeling process, we suggest an adaptive re-weighting type-balanced learning strategy to handle the issue of biased type distribution. We carried out comprehensive experiments on ten CNER settings using three different datasets. The results illustrate that our method significantly outperforms prior state-of-the-art approaches, registering an average improvement of 6.3% and 8.0% in Micro and Macro F1 scores, respectively.

## introduction named entity recognition (ner) is a essential research area in natural language understanding (nlu). its purpose is to assign each token in a sequence with multiple entity types or non-entity type @xcite . recently, the advent of pre-training language models (plms) @xcite has ushered ner into a current task t-2 t-1 t+2 inputs: bin will travel to ximending in taipei in july cl: [o] [o] [o] [o] [o] [o] [gpe] [o] [o] fl: [per] [o] [o] [o] [loc] [o] [gpe] [o] [date] t t+1 fl: [per] [o] [o] [o] [org] [o] [o] [gpe] [o] [date] cl: [o] [o] [o] [o] [o] [o] [o] [gpe] [o] [o] inputs: bin will attend the emnlp meeting in singapore in december (the non-entity type) at the current step t where [gpe] (countries) is the current entity type to be learned, causing the semantic shift problem of the nonentity type (the second row cl). deep learning approaches to cner encounter two primary challenges. the first one is common to all continual learning methods, known as catastrophic forgetting @xcite @xcite . this refers to the tendency of neural networks to forget previously acquired knowledge when learning new information. existing cner methods @xcite @xcite often resort to the widely em-ployed technique of knowledge distillation @xcite to tackle catastrophic forgetting. however, these methods meticulously extract the output probabilities for previous entity types from the old model and transfer them to the new model. consequently, they tend to imbue the model with excessive stability (i.e., retention of old information) at the expense of plasticity (i.e., acquisition of new information). the second one is specific to cner, involving the semantic shift of the non-entity type. in the conventional ner paradigm, tokens are marked as the non-entity type, indicating that they do not belong to any entity type. in contrast, in the cner paradigm, tokens marked as the non-entity type imply that they do not belong to any of the current entity types. this implies that the non-entity type may encompass: the true non-entity type, previously learned old entity types, or future ones not yet encountered. as depicted in figure 1 (the second row cl), the old entity types organization and person (learned in the prior steps such as t-1, t-2, etc.) as well as the future entity type date (will be learned in the future steps such as t+1, t+2, etc.) are all marked as the non-entity type at the current step t. lacking mechanisms to differentiate tokens that pertain to previous entity types from the authentic non-entity type, this semantic shift could worsen catastrophic forgetting. to our knowledge, zheng et al. were the first to recognize the old entity types contained within the non-entity type using the old model to distill causal effect. however, they don't sufficiently handle recognition errors by the old model. in this paper, we present a novel approach named cpfd, an acronym for confidence-based pseudolabeling and pooled features distillation, which utilizes the old model in two significant ways to address the aforementioned challenges inherent in cner. firstly, we introduce a pooled features distillation loss that strikes a judicious trade-off between stability and plasticity, thus effectively alleviating catastrophic forgetting. these features are grounded in the attention weights learned by plms, capturing crucial linguistic knowledge necessary for the ner task, including coreference and syntax information @xcite . secondly, we develop a confidence-based pseudo-labeling strategy to specifically identify previous entity types within the current non-entity type for classification, mitigating the problem of semantic shift. to better re-duce the recognition errors from the old model, we employ entropy as a measure of uncertainty and the median entropy as a confidence threshold, retaining only those pseudo labels where the old model exhibits sufficient confidence. furthermore, we introduce an adaptive re-weighting type-balanced learning strategy to handle the biased distribution between new and old entity types that occurs post pseudo-labeling. this approach adaptively assigns different weights to the tokens of different types based on the number of tokens. our contributions can be summarized as follows: cner traditional ner focuses on the development of various deep learning models aimed at extracting entities from unstructured text @xcite . recently, plms have been widely utilized in ner and have achieved sota performance (kenton and toutanova, 2019; @xcite . however, most existing methods are designed to recognize a fixed set of predefined entity types. in response, cner incorporates the continual learning paradigm with traditional ner @xcite @xcite . extendner @xcite explores the application of knowledge distillation to cner, wherein the new model learns to fit the classifier logits of the old model. l&r (xia et al., 2022) offers a 'learn-and-review' framework for cner. the learning stage mirrors extendner, while the reviewing stage generates synthetic samples of previous entity types to enhance the current dataset. cfner (zheng et al., 2022) introduces a causal framework for cner and distills causal effects from the non-entity type. in spite of significant advancements, these methods strictly distill the output probabilities for previous types from the old model into the new model, resulting in excessive stability but limited plasticity in the models. moreover, these approaches do not adequately address the semantic shift problem. cfner employs the old model to recognize non-entity type tokens belonging to previous entity types for distilling causal effect and utilizes a curriculum learning strategy to mitigate recognition error. however, this curriculum learning strategy requires the manual predefinition of numerous hyper-parameters, limiting its applicability and the effect of error reduction. in contrast, we propose a pooled features distillation loss. this retains crucial linguistic knowledge embedded within attention weights and strikes a suitable balance between the stability and plasticity of the model, thus more efficiently mitigating catastrophic forgetting issue. moreover, we design a confidence-based pseudo-labeling strategy for classification. this strategy employs median entropy as the confidence threshold, eliminating the need for manually predefining any hyper-parameters, better reducing the recognition error from the old model, and dealing with the semantic shift problem. ## preliminary cner aims to train a model across @xmath0, ..., t steps, progressively learning an expanding set of entity ## method in the above formulation, we pinpoint two significant challenges in cner. the first one is the issue of catastrophic forgetting @xcite , which implies that the neural network may entirely and abruptly forget previous entity types e 1:t-1 when learning the current types e t . this issue is further amplified by the second challenge: semantic shift. at step t, tokens labeled as the non-entity type are intrinsically ambiguous, as they could include the true non-entity type, previously learned old entity types, or as-yet-unseen future entity types. to address these challenges in cner, we propose a novel cpfd method, which takes advantage of the previous model in two ways, shown in figure 2 . ## conclusion in this paper, we lay the foundation for future research in cner, an emerging field in nlu. we pinpoint two principal challenges in cner: catastrophic forgetting and the semantic shift problem of the non-entity type. to address these issues, we first introduce a pooled feature distillation loss that carefully establishes the balance between stability and plasticity, thereby better alleviating catas-trophic forgetting. subsequently, we present a confidence-based pseudo-labeling strategy to explicitly extract old entity types contained in the current non-entity type, better reducing the impact of label noise and dealing with the semantic shift problem. we evaluate cpfd on ten cner settings across three datasets and demonstrate that cpfd significantly outperforms the previous sota methods across all settings.