SKD - NER : Continual Named Entity Recognition via Span-based Knowledge Distillation with Reinforcement Learning (2023)

Abstract: Continual learning for named entity recognition (CL-NER) aims to enable models to continuously learn new entity types while retaining the ability to recognize previously learned ones. However, the current strategies fall short of effectively addressing the catastrophic forgetting of previously learned entity types. To tackle this issue, we propose the SKD-NER model, an efficient continual learning NER model based on the span-based approach, which innovatively incorporates reinforcement learning strategies to enhance the model’s ability against catastrophic forgetting. Specifically, we leverage knowledge distillation (KD) to retain memory and employ reinforcement learning strategies during the KD process to optimize the soft labeling and distillation losses generated by the teacher model to effectively prevent catastrophic forgetting during continual learning. This approach effectively prevents or mitigates catastrophic forgetting during continuous learning, allowing the model to retain previously learned knowledge while acquiring new knowledge. Our experiments on two benchmark datasets demonstrate that our model significantly improves the performance of the CL-NER task, outperforming state-of-the-art methods.

## introduction the introduction of continual learning methods has enabled systems to continuously learn from new data and reduce their dependence on initial training data. moreover, these methods facilitate model updates and fine-tuning, enhancing their scalability @xcite . by combining continual learning with ner tasks, systems can significantly improve their ability to perceive the constantly changing real world amidst the emergence of new tasks and data sources, these functionalities can be formulated as paradigms of continual learning @xcite figure 1 : an illustration of the sequence labeling approach in skd-ner. assuming a model learns task1:per and task2:loc in ontonotes5 sequentially, the left side represents the conventional sequence labeling method, while the right side represents the approach we adopted. @xcite . however, continual learning has always faced catastrophic forgetting, which has become a pervasive issue for continual learning ner tasks @xcite @xcite . specifically, simply fine-tuning the ner system based on new data often results in a significant drop in performance on previously learned tasks, which poses a major challenge for achieving human-level intelligence in continual learning for ner @xcite . this is in contrast to the natural ability of humans to learn new entity categories without forgetting previously learned ones. in the context of continual learning, the model training process is typically divided into n cl steps, with each step being specific to the current task. in the case of cl-ner, only new entity types are recognized in each cl step. however, this approach can lead to a situation that is easily overlooked, whereby entity types that are not required to be recognized in the current step (e.g., org) may need to be learned in the future or have been learned in the past. in traditional sequence labeling methods, an entity that is not a required type in the current step is assigned a global o tag to indicate that the model does not need to recognize this type in the current task(refer to figure1). however, this can result in each entity's category needing frequent parameter updates in each different cl step, due to this incoherent optimization. we believe this exacerbates catastrophic forgetting and label noise interference. to test this hypothesis, we conducted experiments, the results of which are presented in figure2. with the goal of addressing the issues of catastrophic forgetting and label noise in cl-ner @xcite , we propose skd-ner. to mitigate the adverse effects of frequent parameter updates due to incoherent optimization in continuous learning, our model first computes an entity classification matrix for each segment of text to be processed, inspired by the multi-headed attention mechanism @xcite . this matrix computes a score for the different entity classes to be recognized, thereby converting the entity recognition problem into a binary classification problem. in the cl setting, we enable the model to perform well in multi-label learning on the span classification, while equipping it with knowledge distillation at the span entity level based on the bernoulli distribution it produces. to address the label noise problem in the incremental learning process, we introduce a reinforcement learning strategy for the student model in the knowledge distillation process (jiang, 2023), using the most suitable knowledge distillation method for the current student model. for model prediction, we introduce a multi-label classification loss function @xcite , which fits well with our sequence labeling approach. this approach offers several advantages: 1) it allows for knowledge distillation and retention of old knowledge during the cl process. 2) the introduction of reinforcement learning effectively reduces label noise while also addressing the issue of catastrophic forgetting that arises due to frequent weight updates for the same entity in different steps based on different categories. 3) the proposed framework of the model can be used as a reusable framework for continuous learning knowledge distillation and applied to other model migration or continuous learning domains. we evaluated our model on two named entity recognition (ner) datasets, namely ontonotes5 @xcite , and fewnerd @xcite . the experimental results demonstrate that our proposed skd-ner model significantly outperforms existing continuous learning ner models and achieves a new state-of-the-art (sota) performance. notably, skd-ner almost eliminates catastrophic forgetting on relatively simple ontonotes, thereby achieving "continuous learning" in the true sense. our contributions can be summarized as follows: • we propose a continual named entity recognition model and a reinforcement learningbased knowledge distillation framework that ensures the model's effectiveness for continuous learning, which can be further leveraged in other fields of continuous learning or knowledge transfer. • innovatively introduce reinforcement learning strategies to support continual named entity recognition, while optimizing traditional sequence labeling methods and loss functions to address catastrophic forgetting and label noise problems in continuous learning. • through extensive experiments, we demonstrate that our approach achieves state-of-theart performance in continual named entity recognition and can be seamlessly integrated as a plug-and-play module to further enhance the performance of other continual named entity recognition models. 2 related work 2.1 continual learning ner recent research has expanded the application of continual learning (cl) from computer vision to natural language processing (nlp) tasks, par-ticularly ner. while most cl-related works in computer vision focus on accuracy-oriented tasks like image classification, their direct application to cl-ner has shown unsatisfactory performance due to challenges in preserving old knowledge in other-class samples. chen and moschitti (2019 ) pioneered the study of knowledge transfer in sequence labeling ner models from source to target domains with new entities, using a neural adapter module to handle diverse entity distributions. following this, addner, extendner @xcite , and lr @xcite were developed under a classincremental setting for cl-ner, employing sequence labeling methods with knowledge distillation. addner uses a multi-head approach, while extendner and lr employ single-head layouts with different strategies for handling o tags and old entity mentions. however, current methods such as addner and extendner still face forward incompatibility issues and require cumbersome cooperation with knowledge distillation. spankl @xcite ) explores the potential of spanbased models for solving cl-ner with excellent forward compatibility to solve this problem. in the context of cl-ner, token-noise is also a very important issue to be concerned about, selftraining @xcite has been a straightforward solution for learning old knowledge from other-class samples. however, this approach suffers from error propagation between models. @xcite have proposed contrastive learning and pretraining techniques to address the problem of token-noise in few-shot ner. cfner @xcite proposes a causal framework in cl-ner that explicitly addresses the challenges posed by token-noise. in contrast, our approach utilizes reinforcement learning strategies to adjust the process of knowledge distillation, optimizing the process and ensuring the forward compatibility of the model, while also addressing the issue of token-noise to some extent. ## our approach in this section, we will first introduce the task setting for continual learning named entity recognition (ner). after presenting the overall structure of the skd-ner model, we provide a detailed explanation of how the model integrates reinforcement learning strategies. ## limitations although the proposed model can partially solve the catastrophic forgetting problem, there is still significant room for improvement in more complex dataset testing. additionally, due to the introduction of knowledge distillation with a reinforcement learning policy, the training time of our model is slightly longer than that of baselines.