Flexible Weight Tuning and Weight Fusion Strategies for Continual Named Entity Recognition (2024)

Abstract: Continual Named Entity Recognition (CNER) is dedicated to sequentially learning new entity types while mitigating catastrophic forgetting of old entity types. Traditional CNER approaches commonly employ knowledge distillation to retain old knowledge within the current model. However, because only the representations of old and new models are constrained to be consistent, the reliance solely on distillation in existing methods still suffers from catastrophic forgetting. To further alleviate the forgetting issue of old entity types, this paper introduces flexible Weight Tuning (WT) and Weight Fusion (WF) strategies for CNER. The WT strategy, applied at each training step, employs a learning rate schedule on the parameters of the current model. After learning the current task, the WF strategy dynamically integrates knowledge from both the current and previous models for inference. Notably, these two strategies are model-agnostic and seamlessly integrate with existing State-Of-The-Art (SOTA) models. Extensive experiments demonstrate that the WT and WF strategies consistently enhance the performance of previous SOTA methods across ten CNER settings in three datasets.

## introduction as a pivotal task in information extraction, named entity recognition (ner) plays a crucial role in various applications, including question answering @xcite , web search queries @xcite . traditional fully-supervised ner endeavors to classify tokens in a sentence into fixed entity types @xcite . however, real-world entity types typically emerge in a streaming manner, as seen in voice assistants like siri, which need to recognize new entity types (e.g., band, song) to understand new user intents (e.g., getmusic) @xcite . one straightforward solution involves retraining the model on the entire dataset by incorporating both old and new entity types. nevertheless, this strategy incurs substantial training costs. another simplistic approach is to fine-tune the previously learned model exclusively on the newly introduced entity types, a scenario known as continual ner (cner) @xcite . however, cner commonly faces catastrophic forgetting, wherein the knowledge acquired from previous entity types is lost after learning new ones @xcite @xcite @xcite @xcite . to address the issue of forgetting old entity types, prior cner methods often employ knowledge distillation to preserve previous knowledge in the current model @xcite @xcite . these approaches maintain output logits or intermediate features to prevent substantial changes in the parameters of the current model. specifically, extendner @xcite distills output logits from the old model to the new model, encouraging the new model to generate logits closely resembling those produced by the old model. l&r @xcite utilizes a twostage learn-and-review framework. in the learning stage, it adopts a similar approach to extendner; while during the reviewing stage, it integrates synthesized samples of old entity types to enhance the current dataset. extending the ideas of extendner, cfner @xcite introduces a causal framework to distill causal effects from the nonentity type. cpfd @xcite presents a feature distillation method to retain linguistic knowledge in attention weights, achieving state-of-the-art (sota) cner performance. however, since only the output or internal representations of old and new models are constrained to be consistent, relying on knowledge distillation alone pro- ## preliminary cner endeavors to learn a model over @xmath0, ..., t steps, progressively acquiring proficiency in an expanding array of entity types ## method to further alleviate the forgetting of old entity types, we introduce model-agnostic wt and wf strategies to augment existing cner methods. the wt strategy implements a learning rate schedule for each continual training step t to adjust the parameters θ t of the current model m t . specifically, during the initial step (@xmath1, the learning rate is set to lr 1 . for subsequent steps (@xmath2, the learning rate is gradually reduced to better retain knowledge of historical entity types. therefore, the learning rate lr t at step t is expressed as: algorithm 1 pseudo code for wt and wf strategies in continual steps (highlighted in red font) require: θ 0 , t, d t , initial learning rate lr 1 , and hyper-parameters α and β following each training step t (@xmath3, the wf strategy, instead of employing the new model directly, produces a fused model for inference. this fused model integrates knowledge from both the new and old models using a dynamic balance factor, establishing a refined equilibrium between old and new knowledge without introducing additional computation costs. the formulation is as follows: the pseudo code for our wt and wf strategies is presented in algorithm 1. ## experiments 4.1 experimental setup to ensure a fair comparison, we adhere to the setup of cpfd @xcite ) method as below: ## conclusion in this paper, we introduce model-agnostic wt and wf strategies for cner to mitigate catastrophic forgetting. wt employs a learning rate schedule within each training step to adjust the parameters of the new model, while wf dynamically fuses the new and old models to maintain a balance between new and old knowledge. extensive experiments conducted on 10 cner settings across 3 datasets illustrate that our strategies further enhance the previous sota methods.