Learn and Review: Enhancing Continual Named Entity Recognition via Reviewing Synthetic Samples (2022)

Abstract: Traditional methods for named entity recognition (NER) classify mentions into a fixed set of pre-defined entity types. However, in many real-world scenarios, new entity types are incrementally involved. To investigate this problem, continual learning is introduced for NER. However, the existing method depends on the relevance between tasks and is prone to inter-type confusion. In this paper, we propose a novel two-stage framework Learn-and-Review (L&R) for continual NER under the type-incremental setting to alleviate the above issues. Specifically, for the learning stage, we distill the old knowledge from teacher to a student on the current dataset. For the reviewing stage, we first generate synthetic samples of old types to augment the dataset. Then, we further distill new knowledge from the above student and old knowledge from the teacher to get an enhanced student on the augmented dataset. This stage has the following advantages: (1) The synthetic samples mitigate the gap between the old and new task and thus enhance the further distillation; (2) Different types of entities are jointly seen during training which alleviates the inter-type confusion. Experimental results show that L&R outperforms the state-of-the-art method on CoNLL-03 and OntoNotes-5.0.

## introduction traditional named entity recognition (ner) aims at extracting mentions from a given text and classifying them into a fixed set of pre-defined entity types such as person, location, organization, etc @xcite . however, in many realworld scenarios, new entity types emerge periodically by demand and the models are required to recognize new types of entities without forgetting the old ones, which can formulate into the paradigm of continual learning (a.k.a. lifelong learning or incremental learning) @xcite . for example, voice assistants such as siri are often expected to grasp new intents (e.g. getmovie) and thus new entity types (e.g. actor, genre) are continually involved. the ability to learn from continuous streams of data after deployment is important for modern ner models in specific scenarios. however, continual learning, as it has long been recognized, suffers severely from catastrophic forgetting, i.e., the loss or disruption of previously learned knowledge when new patterns are learned @xcite @xcite . different from human beings, an ner model (particularly that based on deep neural networks) which stores knowledge by its parameters is vulnerable to catastrophic forgetting of old knowledge while updating parameters to learn new entity types. in order to avoid forgetting old types of entities while learning the new ones, a naive solution is to annotate a dataset for both old and new types and retrain the model from scratch. however, this method is computational-inefficient and labor-extensive, especially when the number of entity types is large. to reduce the cost, @xcite advocate annotating a training set only for new entity types and retaining previously learned knowledge via knowledge distillation (kd) @xcite . in their approach, the current ner model acts as the teacher and the target new ner model the student. the student then learns new entity types by using the new training material and retains knowledge of old entities by imitating the teacher's output on this new training set. despite the initial success, this kd-based approach relies on the co-occurrence of unlabeled old types in the current training data of new types. if the new training set (e.g. annotated only for restaurant) contains little information related to the old entity types (e.g. sport), the knowledge of these old types will be hard to be retained ## method in this section, we first introduce the whole training procedure of our framework which consists of a learning and a reviewing stage. then, we describe the two stages in detail. ## conclusion in this paper, we propose a novel framework introducing the reviewing stage to alleviate the catastrophic forgetting and intra-task confusion issues for ner under the type-incremental setting. after the learning step, we further distill the student and the teacher on the synthetic sample augmented dataset to get an enhanced student. our experiments on the two benchmarks conll-03 and ontonotes-5.0 demonstrate that l&r is less prone to the intra-task confusion and outperforms the state-of-the-art method. order conll-03 ontonotes-5.0