Few-Shot Class-Incremental Learning for Named Entity Recognition (2022)

Abstract: Previous work of class-incremental learning for Named Entity Recognition (NER) relies on the assumption that there exists abundance of labeled data for the training of new classes. In this work, we study a more challenging but practical problem, i.e. , few-shot class-incremental learning for NER, where an NER model is trained with only few labeled samples of the new classes, without forgetting knowledge of the old ones. To alleviate the problem of catastrophic forgetting in few-shot class-incremental learning, we reconstruct synthetic training data of the old classes using the trained NER model, augmenting the training of new classes. We further develop a framework that distills from the existing model with both synthetic data, and real data from the current training set. Experimental results show that our approach achieves significant improvements over existing baselines.

## introduction existing models of named entity recognition (ner) are usually trained on a large scale dataset with predefined entity classes, then deployed for entity extraction on the test data without further adaptation or refinement. in practice, data of new entity classes that the ner model has not seen during training arrives constantly, thus it is desirable that the ner model can be incrementally updated over time with knowledge of data for these new classes. in this case, one challenge is that the training data of old entity classes may not be available due to privacy concerns or memory limitations @xcite . then, the model can easily degrade in terms of the performance on old classes when being fine-tuned with only annotations of new entity classes, i.e., catastrophic forgetting. in addressing this problem, previous work in class-incremental learning for ner (monaikul * corresponding author et al., 2021) regularizes the current model by distilling from the previous model trained on old (existing) classes, using text from the training dataset of new classes. however, this requires abundance of data in the new training dataset being used for distillation. such an assumption is usually unrealistic since the token-level annotations required by ner training are labor-consuming and scarce, especially for the new unseen classes. in this paper, we study a more realistic setting, i.e., few-shot classincremental learning for ner, where the model (i) incrementally learns on new classes with few annotations, and (ii) without requiring access to training data for old classes. there is very limited work in few-shot classincremental learning for ner. such a setting is more challenging compared with class-incremental learning for ner. first, the few-shot datasets in few-shot class-incremental learning may not contain enough information for the trained model to generalize during testing. second, it is more challenging to solve the catastrophic forgetting problem in few-shot class-incremental learning when data for old classes is not available and new data is scarce. in class-incremental learning for ner @xcite , the same training sequence may contain entities of different classes. therefore, when the training dataset for new classes is sufficiently large, its context, i.e., words labeled as not from entities of new classes, may also contain abundant entities of the old classes. that is, the new training data can be regarded as an unlabeled replay dataset of the existing entity classes. in such case, we can simply address the problem of catastrophic forgetting by distilling from the previous model (trained on old classes) to the current one, using text from such a replay dataset @xcite . however, in few-shot classincremental learning, we cannot expect to avoid catastrophic forgetting by distilling with only the few samples from the new training dataset, since there may not exist sufficient (if any) entities of the old classes. in this paper, we propose a framework to enable few-shot class-incremental learning for ner. as mentioned above, since the few-shot dataset may not contain enough entities of old classes as replay data for distilling from the previous model, which leads to catastrophic forgetting, we consider generating synthetic data of the old entity classes for distillation. such data is termed as synthetic replay. specifically, we generate synthetic data samples of old classes by inverting the ner model. given the previous model trained on the old classes, we optimize the token embeddings of the synthetic data, so that predictions from the previous model can contain old entity classes, given the synthetic data as input. in this way, the synthetic data is likely to contain entities of old classes, and distilling from the previous model with such data will thus encourage knowledge preservation of old classes. additionally, to ensure the synthetic (reconstructed) data to be realistic, we propose to leverage the readily available real text data for new classes, via adversarially matching the hidden features of tokens from the synthetic data and those from the real data. note that the synthetic data generated from such adversarial match with real data will contain semantics that are close to the real text data for new classes. consequently, compared with training with only the few samples of new classes, the synthetic data will provide more diverse context that are close to the samples of the few-shot dataset, augmenting the few-shot training for the new classes. further, with the generated synthetic data, we propose a framework that trains the ner model with annotations of the new classes, while distilling from the previous model with both the synthetic data and real text from the new training data. our contributions of this work are summarized as follows: ## conclusion we present the first work of few-shot class-incremental learning for ner to address the problem of catastrophic forgetting, we proposed to reconstruct synthetic training of the old entity classes from the model trained at the previous time step. additionally, the synthetic data allows the model to be trained with a more diverse context, thus less likely to overfit to the few training samples of current step. experimental results showed that our method outperforms the baselines, enabling the ner model to incrementally learning from new classes with few samples.