Serial Contrastive Knowledge Distillation for Continual Few-shot Relation Extraction (2023)

Abstract: Continual few-shot relation extraction (RE) aims to continuously train a model for new relations with few labeled training data, of which the major challenges are the catastrophic forgetting of old relations and the overfitting caused by data sparsity. In this paper, we propose a new model, namely SCKD, to accomplish the continual few-shot RE task. Specifically, we design serial knowledge distillation to preserve the prior knowledge from previous models and conduct contrastive learning with pseudo samples to keep the representations of samples in different relations sufficiently distinguishable. Our experiments on two benchmark datasets validate the effectiveness of SCKD for continual few-shot RE and its superiority in knowledge transfer and memory utilization over state-of-the-art models.

## introduction relation extraction (re) aims to recognize the semantic relations between entities in texts, which is widely applied in many downstream tasks such as language understanding and knowledge graph construction. conventional studies @xcite @xcite mainly assume a fixed pre-defined relation set and train on a fixed dataset. however, they cannot work well with the new relations that continue emerging in some real-world scenarios of re. continual re @xcite @xcite was proposed as a new paradigm to solve this situation, which applies the idea of continual learning @xcite to the field of re. compared with conventional re, continual re is more challenging. it requires the model to learn emerging relations while maintaining a stable and accurate classification of old relations, i.e., the socalled catastrophic forgetting problem @xcite , which refers to the task 1 continually learn new relations . . . task 2 task i learn relations: crosses, movement learn relations: sport, performer learn relations: father, follows . . . . . . . . . severe loss of prior knowledge during the model is learning new tasks. recent continual learning works leverage the regularization-based models, the architecture-based models, and the memorybased models to alleviate catastrophic forgetting. several studies @xcite have shown that the memory-based models are more promising for nlp tasks, and a number of memory-based continual re models @xcite @xcite have made significant progress. in real life, the shortage of labeled training data for relations is an unavoidable problem, especially severe in emerging relations. therefore, the continual few-shot re paradigm @xcite was proposed to simulate real human learning scenarios, where new knowledge can be acquired from a small number of new samples. as illustrated in figure 1 , the continual few-shot re paradigm expects the model to continuously learn new relations through abundant training data only for the first task, but through sparse training data for all subsequent tasks. thus, the model needs to identify the growing relations well with few labeled data for them while retaining the knowledge on old relations without re-training from scratch. as relations grow, the confusion about relation representations leads to catastrophic forgetting. in continual fewshot re, catastrophic forgetting becomes more severe since the few samples of new relations may not be representative for these relations. the possibility of confusion between relation representations greatly increases. since the emerging relations are few-shot, the problem of overfitting becomes another key challenge in the continual few-shot re task. the overfitting for samples in few-shot tasks aggravates the model's forgetting of prior knowledge as well. existing few-shot learning works @xcite @xcite are worthy of reference by continual few-shot re models to ensure good generalization. inspired by knowledge distillation @xcite to transfer knowledge well and contrastive learning @xcite to constrain representations explicitly, we propose sckd, a model built with serial contrastive knowledge distillation for continual few-shot re. through it, we tackle the aforementioned two key challenges. first, how to alleviate the problem of catastrophic forgetting? sckd follows the memory-based methods for continual learning and preserves a few typical samples from previous tasks. furthermore, we present serial knowledge distillation to preserve the prior knowledge from previous models and conduct contrastive learning to keep the representations of samples in different relations sufficiently distinguishable. second, how to mitigate the negative impact of overfitting caused by sparse samples? we leverage bidirectional data augmentation between memory and current tasks to obtain more samples for few-shot relations. the pseudo samples generated in serial contrastive knowledge distillation can help prevent overfitting as well. in summary, our main contributions are twofold: in this section, we review related work on continual re and few-shot re. continual re. the goal of continual learning is to accomplish new tasks sequentially without catastrophically forgetting the acquired knowledge from previous tasks. for continual re, rp-cre (cui et al., 2021) refines sample embeddings for prediction with the generated relation prototypes from memory. however, its relation prototype calculation is sensitive to typical samples. crl (zhao et al., 2022) introduces supervised contrastive learning and knowledge distillation to generate sample representations when replaying memory. it narrows the representations of samples belonging to the same relation through supervised contrastive learning but fails to keep the representations of samples in different relations far away to avoid confusion. besides, knowledge distillation between prototypes calculated by averaging sample representations may lose some features of specific samples. crecl (hu et al., 2022) contrasts a given sample with all the candidate relation prototypes stored in memory by a contrastive network. it faces the same problem as rp-cre on typical samples for computing relation prototypes. conducting contrastive learning only with relation prototypes may not guarantee the differences between sample representations belonging to different relations. kip-framework (zhang et al., 2022) generates knowledge-infused relation prototypes to leverage the relational knowledge from pre-trained language models with prompt tuning. compared with other models, kip-framework needs extra knowledge such as relation descriptions, and its overall procedure is more time-consuming. all these works rely on plenty of training data for learning new relations and large memory for retaining prior knowledge. in contrast, our model only needs a few training samples to learn new relations well through bidirectional data augmentation and the generated pseudo samples from relation prototypes. furthermore, our model can avoid catastrophic forgetting under limited memory through serial knowledge distillation and contrastive learning. as far as we know, erda (qin and joty, 2022) is the only work addressing continual few-shot re. it imposes relational constraints in the embedding space and generates new training data from unlabeled text. however, our model does not need to import extra data like erda. instead, it generates pseudo samples from relation prototypes and augments training data by modifying original samples to alleviate the overfitting problem. few-shot re. few-shot learning aims to leverage only a few novel samples to adapt the model for solving tasks. for few-shot re, its goal is to enable the model to quickly learn the characteristics of relations with very few samples, so as to accurately classify these relations. at present, there are two main lines of work: (1) the metric learning methods @xcite use various metric functions (e.g., the euclidean or cosine distance) learned from prior knowledge to map the input into a subspace so that they can distinguish similar and dissimilar sample pairs easily to assign the relation labels. ## experiments in this section, we assess the proposed sckd and report our results. the datasets and source code for sckd are accessible from github. 1 ## conclusion in this paper, we propose sckd for continual fewshot re. to alleviate the problems of catastrophic forgetting and overfitting, we design the serial contrastive knowledge distillation, making prior knowledge from previous models sufficiently preserved while the representations of samples in different relations remain distinguishable. our experiments on fewrel and tacred validate the effectiveness of sckd for continual few-shot re and its superiority in knowledge transfer and memory utilization. for future work, we plan to investigate how to apply the serial contrastive knowledge distillation to other classification-based continual few-shot learning tasks.