Improving Continual Relation Extraction through Prototypical Contrastive Learning (2022)

Abstract: Continual relation extraction (CRE) aims to extract relations towards the continuous and iterative arrival of new data, of which the major challenge is the catastrophic forgetting of old tasks. In order to alleviate this critical problem for enhanced CRE performance, we propose a novel Continual Relation Extraction framework with Contrastive Learning, namely CRECL, which is built with a classification network and a prototypical contrastive network to achieve the incremental-class learning of CRE. Specifically, in the contrastive network a given instance is contrasted with the prototype of each candidate relations stored in the memory module. Such contrastive learning scheme ensures the data distributions of all tasks more distinguishable, so as to alleviate the catastrophic forgetting further. Our experiment results not only demonstrate our CRECL’s advantage over the state-of-the-art baselines on two public datasets, but also verify the effectiveness of CRECL’s contrastive learning on improving performance.

## introduction in some scenarios of relation extraction (re), massive new data including new relations emerges continuously, which can not be solved by traditional re methods. to handle such situation, continual relation extraction (cre) @xcite was proposed. due to the limited storage and computing resources, it is impractical to store all training data of previous tasks. as new tasks are learned where new relations emerge constantly, the model tends to forget the existing knowledge about old relations. therefore, the problem of catastrophic forgetting damages cre performance severely @xcite . in recent years, some efforts have focused on the alleviating catastrophic forgetting in cre, which can be divided into consolidation-based methods @xcite , dynamic architecture methods @xcite and memory-based methods @xcite @xcite . despite these methods' effectiveness on cre, most of them have not taken full advantage of the negative relation information in all tasks to alleviate catastrophic forgetting more thoroughly, result in suboptimal cre performance. through our empirical studies, we found that the catastrophic forgetting of a model results in the indistinguishability between the data (instances) distributions of all tasks, making it hard to distinguish the relations of all tasks. we illustrate it with the data distribution map after training a relation classification model for a new task, as shown in figure 1 where the dots and crosses represent the data of the old and new task respectively, and different colors represent different relations. it shows that the data points of different colors in either dot group (old task) or cross group (new task) are distinguishable. however, many dots and crosses are mixed, making it hard to discriminate the new task's relations from the old task's relations. therefore, making the data distributions of all tasks more distinguishable is crucial to achieve better cre. to address above issue, in this paper we propose a novel continual relation extraction framework with contrastive learning, namely crecl, which is built with a classification network and a contrastive network. in order to fully leverage the information of negative relations to make the data distributions of all tasks more distinguishable, we design a prototypical contrastive learning scheme. specifically, in the contrastive network of crecl, a given instance is contrasted with the prototype of each candidate relation stored in the memory module. such sufficient comparisons ensure the alignment and uniformity between the data distributions of old and new tasks. therefore, the catastrophic forgetting in crecl is alleviated more thoroughly, resulting in enhanced cre performance. in addition, different to the classification for a fixed (relation) class set as @xcite , crecl achieves an incremental-class learning of cre which is more feasible to real-world cre scenarios. our contributions in this paper are summarized as follows: ## related work in this section, we briefly introduce continual learning and contrastive learning which are both related to our work. continual learning @xcite focuses on the learning from a continuous stream of data. the models of continual learning are able to accumulate knowledge across different tasks without retraining from scratch. the major challenge in continual learning is to alleviate catastrophic forgetting which refers to that the performance on previous tasks should not signif-icantly decline over time as new tasks come in. for overcoming catastrophic forgetting, most recent works can be divided into three categories. 1) regularized-based methods impose constraints on the update of parameters. for example, lwf approach @xcite enforces the network of previously learned tasks to be similar to the network of current task by knowledge distillation. however, lwf depends heavily on the data in new task and its relatedness to prior tasks. ewc @xcite adopts a quadratic penalty on the difference between the parameters for old and new tasks. it models the parameter relevance with respect to training data as a posterior distribution, which is estimated by laplace approximation with the precision determined by the fisher information matrix. wa @xcite ) maintains discrimination and fairness among the new and old task by adjust the parameters of the last layer. 2) dynamic architecture methods change models' architectural properties upon new data by dynamically accommodating new neural resources, such as increased number of neurons. for example, packnet @xcite iteratively assigns parameter subsets to consecutive tasks by constituting pruning masks, which fixes the task parameter subset for future tasks. der @xcite proposes a novel two-stage learning approach to get more effective dynamically expandable representation. 3) memory-based methods explicitly retrain the models on a limited subset of stored samples during the training on new tasks. for example, icarl @xcite focuses on learning in a classincremental way, which selects and stores the samples most close to the feature mean of each class. during training, distillation loss between targets obtained from previous and current model predictions is added into overall loss, to preserve previously learned knowledge. rp-cre @xcite introduces a novel pluggable attention-based memory module to automatically calculate old tasks' weights when learning new tasks. since classification-based approaches require relation schema in the classification layer, classification-based models have an unignorable drawback on class-incremental learning. many researchers leverage metric learning to solve this problem. @xcite utilize sentence alignment model based on margin ranking loss @xcite , while lack the intrinsic ability to perform hard positive/negative memory dropout layer ## conclusion in this paper, we propose a novel cre framework, namely crecl, that consists of a classification network and a contrastive network designed for alleviating the catastrophic forgetting in cre. through the prototypical contrastive learning in crecl, the data distributions of different relations in all tasks are adjusted to be more distinguishable, resulting in cre performance gains. moreover, crecl has the ability of class-incremental learning due to its contrasting-based mechanism of achieving relation classification, which is more practical in real-world cre scenarios than the previous models of classification-based mechanism. our extensive experimental results demonstrate that crecl outperforms the sota cre baselines and obtains the best performance on two benchmark datasets.