Consistent Representation Learning for Continual Relation Extraction (2022)

Abstract: Continual relation extraction (CRE) aims to continuously train a model on data with new relations while avoiding forgetting old ones. Some previous work has proved that storing a few typical samples of old relations and replaying them when learning new relations can effectively avoid forgetting. However, these memory-based methods tend to overfit the memory samples and perform poorly on imbalanced datasets. To solve these challenges, a consistent representation learning method is proposed, which maintains the stability of the relation embedding by adopting contrastive learning and knowledge distillation when replaying memory. Specifically, supervised contrastive learning based on a memory bank is first used to train each new task so that the model can effectively learn the relation representation. Then, contrastive replay is conducted of the samples in memory and makes the model retain the knowledge of historical relations through memory knowledge distillation to prevent the catastrophic forgetting of the old task. The proposed method can better learn consistent representations to alleviate forgetting effectively. Extensive experiments on FewRel and TACRED datasets show that our method significantly outperforms state-of-the-art baselines and yield strong robustness on the imbalanced dataset.

## introduction relation extraction (re) is an essential issue in information extraction (ie), which can apply to many downstream nlp tasks, such as information retrieval @xcite and question and answer @xcite . for example, given a sentence x with the annotated entities pairs e 1 and e 2 , the re aims to identify the relations between e 1 and e 2 . however, traditional relation extraction models *corresponding author @xcite always assume a fixed set of predefined relations and train on a fixed dataset, which cannot handle the growing relation types in real life well. to solve this situation, continual relation extraction (cre) is introduced @xcite @xcite . compared with traditional relation extraction, cre aims to help the model learn new relations while maintaining accurate classification of old ones. @xcite shows that continual relation learning needs to alleviate the catastrophic forgetting of old tasks when the model learns new tasks. because neural networks need to retrain a fixed set of parameters with each training, the most efficient solution to the problem of catastrophic forgetting is to store all the historical data and retrain the model with all the data each time a new relational instance appears. this method can achieve the best effect in continual relation learning, but it is not adopted in real life due to the time and computing power costs. some recent works have proposed a variety of methods to alleviate the catastrophic forgetting problem in continual learning, including regularization methods @xcite @xcite , dynamic architecture methods @xcite , and memory-based methods @xcite . although these methods have been verified in simple image classification tasks, previous works have proved that memory-based methods are the most effective in natural language processing applications @xcite . in recent years, the memory-based continual relation extraction model has made significant progress in alleviating the problem of catastrophic forgetting @xcite @xcite . @xcite proposes a mechanism for embedding sentence alignment in memory mainte-nance to ensure the stability of the embedding space. @xcite introduces a multi-round joint training process for memory consolidation. but these two methods only explore the problem of catastrophic forgetting in the overall performance of the task sequence. @xcite proposes to integrate curriculum learning. although it is possible to analyze the characteristics of each subtask and the performance of the corresponding model, it still fails to make full use of the saved sample information. @xcite introduce an attention network to refine the prototype to better recover the interruption of the embedded space. however, this method will produce a bias in the classification of the old task as the new task continues to learn the classifier, which will affect the performance of the old task. although the above method can alleviate catastrophic forgetting to a certain extent, it does not consider the consistency of relation embedding space. because the performance of the model of cre is sensitive to the quality of sample embedding, it needs to ensure that the learning of new tasks will not damage the embedding of old tasks. inspired by supervised contrastive learning @xcite to explicitly constrain data embeddings, a consistent representation learning method is proposed for continual relation extraction, which constrains the embedding of old tasks not to occur significantly change through supervised contrastive learning and knowledge distillation. specifically, the example encoder first trains on the current task data through supervised contrastive learning based on memory bank, and then uses k-means to select representative samples to storage as memory after the training is completed. to relieve catastrophic forgetting, contrastive replay is used to train memorized samples. at the same time, to ensure that the embedding of historical relations does not undergo significant changes, knowledge distillation is used to make the embedding distribution of the new and old tasks consistent. in the testing phase, the nearest class mean (ncm) classifier is used to classify the test sample, which will not be affected by the deviation of the classifier. in summary, our contributions in this paper are summarized as follows: first, a novel cre method is proposed, which uses supervised contrastive learning and knowledge distillation to learn consistent relation representations for continual learning. second, consistent representation learning can en-sure the stability of the relational embedding space to alleviate catastrophic forgetting and make full use of stored samples. finally, extensive experiments results on fewrel and tacred datasets show that the proposed method is better than the latest baseline and effectively mitigates catastrophic forgetting. ## conclusions and future work this paper proposes a novel consistent representation learning method for the cre task, mainly through contrastive learning and knowledge distillation when replaying memory. specifically, we use supervised contrastive learning based on a memory bank to train each new task so that the model can effectively learn the feature representation. in addition, in order to prevent the catastrophic forgetting of the old task, we contrast and replay the memory samples, and at the same time, make the model retain the knowledge of the relation between the historical tasks through the knowledge distillation. our method can better learn consistent representations to alleviate catastrophic forgetting effectively. extensive experiments on two benchmark data sets show that our method significantly improves the performance of the most advanced technology and demonstrates powerful representation learning capabilities. in the future, we will continue to study cross-domain continual relation extraction to acquire ever-increasing knowledge.