A Spectral Viewpoint on Continual Relation Extraction (2023)

Abstract: Continual Relation Extraction (CRE) aims to continuously train a model to learn new relations while preserving its ability on previously learned relations. Similar to other continual learning problems, in CRE, models experience representation shift, where learned deep space changes in the continual learning process, which leads to the downgrade in the performance of the old tasks. In this work, we will provide an insight into this phenomenon under the spectral viewpoint. Our key argument is that, for each class shape, if its eigenvectors (or spectral components) do not change much, the shape is well-preserved. We then conduct a spectral experiment and show that, for the shape of each class, the eigenvectors with larger eigenvalue are more preserved after learning new tasks which means these vectors are good at keeping class shapes. Based on this analysis, we propose a simple yet effective class-wise regularization that improve the eigenvalues in the representation learning. We observe that our proposed regularization leads to an increase in the eigenvalues. Extensive experiments on two benchmark datasets, FewRel and TACRED, show the effectiveness of our proposed method with significant improvement in performance compared to the state-of-the-art models. Further analyses also verify our hypothesis that larger eigenvalues lead to better performance and vice versa.

## introduction relation extraction (re) problem is the basis of many nlp tasks, such as question answering @xcite , knowledge graph construction @xcite @xcite , definition extraction @xcite and event relation extraction @xcite . in particular, a relation extraction system is expected to classify semantic relation between two entities mentioned in * corresponding author the given context. to address this problem, several methods have been proposed and have achieved remarkable results @xcite @xcite . nevertheless, most previous re studies only considered the traditional setting where the set of relations is pre-defined and fixed during the training and testing phases. this setting is not practical as new relations of interest might emerge during deployment time of re systems in practice, requiring the models to adapt their operations to accommodate new types. recently, continual relation extraction (cre) has attracted considerable attention in the literature @xcite @xcite , aiming to learn new relations from incoming data. cre methods often suffer from catastrophic forgetting (cf) phenomenon where their performance on previous relations reduce significantly when learning new relations. one key issue that causes cf is the representation shift that occurs when models learn new knowledge. existing works have made significant progress in mitigating this phenomenon @xcite @xcite . different from those works, we analyze the representation shift using spectral decomposition. we argue that the eigenvectors play an important role in capturing the shape of each class. as such, if the eigenvectors of a class shape adjust much after exposing new tasks, it means the class shape also changes significantly. we give an intuition for our argument in figure 1 . based on this reasoning, we conduct an experiment to see how eigenvectors change in the continual learning process. we find that the eigenvectors with larger eigenvalue are preserved well when learning new tasks. in other words, these eigenvectors are good at keeping class shapes and hence avoiding the representation shift. based on these experiments, we propose a classwise regularization that aims to boost the eigenvalues. this regularization is inspired by some recent works on dimensional collapse and feature decorrelation @xcite , which try to make different dimensions of the embedding uncorrelated and avoid small eigenvalues effectively. to enhance understanding, we theoretically show why feature decorrelation helps increase the eigenvalues. in summary, our work makes the following contributions: our extensive experiments demonstrate that our model achieves new state-of-the-art performance on two cre benchmarks, fewrel and tacred. we also conduct further analysis to verify the efficacy of our idea in boosting the eigenvalues. ## problem formulation and base model problem formulation: in continual relation extraction, models are trained on a sequence of tasks {t 1 , t 2 , ..., t k }, where the k-th task is one traditional relation extraction task and has its own train-ing set d k and relation set r k . formally, each task t k is a supervised classification task with training set d k containing n samples {(x i , y i )} n @xmath0, where x i is the input sentence and entity pair, and y i is the relation label. as a general goal of continual learning, a cre system is expected to perform well on both the current task and all previous tasks. hence, models are required to classify each relation into a known relation set rk , where @xmath1. we adapt an episodic memory module to store a few examples of historical tasks. each relation has its own memory module, i.e., a memory module for relation r is a set m @xmath2, y i )} o @xmath3, where o is the pre-defined number of samples to be stored. base model and objective function: for modelling details, we use one pretrained language model, i.e., bert @xcite as a feature extractor and a learnable linear classifier on top of the feature extractor. following previous work @xcite , we use the ce loss l ce as the objective function for training task t k : ## conclusion in this work, we conduct a thorough study on the change of distribution shapes in the cre problem through spectral analysis and observe that eigenvectors with larger eigenvalues are less forgettable. based on these findings, we introduce a class-wise feature decorrelation regularization with the goal of boosting eigenvalues. our theoretical analysis shows the effectiveness of our proposed method in handling low eigenvalues. furthermore, our extensive experiments on two benchmark datasets show the superior performance of our method. in the future, we will extend our analysis to other information extraction tasks, such as entity mention detection @xcite and event classification @xcite .