Enhancing Continual Relation Extraction via Classifier Decomposition (2023)

Abstract: Continual relation extraction (CRE) models aim at handling emerging new relations while avoiding catastrophically forgetting old ones in the streaming data. Though improvements have been shown by previous CRE studies, most of them only adopt a vanilla strategy when models first learn representations of new relations. In this work, we point out that there exist two typical biases after training of this vanilla strategy: classifier bias and representation bias, which causes the previous knowledge that the model learned to be shaded. To alleviate those biases, we propose a simple yet effective classifier decomposition framework that splits the last FFN layer into separated previous and current classifiers, so as to maintain previous knowledge and encourage the model to learn more robust representations at this training stage. Experimental results on two standard benchmarks show that our proposed framework consistently outperforms the state-of-the-art CRE models, which indicates that the importance of the first training stage to CRE models may be underestimated. Our code will be released upon acceptance.

## introduction continual relation extraction (cre) @xcite requires models to learn new relations from a class-incremental data stream while avoiding catastrophic forgetting of old relations. to address the problem of catastrophic forgetting, rehearsal-based cre methods store a few typical instances for each relation on memory and replay the memory data in the subsequent learning process. despite the simplicity, rehearsal-based methods have become the state-of-the-art cre methods @xcite @xcite . recent rehearsal-based cre models usually follow a two-stage training paradigm to tackle the data imbalance problem 1 : (1) in stage 1, models 1 if models are directly trained with the mixed data which are trained with only new data to rapidly learn to identify new relations; (2) in stage 2, models are trained on the updating memory data to alleviate catastrophic forgetting. previous cre works mainly focus on stage 2, and propose a variety of replay strategies to better utilize the memory data, such as relation prototypes @xcite , memory network @xcite and contrastive learning @xcite . however, the exploration of the first training stage is still uncharted territory. in this work, we focus on stage 1 and point out that it suffers from two typical biases that harm the model's performance: to alleviate the biases mentioned above, we procontains sufficient current data and few previous data, severe data imbalance will lead models to overfit on the previous data, which harms the model's performance @xcite . training data encoder memory set encoder stage 1 stage 2 frozen (a) empirical initialization [e11] jimmy jam [e12] is the son of [e21] cornbread harris [e22]. bert encoder c 1 c 2 c 3 c 4 c n-2 c n-1 c n ffn 0.02 0.40 0.32 0.05 0.01 0.10 0.03 rel: mother rel: father . . . low lr high lr inner product scores (b) adversarial tuning figure 2: an overall demonstration of our proposed classifier decomposition framework, which includes (a) empirical initialization: the previous knowledge is frozen and transferred to stage 2 via parameter initialization of the previous classifier; (b) adversarial tuning: during stage 1, feedforward nodes in the previous classifier generate adversarial signals, which helps the model learn robust representations. pose a simple yet effective classifier decomposition framework inspired by wang et al. (2021). the framework splits the last ffn layer into the separated previous and current classifiers, and introduces two enhanced strategies: empirical initialization and adversarial tuning. specifically, to alleviate the classifier bias, empirical initialization firstly preserves the well-learned previous classifier nodes and then reuses them after learning current relations. to ease the representation bias, adversarial tuning slows down the update of previous classifier weights. the slowly updating weights can be seen as a series of adversarial signals that induce cre models to learn distinguishable representations. to sum up: (1) we find that cre models suffer from classifier and representation biases when learning the new relations. (2) we propose a simple yet effective classifier decomposition framework with empirical initialization and adversarial tuning to alleviate these two biases. (3) extensive experiments on fewrel and tacred verify the effectiveness of our method. ## task formalization in cre, models are trained on a sequence of tasks {t 1 , t 2 , . . . , t k }, where the k-th task has its own training set d @xmath0, y i )} n @xmath1. each task t k is an independent supervised classification task to identify the instance x i (including the sentence and entity pair) into its corresponding relation label y i . the goal of cre is to learn new tasks while avoiding catastrophic forgetting of previously seen tasks. in other words, the model is evaluated to identify an instance into all c k prev + c k relations, where c k @xmath2. to alleviate catastrophic forgetting in cre, previous rehearsal-based work @xcite @xcite adopts an episodic memory module to store a few representative instances for each previous relation. in the subsequent training process, instances in the memory set will be replayed to balance the model's performance on all seen relations. ## methodology the overall of our proposed classifier decomposition framework is illustrated in figure 2 . following previous work @xcite @xcite , our model architecture contains two main components: (1) an encoder that generates representations of a given instance and (2) a feed-forward network (ffn) that maps the encoded representation into a probability distribution over all seen relations. ## conclusion in this work, we found that the vanilla training strategy adopted by most previous cre models in the first training stage leads to two typical biases: classifier bias and representation bias, which are important factors causing catastrophic forgetting. to this end, we propose a simple yet effective classifier decomposition framework with two enhanced strategies to help models alleviate those biases at the first training stage. experimental results on two benchmarks show that our framework consistently outperforms previous state-of-the-art cre models, which indicates that the value of this training stage to cre models may be undervalued. further analysis shows the effectiveness of our proposed classifier decomposition framework. limitations as a preliminary study, our proposed classifier decomposition framework focuses on the first training stage of cre models with a lack of explorations on stage 2. besides, more experiments can be conducted by combining our framework with previous leading cre models, which we leave for future research. in addition, our work only focuses on strategies with the ffn layer. as the bert encoder is the main component of cre models, we call for more attentions to the research of improving encoder representations in the first training stage. pages 35-45. association for computational linguistics. kang zhao, hua xu, jiangong yang, and kai gao. 2022. consistent representation learning for continual relation extraction. in findings of the association for computational linguistics: acl 2022, dublin, ireland, may 22-27, 2022, pages 3402-3411. association for computational linguistics.