An Ensemble-of-Experts Framework for Rehearsal-free Continual Relation Extraction (2024)

Abstract: Continual relation extraction (CRE) aims to continuously learn relations in new tasks without forgetting old relations in previous tasks.Current CRE methods are all rehearsal-based which need to store samples and thus may encounter privacy and security issues.This paper targets rehearsal-free continual relation extraction for the first time and decomposes it into task identification and within-task prediction sub-problems. Existing rehearsal-free methods focus on training a model (expert) for within-task prediction yet neglect to enhance models’ capability of task identification.In this paper, we propose an Ensemble-of-Experts (EoE) framework for rehearsal-free continual relation extraction. Specifically, we first discriminatively train each expert by augmenting analogous relations across tasks to enhance the expert’s task identification ability. We then propose a cascade voting mechanism to form an ensemble of experts for effectively aggregating their abilities.Extensive experiments demonstrate that our method outperforms current rehearsal-free methods and is even better than rehearsal-based CRE methods.

## introduction relation extraction (re) aims to identify the relation between two entities mentioned in the sentence, which is essential for many tasks like knowledge graph construction @xcite . traditional re methods based on pre-defined relation sets have achieved remarkable performance but cannot handle emerging relations in the real world @xcite . hence, continual relation extraction (cre) is proposed @xcite to deal with such a problem, with the goal of continuously learning new relations while not forgetting previously learned ones. existing cre methods need to store a few typical samples for each learned relation and then utilize memory replay to avoid the catastrophic forgetting problem in continual learning. however, due to privacy and security issues, storing training data is not allowed in sensitive areas such as finance and biology @xcite . in light of this, we consider the rehearsal-free continual relation extraction (rfcre) problem for the first time, for which we expect to incrementally learn new relations without storing any training data. following the pioneering work in continual learning @xcite , we decompose rfcre into two sub-problems: task identification and within-task prediction. benefiting from parameterefficient tuning @xcite @xcite , the within-task prediction problem has been well addressed by saving the task-specific parameters of each task (named as an expert). then, the core challenge lies in the task identification problem, i.e., how to identify the task-id for each test sample. along this line, @xcite employs a pre-trained language model (plm) @xcite based method for task identification in continual text classification. however, when applying this method to the re task, we find its performance is not very satisfying. the main reason is that only one plm is involved in task identification, as shown in fig. 1(a ). the knowledge in plm might be enough for the text classification task but is insufficient for the re task, whose performance relies heavily on task-related knowledge. to address this problem, we propose an ensemble-of-experts (eoe) framework for rehearsal-free continual relation extraction. our goal is to aggregate the experts of different tasks from different stages during the continual learning who already possess rich task-related knowledge for both within-task prediction and task identification, as shown in fig. 1 (b ). despite the appealing target, we now face two new research questions. for [rq1], we find the challenge in task identification mainly comes from the analogous relations across tasks, e.g., father and sibling, and it is pretty hard for an expert to discern such relations. in view of this, we introduce a discriminative training @xcite approach to increase each expert's task identification ability. specifically, we develop a novel relation augmentation method to generate two types of analogous relations. one is to reverse the positions of two entities in an old relation @xcite . the other is to remove the context between two entities in an old relation. since these augmented relations have not been seen by the expert of the current task, they will force the expert to distinguish old relations from the augmented ones during the discriminative training process. for [rq2], we find that directly aggregating the votes from all experts will bring about a transboundary risk. this is because the scope of task identification varies among different experts due to the temporal nature of continuous learning. for example, the t th expert has not seen the (t -1) th task, so the (t -1) th task is out of the scope of the t th expert. to handle this, we propose a cascade voting mechanism consisting of a two-phase voting procedure. specifically, in the first phase, we use the first expert and the plm to make an initial decision. then, in the second phase, we dynamically aggregate votes from qualified experts for the final decision. our main contributions are as follows: ## related work continual learning (cl) the goal of continual learning is to extend knowledge from a continuous data stream without catastrophic forgetting @xcite . existing cl methods can be roughly categorized into three groups: (a) regularization-based methods mitigate catastrophic forgetting by introducing parameter regularization terms @xcite or knowledge distillation loss @xcite , (b) rehearsal-based methods store a small set of old training samples to assist the model in replaying learned knowledge. current cre methods @xcite @xcite @xcite @xcite @xcite @xcite ) mainly focus on retaining or recovering the old relation knowledge during the memory replay stage, such as prototype learning @xcite , curriculum learning @xcite , knowledge distillation @xcite , contrastive learning @xcite . (c) network-based methods dynamically expand or split the network parameters to learn new tasks. thanks to the development of parameter-efficient tuning (pet) methods @xcite @xcite , several studies @xcite try to assign task-specific parameters to each task and then query the task-id for the given test sample. in contrast to existing rehearsal-based cre methods, we target rehearsal-free cre, which is the first attempt in this field. ## problem formulation relation extraction aims to determine whether the relation y holds for the head and tail entity in the given instance x. continual relation extraction (cre) aims to train the model from a series of class-incremental relation extraction tasks {t 1 , t 2 , • • • , t n }, where n is the number of tasks, each task t k contains its training set ## methodology as shown in fig. 2 , when the new task t k comes, the overall framework is divided into three steps: ## analysis 6.1 ablation study to validate the effectiveness of each component in our proposed framework, we conduct an ablation study. specifically, (1) w/o dt denotes we remove the discriminative training, (2) w/o ur denotes ## conclusion in this work, we make the first attempt on the rehearsal-free continual relation extraction problem. we propose an ensemble-of-experts framework consisting of discriminative training and cascade voting. specifically, we first introduce discriminative training to enhance the identification ability of experts when facing inter-task analogous relations. we then propose a cascade voting mechanism to aggregate experts' abilities from different stages. extensive experimental results show that our method significantly outperforms existing rehearsal-free and rehearsal-based continual relation extraction methods.