Enhancing Contrastive Learning with Noise-Guided Attack: Towards Continual Relation Extraction in the Wild (2024)

Abstract: The principle of continual relation extraction (CRE) involves adapting to emerging novel relations while preserving old knowledge. Existing CRE approaches excel in preserving old knowledge but falter when confronted with contaminated data streams, likely due to an artificial assumption of no annotation errors. Recognizing the prevalence of noisy labels in real-world datasets, we introduce a more practical learning scenario, termed as noisy-CRE . In response to this challenge, we propose a noise-resistant contrastive framework called Noise-guided Attack in Contrastive Learning (NaCL), aimed at learning incremental corrupted relations. Diverging from conventional approaches like sample discarding or relabeling in the presence of noisy labels, NaCL takes a transformative route by modifying the feature space through targeted attack. This attack aims to align the feature space with the provided, albeit inaccurate, labels, thereby enhancing contrastive representations. Extensive empirical validations demonstrate the consistent performance improvement of NaCL with increasing noise rates, surpassing state-of-the-art methods.

## introduction alongside the predictive wins of relation extraction (re) on various benchmarks @xcite , the need for the ability to acquire sequential experience in dynamic environments stands out the significance. catering to the real-world learning requirement, a new re formulation, namely continual relation extraction (cre), has been proposed @xcite . under this topic, catastrophic forgetting @xcite where previous knowledge is overwritten as new concepts are learned, remains a key challenge. to prevent forgetting, a variety of sophisticated methods are developed by memory replay @xcite , weight regularization @xcite or architecture expansion @xcite . @xcite explicitly store past experiences into a limited memory and replay them to complement new tasks learning. in comparison to exemplars storage, @xcite impose constraints on the update of the important network weights for old knowledge consolidation. as for architecturebased method, it dynamically changes model architectures to acquire new information while remembering previous knowledge @xcite . despite the effectiveness, all of these methods implicitly assume the correctness of the labels for the streaming data. in practice, such an assumption is rather artificial even impossible to satisfy since label shifts are inevitable in real-world scenarios. worse still, official statistics in the table of figure 1 reveal that the widely used benchmarks with elaborate human annotations, likewise, contain a certain proportion of noisy labels. due to the ignorance of noisy labels over data streams, it is clear to see in figure 1 that state-of-the-art cre models fail to defend against label inconsistency, resulting in significant performance drops. to break the impractical structure of current cre setup and to enhance the noise-resistant capacity of models, in this paper, we present a more generalized learning setting coined as noisy-cre. in this challenging scenario, there is a potential for mislabeled samples to contaminate the sequential stream in every incremental task. we assume that models trained under the noisy-cre setting can reflect their ability to adapt to new relations in the real world. in the face of the great challenge, in this paper, we propose a robust contrastive framework as noise-guided attack contrative learning (nacl) for noisy-cre. generally, handling noisy labels can be relaxed to a subsequent process of clean sample selection and noisy sample correction. in nacl, we introduce an auxiliary model to play the two roles. first, at each new task, the auxiliary model will be re-initialized to train for new relations learning. intriguingly, we term it as reboot, which can make the model escape the interference of prior knowledge so that its logit outputs can be a measure of clean sample selection for current task. second, this model will translate a novel sight into feature space for correction by performing noiseguided attack. this attack can actively drive the feature distribution of noisy negatives more aligned with their given labels. to demonstrate the effectiveness of nacl, we design two benchmarks based on fewrel and ta-cred. empirical results and in-depth analyses show that our nacl can achieve consistent improvements when noise rates vary from light to heavy, and it outperforms all state-of-art baselines far ahead. in summary, the contributions of this work are three-fold: ## noisy-cre setting formulation continual relation extraction is defined as training models on non-stationary data from sequential tasks. in the setup of noisy-cre, we first define a sequence of tasks @xmath0, • • • , t n ). for the k-th task t k , its training dataset is denoted as d k @xmath1, y i )} n k @xmath2, where y has a probability of rate to be corrupted. our goal is to train a single model f θ : @xmath3, such that it predicts the label @xmath4. protocols for label corruption. in an ideal cre mode, each task has independent relation space y. however, for noisy-cre, due to the inevitable label corruption, this assumption does not hold in the training set. as shown in figure 2 , the relation space y k of the k-th task can be contaminated arbitrarily by samples from label space y i with @xmath5, • • • , k -1, k +1, • • • , n}, thus leading to two kinds of noisy labels. when @xmath6, we term these noisy labels as closed-set ones, since their gold relations are embedded in the model knowledge and can be recovered. in contrast, when @xmath7, the gold relations of the noisy ones are unreachable and formed as open-set noise. ## nacl: towards noise-resistant cre in this section, we present nacl, our noiseresistant contrastive learning framework designed to simultaneously handle closed-set and open-set noisy labels in the noisy-cre scenario. ## conclusion building on the recent wave of learning without forgetting, in this paper, we demonstrate current con-tinual learners are vulnerable under natural label shifts. hence, we propose a novel noise-resistant contrastive learning framework nacl to correct the false contrastive pairs brought by the co-existence of closed-set and open-set label noise. comprehensive experiments and analyses validate that our method can achieve the triple wins that boost old knowledge, new task learning and noisy label robustness in one integrated algorithm.