Preserving Generalization of Language models in Few-shot Continual Relation Extraction (2024)

Abstract: Few-shot Continual Relations Extraction (FCRE) is an emerging and dynamic area of study where models can sequentially integrate knowledge from new relations with limited labeled data while circumventing catastrophic forgetting and preserving prior knowledge from pre-trained backbones. In this work, we introduce a novel method that leverages often-discarded language model heads. By employing these components via a mutual information maximization strategy, our approach helps maintain prior knowledge from the pre-trained backbone and strategically aligns the primary classification head, thereby enhancing model performance. Furthermore, we explore the potential of Large Language Models (LLMs), renowned for their wealth of knowledge, in addressing FCRE challenges. Our comprehensive experimental results underscore the efficacy of the proposed method and offer valuable insights for future work.

## introduction continual relations extraction (cre) @xcite @xcite is a learning scenario that requires a model to identify emerging relationships between entities or objects in texts @xcite @xcite while maintaining the accuracy of existing classifications and avoiding the problem of catastrophic forgetting @xcite @xcite @xcite . in many real-world situations, models must learn from a few new samples due to the limited availability of labeled training data for relations. as a result, few-short continual relation extraction (fcre) methods have been proposed @xcite to enable models to solve new tasks where each new relation has only a minimal number of corresponding samples. however, due to the lack of data, fcre models are often biased towards the current task compared to related scenarios, which can lead to forgetting previous knowledge and losing highly general priori from the pre-trained backbone. thus, the challenge of fcre is not only catastrophic forgetting but also severe overfitting. recent works @xcite @xcite tackles these issues by employing memory-based approaches inspired by traditional continual learning methods @xcite @xcite @xcite , along with various strategies to enhance the model's ability to distinguish relation representations. nevertheless, these methods solely fine-tune pre-trained bert-based backbones for few-shot tasks, which leads to eroding prior knowledge from the pre-trained model and hindering the final performance. additionally, these methods often neglect the pre-trained lm head in favor of training a new classifier from scratch, even though this component contains rich and general knowledge that remains untapped. therefore, we propose our mutual information maximization (mim) strategy that leverages pre-trained lm heads during training fcre models for the first time. our proposed strategy not only helps preserve the knowledge on the backbone but also assists in aligning the main classifier to improve representation learning. extensive experimental results on benchmark datasets demonstrate the effectiveness of our novel approach in preserving the pre-trained lm's generalization capability and reducing forgetting, leading to remarkable results. furthermore, pre-trained large language models (llms) @xcite with billions of parameters are known for their excellence in autoregressive text generation tasks. they have also been extensively studied in text classification and information extraction @xcite . however, these models often underperform compared to discriminative encoder models like bert due to their generation-focused mechanism. to address this, recent work @xcite proposed replacing ineffective llm heads with classification heads in the restricted space of the classification problem. this approach has shown promise, but the potential of llms in cl, specifically in fcre, remains underexplored. therefore, we conduct extensive experiments to answer: how the performance would llms yeild for fcre? how will limited data in this scenario impact the generalization of llms? we also assess the effectiveness of our mim strategy when using llm heads, which were eliminated due to their unsuitability. the results offer valuable insights for the community. to sum up, our main contributions are twofold: ## related work continual learning (cl) is a learning scenario that requires models to continually acquire new knowledge from a sequence of tasks while preventing the loss of previously learned information. the main challenge in cl is catastrophic forgetting @xcite . to address this problem, memorybased approaches prove to be effective methods for both machine learning @xcite and nlp problems @xcite . in particular, models need to save a few representative samples from the current task in a memory buffer and replay these samples when learning new tasks to review old knowledge. fewshot continual relation extraction is a challenging scenario, which was introduced by @xcite for relation extraction problems. this challenge arises due to the limited availability of data for new tasks, coupled with the high cost and time involved in obtaining high-quality data. recent work like wang et al. (2023); chen et al. (2023); ma et al. (2024) propose memorybased solutions, which suggest imposing objective functions on the embedding space and classification head. specifically, wang et al. (2023) employs serial objective functions based on contrastive and distillation, qin and joty (2022) leverage extra training data from unlabeled text, and chen et al. ## proposed method in this section, we first present our efficient strategy in section 4.1 that can flexibly adapt to the existing fcre methods and enhance model performance. after that, in section 4.2, we explain in detail the motivation and research questions when investigating llms in fcre. ## experimental results in this part, we first present the experiment setup in section 5.1, followed by the results that demonstrate the effectiveness of our proposed method (section 5.2) when using bert-based backbones. we then discuss the investigation results of using pre-trained llms for fcre tasks in section 5.3. ## conclusion in this work, we introduce a novel method that utilizes pre-trained language model heads to maintain the generalization of lms in fcre problems. by making use of this often ignored component through a mutual information strategy, our approach also significantly improves the comprehensiveness of the representation on the main classifier. additionally, we present comprehensive experimental results that demonstrate the impact of using llms for fcre and provide valuable insights to the community.