Enhancing Discriminative Representation in Similar Relation Clusters for Few-Shot Continual Relation Extraction (2025)

Abstract: Few-shot Continual Relation Extraction (FCRE) has emerged as a significant challenge in information extraction, necessitating that relation extraction (RE) systems can sequentially identify new relations with limited labeled samples. While existing studies have demonstrated promising results in FCRE, they often overlook the issue of similar relations, which is a critical factor contributing to catastrophic forgetting. In this work, we propose Sirus–a novel method that utilizes relation descriptions and dynamic clustering on these descriptions to identify similar relations. Leveraging this information, we introduce innovative loss functions specifically designed to enhance the distinction between relations, with a focus on learning to differentiate similar ones. Experimental results show that our approach can effectively mitigate the problem of catastrophic forgetting and outperforms state-of-the-art methods by a large margin. Additionally, we explore the potential of Large Language Model Embeddings (LLMEs) with representation learning and embedding capabilities, demonstrating their promise for advancing FCRE systems.

## introduction relation extraction (re) is a fundamental task in natural language processing (nlp) that involves recognizing relationships between entities from underlying content. traditional relation extraction approaches commonly require substantial labeled datasets and assume a static collection of predefined relationships @xcite @xcite . in real-world settings, various specialized fields, such as scientific research @xcite , medicine @xcite , or law @xcite , are rapidly advancing, leading to a continuous expansion in the diversity of relationships @xcite . consequently, re systems are required to exhibit adaptability to handle these evolving changes effectively. besides, another challenge in developing re models is the scarcity of annotated data for emerging relations. to this end, the concept of few-shot continual relation extraction (fcre) has been proposed @xcite to enable the continuous learning of new relations from a limited number of samples. fcre is a branch of continual information extraction @xcite @xcite . however, due to the continual learning process with limited data, fcre models are often biased toward the current task, facing the challenges of overfitting @xcite and catastrophic forgetting @xcite @xcite @xcite . several methods have been introduced to address these issues, with memory-based approaches integrated with contrastive learning emerging as a prominent paradigm @xcite @xcite @xcite . these methods typically involve retaining a few representative samples from previous tasks and applying contrastive learning to ensure that the representations of samples across different relations remain sufficiently distinguishable. however, none of the mentioned methods have considered the confusion between similar relations, which has been identified as a significant factor contributing to catastrophic forgetting in continual relation extraction @xcite . this phenomenon becomes even more critical in the fcre scenario, where the few available samples for each relation may not sufficiently represent the relations causing models to ignore subtle distinctions between similar relations, leading to catastrophic forgetting. a recent method, conpl, introduced by @xcite this is-sue by identifying similar classes based on the distance between relation prototypes and examples, and employs focal loss to emphasize the distinctions between these similar classes. the relation prototypes are computed by averaging the representations of entities within the same relations; however, a limited number of samples may fail to produce sufficiently representative prototypes @xcite . moreover, the problem of inconsistencies resulting from varying context sentences can render these prototypes unstable @xcite , reducing the effectiveness of this approach in identifying similar classes in the fcre scenario. recent studies have leveraged label descriptions @xcite @xcite for few-shot relation extraction, demonstrating their effectiveness in enriching representations and stabilizing label prototypes. especially, @xcite utilizes llms to augment data by generating additional label descriptions and multiple samples for each relation type. however, in this work, we focus solely on exploiting the original relation descriptions without using an llm-based data augmentation mechanism. we propose sirus, a novel method to enhance the discriminative representation in similar relation clusters, involves utilizing relation descriptions for label representation and employing dynamic clustering. as a result, the learning process enhances the differentiation of samples across relations, with a particular focus on similar relations, thereby reducing the phenomenon of catastrophic forgetting. furthermore, pre-trained large language models (llms) with billions of parameters excel in autoregressive text generation tasks @xcite and demonstrate strong performance on downstream tasks with only a few examples (brown, 2020; @xcite , making them a promising approach for application in fcre. they have also been explored in text classification and information extraction @xcite ; however, they often underperform compared to discriminative encoder models like bert due to their generation-focused mechanism potentially makes them less effective for text representation learning. in the fcre scenario, a recent study by @xcite has explored the capabilities of llms; however, it retained the use of causal language modeling (clm) and applied a classification head to the last token, which may not fully exploit the embedding ability of llms. several recent studies @xcite @xcite have investigated the capabilities of llms in representation learning by removing the causal mask and finetuning llms with contrastive learning, referred to as large language model embeddings (llmes), demonstrating promising results in retrieval and classification tasks. however, their ability for continual learning, particularly in the context of fcre, remains unexplored. therefore, we conduct comprehensive experiments on these llmes, offering valuable insights into the forgetting phenomenon within these models and their performance outcomes in fcre settings. in summary, our contributions are as follows: ## methodology in this section, we present our method aimed at enhancing fcre by tackling the problem of similar relations. specifically, we proposed clustering relations via label description (crld) to identify groups of similar relations. accordingly, we propose three loss functions designed to differentiate samples that are semantically similar but belong to distinct relations, using the clustering information. ## conclusion in conclusion, our novel approach to few-shot continual relation extraction (fcre) effectively addresses the challenge of similar relations, which often leads to catastrophic forgetting. by leveraging relation descriptions and dynamic clustering, we enhance the distinction between relations through innovative loss functions. our experimental results indicate that our approach achieves superior performance, surpassing state-of-the-art methods. moreover, our comprehensive investigation of large language model embeddings (llmes) demonstrates superior performance over both bert and decoder-only llms in all cases, emphasizing their potential to advance fcre systems. this work paves the way for more robust and accurate relation extraction systems, contributing to the broader field of information extraction.