Refining Sample Embeddings with Relation Prototypes to Enhance Continual Relation Extraction (2021)

Abstract: Continual learning has gained increasing attention in recent years, thanks to its biological interpretation and efficiency in many real-world applications. As a typical task of continual learning, continual relation extraction (CRE) aims to extract relations between entities from texts, where the samples of different relations are delivered into the model continuously. Some previous works have proved that storing typical samples of old relations in memory can help the model keep a stable understanding of old relations and avoid forgetting them. However, most methods heavily depend on the memory size in that they simply replay these memorized samples in subsequent tasks. To fully utilize memorized samples, in this paper, we employ relation prototype to extract useful information of each relation. Specifically, the prototype embedding for a specific relation is computed based on memorized samples of this relation, which is collected by K-means algorithm. The prototypes of all observed relations at current learning stage are used to re-initialize a memory network to refine subsequent sample embeddings, which ensures the model’s stable understanding on all observed relations when learning a new task. Compared with previous CRE models, our model utilizes the memory information sufficiently and efficiently, resulting in enhanced CRE performance. Our experiments show that the proposed model outperforms the state-of-the-art CRE models and has great advantage in avoiding catastrophic forgetting. The code and datasets are released on https://github.com/fd2014cl/RP-CRE .

## introduction as one of the most important tasks in information extraction (ie), relation extraction (re) has been * corresponding author widely applied in many downstream tasks, such as knowledge base construction and completion @xcite . the goal of re is to recognize a relation predefined in knowledge graphs (kgs) for an entity pair in texts. for example, given the entity pair @xcite in the sentence "interstellar is an epic science fiction film directed by christopher nolan", the relation thedirector-of should be recognized by an re model. conventional re models @xcite @xcite always assume a fixed pre-defined set of relations and perform once-and-for-all training on a fixed dataset. therefore, these models can not well handle the learning of new relations, which often emerge in many realistic applications given the continuous and iterative nature of our world @xcite . to adapt to such a situation, the paradigm of continual relation extraction (cre) is proposed @xcite @xcite . compared with conventional re, cre focuses more on helping a model keep a stable understanding of old relations while learning emerging relations, which in fact could be precisely modeled by continual learning. continual learning (or lifelong learning) systems are defined as adaptive algorithms capable of learning from a continuous stream of information @xcite , where the information is progressively available over time and the number of learning tasks is not pre-defined. continual learning remains a long-standing challenge for machine learning and deep learning @xcite , as its main obstacle is the tendency of models to forget existing knowledge when learning from new observations @xcite , which is called as catastrophic forgetting. recent works try to address the problem of catastrophic forgetting in three ways, including consolidation-based methods @xcite , dynamic archi-tecture @xcite and memory-based methods @xcite @xcite , in which memory-based methods have been proven promising in nlp tasks. in recent years, some memory-based cre models have made significant progress in overcoming catastrophic forgetting while learning new relations, such as ea-emr @xcite , mllre @xcite , cml @xcite and emar @xcite . despite of their effectiveness, there are some challenges remaining in current cre. one noticeable challenge is how to restore the sample embedding space disrupted by the learning of new tasks, given that re models' performance is very sensitive to the quality of sample embeddings. another challenge is that most existing cre models have not fully exploited memorized samples. in order to enhance re performance and overcome the overfitting problem caused by high replay frequency, the samples memorized in these models usually have the same magnitude as the original training samples @xcite , which is unrealistic in real-world tasks. inspired by prototypical networks @xcite for few-shot classification, we employ relation prototypes to represent different relations in this paper, which help the model understand different relations well. furthermore, these prototypes are used to refine sample embeddings in cre. this process is named as prototypical refining in this paper. specifically, the prototype for a specific relation is the average embedding of typical samples labeled with this relation, which are collected by kmeans and memorized by our model for future use. the prototypical refining can help our model recover from the disruption of embedding space and avoid catastrophic forgetting during learning new relations, thus enhance our model's cre performance. another advantage of prototypical refining is the efficient utilization of memorized samples, resulting in our model's less dependence on memory size. our contributions in this paper are summarized as follows: ## related works conventional studies in relation extraction (re) mainly focus on designing and utilizing various deep neural networks to discover the relations between entities given contexts, including: (1) convolutional neural networks (cnns) @xcite @xcite @xcite can effectively extract local textual features. (2) recurrent neural networks (rnns) @xcite @xcite are particularly capable of learning long-distance relation patterns. (3) graph neural networks (gnns) @xcite @xcite build word/entity graphs for cross-sentence reasoning. recently, pre-trained language models @xcite have also been extensively used in re tasks @xcite @xcite , and have achieved state-of-the-arts performance. however, most of these models can only extract a fixed set of pre-defined relations. hence, continual relation learning, i.e., cre, has been proposed to overcome this problem. existing continual learning methods can be divided into three categories: (1) regularization methods @xcite @xcite alleviate catastrophic forgetting by imposing constraints on updating the neural weights important to previous tasks. (2) dynamic architecture methods @xcite change architectural properties in response to new information by dynamically accommodating novel neural resources. (3) memory-based methods @xcite @xcite remember a few examples in previous tasks and continually replay the memory with emerging new tasks. for cre, the memory-based methods have been proven most promising @xcite . in addition, in order to accurately represent relations with limited samples, the idea of prototypical networks is intro-duced into re @xcite . there are also many memory networks proposed to remember information of long periods, such as lstm @xcite and memory-augmented neural networks @xcite . besides, a new memory module @xcite has demonstrated its success in relational reasoning, which employs multi-head attention to allow memory interaction. ## methodology in this section, we introduce our cre model in details. at first, we formalize the problem of cre and the memory module used in our model. ## end rk-1 by averaging the embeddings of memorized samples in m r with sample encoder e (section 3.3). these prototypes constitute a prototype set p k , which is used to memorize model's embedding space before training on t k . note that the encoder e is continuously changing with tasks, the prototypes of relations need to be regenerated at bert [cls] [e 11 ] [e 12 ] w 1 w 2 … w p … [e 21 ] [e 22 ] w q … … w n … entity 1 entity 2 concat. fully connected layer + layer normalization encoder output encoder input h 11 h 21 h figure 1: the structure of sample encoder e. the beginning of each task. initial training (line 16 ∼ 18): the parameters in sample encoder e and relation classifier c are tuned with the training samples in d k (section 3.4). ## conclusion in this paper, we propose a novel cre model obtaining enhanced performance through refining sample embeddings. in our model, the sample embeddings are refined by an attention-based memory network fed with relation prototypes, that are generated from memorized samples. the comparison experiments show that our model significantly outperforms current state-of-the-art cre models. as most current cre models are memory-based, we further explore the long-term effectiveness of episodic memory. the results show that our model has great advantages in maintaining diversity of memorized samples and performs well in avoiding catastrophic forgetting of old relations (tasks). because of the efficiency in memory mechanism, our model depends less on memory size. in future work, we will explore whether the mechanism of refining sample embeddings with prototypes can be used in other classification-based continual learning tasks.