Improving Continual Relation Extraction by Distinguishing Analogous Semantics (2023)

Abstract: Continual relation extraction (RE) aims to learn constantly emerging relations while avoiding forgetting the learned relations. Existing works store a small number of typical samples to re-train the model for alleviating forgetting. However, repeatedly replaying these samples may cause the overfitting problem. We conduct an empirical study on existing works and observe that their performance is severely affected by analogous relations. To address this issue, we propose a novel continual extraction model for analogous relations. Specifically, we design memory-insensitive relation prototypes and memory augmentation to overcome the overfitting problem. We also introduce integrated training and focal knowledge distillation to enhance the performance on analogous relations. Experimental results show the superiority of our model and demonstrate its effectiveness in distinguishing analogous relations and overcoming overfitting.

## introduction relation extraction (re) aims to detect the relation between two given entities in texts. for instance, given a sentence "remixes of tracks from persona 5 were supervised by kozuka and original composer shoji meguro" and an entity pair (persona 5, shoji meguro), the "composer" relation is expected to be identified by an re model. conventional re task assumes all relations are observed at once, ignoring the fact that new relations continually emerge in the real world. to deal with emerging relations, some existing works @xcite @xcite @xcite @xcite study continual re. in continual re, new relations and their involved samples continually emerge, and the goal is to classify all observed relations. therefore, a continual re model is expected to be able to learn new relations while retaining the performance on learned relations. existing works primarily focus on storing and replaying samples to avoid catastrophic forgetting @xcite of the learned relations. on one hand, considering the limited storage and computational resources, it is impractical to store all training samples and re-train the whole model when new relations emerge. on the other hand, replaying a small number of samples every time new relations emerge would make the model prone to overfit the stored samples @xcite . moreover, existing works simply attribute catastrophic forgetting to the decay of previous knowledge as new relations come but seldom delve deeper into the real causation. we conduct an empirical study and find that the severe decay of knowledge among analogous relations is a key factor of catastrophic forgetting. to address the above issues, we propose a novel continual extraction model for analogous relations. specifically, we introduce memory-insensitive relation prototypes and memory augmentation to reduce overfitting. the memory-insensitive relation prototypes are generated by combining static and dynamic representations, where the static representation is the average of all training samples after first learning a relation, and the dynamic representation is the average of stored samples. the memory augmentation replaces entities and concatenates sentences to generate more training samples for replay. furthermore, we propose integrated training and focal knowledge distillation to alleviate knowledge forgetting of analogous relations. the integrated training combines the advantages of two widely-used training methods, which contribute to a more robust feature space and better distinguish analogous relations. one method uses contrastive learning for training and generates prototypes for relation classification, while the other trains a linear classifier. the focal knowledge distillation assigns high weights to analogous relations, making the model more focus on maintaining their knowledge. our main contributions are summarized below: ## related work continual learning studies the problem of learning from a continuous stream of data @xcite . the main challenge of continual learning is avoiding catastrophic forgetting of learned knowledge while learning new tasks. existing continual learning models can be divided into three categories: regularization-based, dynamic architecture, and memory-based. the regularization-based models @xcite impose constraints on the update of parameters important to previous tasks. the dynamic architecture models @xcite dynamically extend the model architecture to learn new tasks and prevent forgetting previous tasks. the memory-based models @xcite @xcite store a limited subset of samples in previous tasks and replay them when learning new tasks. in continual re, the memory-based models @xcite @xcite @xcite @xcite are the mainstream choice as they have shown better performance for continual re than others. to alleviate catastrophic forgetting, previous works make full use of relation prototypes, contrastive learning, multi-head attention, knowledge distillation, etc. ea-emr @xcite introduces memory replay and the embedding aligned mechanism to mitigate the embedding distortion when training new tasks. cml @xcite combines curriculum learning and meta-learning to tackle the order sensitivity in continual re. rp-cre @xcite and kip-framework @xcite leverage relation prototypes to refine sample representations through multi-head attention-based memory networks. additionally, kip-framework uses external knowledge to enhance the model through a knowledge-infused prompt to guide relation proto-augmentative memory encoder linear linear linear relu contrastive softmax sim (•) linear probability 𝑃𝑃(𝜃𝜃 𝑘𝑘 ) 𝐿𝐿 c_cls 𝐿𝐿 l_cls softmax encoder new task data classifier 𝐿𝐿 new k-means relation prototype typical samples dynamic static … [e11] 𝑒𝑒 𝑖𝑖𝑖 [e12] … [e21] 𝑒𝑒 𝑖𝑖𝑖 [e22] … memory augmentation focal knowledge distillation integrated training augmentative memory encoder encoder sim (•) sample representation (1 -𝑃𝑃 𝜃𝜃 𝑘𝑘 ) 𝛾𝛾 probability 𝑃𝑃(𝜃𝜃 𝑘𝑘-𝑖 ) 𝐿𝐿 fkd ⨀ lin./cont. lin./cont. similarity weight 𝑠𝑠 𝑖𝑖,𝑗𝑗 probability 𝑃𝑃(𝜃𝜃 𝑘𝑘 ) [e11] 𝑒𝑒 𝑗𝑗𝑖 [e12] … [e11] 𝑒𝑒 𝑗𝑗𝑖 [e12] … … … 𝑒𝑒 𝑚𝑚𝑖 … 𝑒𝑒 𝑚𝑚𝑖 … … [e11] 𝑒𝑒 𝑖𝑖𝑖 [e12] … [e21] 𝑒𝑒 𝑖𝑖𝑖 [e22] … … 𝑒𝑒 𝑛𝑛𝑖 … 𝑒𝑒 𝑛𝑛𝑖 … … 𝑥𝑥 𝑖𝑖𝑗𝑗 𝑟𝑟 𝑥𝑥 𝑖𝑖-𝑚𝑚 𝑟𝑟 𝑥𝑥 𝑖𝑖𝑗𝑗-𝑛𝑛 𝑟𝑟 𝑥𝑥 𝑖𝑖 𝑟𝑟 [e11] 𝑒𝑒 𝑗𝑗𝑖 [e12] … [e11] 𝑒𝑒 𝑗𝑗𝑖 [e12] … type generation. emar @xcite , crl @xcite , and crecl @xcite leverage contrastive learning for model training. besides, knowledge distillation is employed by crl to maintain previously learned knowledge. aca @xcite is the only work that considers the knowledge forgetting of analogous relations ignored by the above works and proposes an adversarial class augmentation strategy to enhance other continual re models. all these models do not explicitly consider the overfitting problem @xcite , which widely exists in the memory-based models. as far as we know, a few works @xcite in other continual learning fields have tried to reduce the overfitting problem and achieve good results. we address both the problems of distinguishing analogous relations and overfitting to stored samples, and propose an end-to-end model. ## task definition a continual re task consists of a sequence of tasks @xmath0, t 2 , . . . , t k }. each individual task is a conventional re task. given a sentence, the re task aims to find the relation between two entities in this sentence. the dataset and relation set of t @xmath1, respectively. d k contains separated training, validation and test sets, denoted by d train k , d valid k and d test k , respectively. r k contains at least one relation. the relation sets of different tasks are disjoint. continual re aims to train a classification model that performs well on both current task t k and previously accumulated tasks tk-1 = k-1 @xmath2. in other words, a continual re model is expected to be capable of identifying all seen relations @xmath3 ## experiments and results in this section, we report the experimental results of our model. the source code is accessible online. ## conclusion in this paper, we study continual re. through an empirical study, we find that knowledge decay among analogous relations is a key reason for catastrophic forgetting in continual re. furthermore, the overfitting problem prevalent in memorybased models also lacks consideration. to this end, we introduce a novel memory-based model to address the above issues. specifically, the proposed memory-insensitive relation prototypes and memory augmentation can reduce overfitting to typical (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (1) (2) (3) (4) (5) (6) (7) (8) (9) (10)