A Two-phase Prototypical Network Model for Incremental Few-shot Relation Classification (2020)

Abstract: Relation Classification (RC) plays an important role in natural language processing (NLP). Current conventional supervised and distantly supervised RC models always make a closed-world assumption which ignores the emergence of novel relations in open environment. To incrementally recognize the novel relations, current two solutions (i.e, re-training and lifelong learning) are designed but suffer from the lack of large-scale labeled data for novel relations. Meanwhile, prototypical network enjoys better performance on both fields of deep supervised learning and few-shot learning. However, it still suffers from the incompatible feature embedding problem when the novel relations come in. Motivated by them, we propose a two-phase prototypical network with prototype attention alignment and triplet loss to dynamically recognize the novel relations with a few support instances meanwhile without catastrophic forgetting. Extensive experiments are conducted to evaluate the effectiveness of our proposed model.

## introduction relation classification (rc) is a fundamental task in natural language processing (nlp), aiming to assign semantic relations to the entity pairs mentioned in sentences. currently, conventional supervised @xcite or distantly supervised @xcite rc models are widely used and achieve remarkable performance. they are always based on a closed-world assumption that the relations expressed in query instances must have appeared in the pre-defined relation set. it is clearly limited in many realistic scenarios, especially in a dynamic or open environment. as shown in in contrast, it is intuitive that the human can learn new knowledge after being taught just few instances @xcite . based on this intuition, a series of few-shot rc models @xcite @xcite are proposed and can effectively recognize the novel relations with only a few (e.g., 1 or 5) support instances. nevertheless, current few-shot rc models only focus on learning novel relations and ignore a fact that many common relations (i.e., base relations) are readily available in large datasets. they neglect the existence of large-scale training data for base relations and still learn the base relations in the low-resource setting (i.e., each base relation is given only a few support instances), which cannot fully capture or model the features of base relations. to tackle this limitation of current few-shot rc models, our work in this paper considers a more realistic relation set sentence sample capital of s1:[london]e1 is the capital of the [u.k]e2 data of birth s2: [mark twain]e1 was born in [1835]e2 member of s3:[newton]e1 served as the president of the [royal society]e2 query set q:[gerard]e1 was the father of [gottfried]e2 for the closed-world assumption based rc models, the set of predefined relations (i.e., capital of, data of birth and member of ) is fixed after model training. in both fields of deep supervised learning and few-shot learning, prototypical networks @xcite obtain better performance on several benchmarks. they conduct classification by learning the distance distribution among relations. however, limited by the closed-world assumption, they only focus on the feature embedding learning for the base relations. when the novel relations come in, the feature spatial distributions of novel relations might be distorted and become incompatible with those of base relations. as shown in figure 1 , the base relations are well-distinguished in the feature embedding spaces. nevertheless, as the novel relations come in, the feature spatial distributions of novel relations are extremely wider than those of base ones and even overlap the spatial distributions of the base relations. it becomes infeasible to conduct classification simultaneously for base and novel relations. to solve this incompatible feature embedding problem, the prototype attention alignment (protoatt-alignment) and triplet loss function are designed in our proposed model. they aim to force the prototypical network to narrow down the feature spatial distributions of novel relations and meanwhile to enlarge the distances among different relations in the same embedding space. in our paper, we propose a two-phase prototypical network model with protoatt-alignment and triplet loss for incremental few-shot relation classification. the whole framework is shown in figure 2 . in the first phase, a deep prototypical network is proposed to learn the feature embedding space of base relations in a supervised learning manner, following @xcite . each base relation is represented as the center (base prototype) of its training instances. to dynamically recognize the novel relations, the novel prototype generator is designed to learn the representations for novel relations (novel prototype) with only a few support data. then, an incremental prototypical network with novel prototype generator is proposed in the second phase and classification is conducted by comparing the distances between query instance and each prototype (i.e., both the base and novel prototypes). the main contributions of this paper can be summarized as follows: (1) we explore a problem of incremental few-shot relation classification and propose a two-phase prototypical network model to dynamically recognize the novel relations with a few support data meanwhile without catastrophic forgetting. to the best of our knowledge, our work is the first study focusing on incremental few-shot relation classification. ## related work relation classification (rc) is one of the most important techniques in natural language processing (nlp) and has various applications such as information retrieval @xcite , question answering @xcite and dialogue systems @xcite . currently, conventional deep supervised @xcite and distantly supervised @xcite @xcite ) rc models are widely used and achieve remarkable performance. they are always based on the closed-world assumption @xcite that the relation expressed in the query instances must have appeared in the pre-defined relation set. the set of relations which rc models can recognize is fixed after training. however, it is often violated and limited a lot in many realistic scenarios, especially in a dynamic or open environment. novel relations can emerge dynamically in an open-world scenario. to dynamically expand the fixed relation set, two solutions can be concluded. firstly, the base and straightforward method is re-training @xcite . every time the novel relations come in, we need to collect training data for novel relations and then train from scratch on the enhanced training data, aiming to avoid catastrophic forgetting @xcite . however, the repeating training process is computationally expensive and time-consuming. recently, two lifelong learning based rc models @xcite are proposed to alleviate the expensive re-training process. nevertheless, both the solutions still suffer from the lack of large-scale of training data for novel relations. without enough training data for novel relations, both the above two solutions risk overfitting on the recognition of novel relations and even suffer from catastrophic forgetting on base relations. in contrast, humans have the ability to perform even one-shot classification, where only one example of each new category is given. based on this intuition, a series of few-shot rc models are proposed. they can be classified into two categories: meta-learning based models @xcite @xcite and metric-learning based models @xcite @xcite @xcite @xcite . however, the few-shot rc models only focus on novel relations learning, but ignore a fact that many common relations are readily available in large datasets. to tackle this problem, we consider a more realistic setting where the relations learning system can not only learn the base relations from the large-scale training data, but also dynamically recognize the novel relations with only a few support examples (termed as incremental few-shot relation classification). currently, several related works @xcite @xcite are proposed in computer vision field and they concentrate on image classification task. different from images, the text is more diverse and noisy. it is hard to directly generalize to nlp applications @xcite . in this paper, we propose a two-phase prototypical network model for incremental few-shot relation classification. extensive experiments are conducted to evaluate the effectiveness of our proposed model. ## model to address the problem of incremental few-shot relation classification, we propose a two-phase prototypical network model with the prototype attention alignment and an auxiliary triplet loss (increprotonet). in a dynamic and open environment, novel relations can emerge in test stage. however, current conventional supervised rc models are based on the closed-world assumption and neglect the emergence of novel relations. although current few-shot rc models achieve remarkable performances on recognizing novel relations with a few support instances, they ignore a fact that the common relations (i.e., base relations) are readily available in large datasets. to simultaneously learn the base relations with large training data and dynamically recognize the novel relations with only a few support data, a two-phase increprotonet is proposed, as shown in figure 2 . the first phase, named deep prototypical network, is designed to pre-train a base model for base relations in a deep supervised manner. the second phase, named incremental prototypical network, is proposed to dynamically recognize the novel relations with only a few support instances meanwhile do not forget the base relations. 3.1 problem definitions and notations the incremental few-shot relation classification can be defined as a task as follows. we assume there exist a large dataset d @xmath0,@xmath1,i , h b,i , t b,i , r b )} k b @xmath2, where k b is the number of training instances of r b base relation and i b,i means entity pair (h b,i , t b,i ) mentioned in sentence x b,i which expresses the semantic relation r b . using this large training data of base relations, our work aims to effectively learn the base relations and meanwhile to dynamically recognize the novel relations with only a few (e.g., 1 or 5) support instances. therefore, given a support set s for n novel novel relations, the model can classify the entity pair (h, t) mentioned in query instance q into the most possible relation r @xmath3, where r @xmath4 ## conclusion in this paper, we propose a two-phase prototypical network model for incremental few-shot relation classification. current conventional supervised rc models are always based on the closed-world assumption that the relations expressed in query instances must have appeared in the pre-defined relations. however, novel relations often emerge in the dynamic or open-world environment. although current few-shot rc models effectively recognize the novel relations with only a few support instances, they ignore a fact that the common relations (i.e., base relations) are readily available in large datasets. to simultaneously learn the base relations with large-scale training data and the novel relations with a few support data, an incremental few-shot relations learning model is proposed in our paper. the extensive experimental results and visualization analysis show that our proposed model can effectively recognize the novel relations with a few support data and maintain high recognition accuracy on base relations.