Improving Few-Shot Relation Classification by Prototypical Representation Learning with Definition Text (2022)

Abstract: Few-shot relation classification is difficult because the few instances available may not represent well the relation patterns. Some existing approaches explored extra information such as relation definition, in addition to the instances, to learn a better relation representation. However, the encoding of the extra information has been performed independently from the labeled instances. In this paper, we propose to learn a prototype encoder from relation definition in a way that is useful for relation instance classification. To this end, we use a joint training approach to train both a prototype encoder from definition and an instance encoder. Extensive experiments on several datasets demonstrate the effectiveness and usefulness of our prototype encoder from definition text, enabling us to outperform state-of-the-art approaches.

## introduction relation classification (rc) aims to determine the relation expressed between two entities in a sentence. typical approaches to rc train a classification model or a prototype from labeled sentences, which are intended to represent the typical relation patterns including various syntactic, semantic and contextual features. in practical situations, we may have only a few labeled examples, which are insufficient for training a good classification model. in some cases, the labeled examples may also be atypical. for example, if a relation is exemplified by sentences describing obama's presidency in the us, the resulting prototype would be largely tuned toward a political context, which would be difficult to apply outside the political context. even for a human annotator, with only a relation id and some examples, it would be difficult to learn to classify a relation correctly. this is exactly the same problem faced by a few-shot rc system: the best guess based on the limited labeled examples may fail. if, however, the human annotator is told that the relation is named "position held", then he/she would have a better understanding of the relation to better generalize the examples to other instances. in this case, the human annotator indeed exploits the prior knowledge about the relation (from its name). if more information such as a description of the relation is available, then the annotator would do an even better job. the definition of a relation, being it the name or the description, is important to help the human annotator understand the relation. it is the same for an automatic rc model. several previous studies have explored using such extra knowledge about the relation @xcite @xcite . for example, relation names have been used for better initializing the relation prototype before being trained with labeled instances @xcite . however, dong et al. only considered a limited number of relations in the training data, making the few-shot rc model prone to over-fitting. inspired by these studies and human behaviors, in this paper, we leverage relation definitions to build high-level prototype representations of the relations. a second problem we consider in this paper is the ability to construct prototypes for new relations, i.e. zero-shot learning. in this case, if the new relation has a description, then one would be able to apply the same prototype encoder learned on the other relations to the new relation. in other words, the mechanism of building relation prototypes from their definitions could be generalized and transferred to a new relation. some previous work @xcite has considered prototype transfer based on a knowledge graph which provides relationships between relations. it is assumed that a relation's prototype can be partly transferred to a related relation in the graph. while this could be a possible way to generate a more reasonable prototype for a relation, we believe that relation definitions provide a better basis for constructing a relation prototype. we propose to learn a general mapping function from relation definitions to their prototypes. the advantages of the approach come from the fact that the definition text expresses the intrinsic semantics of relation, which is not explicitly covered by labeled instances. for example, as shown in mapping a relation definition to a prototype vector could be done naively using a pre-trained language model such as bert @xcite , but the resulting prototype may not be the most useful for relation instance classification. we believe that a good prototype encoded from a definition should be the one that helps rc. therefore, we propose to train a prototype encoder together with rc of some examples. to tackle the problem of limited labeled examples, inspired by the rc-oriented pre-training work @xcite , we use abundant distantly labeled data to help train a prototype representation. the pre-training will generate the relation prototype vector that can best classify the weakly labeled data. to integrate the prototype representation learned from relation definitions with the given limited hand-labeled instances, we adapt the bayesian meta-learning approach @xcite to learn a posterior distribution of the prototype vectors of relations based on both the initial prototype representations and the labeled instances. this process helps adapt the prototype to the labeled instances. we test our approach on two few-shot rc datasets. it outperforms previous competitive models that apply pre-trained instance encoder or rela-tion definition text. we also show that the encoder can be easily generalized to new relations in zeroshot rc setting. experimental results demonstrate the effectiveness and generalization ability of our pre-trained prototype encoder. our main contribution in this paper is twofold: 1. we propose a new relation prototype construction method from relation definition; 2. we experimentally show that the approach is effective in few-shot rc and can be generalized to new relations in zeroshot rc. ## problem definition rc aims to predict whether a sentence (instance) x expresses a pre-defined relation between two given entities (e 1 , e 2 ). neural rc models usually contain an instance encoder that encodes the relation expressed by an instance into a dense vector and a classification layer to classify the dense vector to the relation which has the most similar prototype vector. in this work, we leverage the definition text denoted as y to help learn the relation prototype. the prototype encoder is trained to help the classification of relation instances. some training instances are required. inspired by the rc-oriented pre-training work @xcite , we leverage a large set of distantly labeled data. the distant labeling @xcite is done as follows: given a known relation containing a pair of entities, any sentence mentioning the same entity pair is labeled by the relation. let us denote the set of distantly labeled instances as d : {(x i , e 1i , e 2i , r i )} z @xmath0: {(y t , r t )} n @xmath1. the instance encoder (instance relation encoder) relenc φ , parameterized by φ, produces an instance embedding @xmath2, e 1 , e 2 ), where d is the vector dimension of prototype representation. the prototype encoder protoenc θ , parameterized by θ, produces a set of definition representations {v t } n @xmath3, where v @xmath4. the training goal is to map definition representations to relation prototypes useful for classifying instances. ## methodology in this section, we first explain how a relation prototype is learned with the help of a set of distantly labeled instances. then we describe how the learned prototype is further enhanced by the limited labeled instances in a bayesian meta-learning framework. ## experiments pre-training details to run experiments with limited computation resources, we use bert base 2 as the backbone of instance encoder and a fourlayer tinybert foot_2 for the prototype encoder. we obtain a large-scale distantly-labeled dataset by processing the largest available alignments -t-rex @xcite , which align the documents from wikipedia and triplets from wikidata. we removed the relations that do not have the textual definition, i.e., all the relations in the distantly labeled data have the textual definition. after removing the repetitive instances, the final dataset contains 636 relations and ∼ 8m instances. we get the relation name and descriptions from wikidata. the hyper-parameters we used during the pretraining process are: batch size is 96; the number of training epochs is 3; optimizer is adam with a learning rate of 1e-4, which decays by 0.8 per epoch. our pre-training takes about 68 hours on four v100 gpus. ## related work relation classification (rc) is pivotal for natural language understanding and has been studied for a long time @xcite . supervised machine learning approaches achieve remarkable progress on rc @xcite , but rely on high-quality labeled data. to relieve the heavy burden of manual annotation, researchers study rc under distant supervision @xcite or few-shot rc @xcite . the former focuses on robust classifier training with automatically labeled noisy data @xcite . our work belongs to the latter, which aims to learn general knowledge transferable to new relations. few-shot learning methods have been well studied for image classification @xcite and some classical approaches such as prototype network @xcite , model-agnostic meta-learning @xcite have been applied for rc @xcite . two types of efforts have been devoted to improving few-shot rc. firstly, some approaches @xcite @xcite @xcite design specific model architectures such as using attention mechanism to model complex interactions between labeled instances. however, these approaches are still limited when the few labeled instances are atypical and does not reflect the general patterns of the relation. secondly, researchers leverage extra information to complement the insufficient labeled data @xcite . our method belongs to this line. some methods @xcite leverage extra knowledge from kbs but they cannot deal with relations not covered by the kbs, showing limited applicability. similar to our work, some studies @xcite use relation names or descriptions as extra information. they design specific modules to integrate the extra information, but the whole few-shot rc model suffers from over-fitting and unstable performance due to the limited number of training relations. in contrast, our method adopts pre-training techniques to learn a general mapping function, which is more applicable and proven effective for domain adaptation. some recent studies @xcite @xcite conduct rcoriented pre-training to learn a general-purpose instance encoder. such instance encoders improve rc on both supervised and few-shot learning settings. we adopt the same idea of pre-training but focus on learning the general function of mapping relation definitions to the prototype space for relation classification. ## conclusion this paper studies prototypical representation learning for few-shot relation classification, and the key idea is to encode the definition text as prior knowledge to help classify new relations. we proposed to train a general-purpose prototype encoder that could encode the definition text of any relation into the prototype space. an instance encoder and the prototype encoder are trained jointly with a multi-instance learning method on distantly labeled data. applying our prior prototypes with a bayesian meta-learning approach, our method outperforms previous state-of-the-art models by using pre-trained instance encoder on two datasets, verifying its wide applicability. our prototype model also presents competitive performance on the zeroshot learning setting.