Better Few-Shot Relation Extraction with Label Prompt Dropout (2022)

Abstract: Few-shot relation extraction aims to learn to identify the relation between two entities based on very limited training examples. Recent efforts found that textual labels (i.e., relation names and relation descriptions) could be extremely useful for learning class representations, which will benefit the few-shot learning task. However, what is the best way to leverage such label information in the learning process is an important research question. Existing works largely assume such textual labels are always present during both learning and prediction. In this work, we argue that such approaches may not always lead to optimal results. Instead, we present a novel approach called label prompt dropout, which randomly removes label descriptions in the learning process. Our experiments show that our approach is able to lead to improved class representations, yielding significantly better results on the few-shot relation extraction task.

## introduction enabling machines to comprehend sentences and extract relations between entities has been a crucial task in natural language processing (nlp). conventional methods frame this task as a multiclass classification problem, trying to solve it through large-scale supervised training with lstm @xcite or bert @xcite as the backbone @xcite @xcite . such an approach has shown great effectiveness. however, one problem left unsolved is to identify novel relations with only a handful of training examples. therefore, recent studies @xcite introduce the task of few-shot relation extraction @xcite to study this data scarcity problem. aligned with the success of few shot learning in computer vision @xcite ; sator-figure 1 : an example of 2-way-1-shot learning using label prompt dropout (lpd). top: instead of assuming textual labels are always present for support instances, lpd randomly drops out such textual labels. here the textual label "country of origin" for the second instance is droppoed out. bottom: lpd directly concatenates the textual label and the context sentence. the textual label serves as a prompt to guide bert to derive a better class prototype. note that for simplicity we use the relation names here, while in our implementation we use relation descriptions, which are lengthier and more complex. while the bert-based prototypical networks @xcite have shown impressive performance on fsre, the class prototypes are only constructed through the average representation of support instances of each class, neglecting the textual labels that may provide additional useful information. therefore, recent efforts try to modify the prototypical network such that it can use the label information as well. @xcite insert both entity type information and relation descriptions to the model. @xcite use a relation encoder to generate relation representation besides the sentence encoder. @xcite propose a hybrid prototypical network that can generate hybrid prototypes from context sentences and relation descriptions. nonetheless, these methods largely assume that every support instance is provided with a corresponding textual label in the support set during both learning and prediction. we argue that injecting textual labels to all support instances may render the training task unchallenging, because the model can largely rely on the textual labels during training, and thus results in poor performance during testing when faced with unseen relations and textual labels. ideally, textual labels should be treated as additional source of information, such that the model can work with or without the textual labels, as shown in the top part in figure 1 . in this work, we propose a novel approach called label prompt dropout (lpd). we directly concatenate the textual label and the context sentence, and feed them together to the transformer encoder @xcite . the textual label serves as a label prompt 2 to guide and regularize the transformer encoder to output a label-aware relation representation through self-attention. during training, we randomly drop out the prompt tokens to create a more challenging scenario, such that the model has to learn to work with and without the relation descriptions. experiments show our approach achieves significant improvement on two standard fsre datasets. extensive ablation studies are conducted to demonstrate the effectiveness of our approach. furthermore, we highlight a potential issue with the evaluation setup of previous research efforts, in which the pre-training data contains relation types that actually overlap with those in the test set. we argue that this may not be a desirable setup for few-shot learning, and show that the performance gain of existing efforts may be partly due to this "knowledge leakage" issue. we 2 in this work, we use the terms label prompt, relation description, and textual label interchangeably. however, our method differs from the conventional prompt-based model in which a verbalizer @xcite is needed. we use relation description to construct a natural language sentence for each instance to better make use of implicit knowledge acquired by language models during pre-training. this goal is similar to that of the conventional prompt-based method. this is why we call call our method label prompt dropout. figure 2 : the framework of lpd. we prepend label prompt at the front of context sentences, and dropout the label prompt with probability α (α pre-train , α train , α test for the pre-training, training, and testing stage, respectively). top: we follow @xcite to use a knowledge graph to distantly annotate the pre-training corpus. bottom left: during training, label prompt in the support set is randomly dropped out, while there is no label prompt for the query instance. bottom right: during testing,α test is set to zero, meaning that all support instances are equipped with label prompts. ## task definition for an fsre task, each instance (x, e, y) is composed of a context sentence @xmath0, x 2 , x 3 , ..., x m }, where x i stands for the input token of position i; entity positions @xmath1, e tail }, where e head refers to the head entity span and e tail refers to the tail entity span; and a label @xmath2, y num }, where y text is the textual label and y num is the numerical label. let e train , e val , e test be the training, validation, and test dataset with mutually exclusive label sets. under the meta-learning paradigm, each dataset consists of multiple episodes, each with a support set s and query set q. for n -way-k-shot learning, the support set @xmath3; @xmath4, ..., n, @xmath5, ..., k} contains n different classes. inside each class there are k different support instances. our job is to predict the correct label @xmath6, ..., y n } for each query instance q in the query set. in this work, we will follow the continued pre-training setup @xcite , so there is another dataset e pretrain . note that this e pretrain is not the dataset used for the masked language modeling but for the domain-specific pre-training in re. ## conclusion this paper proposes a novel label prompt dropout approach that directly concatenates the label prompt with the context sentence for few-shot relation extraction. the label prompt is randomly dropped out during pre-training and training to create a more challenging learning setup, leading to better use of the relation descriptions. in the experiments, we discover a "knowledge leakage" issue in the previous works' experimental setup. we propose a stricter setup for more rigorous evaluations in fsre by filtering out all overlapping relations. our method has demonstrated significant improvements on both evaluation settings. ablation studies show that lpd shares some similar and interesting properties to the neural dropout operation and prompt based methods. one possible direction of future work is to generalize this idea to other text classification tasks such as intent classification @xcite .