Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes (2021)

Abstract: We explore few-shot learning (FSL) for relation classification (RC). Focusing on the realistic scenario of FSL, in which a test instance might not belong to any of the target categories (none-of-the-above, [NOTA]), we first revisit the recent popular dataset structure for FSL, pointing out its unrealistic data distribution. To remedy this, we propose a novel methodology for deriving more realistic few-shot test data from available datasets for supervised RC, and apply it to the TACRED dataset. This yields a new challenging benchmark for FSL-RC, on which state of the art models show poor performance. Next, we analyze classification schemes within the popular embedding-based nearest-neighbor approach for FSL, with respect to constraints they impose on the embedding space. Triggered by this analysis, we propose a novel classification scheme in which the NOTA category is represented as learned vectors, shown empirically to be an appealing option for FSL.

## introduction we consider relation classification-an important sub-task of relation extraction-in which one is interested in determining, given a text with two marked entities, whether the entities conform to one of pre-determined relations, or not. while supervised methods for this task exist and work relatively well @xcite @xcite , they require large amounts of training data, which is hard to obtain in practice. we are therefore interested in a data-lean scenario in which users provide only a handful of training examples for each relation they are interested in. this has been formalized in the machine learning community as few-shot learning (fsl) ( §2). fsl for rc has been recently addressed by the work of @xcite and @xcite , who introduced the fewrel 1.0 and shortly after the fewrel 2.0 challenges, in which researchers are provided with a large labeled dataset of background relations, and are tasked with producing strong few-shot classifiers: classifiers that will work well given a few labeled examples of relations not seen in the training set. the task became popular, with scores on fewrel 1.0 achieving an accuracy of 93.9% @xcite , surpassing the human level performance of 92.2%. results on fewrel 2.0 are lower, at 80.3% for the best system @xcite , but are still very high considering the difficulty of the task. is few-shot relation classification solved? we show that this is far from being the case. we argue that the evaluation protocol in fewrel 1.0 is based on highly unrealistic assumptions on how the models will be used in practice, and while fewrel 2.0 tried to amend it, its evaluation setup remains highly unrealistic ( §3.1). therefore, we propose a methodology to transform supervised datasets into corresponding realistic few-shot evaluation scenarios ( §3.2). we then apply our transformation on the supervised tacred dataset @xcite to create such a new few-shot evaluation set ( §3.3). our experiments ( §6.2) reveal that indeed, moving to this realistic setup, the performance of existing state-of-the-art (sota) models drop considerably, from scores of around 80 f1 (as well as accuracy) to around 30. a core factor in a realistic few-shot setup is the nota (none-of-the-above) option; allowing a case where a particular test instance does not conform to any of the predefined target relations. triggered by presenting an analysis of possible decision rules for handling the nota category ( §5), we propose a novel enhancement that models nota by an explicit set of vectors in the embedding space ( §5.2). this explicit ''nota as vectors'' approach achieves new sota performance for the fewrel 2.0 dataset, and outperforms other models on our new dataset ( §6). yet, the realistic scenario of our tacredderived dataset remains far from being solved, calling for substantial future research. we release our models, data, and, more importantly, our data conversion procedure, to encourage such future work. ## desired versus existing few-shot relation classification datasets a fsl system is intended to be used in a real-life scenario. thus, evaluation procedures for fsl should attempt to mimic the conditions under which the fsl system will be applied in practice. in a realistic fsl scenario, the user has a set of relations of interest (''target relations''), and can come up with a handful of examples for each. the relations in the set are often related to each other. the user may potentially have access to a labeled dataset of a different set of relations (''background relations''), which they may want to use to train, or to improve, their fsl system. the resulting classifier will then be applied to unlabeled data aiming to detect new target relations, in which, realistically: as we show below, existing fsl-rc datasets do not conform to these properties, resulting in artificial-and substantially easier-classification tasks. this in turn leads to inflated accuracy numbers that are not reflective of the real potential performance of a system. we propose a refined sampling procedure that adheres to the realistic setting, and results in a substantially more realistic evaluation set, while conforming to the same n-way k-shot protocol. as we show in the experiments section ( §6), this setup proves to be substantially more challenging for existing algorithms. we propose to use this procedure for future evaluation of fsl-rc algorithms, and release the corresponding code and data. ## background: prior few-shot rc models as mentioned earlier, only a handful of examples are provided for the target classes in the few-shot setting. it is therefore quite challenging to utilize these examples effectively for learning or updating model parameters. consequently, quite a few existing few-shot models, in the machine learning literature as well as in nlp @xcite @xcite , perform a representation learning phase (typically known as embedding learning or metric learning), followed by nearest neighbor classification. here, model parameters are first learned over the background classes, for which substantial training is available. then, classification of test instances is based on the trained model, with the hope that this model would generalize reasonably well for the target classes. 4 in the nearest neighbor approach, classification is done via a scoring function score(q, c i ), which assigns a score for a query instance, q, and a target class, c i . because the class is represented by its support set, σ c i , the scoring function can be a similarity function between the query and the class's support set: most often, an embedding-based approach is taken to compute similarity, decomposing the process into two separate components @xcite @xcite . first, instances are embedded into an explicit, typically dense, vector space, by an embedding function. then, query-support similarity is measured over embedded vectors. specifically, the prototypical network of @xcite represents a target class c i by a class prototype vector μ i , which is the average embedding of the k instances in the support set of the class. the similarity between the query and each support set, sim(q, σ c i ), is then measured as the similarity between the query and the corresponding prototype vector, assuming some similarity function between vectors in the embedding space: this approach was adopted in the state-of-the-art method @xcite for few-shot rc (fewrel 1.0, excluding the nota category), as well as by several other works for fsl in nlp @xcite . nearest-neighbor classification rule similarity is computed between a test instance and each support set, selecting the nearest class: @xcite further conducted an empirical analysis of embedding functions for few-shot rc. their most effective embedding method augments the sentence with special tokens at the beginning and at the end of each of the two entities of the relation instance. the instance representation is then obtained by concatenating the two corresponding start tokens from bert's last layer @xcite . in our experiments, we adopt this embedding function, denoted bert em (bertbased entity marking), as well as the use of dot product as the vector similarity function (after we reassessed its effectiveness as well). ## classification rules: analysis and extension in this section, we provide an analytic perspective on the bias that different nearest-neighbor classification rules impose on the learned embedding space. we start with an analysis of the classification rule for the basic few-shot rc setting, without the nota category, as was applied in prior work (section 4). this analysis follows directly the constraint presented in the influential work of @xcite , and utilized in subsequent work (e.g., @xcite . we then extend this analysis to the setting that does include the nota category. first, we analyze the straightforward thresholdbased approach for this setting. then, inspired by this analysis, we propose an alternative approach, with a corresponding constraint, which represents the nota category by one or more explicit learned vectors. as shown in subsequent sections, this new approach performs consistently better than other methods on both the fewrel 2.0 and our new few-shot tacred benchmarks, and is thus suggested as an appealing approach for few-shot relation classification. ## experiments and results in this section, we assess our two main contributions. with respect to our few-shot tacred dataset, we show that models that perform well on fewrel 2.0 perform poorly on this much more realistic setting, leaving a huge gap for improvement by future research. with respect to our proposed nav modeling approach, we show that it is a viable, and advantageous, alternative to the threshold approach. implemented models we conduct our investigation in the framework of the common embedding based approach to fsl, with respect to the mnav, nav, and threshold-based methods described in §5. these methods are implemented following the best-performing embedding and similarity methods identified for the state-ofthe-art method on fewrel 1.0 @xcite , namely, bert em applied using bert base , and dot product similarity ( §4). in addition, we train and evaluate the baseline sentence-pair model, described in §4.1. to select the number of nota vectors in the mnav model, we experimented with 5 different values, ranging from 1 to 20. in practice, the choice of the number of vectors had rather little impact on the results (less than one f1 point). we use the best performing value for this hyperparameter, which was 20. in terms of memory utilization, as 5-way 5-shot episodes require feeding the 25 instances of the support set in addition to the query instances into bert simultaneously, they often occupy nearly the entire 32gb of gpu memory. to leverage the memory taken by the support set instances, we include as many queries as we can fit into the gpu's memory. in our experiments, we construct 3 episodes for each sampled support set (by sampling 3 different queries for it), which fully utilizes the gpu capacity. since these episodes occupy the entire gpu memory, we use a single episode per batch. we further note that it may be possible to perform the n-way classification by transforming it into a pair-wise classification, repeated n times (both in training and evaluation). this technique would allow to reduce the memory usage but would increases the run-time. as we managed to fit the entire episode to our gpu memory, we followed the standard n-way approach, for faster computation, as was previously done by @xcite . ## conclusions in this work, we offer several required criteria for realistic fsl datasets, while proposing a methodology to derive such benchmarks from available datasets designed for supervised learning. we then applied our methodology on the tacred relation classification dataset, creating a challenging benchmark for future research. indeed, previous models that achieved impressive results on fewrel, a synthetic dataset for fsl, failed miserably on our naturally distributed dataset. these results call for better models and loss functions for fsl, and indicate that we are far from having satisfying results on this setup. our methodology may be further applied to additional datasets, enriching the availability of realistic datasets for fsl. next, we analyzed the constraints imposed embedding functions by nearest-neighbor classification schemes, common for fsl. this analysis led us to derive a new method for representing the nota category as one or more explicit learned vectors, yielding a novel classification scheme, which achieves new state-of-the-art performance. we suggest that our analysis may further inspire additional innovations in few-shot learning.