On Measuring the Intrinsic Few-Shot Hardness of Datasets (2022)

Abstract: While advances in pre-training have led to dramatic improvements in few-shot learning of NLP tasks, there is limited understanding of what drives successful few-shot adaptation in datasets. In particular, given a new dataset and a pre-trained model, what properties of the dataset make it few-shot learnable, and are these properties independent of the specific adaptation techniques used? We consider an extensive set of recent few-shot learning methods and show that their performance across a large number of datasets is highly correlated, showing that few-shot hardness may be intrinsic to datasets, for a given pre-trained model. To estimate intrinsic few-shot hardness, we then propose a simple and lightweight metric called Spread that captures the intuition that few-shot learning is made possible by exploiting feature-space invariances between training and test samples. Our metric better accounts for few-shot hardness compared to existing notions of hardness and is ~8-100x faster to compute.

## introduction a growing body of recent work has shown impressive advances in few-shot adaptation of pretrained transformers @xcite @xcite @xcite . despite this progress, there is no concrete understanding of when and why few-shot learning may be successful for a given pre-trained model. indeed, no free lunch style arguments necessitate the existence of tasks that are not few-shot learnable by a given pre-trained model, regardless of the adaptation method, and in practice, very similar datasets exhibit varying levels of success when state-of-the-art few-shot adaptation methods are applied (fig. 1a ). ## background consider a labeled dataset @xmath0, z (2) , . . .}, split into a training set d tr and a test set d ts , where each example z (k) is a tuple consisting of an input x (k) and a label y (k) . typically, in k-way l-shot learning (l is typically less than 128), d tr consists of l examples of each of the k labels. given some pre-trained model f , a few-shot adaptation method m uses d tr to modify f , outputting an "adapted model" that can make predictions on d ts . state-of-the-art approaches for such adaptation typically involve either recasting the task into the pre-training objective of the model @xcite , or using lightweight / parameter efficient finetuning @xcite @xcite @xcite . for this work, we experiment with an extensive set of recently proposed few-shot adaptation methods that we further categorize into prompt-based methods which includes lmbff @xcite , adapet @xcite , null prompts @xcite and prompt-bitfit (ben zaken et al., 2022) 1 , and light-weight finetuning methods which includes 1 as demonstrated by @xcite , we use trigger prompts in bitfit to improve few-shot performance. prefix tuning (li and liang, 2021) and compacter @xcite . ## intrinsic few-shot hardness for a fixed method and dataset, we define "method specific few-shot hardness" (mfh) as a function that takes a dataset d and some adaptation method m, and outputs a real number that captures fewshot hardness of d with respect to f and m. concretely, for this work, we define mfh(d, f, m) as the classification accuracy of the adapted model, normalized against the classification accuracy of the majority baseline, on d ts . ## automatic metrics for intrinsic few-shot hardness noting that the experiments of section 3 suggest that few-shot hardness may be intrinsic to datasets, we define the "intrinsic few-shot hardness" (ifh) of a dataset as the mean mfh values across a collection of adaptation method @xmath1, m 2 , . . .}, clearly, computing ifh values for some dataset is computationally intensive since it requires running multiple few-shot adaptation methods on f . thus, to automatically estimate ifh, an automatic hardness metric h takes f and d and outputs a (real-valued) score h(f, d) that is well-correlated with ifh(f, d) across some collection of datasets {d 1 , d 2 , . . .}. ## decreasing few-shot hardness via dataset decomposition we conclude experiments with a simple application of spread by proposing a training set sampling strategy that minimizes spread values for improved few-shot performance. our strategy is method ours control lmbff 17.1 19.7 adapet 32.3 8.5 null prompt 3.3 -7.7 prompt-bitfit 2.9 -9.0 few-shot avg. 13.9 2.9 in particular, we run k-means clustering on input features from the pre-trained model to create p clusters. then, we train p distinct models on fewshot training sets sampled from each of the clusters. at test time, examples are classified into one of the p clusters, and the corresponding model is used to make predictions. to account for any effects due to ensembling, we compare with an approach where we randomly sample a model to make predictions, instead of using the model corresponding to the cluster. results. we experiment with the sampling strategy described above on a named entity classification based dataset from fs-nli. from ## related work and discussion most notions of what makes datasets challenging are either model agnostic-example length @xcite , order sensitivity @xcite etc, or indirect @xcite . on the other hand, intrinsic hardness as defined in this work, is both model-centric and directly measures the ability of a specific model family to make good generalizations from a training set. while measuring hardness of datasets has seen some recent traction @xcite , to the best of our knowledge, we are the first to study hardness in the context of few-shot adaptation. we show that few-shot hardness of datasets (as measured by test set performance normalized against a random baseline) among an extensive set of recently proposed methods is highly correlated, suggesting that few-shot hardness may be a property intrinsic to datasets. we then propose a simple hardness metric called spread, based on the in-tuition that a test set with input features close to the few-shot training set is easy, since it allows a model to exploit feature-space invariances. compared to other metrics, spread provides hardness judgements that are much better correlated with intrinsic few-shot hardness while being 8-100x faster to compute, compared to prior hardness metrics. metrics for predicting fewshot hardness have several applications. for the nlp practitioner, spread could be used as a simple plug-and-play estimator of whether few-shot adaptation of a pretrained model might be successful for a given use case. for the dataset developer, spread could be used to adversarially curate harder few-shot benchmarks. finally, a model developer can use spread to inform better training set sampling strategies to improve test set performance in the few-shot training regime, as well as for model selection by selecting models with lower spread. ## acknowledgements sm was funded by a gift from apple inc. we are grateful to john hewitt, eric mitchell, roy schwartz and the anonymous reviewers for helpful comments. ## reproduciblity our code is available at: https://github.com/ colinzhaoust/intrinsic_fewshot_hardness . ## limitations instance-level analysis. we focus on discovering and measuring intrinsic few-shot hardness at the dataset level and do not study instance-level hardness quantitatively or qualitatively. understanding hardness at the instance level can further help understand the recent successes behind fewshot learning in nlp. base model selection. we compare few-shot performance between a variety of methods with two base models that have very different pre-training objectives, yet have correlated few-shot behaviors across tasks and adaptation methods. of course, there exist a much wider range of pre-trained models with very diverse pre-training data and objectives. while beyond the scope of this work, we believe that studying factors such as the relationship between few-shot performance and pre-training data / objectives is worth further investigation.