F ew NLU : Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding (2022)

Abstract: The few-shot natural language understanding (NLU) task has attracted much recent attention. However, prior methods have been evaluated under a disparate set of protocols, which hinders fair comparison and measuring the progress of the field. To address this issue, we introduce an evaluation framework that improves previous evaluation procedures in three key aspects, i.e., test performance, dev-test correlation, and stability. Under this new evaluation framework, we re-evaluate several state-of-the-art few-shot methods for NLU tasks. Our framework reveals new insights: (1) both the absolute performance and relative gap of the methods were not accurately estimated in prior literature; (2) no single method dominates most tasks with consistent performance; (3) improvements of some methods diminish with a larger pretrained model; and (4) gains from different methods are often complementary and the best combined model performs close to a strong fully-supervised baseline. We open-source our toolkit, FewNLU, that implements our evaluation framework along with a number of state-of-the-art methods.

## introduction few-shot learning for natural language understanding (nlu) has been significantly advanced by pretrained language models (plms; @xcite . with the goal of learning a new task with very few (usually less than a hundred) samples, few-shot learning benefits from the prior knowledge stored in plms. various few-shot methods based on plms and prompting have been proposed @xcite @xcite . although the research of few-shot nlu is developing rapidly, the lack of a standard evaluation protocol has become an obstacle hindering fair comparison between various methods on a common ground and measuring progress of the field. while some works @xcite experimented with a fixed set of hyper-parameters, prior work @xcite ) noted that such a setting might be exposed to the risk of overestimation . 3 other works @xcite @xcite proposed to use a small development set to select hyper-parameters, but their evaluation protocols vary in a few key aspects (e.g., how to construct data splits), which in fact lead to large differences as we will show (in section 4.2). the above phenomena highlight the need for a common protocol for the evaluation of few-shot nlu methods. however, the fact that few-shot learning is extremely sensitive to subtle variations of many factors @xcite poses challenges for designing a solid evaluation protocol. in this work, aiming at addressing the aforementioned challenge, we propose an evaluation framework for few-shot nlu. the evaluation framework consists of a repeated procedure-selecting a hyperparameter, selecting a data split, training and evaluating the model. to set up a solid evaluation framework, it is crucial to specify a key design choice-how to construct data splits for model selection. we conduct a comprehensive set of experiments to answer the question. specifically, we propose a "multi-splits" strategy, which randomly splits the available labeled samples into training and development sets multiple times, followed by aggregating the results from each data split. we show that this simple strategy outperforms several baseline strategies in three dimensions: (1) the test set performance of the selected hyper-parameters; we then take a step further to re-evaluate recent state-of-the-art few-shot nlu methods under this common evaluation framework. our re-evaluation leads to several key findings summarized in section 2. to aid reproducing our results and benchmarking few-shot nlu methods, we open-source fewnlu, a toolkit that contains implementations of a number of state-of-the-art methods, data processing utilities, as well as our proposed evaluation framework. to sum up, our contributions are as follows. 1. we introduce a new evaluation framework of few-shot nlu. we propose three desiderata of few-shot evaluation and show that our framework outperforms previous ones in these aspects. thus our framework allows for more reliable comparison of few-shot nlu methods. 2. under the new evaluation framework, we benchmark the performance of recent methods individually as well as the best performance with a combined approach. these benchmarks reflect the current state of the art and will serve as important baselines for future research. 3. throughout our exploration, we arrive at several key findings summarized in section 2. 4. we open-source a toolkit, fewnlu, to facilitate future research with our framework. ## summary of findings for reference, we collect our key findings here and discuss each of them throughout the paper. finding 1. our proposed multi-splits is a more reliable data-split strategy than several baselines with improvements in (1) test performance, (2) correlation between development and test sets, and (3) stability w.r.t. the number of runs. finding 2. the absolute performance and the relative gap of few-shot methods were in general not accurately estimated in prior literature. it highlights the importance of evaluation for obtaining reliable conclusions. moreover, the benefits of some fewshot methods (e.g., adapet) decrease on larger pretrained models. finding 3. gains of different methods are largely complementary. a combination of methods largely outperforms individual ones, performing close to a strong fully-supervised baseline with roberta. finding 4. no single few-shot method dominates most nlu tasks. this highlights the need for the development of few-shot methods with more consistent and robust performance across tasks.