A Few More Examples May Be Worth Billions of Parameters (2022)

Abstract: We investigate the dynamics of increasing the number of model parameters versus the number of labeled examples across a wide variety of tasks. Our exploration reveals that while scaling parameters consistently yields performance improvements, the contribution of additional examples highly depends on the task’s format. Specifically, in open question answering tasks, enlarging the training set does not improve performance. In contrast, classification, extractive question answering, and multiple choice tasks benefit so much from additional examples that collecting a few hundred examples is often “worth” billions of parameters. We hypothesize that unlike open question answering, which involves recalling specific information, solving strategies for tasks with a more restricted output space transfer across examples, and can therefore be learned with small amounts of labeled data.

## introduction recent work on few-shot learning for natural language tasks explores the dynamics of scaling up either the number of model parameters @xcite or labeled examples (le scao and rush, 2021), while controlling for the other variable by setting it to a constant. for example, @xcite focus on in-context learning from roughly 32 to 64 examples, a practice that was adopted by fine-tuning approaches as well @xcite @xcite ; however, there are many practical few-shot scenarios where hundreds of examples can be collected at a relatively low effort. 2 other work experiments with single-size models @xcite @xcite ; figure 1 : open qa tasks (e.g. triviaqa) benefit from additional parameters exclusively, while extractive qa tasks (e.g. squad 2) benefit from both larger models and more labeled data. @xcite , even though larger (or smaller) models may exhibit different behavior. furthermore, much of the literature focuses on classification tasks @xcite @xcite , leaving it unclear whether their conclusions generalize to tasks with less restricted output spaces. in this paper, we conduct a systematic exploration of few-shot learning for language tasks, where we investigate the dynamics of increasing the number of model parameters (using different sizes of the self-supervised t5 @xcite ) versus the number of target-task labeled examples (from 32 to 2048) across a variety of tasks, including not only classification, but also extractive, multiple-choice, and open question answering. overall, we evaluate 192 scenarios by training 7,680 models to control for hyperparameters and random seeds. our experiments show that, surprisingly, the contribution of additional parameters versus additional labeled examples highly depends on the format of the task. for open qa tasks, such as the opendomain version of natural questions @xcite , which require the model to recall specific information seen during pretraining, enlarging the training set does not improve performance. by contrast, increasing the number of model parameters results in substantial gains (see triviaqa @xcite in figure 1 ). hence, when dealing with open qa, model parameters are of immense value, and cannot be replaced by increasing the number of labeled examples. on the other hand, we observe a completely different trend for classification, extractive qa, and multiple-choice tasks. these tasks benefit from enlarging both the training set and the model (see squad 2 @xcite in figure 1 ). we observe that hundreds of examples are often "worth" billions of parameters; t5-l fine-tuned on 4 times more data is roughly competitive with t5-xl, which has 4 times the number of parameters. moreover, some tasks benefit so much from labeled examples, that collecting even 512 data points can make a fine-tuned t5-l (800m parameters) outperform gpt-3 (175b parameters). finally, we hypothesize that unlike open qa, formats with restricted output spaces have solving strategies (such as elimination) that can be learned from small amounts of labeled data. this hypothesis also provides a possible explanation as to why lean retrieve-and-read approaches (such as drqa @xcite , orqa @xcite , and dpr @xcite )) appear to be more robust than multi-billion-parameter closedbook models @xcite ) when tested on non-overlapping data @xcite . ## experiments we describe the tasks (section 2.1), models (section 2.2), data regimes (section 2.3), and implementation details (section 2.4) of our systematic experiment suite. in total, we experiment with 12 tasks, 4 models, 4 data regimes (with 5 samples each), and 8 hyperparameter configurations; these amount to 7,680 trained models, evaluated across 192 task-model-data scenarios. ## results for each task in our experiment suite, we present a heatmap of the model's performance as a function of model size and the number of labeled examples. these heatmaps expose that most tasks benefit from both larger models and more training data, to a point where enlarging the dataset will result in similar gains as increasing the number of parameters. however, this trend does not apply to open qa tasks, whose performance only improves with additional model parameters (section 3.1). furthermore, we show that converting multiple choice datasets to the open qa format disables the benefits of additional training data, whereas converting in the opposite direction -from open qa to extractive or multiple-choice qa -enables models to improve with more examples (section 3.2). next, we describe a method for quantifying the relative benefit from parameters versus training data, which confirms the observed trends (section 3.3). we then show that collecting a few hundred examples allows the much smaller t5-l to outperform gpt3, but not in open qa tasks, where the massive amount of parameters is the prime contributor (section 3.4). finally, we suggest a hypothesis to explain the observed trends (section 3.5). ## related work few-shot learning has been a subject of interest for several decades @xcite @xcite @xcite . within nlp, "few-shot learning" has traditionally focused on quickly learning new classes and domains within the context of a single task (see @xcite for a recent survey). recently, there has been a surge of interest in few-shot learning, following the release of gpt3 @xcite . here, the few-shot learning paradigm has shifted subtly, and refers to building models capable of tackling a range of standard nlp tasks, albeit using very restricted training sets, usually sub-sampled from the full training set. a great deal of work has recently been produced in this area, and we provide a necessarily incomplete summary below. ## conclusions in this work, we present an empirical investigation on the relationships between (1) a task's format, (2) the number of labeled examples available for said task, and (3) the number of parameters the model tackling the task has. through our extensive experiments, we determine that task format greatly affects the relative performance improvement that can be expected from increased training set size and parameter count. for tasks that do not require the recollection of specific external information -i.e. classification, multiple choice, and extractive qa -we find that more labeled data and larger models both reliably improve performance. in fact, for some of these tasks, adding a few hundred labeled examples is more beneficial than scaling up the model size by billions of parameters. it seems then, from a practitioner's perspective, that for many tasks where data is very sparse, the tried-and-true strategy of simply collecting more training data will often be a more effective strategy than attempting to scale to larger, more computationally-demanding models. however, the picture is very different for open qa tasks; for such tasks, we find that increasing the size of the training data barely improves performance, leaving parameter inflation as the only reliable approach to improve accuracy. finally, we provide a hypothesis to explain these results and conclude with a practical corollary -when possible, changing the format from open qa into a more "self-contained" one will allow labeled data to bridge performance gaps between moderately-sized models and much larger ones.