Few-shot Adaptation Works with U npredic T able Data (2023)

Abstract: Prior work on language models (LMs) shows that training on a large number of diverse tasks improves few-shot learning (FSL) performance on new tasks. We take this to the extreme, automatically extracting 413,299 tasks from internet tables - orders of magnitude more than the next-largest public datasets. Finetuning on the resulting dataset leads to improved FSL performance on Natural Language Processing (NLP) tasks, but not proportionally to dataset scale. In fact, we find that narrow subsets of our dataset sometimes outperform more diverse datasets. For example, finetuning on software documentation from support.google.com raises FSL performance by a mean of +7.5% on 52 downstream tasks, which beats training on 40 human-curated NLP datasets (+6.7%). Finetuning on various narrow datasets leads to similar broad improvements across test tasks, suggesting that the gains are not from domain adaptation but adapting to FSL in general. We do not observe clear patterns between the datasets that lead to FSL gains, leaving open questions about why certain data helps with FSL.

## why is unpredictable helpful? to understand why unpredictable is helpful training data, we construct subsets of the dataset varying features we wish to study. for each subdataset, we finetune on that dataset individually following the setup as before (appendix c) and measure fsl performance on metaicl test tasks from all categories (52 total). all experiments are repeated for 3 random seeds to minimize the effects of random task sampling in each dataset. we report the mean accuracy from each experiment in fig. 3 . ## related work we focus on the fsl setting where few training samples are available. pretrained lms can learn from few-shot examples in-context (brown et al., ag_news dbpedia_14 commonsense_qa sciq amazon_polarity qasc financial_phra... tweet_eval-sta... yelp_polarity ethos-race test tasks unpredictable-5k nlp-1250 cluster 7 cluster 8 cluster 23 support.google.com w3.org wiki.openmoko.org numer_sense spider lama-trex train datasets : when finetuning on train dataset, test task is its top-10 most improved tasks. figure 6 : finetuning on different datasets leads to broadly similar improvements. for example, finetuning on wiki.openmoko.org (software documentation) and lama-trex (factual knowledge) lead to 8 of the same test tasks being in their respective top-10 mostimproved test tasks. (out of 52 total test tasks) 2020; scao and rush, 2021) but have weaknesses including prompt sensitivity @xcite and miscalibration @xcite . @xcite and chen et al. (2021b) alleviate these issues with fsl adaptation -fine-tuning lms to predict the target given few-shot examples in the prompt. we adopt metaicl @xcite training for our main experiments and support our results with additional few-shot benchmarks, cross-fit @xcite and flex @xcite . our work connects with other work in domain adaptation. @xcite show that finetuning on domains related to the downstream task leads to performance gains. more recent examples include chen et al. (2021a) for coding tasks and @xcite for mathematics tasks. @xcite demonstrate finetuning on value-aligned text to generate text in accordance with intrinsic human values. in contrast, we show that lms can be finetuned on unrelated domains to improve on new tasks. other work adapt to task formats: @xcite al. (2020) 2020 ) finetune models to perform tasks within predetermined prompt templates. more generally, lms have been finetuned to follow instructions @xcite which allows for diverse task formats. fsl adaptation can be seen as adaptation to the fsl prompt format, though the tasks can be diverse in domain and structure. multi-task literature show that training on a wide variety of tasks improves generalization to new tasks, which motivates our exploration of a large scale task dataset. 2022 ) extended this result to more than 1,000 tasks. we were inspired by these results to obtain a training dataset with 100x more tasks, but found certain narrow datasets are more helpful than diverse ones. @xcite showed that some training tasks negatively impact downstream performance, which could explain why mixing diverse tasks might underperform. this begs the question of how to select training datasets to improve downstream task performance. @xcite show that domain similarity can be used as a predictor for successful transfer, but our results suggest there may be domain-agnostic improvements to be gained from training on tasks unrelated to the test tasks. others study the effect of pretraining data on fsl, including @xcite and @xcite who find that fsl emerges when the training data exhibits particular distributional properties. our use of structured datasets to generate training tasks is inspired by other work, though others have focused on a limited set of task types. yoran et al. (2021) also turn tables into tasks, using handwritten templates to extract question-answer pairs from tables. aghajanyan et al. (2021b) train lms to predict masked spans in html webpages, then use html markup to prompt language models to do summarization and classification tasks. chen et al. (2022) transform ordinary (non-table) text into sentence completion, masked phrase prediction, and classification tasks. in contrast, our approach captures any tasks that occur naturally in tables. ## limitations & future work the unpredictable dataset may contain inaccuracies, biases, and inappropriate content. we do not recommend using this dataset to train models for deployment, but release this primarily as a research resource. we do not introduce any new model capabilities that lead to different risks than the usual risks associated with model usage. our work highlights the unpredictability of model behavior given various training datasets which calls for heightened vigilance for behavior changes after finetuning. our design choices in using table data for fsl training led to a dataset that is quite different than typical nlp datasets, so specific results from training on our dataset may not fully generalize to other kinds of datasets. further work may consider other methods for converting tables to tasks, other sources of tables besides wtc, or other structured datasets besides tables. our experiments focused on modestly-sized models (gpt-2 large, 750m parameters) so our conclusions may not hold for larger models. our evaluations are limited to multiple-choice tasks. future work may extend our analyses with larger models and other tasks including freeform generation. ## conclusion we produced unpredictable, a dataset of 413,299 diverse few-shot learning tasks from internet tables. finetuning on unpredictable improves the fsl ability of lms. however, the size of our dataset is not the key factor in its success. we find that certain narrow datasets (even ones made of trivia) are even more helpful than diverse, curated nlp datasets. finetuning on these narrow datasets leads to strong improvements on the same test tasks as finetuning on diverse, curated nlp datasets. this suggests that finetuning on these datasets cause domain-agnostic fsl gains, though we were unable to find clear patterns to explain why this happens for some data and not others. our results question common wisdom that task diversity is necessary for adapting lms to fsl. we hope our work spurs investigation on what data causes few-shot learning to emerge, both to develop better datasets and to better understand how training data leads to unexpected behaviors or failures.