Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation (2023)

Abstract: Few-shot fine-tuning and in-context learning are two alternative strategies for task adaptation of pre-trained language models. Recently, in-context learning has gained popularity over fine-tuning due to its simplicity and improved out-of-domain generalization, and because extensive evidence shows that fine-tuned models pick up on spurious correlations. Unfortunately, previous comparisons of the two approaches were done using models of different sizes. This raises the question of whether the observed weaker out-of-domain generalization of fine-tuned models is an inherent property of fine-tuning or a limitation of the experimental setup. In this paper, we compare the generalization of few-shot fine-tuning and in-context learning to challenge datasets, while controlling for the models used, the number of examples, and the number of parameters, ranging from 125M to 30B. Our results show that fine-tuned language models can in fact generalize well out-of-domain. We find that both approaches generalize similarly; they exhibit large variation and depend on properties such as model size and the number of examples, highlighting that robust task adaptation remains a challenge.

## introduction adapting a pre-trained language model to a target task is of high practical importance to the natural language processing (nlp) community (as seen in @xcite @xcite . among the commonly used task adaptation strategies, two stand out: fine-tuning (ft) and in-context learning (icl). foot_1 icl -16 samples ft -16 samples 0.4 0.5 0.6 0.7 0.8 0.9 figure 1: in-domain (rte) and out-of-domain performance (hans) for in-context learning (icl) and finetuning (ft) with opt models of various sizes. we fine-tune models using pattern-based fine-tuning. we report results using 10 different data seeds. when using 16 samples, icl's performance with a 30b model is comparable to that of ft with smaller models (6.7b) and for most model sizes, ft outperforms icl (see both approaches come with pros and cons: icl reuses a single pre-trained model for various downstream tasks, allows specifying the desired behavior via natural language, and has recently shown impressive results on challenging reasoning tasks @xcite @xcite . however, the model's context size limits the number of demonstrations that can be used. for instance, using 32 randomly selected examples from the rte dataset @xcite already exceeds the context size of opt models @xcite . 3 in addition, icl is highly sensitive to the format and order of its inputs @xcite . ft, on the other hand, typically results in a single specialized model per task, 4 and can be applied to training sets of arbitrary size. however, such models are sensitive to initialization @xcite and can suffer from instability during training @xcite . for text classification tasks, where both strategies often lead to similar performance on in-domain data (when using the same amount of data), recent works have argued that icl leads to better outof-domain (ood) generalization @xcite . however, these comparisons of generalization abilities were not conducted under equal conditions. most studies compare the icl abilities of large models (e.g. gpt-3, 175b; @xcite to the ft abilities of much smaller models (e.g. roberta-large, 350m; @xcite . these comparisons raise the question of whether ft indeed leads to weaker ood generalization than icl, or whether this is just a byproduct of the experimental setup. in figure 1 , we show this is indeed the case: when given only 16 examples, fine-tuning a 6.7b parameters model already achieves similar results to icl with a 30b model, and ft performance keeps improving with larger models. 5 moreover, we show in section 4.1 that fine-tuning performance improves even further when training on more data. in this paper, we compare icl and ft on an equal footing ( §3). we compare both strategies using the same model (opt; @xcite , the same number of parameters (from 125m to 30b), and the same number of examples. our results and analyses ( §4) show that both approaches often achieve comparable results. both methods are unstable and can perform badly on in-domain and ood data due to training instability, or prompt choice. we also find that both approaches improve as we increase model size, and that, for the models and datasets we consider, ft often generalizes even better than icl. notably, this is in contrast to prior work ( §7), highlighting the need for fair comparisons of task adaptation strategies. based on our findings, we discuss the strengths and limitations of ft and icl ( §6), which can inform when to use and how to get the most out of each method. ## a fair comparison of ft and icl we perform a fair comparison of task adaptation via ft and icl, focusing on in-domain and ood generalization. we compare them in the few-shot setting using the same models. in the following paragraphs, we provide details about our setup. ## results we present the results for in-domain and ood model performance in figure 2 , comparing both icl and ft. we perform task adaptation using 16 examples for both strategies. for icl, we provide additional results that demonstrate the importance of choosing the right pattern and number of demonstrations in appendix b.2. for ft, we provide more details, ablations and discussion of various choices later in this section. in-domain performance for mnli and rte, both icl and ft exhibit in-domain performance above the majority baseline for most model sizes. focusing on icl, mnli and rte in-domain performance improves as model size increases. on mnli the largest model (30b) obtains an average performance of 71.4% and a maximum performance of 74.9%. on rte, icl with the same model achieves an average and maximum performance of 61.7% and 66.8% respectively. on qqp, the trend of improved performance with increasing model size for ft, we similarly observe that in-domain performance increases with model size. moreover, across all datasets and model sizes, ft with just 16 examples leads to similar in-domain performance as icl (see tables 8 and 9 in appendix b.1 for statistical tests comparing in-domain performance of ft and icl on rte and mnli). on qqp, we again observe no clear relationship between model size and performance. only 10 out of 70 models perform better than the majority baseline. out-of-domain performance turning to ood performance, we find that for mnli and qqp most of the icl models perform close to the majority baseline. on mnli, only the largest model (30b) shows good ood generalization for 4 out of 10 runs. on rte, in-domain and ood performance of the 30b model mostly overlap, which is consistent with the findings of @xcite . in particular, when comparing the relationship between the indomain and ood performance of the 30b model to the smallest fine-tuned models (125m and 350m) one might conclude that icl leads to better ood performance; for ft on mnli and rte, indeed, the smallest models have poor ood performance. however, as model size increases, ood performance increases as well, demonstrating that even in the challenging few-shot setting, fine-tuned models can generalize ood. focusing on the largest ## discussion our findings in the previous section demonstrate that fine-tuned language models can generalize ood too, highlighting the importance of comparing adaptation approaches fairly. in this section, we present further insights from our experiments and provide a high-level comparison of the pros and cons of adaptation via icl and ft. what signal to learn from? both our icl and ft results exhibit a large variance in both in-domain and ood performance. our results show different ood behavior during ft when varying only the data seed. in addition, as previous work has shown, the choice of patterns and ver-balizers impact both icl and pbft performance in unintuitive ways. for instance, webson and pavlick (2022) find that pattern-based fine-tuned models perform well even when using misleading patterns. here, we find that icl's generalization is heavily dependent on the choice of pattern and verbalizer. this shows the importance of the choice of training data and patterns for task adaptation. ## comparing ft and icl this section examines the key features for task adaptation and compares ft and icl. we summarize our findings in next, we compare technical details regarding the training and inference of such approaches. while ft requires training (which when dealing with large models can become expensive), icl does not. on the other hand, the inference time of fine-tuned models is much smaller than icl, since it only includes the time that it takes to process the minimal pattern and the test instance. when using icl, each test instance has to include all of the demonstrations as well, which increases the inference time. the fixed context size of the model also limits the number of demonstrations that can be used foot_15 , while ft allows for unlimited training examples. we show in this work that both methods can achieve strong performance on both in-domain and ood datasets. both approaches improve with model size, but ft benefits more from additional samples than icl does, as was also shown in previous work @xcite . finally, we highlight that both methods are relatively recent: vanilla ft was invented in 2018 @xcite and icl in 2020 @xcite . 18 as such, these methods are still poorly understood, and more research is required on both approaches to better understand their strengths and weaknesses. while we focus on the task adaptation of decoder-only models, utama et al. (2021) investigate the ood generalization of encoder-only models adapted via pattern-based few-shot ft. for mnli and hans, they find that these models adopt similar inference heuristics to those trained with vanilla ft and hence perform poorly ood. they observe that models rely even more on heuristics when fine-tuned on more data. this is in contrast to our results where we find that pattern-based few-shot ft can lead to good ood generalization, and ood generalization improves as we train on more data. we attribute this to the fact that they experiment with a smaller model (roberta-large; 350m). 19 lastly, @xcite show that masked language models can generalize well on hans if fine-tuned for a sufficient number of steps. while they focus on fine-tuning on the entire dataset, their findings provide additional evidence that fine-tuned language models can generalize well ood. ## conclusion we perform a fair comparison between in-domain and ood generalization of two alternative task adaptation strategies: few-shot icl and ft. we compare opt models @xcite ranging from 125m to 30b parameters on three classification datasets across two tasks. we find that for both approaches, performance improves as models become larger. for the largest models we experiment with (opt-30b), we find that ft outperforms icl on both in-domain and ood performance and even improves further as we train on more data. however, our results also demonstrate that the performance of both ft and icl exhibits high variance, highlighting that truly robust task adaptation remains an open challenge. we end by providing a high-level comparison between the two approaches, listing the benefits and limitations of each, and discussing some future directions.