Continued Pretraining for Better Zero- and Few-Shot Promptability (2022)

Abstract: Recently introduced language model prompting methods can achieve high accuracy in zero- and few-shot settings while requiring few to no learned task-specific parameters. Nevertheless, these methods still often trail behind full model finetuning. In this work, we investigate if a dedicated continued pretraining stage could improve “promptability”, i.e., zero-shot performance with natural language prompts or few-shot performance with prompt tuning. We reveal settings where existing continued pretraining methods lack promptability. We also identify current methodological gaps, which we fill with thorough large-scale experiments. We demonstrate that a simple recipe, continued pretraining that incorporates a trainable prompt during multi-task learning, leads to improved promptability in both zero- and few-shot settings compared to existing methods, up to 31% relative. On the other hand, we find that continued pretraining using MAML-style meta-learning, a method that directly optimizes few-shot promptability, yields subpar performance. We validate our findings with two prompt tuning methods, and, based on our results, we provide concrete recommendations to optimize promptability for different use cases.

## introduction conditioning language models (lms) on manuallywritten or learned continuous prompts allows them to solve tasks with high accuracy and minimal parameter overhead @xcite @xcite . however, prompting performance often still lags behind traditional full finetuning. natural language prompts usually underperform trained models even when manually curated @xcite . similarly, while learned prompts yield higher accuracy, they do not work as well when the training data is scarce @xcite , when the model is small or moderately sized @xcite , and when the tasks are difficult @xcite . to reduce the gap between prompt and full model tuning, past work has shown that continued pretraining on data that resembles the downstream prompting setup induces better "promptability", i.e., zero-shot performance with natural language (nl) prompts and few-shot performance of prompt tuning @xcite . however, in this paper, we identify several shortcomings of these methods. first, continued pretraining on nl prompts @xcite sometimes causes performance degradation with prompt tuning. second, continued pretraining approaches that learn only a universal prompt initialization @xcite bring only marginal improvement on the p3 datasets @xcite . to further improve zero-shot and few-shot promptability, we investigate gaps in existing methods with different parameter configurations and training procedures. first, we explore the effect of incorporating a learned continuous prompt into multi-task learning (mtl), and find it to significantly improve zero-and few-shot promptability across the board. in addition, we explore mamlstyle meta-learning @xcite as an alternative to the standard continued pretraining paradigm, but find that it underperforms simple mtl, despite its previous success on fewshot learning tasks @xcite @xcite . we perform an analysis of this phenomenon and present several explanations. through large-scale experiments, each involving continued pretraining on over 9b tokens ( §a), we make several contributions: (1) we thoroughly evaluate continued pretraining methods, both existing and our proposed ones, in many setups; (2) we demonstrate that a simple continued pretraining recipe improves over existing methods by up to 31%; (3) we show that maml-style meta-learning underperforms multi-task learning and provide explanations; (4) we provide concrete recommendations to improve promptability in various use cases. ## prompting we review two types of prompting that we use: natural language (nl) prompting and prompt tuning. traditionally, nlp tasks are solved by taskspecific models that predict label @xmath0. we can consider lms as functions that score any source and target text pair, lm : v * × v * → r with vocabulary v. foot_1 past work found that large lms can be repurposed to solve many tasks by casting x, y into a text format using a template function f : x ∪ @xmath1, f (y ′ )). nl prompts, or instructions, are manually constructed f (•). without task-specific training, they have been successfully used to elicit predictions from lms to perform tasks with high accuracy @xcite . sharing the motivation, prompt tuning learns a continuous prompt to condition the model. it takes the source text embedded by the lm input embeddings, @xmath2, and prepends learnable embeddings @xmath3, where l is a hyperparameter, to obtain a new (l + n )-lengthed embedded sequence. we consider hybrid prompt tuning, where s is the embedding of the templatized f (x), i.e., prompt tuning is always performed in addition to nl templates. this has been widely adopted due to demonstrated better performance @xcite . we also study a variant of prompt tuning, sometimes called prefix tuning @xcite , where the learnable vectors are added not only to the input but all transformer layers. see @xcite and @xcite for more details on these methods. following the terminology of @xcite , we refer to the input-level method as shallow prompt tuning and the layer-specific method as deep prompt tuning. ## improving promptability in this section, we describe existing methods to improve promptability and a new paradigm that combines their advantages. while prompt tuning sometimes performs close to full model finetuning @xcite , there is often still a substantial gap, such as with limited training data @xcite , non-gigantic models @xcite , or challenging tasks @xcite . we therefore study ways to improve lms' "promptability." we focus on a low resource setup and consider zero-shot nl prompts and few-shot learned prompts (which, again, are in conjunction with nl prompts; §2). for the former, better promptability increases the performance when lms face textual prompts of new tasks. for the latter, it more effectively leverages limited training examples for higher accuracy. we investigate if promptability can improve with a continued pretraining stage after lm pretraining (or lm adaptation for lm-adapted t5 @xcite ) and before task-specific finetuning. the model is trained on a collection of tasks that have nl prompts and evaluated on unseen tasks. the methods that we explore below differ in how the continued pretraining stage is performed. we use the notation mtl-t_p_ to abbreviate those methods that are based on multi-task learning, where the blanks _ specify different configurations of the transformer (t) and the prompt (p) components during mtl. architecturally, a method may continue to pretrain only the t5 model without prompt parameters, in which case we use p✗ to denote the lack of them; otherwise, both transformer and prompt parameters exist during mtl. we use and to denote if the corresponding component is trained or frozen in mtl, respectively. this notation describes the continued pretraining stage only: in the final finetuning stage, all methods include both the transformer and prompt components, but only the latter is updated. continued pretraining has been studied in limited settings. @xcite proposed t0 by multi-task training a t5 model @xcite as continued pretraining. they updated t5 parameters through learning on continued pretraining tasks, not including a prompt component, and showed that this training improves zero-shot nl promptability. following our nomenclature, we refer to this paradigm as mtl-t p✗. additionally, @xcite employed a similar stage, incorporating and multi-task training a shallow prompt as continued pretraining, while freezing the transformer parameters in this stage. they showed that this strategy helps few-shot promptability during finetuning. we refer to this paradigm as mtl-t p . in this work, we study the gains of the previous two continued pretraining approaches, as well as a model that synthesizes them, mtl-t p , which we are the first to propose. for few-shot downstream tuning, the learned prompt can act as a good initialization compared to mtl-t p✗. in the zeroshot setup, prior work has discovered that including certain text in a prompt, such as "let's think step by step," can adjust the reasoning of lms to yield substantially improved performance across tasks @xcite . the learned prompt here could function analogously. compared to mtl-t p , on the other hand, the additional capacity brought by more updatable parameters could further boost model performance. maml-style meta-learning @xcite ) directly optimizes for the downstream updates and can outperform mtl for full model finetuning @xcite ). yet, it similarly remains unexplored for prompting. we examine first-order maml (fomaml; finn et al., 2017), performing t steps of prompt tuning in the inner loop and updating all parameters in the outer loop. we also evaluate a version of reptile @xcite adapted for our setting that performs t steps of prompt tuning followed by one step of full model tuning, and use the resulting reptile gradient for model updates. they have the same architecture as mtl-t p and all parameters are trainable too. we provide a detailed description and theoretical discussion of these processes in §b. see the original papers for more details. ## experimental setup we use p3, a collection of nl-templatized examples for a variety of datasets, for training and evaluation using the standard splits in @xcite . not only is there no dataset overlap between training and evaluation, but no task overlap either (e.g., sentiment vs. qa), making it challenging. we report dataset statistics in §a. we perform continued pretraining for one epoch over all training datasets. each dataset has multiple templates, each evaluated with accuracy. as different datasets have different numbers of answer choices and hence different baseline accuracy, we report average relative gain (arg; @xcite as a single summary metric by averaging across all templates the relative ac-curacy improvement over a random baseline. we perform significance testing using bootstrap with 1,000 iterations, in each iteration randomly sampling evaluation examples and comparing the two models in question. §d reports per-dataset results. following @xcite , we initialize the continued pretraining stage from t5 finetuned with an lm objective @xcite , making it more amenable to prompting. we experiment with two sizes: t5-large with 770m parameters and t5-xl with 3b parameters. we retrain t0 @xcite , i.e. mtl-t p✗, to eliminate confounding factors in the training procedure. we also reproduce @xcite 's experiment in our setup, i.e. mtl-t p , pretraining a shallow prompt with other parameters frozen. during few-shot finetuning, we train on the same 16 examples for 100 epochs. §c reports additional hyperparameters. ## results existing methods for continued pretraining have their drawbacks. in contrast to @xcite , we found that mtl-t p with a shallow prompt does not substantially perform above random. we attribute this to (1) their simpler evaluation tasks which, unlike ours, have decent prompt-tuned performance without continued pretraining; and (2) their hand-designed pretraining tasks that match their evaluation tasks, while p3 conversely avoids training-evaluation task overlap, requiring generalizability. @xcite also found mtl-t p to be effective, though with high resources. we also compare with t0, i.e. mtl-t p✗, where both the official model and our reproduction suffer from degraded performance when few-shot shallow prompt tuned (compared to 0-shot), likely because the prompt added during finetuning is intrusive, and the limited gradient updates are not sufficient to re2.8 2.9 ---1.7 -1.6 --mtl-t p✗ @xcite cover from it. we note that the official t0 model is not well-optimized: even without hyperparameter tuning, our implementation is significantly better (@xmath4.001 for all). mtl-t p significantly outperforms mtl-t p✗, the strongest existing method we examine, across all settings (@xmath5.005 for all) except for few-shot deep prompt tuning on t5-xl (@xmath6.21). for zero-shot nl promptability, the improvement could be due to the extra model capacity, or the multi-task trained prompt adjusting the reasoning of the lm, analogous to the text-based "let's think step by step" effect @xcite . for fewshot shallow prompt tuning, unlike mtl-t p✗, mtl-t p does not degrade in performance, resulting in 31% higher arg than mtl-t p✗ on t5-large. this is likely because of the model's familiarity with the prompt, though the limited capacity of shallow prompt tuning does not yield benefits either. nevertheless, with deep prompt tuning, which gives the model sufficient conditioning capacity, few-shot tuning does lead to performance increase, again outperforming mtl-t p✗. here, mtl-t p provides a good prompt initialization and alleviates its intrusiveness. these results emphasize the importance of continued pretraining being aware of the downstream finetuning process. interestingly, however, the gap between these two models shrinks as the model size increases, no longer significant at t5-xl (@xmath7.21). also, notably, pretraining with a shallow prompt has better 0-shot performance than a deep prompt. this high-lights that higher pretraining capacity is not always beneficial, and matches our motivation from textbased conditioning which also happens at the input level. fomaml and reptile surprisingly underperform mtl-t p in few-shot prompt tuning, even though they specifically optimize for this procedure and have demonstrated success in nlp for full model finetuning @xcite @xcite , i.a.) and few-shot learning @xcite @xcite . @xcite also found fomaml to underperform mtl, they sub-optimally only performed one inner loop update. here, we show that this comparison holds for more appropriate hyperparameters. this could be due to the fewer number of gradient updates: to perform one gradient update, mtl uses one training batch, while fomaml with t inner loop steps or reptile with t prompt tuning steps use t + 1 batches. not only might this be an inefficient use of training examples, but compute flops too, since each inner loop/prompt tuning step involves a full forward-backward pass. we attempt using a t + 1 times smaller meta batch size (see §b for more detail) to pretrain a deep t5-large-sized reptile. when prompt-tuned, it achieves 22.8 arg, which is even lower, possibly due to higher gradient estimation noise. alternatively, other factors could affect the performance of meta-learning. it is, for example, well-known that maml-style meta-learning can be unstable and sensitive to architectures and hyperparameters @xcite . this instability is likely amplified by our large heterogeneous multi-task setup and our inability to afford hyperparameter search. furthermore, its theoretical foundation has mostly only been examined through simple optimizers, predominantly sgd @xcite . how it interacts with optimizers more common in modern nlp, such as adafactor (which we use), remains to be explored. ## conclusion we demonstrated that the simple recipe of continued pretraining with a prompt significantly improves zero-shot nl promptability and fewshot learned promptability. maml-based metalearning, on the other hand, obtains worse performance, for which we provided several explanations. nonetheless, we believe future efforts to leverage their conceptual advantage could be fruitful, perhaps aided by our observations. we also hope to study the effect of continued pretraining with other parameter injection methods @xcite @xcite .