Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation (2022)

Abstract: Large pretrained language models (PLMs) are often domain- or task-adapted via finetuning or prompting. Finetuning requires modifying all of the parameters and having enough data to avoid overfitting while prompting requires no training and few examples but limits performance. Instead, we prepare PLMs for data- and parameter-efficient adaptation by learning to learn the difference between general and adapted PLMs. This difference is expressed in terms of model weights and sublayer structure through our proposed dynamic low-rank reparameterization and learned architecture controller. Experiments on few-shot dialogue completion, low-resource abstractive summarization, and multi-domain language modeling show improvements in adaptation time and performance over direct finetuning or preparation via domain-adaptive pretraining. Ablations show our task-adaptive reparameterization (TARP) and model search (TAMS) components individually improve on other parameter-efficient transfer like adapters and structure-learning methods like learned sparsification.

## introduction finetuning large pretrained language models (plms) on task-specific supervised data has become the default strategy to produce performant models for various nlp tasks @xcite @xcite , inter alia), provided a task has enough training data to be adapted to without overfitting. for few-shot tasks, very large plms like the 175b-parameter @xcite do surprisingly well without training using prompts, where task-specific examples (x j , y j ) are presented as text to condition the plm before a test input x test is given. our work considers an important middle ground: minimizing the computational cost of finetuning while improving on its performance in low-resource and fewshot settings. in general, self-supervised objectives used for plms assume little about the nature of downstream tasks. earlier works suggested that task-awareness is unnecessary for plms of sufficient scale; for example, @xcite found that multi-task learning underperformed pretrain-finetune for the largest t5 models on multi-format question answering. however, @xcite showed that further pretraining on unlabeled text from the downstream task (task-adaptive pretraining, or tapt) or a related domain (dapt) consistently improved adaptation performance. @xcite revisited @xcite found that by greatly improving the number and balance of tasks, one can utilize a multitask objective after pretraining and achieve gains in proportion to the number of tasks. as for even larger models, @xcite argue that the impressive few-shot prompting ability of gpt-3 comes from ''implicit'' meta-learning @xcite which they term in-context learning, where the outer loop is performed by self-supervised pretraining, and the inner loop is performed by forward passes on implicit examples in unlabeled texts. these works motivate that exposure to broad information about downstream tasks remains useful in preparing a large plm for adaptation. hence, we propose explicit meta-learning for preparing large plms for data-efficient adaptation; a visual comparison is in figure 1 . to also achieve parameter efficiency and performance, we adapt meta-transfer learning @xcite to large plms in two proposed ways: an inner loop optimizing a low-rank task-adaptive reparameterization (tarp) of weights, and an outer loop learning an architecture controller for searching task-adaptive model structures (tams). these improve over general finetuning figure 1 : comparison between (top) implicit meta-learning from text corpora that incidentally contain task @xcite , and (bottom) explicit meta-learning the transformation of a plm's weights and sublayers for a distribution of tasks. and even dapt-prepared lms on generative and unconditional few-shot and low-resource settings, such as multi-domain abstractive summarization (adaptsum; @xcite and language modeling. furthermore, our analysis shows that each component of our task distribution-aware strategy independently improves over prior work: (1) meta-transfer learning improves over model-agnostic meta learning @xcite even after multitask learning on the same data, setting a new state-of-the-art on few-shot persona-chat dialog personalization @xcite ; (2) our proposed dynamic low-rank tarp outperforms recent methods such as mam adapters @xcite and alternate reparameterizations like kronecker products @xcite ; (3) our lightweight controller for generating task-aware architectures in tams extends improvements into higher resource tasks and rediscovers task-specific modifications like 1d convolutions for transformers. our proposal is summarized in figure 2 , with pseudocode in algorithm 1 at the end of the next section. we publicly release the code for our experiments and our reference library online. ## methodology our goal is to explicitly optimize a plm for efficient adaptation to any task t i sampled from a distribution of low-resource nlp tasks p(t ). each task consists of a training set d train i , a test set d test i , and a loss function l i . the prevailing approach for efficiently optimizing a base model f θ on a (relatively) small task-specific dataset is to use model-agnostic meta-learning (maml; @xcite . this is a bi-level optimization process that uses a stochastic gradient-based strategy to sample a batch of tasks {t i } b maml, however, is not generally used in nlp as a competitive alternative to pretrain-thenfinetune methods for low-resource and few-shot settings. to rectify maml's limitations, we propose a meta-learning the difference (mltd) framework to optimize plms for fast and data-efficient adaptations with the following contributions: maml after pretraining. earlier work performed maml using random initializations, or at best with pretrained token embeddings @xcite , which was shown to underperform the pretrain-finetune paradigm. with the increased prevalence of large-scale pretraining, recent work has begun to initialize maml with plms @xcite . we continue this approach, but further show that pretraining + maml, even when labeled (i.e., multitask) and performed only on the meta-training data (i.e., no external text), improves performance and mitigates overfitting versus pretraining alone or maml alone (section 4), suggesting that pretraining produces a better initialization that promotes generalization in later meta-learning. parameter-efficient transfer. adaptation data is typically limited, making it easy for large models figure 2 : overview of our proposed method, which learns to transform a small set of weights φ i (tarp learning) and modify sublayer modules α i (tams learning) in a task-specific, data-efficient, and parameter-efficient manner. first, we initialize with a base plm (top left). in each meta-iteration, we sample a batch of tasks from a task distribution (left). in the inner loop (middle), independent sets of dynamic low-rank reparameterizations are initialized, and an architecture controller generates independent task-specific sublayer modules, all of whose weights are adapted to the task's training set. each task model is evaluated on the corresponding task's test set. in the outer loop (right), these task losses are summed up to produce the overall meta-loss, and the backward path optimizes the base model, the initial reparameterization, and the architecture controller. to overfit. previous work uses very shallow cnns @xcite , only adapt scale-and-shift parameters atop the original model @xcite , or apply various general regularization techniques such as weight decay, label smoothing, dropout, early stopping, and ℓ 1 regularization @xcite . in contrast, we propose learning dynamic low-rank reparameterizations g φ i (section 2.1) of the base model such that θ i (x) = g φ i (θ lm , x) for task t i . here, φ is a small set of new parameters that are adapted into task-specific φ i when finetuning. notably, we modify maml to incorporate these parameter-efficient modules, so that during task adaptation in both meta-training (the inner loop) and meta-testing (novel tasks) t i , we only adapt φ → φ i instead of θ → θ i , speeding up both phases and improving overall performance. though some work explores the benefits of joint training or fusion of parameter-efficient modules @xcite @xcite , prior work has not explored meta-learning to learn these adaptations in a task distribution-aware setting. architecture adaptation. while the transformer has proven to be a robust general-purpose architecture, recent work has shown that the optimal attention-then-ffn sublayer structure can vary across tasks (e.g., sandwich transformers; @xcite . however, previous data-driven sublayer searches are often task-agnostic (e.g., @xcite , where the sublayer search is implemented before pretraining. meta-learning enables learning data-driven sublayers after pretraining, in a differentiable, task-adaptive manner (section 2.2). instead of a separate search per task as in previous methods (darts; @xcite , we propose meta-learning a task-aware architecture controller to help it generalize to new tasks when searching neural architectures, by learning to directly generate task-specific sublayer structures from the dataset. by exploiting architectural knowledge learned over the task distribution, our task-adaptive model structure approach improves test-time performance. a related work in customizing model structure is cmaml @xcite , which applies a sparse pruning algorithm to obtain task-specific weight masks. our method differs in that we consider generalization over a distribution of tasks (instead of a single task), and it has a richer search space with different operations, numbers of layers, and widths of layers, so that our method provides architecture diversity to accommodate to the different task data. in all, we employ meta-learning to improve upon initializing from a pretrained θ lm , allowing better downstream finetuning on tasks. by learning only the transformation weights φ i and (optionally) the task-specific architecture α i for new tasks t i , our method ''learns to learn the difference'' between a plm and a task-specific lm in a training-efficient way. ## main results to demonstrate the overall benefit of our method, we compare our results to other approaches algorithm 1: meta-learning the difference (mltd) with tarp and tams. ## conclusion we have shown that explicit meta-learning is a useful preparation step on top of plms to improve later finetuning. specifically, our mltd framework incorporating dynamic taskadaptive reparameterization (tarp) and taskadaptive model search (tams) enable data-and parameter-efficient adaptation to a family of lowresource tasks. future avenues include applying our method in other modalities like vision and speech, as well as exploring better model formulations for tarp and tams.