Parameter-Efficient Transfer Learning with Diff Pruning (2021)

Abstract: The large size of pretrained networks makes them difficult to deploy for multiple tasks in storage-constrained settings. Diff pruning enables parameter-efficient transfer learning that scales well with new tasks. The approach learns a task-specific “diff” vector that extends the original pretrained parameters. This diff vector is adaptively pruned during training with a differentiable approximation to the L0-norm penalty to encourage sparsity. As the number of tasks increases, diff pruning remains parameter-efficient, as it requires storing only a small diff vector for each task. Since it does not require access to all tasks during training, it is attractive in on-device deployment settings where tasks arrive in stream or even from different providers. Diff pruning can match the performance of finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model’s parameters per task and scales favorably in comparison to popular pruning approaches.

## introduction task-specific finetuning of pretrained deep networks is the dominant paradigm in contemporary nlp, achieving state-of-the-art results across a suite of natural language understanding tasks @xcite @xcite . while straightforward and empirically effective, this approach is difficult to scale to multi-task, memory-constrained settings (e.g. for on-device applications), as it requires shipping and storing a full set of model parameters for each task. inasmuch as these models are learning generalizable, task-agnostic language representations through self-supervised pretraining, finetuning the entire model for each task seems especially profligate. code: https://github.com/dguo98/diffpruning a popular approach to parameter-efficiency is to learn smaller compressed models for each task @xcite @xcite . such approaches face a steep sparsity/performance tradeoff and keep a substantial amount of nonzero parameters per task (e.g. 10%-30%). multi-task learning and featurebased transfer allow for more parameter-efficient transfer learning per task @xcite @xcite . these methods train a small number of additional parameters (e.g. a linear layer) on top of a shared model. however, multi-task learning generally requires access to all tasks during training to prevent catastrophic forgetting @xcite , while feature-based transfer learning (e.g. based on task-agnostic sentence representations) is typically outperformed by finetuning @xcite . an appealing middle ground is to finetune an extension of the base model for specific tasks. this approach captures the training benefits of finetuning while maintaining the task modularity of feature-based transfer. for example, adapters @xcite use smaller, task-specific modules that are inserted between layers of a model this approach does not require access to all tasks during training, targeting realistic settings where as new tasks arrive in stream @xcite . @xcite find that adapter layers can match the performance of fully finetuned bert on the glue benchmark while requiring 3.6% additional parameters (on average) per task. diff pruning is a new extension to pretrained models with the goal of even more parameterefficient transfer learning. instead of modifying the architecture of the model, diff pruning extends the base model through a task-specific difference vector. in order to learn this vector, we reparameterize the task-specific model parameters as θ @xmath0, where the pretrained parameter vector θ pretrained is fixed and the task-specific diff vector δ task is finetuned. the diff vector is regularized with a differentiable approximation to the l 0 -norm penalty @xcite to encourage sparsity. diff pruning can become extremely parameterefficient, as it only requires storing the nonzero positions and weights of the diff vector for each task. the cost of storing the shared pretrained model remains constant and is amortized across multiple tasks. on the glue benchmark @xcite , diff pruning can match the performance of the fully finetuned bert baselines while finetuning only 0.5% of the pretrained parameters per task. as the number of tasks increase, diff pruning outperforms popular pruning-based methods in amount of storage required. ## background: transfer learning transfer learning in nlp mostly uses a pretrainand-finetune paradigm, which initializes a subset of the model parameters for all tasks from a pretrained model and then finetunes on a task-specific objective. pretraining objectives include context prediction @xcite , autoencoding @xcite , machine translation @xcite , and more recently, variants of language modeling @xcite @xcite objectives. here we consider applying transfer learning to multiple tasks. we consider a setting with a potentially unknown set of tasks (which may arrive in stream), where each task τ ∈ t has an associated training set for all tasks, the goal is to produce (possibly tied) model parameters θ τ to minimize the empirical risk, we can use the pretrain-finetune approach by simply learning independent parameters for each task. however, the large size of pretrained models makes this approach exceedingly parameter inefficient. for example, widely-adopted models such as bert base and bert large have 110m and 340m parameters respectively, while their contemporaries have parameter counts in the billions @xcite @xcite . storing the fully finetuned models therefore becomes difficult even for a moderate number of tasks. 2 a classic approach to tackling this parameter-inefficiencyis to train a single shared model (along with a task-specific output layer) against multiple tasks through joint training @xcite . however, the usual formulation of multi-task learning requires the set of tasks t to be known in advance in order to prevent catastrophic forgetting @xcite ), foot_2 making it unsuitable for applications in which the set of tasks is unknown or when tasks arrive in stream. ## diff pruning diff pruning formulates task-specific finetuning as learning a diff vector δ τ that is added to the pretrained model parameters θ, which remain fixed. we first reparameterize the task-specific model parameters, this trivial reparameterization shows that the cost of storing the pretrained parameters θ is amortized across tasks, and the only marginal cost for new tasks is the diff vector. if we can regularize δ τ to be sparse such that δ τ 0 θ 0 , then this approach can become more parameter-efficient as the number of tasks increases. we can specify this goal with an l 0 -norm penalty on the diff vector, r(θ + δ τ ) = δ τ 0 = d @xmath1,@xmath2. ## analysis 6.1 varying the target sparsity in figure 1 (left), we plot results on the glue validation set averaged across all tasks at target sparsity 14 comparing storage costs is a bit more challenging as it is implementation-specific. diff pruning incurs additional storage cost due to storing the nonzero positions of the diff vector. see section 6.6 for storage comparison against adapters assuming float32 for weights and int32 for positions. rates of 0.1%, 0.25%, 0.5%, 1.0% for the different baselines. structured diff pruning consistently outperforms non-structured and and non-adaptive variants across different sparsity rates. the advantage of adaptive methods becomes more pronounced at extreme sparsity rates. in ## related work multi-task learning multi-task learning @xcite , broadly construed, aims to learn models and representations that can be utilized across a diverse range of tasks, and offers a natural approach to training parameter-efficient deep models. several works have shown that a single bert model can obtain good performance across multiple tasks when jointly trained @xcite @xcite ). an alternative approach to multi-task learning that does not require access to all tasks during training involve training smaller task-specific layers that interact with a fixed pretrained model @xcite . in particular, adapters @xcite , which learn to read and write to layers of a shared model, have been applied to obtain parameter-efficient bert models @xcite . in recent work, li & liang (2021) and @xcite explore the use of learned prompts on top of pretrained models to obtain task-specific models. yet another line of work targets extreme parameterefficiency through task-agnostic sentence representations that can be used without finetuning for downstream tasks @xcite @xcite @xcite @xcite @xcite @xcite . these feature-based transfer learning methods are however generally outperformed by fully finetuned models @xcite . ## conclusion we propose diff pruning as a simple approach for parameter-efficient transfer learning with pretrained models. experiments on standard nlp benchmarks and models show that diff pruning can match the performance of fully finetuned baselines while requiring only a few additional parameters per task, and can sometimes have a regularization effect and improve upon regular finetuning. we also propose a structured variant of diff pruning which provides further improvements. avenues for future work include (i) injecting parameter-efficiency objectives directly into the pretraining process (to pretrain models that are better suited towards sparse transfer learning), and (ii) combining diff pruning with other techniques (e.g. adapters, model compression) to achieve even greater parameter-efficiency.