Parameter-Efficient Fine-Tuning without Introducing New Latency (2023)

Abstract: Parameter-efficient fine-tuning (PEFT) of pre-trained language models has recently demonstrated remarkable achievements, effectively matching the performance of full fine-tuning while utilizing significantly fewer trainable parameters, and consequently addressing the storage and communication constraints. Nonetheless, various PEFT methods are limited by their inherent characteristics. In the case of sparse fine-tuning, which involves modifying only a small subset of the existing parameters, the selection of fine-tuned parameters is task- and domain-specific, making it unsuitable for federated learning. On the other hand, PEFT methods with adding new parameters typically introduce additional inference latency. In this paper, we demonstrate the feasibility of generating a sparse mask in a task-agnostic manner, wherein all downstream tasks share a common mask. Our approach, which relies solely on the magnitude information of pre-trained parameters, surpasses existing methodologies by a significant margin when evaluated on the GLUE benchmark. Additionally, we introduce a novel adapter technique that directly applies the adapter to pre-trained parameters instead of the hidden representation, thereby achieving identical inference speed to that of full fine-tuning. Through extensive experiments, our proposed method attains a new state-of-the-art outcome in terms of both performance and storage efficiency, storing only 0.03% parameters of full fine-tuning.

## introduction pre-trained language models (plms) have served as a cornerstone for various natural language processing applications, favoring downstream tasks by offering a robust initialization @xcite @xcite @xcite (zoom in for a better experience.) marker size denotes the storage requirement. hiwi stores the least and its storage is invariant to the number of trainable parameters. specific values are shown in with a pre-trained checkpoint, a model can achieve significantly better performance on tasks of interest than the one from scratch. the most historically common way to adapt plms to downstream tasks is to update all pre-trained parameters, full fine-tuning. while full fine-tuning produces numerous state-of-the-art results, it is impractical for storage-constrained and communication-frequent cases, like federated learning @xcite , since it requires a full copy of the fine-tuned model for each task. this issue becomes more severe when plms are large-scale @xcite @xcite @xcite , the number of tasks in interest grows, or data are privately saved on hundreds of servers for federated learning. an alternative approach popularized by @xcite is parameter-efficient fine-tuning (peft), where a small number of task-specific parameters is updated and the majority of plm's parameters is frozen. in this way, only one general plm alongside the modified parameters for each task is saved or transferred. except for saving memory and training cost, peft matches the performance of full fine-tuning with only updating less than 1% of the plm parameters, quickly adapts to new tasks without catastrophic forgetting @xcite and often exhibits robustness in out-ofdistribution evaluation @xcite . these compelling advantages have sparked considerable interest in the adoption of peft. peft methods can be split into two categories: sparse and infused fine-tuning. sparse fine-tuning tunes a small subset of existing parameters without introducing new parameters. one typical example is bitfit @xcite , where only the biases are updated. nonetheless, bitfit is not scalable because of the fixed bias terms. diff pruning @xcite and fish mask @xcite alleviate this issue by learning and updating task-specific masked parameters with a specified sparsity ratio. however, different masks are learned under different tasks, making these two methods unsuitable for the federated learning setting, where data is rarely i.i.d. across servers. infused fine-tuning introduces new parameters to plms, and only updates these parameters during training. for example, adapter fine-tuning @xcite inserts adapters to each layer of the plm. other methods, like prefix tuning @xcite and prompt tuning @xcite , append trainable vectors to input or hidden layers. however, inference latency is typically introduced by the newly added parameters and is nonnegligible for some complex tasks, like machine translation (mt) and summarization that add more than 4% of the plm parameters @xcite . in this paper, we address the above-mentioned challenges from sparse and infused fine-tuning by proposing two methods, pafi and hiwi (illustrated in figure 2 ). pafi is a sparse fine-tuning method that selects trainable parameters in a task-agnostic way. i.e., we have the same mask for various downstream tasks. the mask generation of pafi is also data-less. it doesn't require any training on any data. hiwi is an infused fine-tuning method that applies the adapters directly to pre-trained weights or biases instead of to hidden representations. after training, the adapters are abandoned, therefore sharing the same inference speed as full fine-tuning. our main contributions in this paper are: (1) we introduce two novel transfer learning methods that solve the above-mentioned key challenges of sparse and infused fine-tuning. (2) we empirically evaluate pafi on the glue benchmark and show its effectiveness over existing sparse fine-tuning methods. (3) we compare our methods to a wide range of baselines on a newly constructed benchmark that contains tasks in different types and resources. hiwi outperforms all baselines and full fine-tuning, while requiring the minimum storage (see figure 1 ). (4) our proposed methods still show their effectiveness on a complex task, i.e. machine translation. and all pafi and hiwi share the same inference speed as full fine-tuning. ## preliminaries in this section, we give an overview of sparse finetuning and adapter fine-tuning, and highlight the key challenges of these peft methods. sparse fine-tuning. sparse fine-tuning formulates a task-specific fine-tuning as a two-phase learning problem. in the first phase, one needs to determine which subset of the pre-trained parameters θ (0) can be modified by generating a sparse mask @xmath0, 1} after obtaining the mask, the plm is finetuned and only the masked parameters are updated whereas the others are frozen. the learning procedure of the second phase is defined as with θ initialized by θ (0) , where l and d are the objective and data of downstream task, respectively. adapter fine-tuning. adapter fine-tuning methods @xcite insert one or multiple small mlp modules into each layer of the plm. this mlp module consists of a down (w @xmath1, where r is the bottleneck dimension, d is the dimension of hidden representation and r ≪ d. most adapter fine-tuning methods can be fed into the formula of the adapter fine-tuning methods in the formula of equation 2 are module-wise, which means they consider the attention module or the feed-forward module as a unit and insert the adapter in between or after these units. in contrast, lora @xcite inserts the adapter layer-wise as: a unified framework for sparse and adapter fine-tuning. normally, we initialize w down and w up (or at least one of them) close to 0 @xcite , so the initial state of h is close to the original state h of plm, which makes the fine-tuning empirically perform better. this initialization is important for peft in case of the catastrophic forgetting of the pre-training knowledge. supposed we defined the newly added parameters (w down s and w up s) for adapter finetuning as δ, the initialization of δ (i.e. straightforwardly, we can combine the second phase of sparse fine-tuning (equation 1 ) and the adapter fine-tuning as: for sparse fine-tuning, the 1s in z only locates for the trainable parameters in θ (0) , whereas all locations for δ (0) in z are 1s for adapter fine-tuning. in a word, the subset of the trainable parameters in ϕ is fixed for adapter fine-tuning, but task-specific for sparse fine-tuning. key challenges. sparse fine-tuning normally gains less attention than adapter fine-tuning. the reasons are two-fold: (1) in the first phase of sparse fine-tuning, the generation of the sparse mask is task-specific, which means different downstream tasks or the same task with different domain data might have different masks, whereas adapter finetuning always has fixed positions for adapters; (2) one needs some tricks to generate these masks, like a differential version of l 0 norm @xcite or fisher information @xcite , which requires more computation than a normal full fine-tuning for the same iterations. compared to sparse fine-tuning, adapter finetuning typically introduces additional inference latency from the newly added parameters. though lora doesn't have this issue, one can't apply a nonlinear function in the adapter for lora since ## methodologies motivated by the key challenges stated in section §2, we propose two methods in this section and illustrate them in figure 2 : one for sparse fine-tuning that generates a universal mask for various tasks without any training, and another one for adapter fine-tuning that has the same inference speed as full fine-tuning while requiring even less storage than bitfit @xcite . ## result and discussion in this section, we present the results of baselines and our proposed methods on glue, various-glue and translation tasks. ## conclusion in this work, we first propose pafi as a novel but simple method for computing the sparse mask without any training and in a task-agnostic way. it selects the parameters with the lowest absolute magnitude from plms and tunes them, with keeping others frozen. we demonstrate the effectiveness of pafi on the glue benchmark and translation task. secondly, we propose hiwi as an adapter finetuning method that feeds pre-trained parameters instead of hidden representations to the adapters. it doesn't introduce any inference latency. furthermore, it requires the lowest storage while outperforming other strong baselines. in the future, we will try to compute the sparse mask with fisher information and estimate our methods in a realistic few-shot learning setting.