B it F it: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2022)

Abstract: We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.

## introduction large pre-trained transformer based language models, and in particular bidirectional masked language models from the bert family @xcite @xcite , are responsible for significant gains in many nlp tasks. under the common paradigm, the model is pre-trained on large, annotated corpora with the lm objective, and then finetuned on task-specific supervised data. the large size of these models make them expensive to train and, more importantly, expensive to deploy. this, along with theoretical questions on the extent to which finetuning must change the original model, has led researchers to consider finetuning variants where one identifies a small subset of the model parameters which need to be changed for good performance in end-tasks, while keeping all others intact ( §2). we present a simple and effective approach to fine tuning ( §3), which has the following benefits: specifically, we show that freezing most of the network and fine-tuning only the bias-terms is surprisingly effective. moreover, if we allow the tasks to suffer a small degradation in performance, we can fine-tune only two bias components (the "query" and "middle-of-mlp" bias terms), amounting to half of the bias parameters in the model, and only 0.04% of all model parameters. this result has a large practical utility in deploying multi-task fine-tuned models in memoryconstrained environments, as well as opens the way to trainable hardware implementations in which most of the parameters are fixed. additionally, it opens up a set of research directions regarding the role of bias terms in pre-trained networks, and the dynamics of the fine-tuning process. ## background: fine-tuning and parameter-efficient fine-tuning in transfer-learning via model fine-tuning, a pretrained encoder network takes the input and produces contextualized representations. then, a taskspecific classification layer (here we consider linear classifiers) is added on top of the encoder, and the entire network (encoder+task specific classifiers) is trained end-to-end to minimize the task loss. desired properties. while fine-tuning per-task is very effective, it also results in a unique, large model for each pre-trained task, making it hard to reason about what was changed in the fine-tuning process, as well as hard to deploy, especially as the number of tasks increases. ideally, one would want a fine-tuning method that: (i) matches the results of a fully fine-tuned model; learning vs. exposing. the feasibility of fulfilling the above requirements depends on a fundamental question regarding the nature of the fine-tuning process of large pre-trained lms: to what extent does the fine-tuning process induces the learning of new capabilities, vs. the exposing of existing capabilities, which were learned during the pre-training process. existing approaches. two recent works have demonstrated that adaptation to various end-tasks can in fact be achieved by changing only a small subset of parameters. the first work, by @xcite ("adapters"), achieves this goal by injecting small, trainable task-specific "adapter" modules between the layers of the pre-trained model, where the original parameters are shared between tasks. the second work, by @xcite ("diff-pruning"), achieves the same goal by adding a sparse, task-specific difference-vector to the original parameters, which remain fixed and are shared between tasks. the difference-vector is regularized to be sparse. both methods allow adding only a small number of trainable parameters per-task (criteria ii), and each task can be added without revisiting previous ones (criteria iii). they also partially fulfill criteria (i), suffering only a small drop in performance compared to full fine-tuning. the adapter method, but not the diff-pruning method, also supports criteria (iv). however, diff-pruning is more parameter efficient than the adapter method (in particular, it adds no new parameters), and also achieves better task scores. we compare against diff-pruning and adapters in the experiments section, and show that we perform favorably on many tasks while also satisfying criteria (iv). ## bias-terms fine-tuning (bitfit) we propose a method we call bitfit foot_0 (bias-term fine-tuning), in which we freeze most of the transformer-encoder parameters, and train only the bias-terms and the task-specific classification layer. bitfit has three key properties: (i) match the results of fully fine-tuned model. (ii) enable tasks to arrive in a stream, this way it does not require simultaneous access to all datasets. (iii) fine-tune only a small portion of the model's parameters. the approach is parameter-efficient: each new task requires storing only the bias terms parameter vectors (which amount to less than 0.1% of the total number of parameters), and the task-specific final linear classifier layer. concretely, the bert encoder is composed of l layers, where each layer ℓ starts with m selfattention heads, where a self attention head (m, ℓ) has key, query and value encoders, each taking the form of a linear layer: where x is the output of the former encoder layer (for the first encoder layer x is the output of the embedding layer). these are then combined using an attention mechanism that does not involve new parameters: the collection of all matrices w the bias terms are additive, and correspond to a very small fraction of the network, in bert base and bert large bias parameters make up 0.09% and 0.08% of the total number of parameters in each model, respectively. we show that by freezing all the parameters w (•) and g (•) and fine-tuning only the additive ## experiments and results datasets. we evaluate bitfit on the glue benchmark @xcite ). foot_2 consistent with previous work (houlsby et al., 2019; guo et al., 2020) we exclude the wnli task, on which bert models do not outperform the majority baseline. models and optimization. we use the publicly available pre-trained bert base , bert large (devlin et al., 2018) and roberta base (liu et al., 2019) models, using the huggingface (wolf et al., 2020) interface and implementation. appendix §a.2 lists optimization details. comparison to diff-pruning and adapters ( on validation set, bitfit outperforms diff-pruning on 4 out of 9 tasks, while using 6x fewer trainable parameters foot_3 . as for test-set results, two clear wins compared to diff-pruning and 4 clear wins compared to adapters while using 45x fewer trainable parameters. different base-models ( are bias parameters special? are the bias parameters special, or will any random subset do? we randomly sampled the same amount of parameters as in bitfit from the entire model, and fine-tuned only them ("rand uniform" line in fewer bias parameters ( we define the amount of change in a bias vector b to be 1 dim(b) ∥b 0 -b f ∥ 1 , that is, the average absolute change, across its dimensions, between the initial lm values b 0 and its fine-tuned values b f . figure 1 shows the change per bias term and layer, for the rte task (other tasks look very similar, see appendix §a.4). the 'key' bias b k has zero ## related work the problem of identifying the minimal set of parameters that need to be fine-tuned to achieve good performance in end-tasks relates both to practical questions of model compression, and also to more fundamental question on the nature of the pre-training and finetuning process, the "linguistic knowledge" induced by each of them, and the extent to which it generalizes to different tasks. over-parameterization large lm models were shown to be over-parameterized: they contain more parameters than needed in inference @xcite @xcite @xcite @xcite @xcite . @xcite have demonstrated that overparmeterization can be exploited in finetuning: pruned network perform well in transfer setting. we work in a complementary setting, where the entire model is kept, but only some parameters are updated. the remarkable success of those works have sparked interest the lottery-ticket hypothesis @xcite @xcite : the conjecture that large models are needed in pretraining only to induce (in high probability) the existing of sub-networks initialized with the correct inductive bias for learning, and the findings that those sparse networks often transfer well to different tasks. bias terms bias terms and their importance are rarely discussed in the literature. 5 zhao et al. (2020) describe a masking-based fine-tuning method, and explicitly mention ignoring the bias terms, as handling them "did not observe a positive effect on performance". an exception is the work of @xcite who analyzed bias terms from the perspective of attribution method. they demonstrate that the last layer bias values are responsible for the predicted class, and propose a way to back-propagate their importance. @xcite finetuned the biases of the output softmax in an nmt systems, to personalize the output vocabulary, and @xcite have demonstrated that randomly-initialized cnns achieve reasonable accuracy after training the batch-norm layers alone. finally, and closest to our work, @xcite demonstrate that bias-only fine-tuning similar to ours is effective also for adaptation of pre-trained computer vision models. our work empirically shows the importance and power of the bias parameters to substantially change the networks' behavior, calling for further analysis and attention on the bias terms. ## conclusions we propose bitfit, a novel method for localized, fast fine-tuning of pre-trained transformers for endtasks. the method focuses the finetuning on a specific fraction of the model parameters-the biasesand maintains good performance in all glue tasks we evaluated on. the focus on modifying a small group of parameters eases deployment, as the vast majority of the parameters of the model are shared between various nlp tasks. it also allows for efficient hardware implementations that hard-wire