An Empirical Study on the Transferability of Transformer Modules in Parameter-efficient Fine-tuning (2022)

Abstract: Parameter-efficient fine-tuning has garnered lots of attention in recent studies.On this subject, we investigate the capability of different transformer modules in transferring knowledge from a pre-trained model to a downstream task. Our empirical results suggest that every transformer module is a winning ticket such that fine-tuning the specific module while the rest of the network is frozen achieves a comparable performance to the full fine-tuning case. Among different modules in LMs, LayerNorms exhibit a significant capacity for transfer learning to the extent that with only 0.003% updateable parameters in the layer-wise analysis, they can show acceptable performance on various target tasks.We argue that the performance of LayerNorms could be attributed to their high-magnitude weights compared to other components in a pre-trained model.

## introduction fine-tuning is widely used as a procedure to employ the knowledge learned during pre-training of language models for specific tasks @xcite @xcite . however, finetuning can be a computationally expensive process, given that it usually involves updating all the parameters in transformer-based models which are often massive in size. parameter-efficient finetuning methods try to ameliorate this by reducing the number of updatable parameters during finetuning. * the authors contributed equally to this work. adapters @xcite @xcite @xcite try to circumvent this issue by inserting lightweight modules in the transformer blocks, tuning of which usually results in comparable performance to the full fine-tuning (while the number of updatable parameters is significantly lower). nevertheless, introducing new parameters to an already-large model can be considered a drawback. another category of parameter-efficient fine-tuning methods is based on the lottery ticket hypothesis @xcite , where the goal is to find a small subset of parameters that can compete with the full fine-tuning setting. various subsets of network parameters have been suggested as the winning ticket, including the connections with high magnitudes @xcite , identity mappings @xcite , and dominant dimensions @xcite . in this paper, we study the ability of different modules of a transformer block in knowledge transfer. our experiments provide a more comprehensive analysis than the existing work, which usually suggests specific modules as the winning ticket, such as the bias terms @xcite . through module-wise fine-tuning, we check if the winning ticket is a property that can be associated only with some specific modules in the transformer block. our results suggest that all individual modules possess this property to some extent. among these, layernorms prove to be highly reliable for knowledge transfer: fine-tuning only 37k layernorm weights (out of 110m parameters in bert-base) is often on par with full fine-tuning on various downstream tasks. extending this analysis, we show that tuning even only one layernorm can yield comparable performance and that the middle layers are the best in terms of transferability. we also investigate the reasons behind the effectiveness of layernorm tuning. our experiments suggest that this could be due to the relatively high-magnitude weights in these modules. in fact, we show that tuning just a tiny fraction of high-magnitude dimensions (usually referred to as outliers) can lead to competitive performance on various tasks. ## winning modules according to the lottery ticket hypothesis, there are small sub-networks whose performance is comparable to the over-parameterized model on different tasks @xcite . several studies have been carried out to identify subnetworks across the model that can provide the best transferability @xcite @xcite @xcite . nonetheless, finding the winning sub-network usually requires extra computation, which is costly in terms of time and memory. in this section, we take another look at the transformer block of bert and focus on the ability of its different modules to transfer knowledge to various downstream tasks. more specifically, we aim to find the winning module among the different modules in the transformer-based architecture of the pre-trained bert. ## analysis in the previous section, we have shown that different modules of a transformer block can play as the winning tickets, since they all have the potential for transferring knowledge to the selected downstream tasks. among different modules, layer-norms have proven to be the most reliable in finetuning. in this section, we search for the reasons behind the effectiveness of these modules. to this end, we focus on the magnitude of every weight and how they change during full fine-tuning across all layers. as a first step, in figure 1 , we visualize the distribution of weights for different bert modules on rte and sts-b (more tasks can be found in the appendix). in general, the distribution of weights is similar across feed-forward and multi-head modules. nevertheless, layernorms tend to have a bimodal distribution, with one of the modes having significantly higher magnitudes. the pattern is consistent across layernorms a and layernorms f . we hypothesize that these highmagnitude weights are the reason behind the effectiveness of layernorms and, in what follows, 2 for sts-b, the number of classifier parameters is 769. check our hypothesis by restricting our experiments to only high-magnitude dimensions of lay-ernorms. ## conclusions our results pave the way for better parameterefficient fine-tuning of large language models without the need for costly algorithms to determine the optimum sub-network or introduce additional parameters for knowledge transfer.