Exploring the Impact of Model Scaling on Parameter-Efficient Tuning (2023)

Abstract: Parameter-efficient tuning (PET) methods can effectively drive extremely large pre-trained language models (PLMs) by training only minimal parameters. Different PET methods utilize different manually designed tunable modules. In small PLMs, there are usually noticeable performance differences among PET methods. Nevertheless, as the model scale increases, the performance differences become marginal. Hence, we hypothesize that model scaling mitigates the impact of design differences on PET methods. To investigate this hypothesis, we introduce a more flexible PET method called Arbitrary PET (APET) method. The APET method is compatible with a tunable module, which consists of any number of parameters distributed in arbitrary positions. Then, we utilize it and conduct experiments on 11 NLP tasks across 3 representative PLMs. Our investigations reveal that model scaling (1) mitigates the effects of the positions of tunable parameters on performance, and (2) enables tuning methods to achieve performance comparable to full-parameter fine-tuning by optimizing fewer tunable parameters. Intriguingly, we also observe that tuning methods optimize the similar number of tunable parameters to exceed random guess performance on different tasks. We collectively discuss this phenomenon and the two aforementioned findings from an optimization perspective to understand the underlying mechanisms. These conclusions enhance our understanding of the impact of model scaling on PET and assist in designing more effective and efficient PET methods for PLMs of different scales. The source code can be obtained from this GitHub repository: https://github.com/yushengsu-thu/PET_Scaling .

## introduction pre-trained language models (plms), such as gpt @xcite , bert @xcite , * the first two authors contributed equally. † corresponding author: z.liu and m.sun. ## related work parameter-efficient tuning (pet) methods with larger plms continuously being developed, fine-tuning all of the parameters and storing the adapted weights become increasingly cumbersome. to address the issue, researchers propose pet methods which keep most of the parameters of plms frozen and optimize only a tunable module consisting of a few parameters during downstream adaptation. over the recent years, many different designs of pet methods have emerged. for instance, some pet methods insert the external tunable modules after the feed-forward and attention layers in a plm @xcite @xcite ; others prepend the tunable modules into attention layers @xcite or the embedding layer @xcite . another line of pet method selects the existing parameters in a plm @xcite as the tunable module to optimize. to further enhance the performance of pet methods, some works propose automatic selection strategies @xcite @xcite for tunable parameters. ## preliminary in this section, we first introduce the transformer framework ( § 3.1) and the most representative pet ( § 3.2). ## main experiments to explore the impact of model scaling on these pet methods, we first introduce the investigated tasks, plms, and settings of the existing representative pet methods in the experiments ( § 4.1), and then report the main experimental results ( § 4.2). ## ablation analyses the design differences among the pet methods mainly lie in the tunable module's parameter position and parameter quantity. to further verify whether the model scaling will respectively remove the effects of the above differences on pet methods, we conducted two ablations to investigate whether model scaling can mitigate (1) the impact of tunable parameter position and (2) the impact of tunable parameter quantity. however, only investigating the above four respective pet methods is insufficient to cover enough variations of parameter position for ablation study (1). this limitation makes us hard to preciously control the number of tunable parameters at the fine-grained (parameter level) in ablation study (2). hence, we develop a more flexible pet method, arbitrary parameter-efficient tuning (apet) method. its tunable module can be arbitrary structure ( § 5.1) that facilitates us to explore various parameter positions in the ablation study ( § 5.2) and easier control the number of tunable parameters in the ablation study ( § 5.3). ## discussing the ablation results from the optimization perspectives the objectives of all parameter-efficient tuning methods (pet, apet) can be expressed as min θ l(m (φ,θ) (x), y ) as introduced in equation ( 3 ), where θ is a tunable module. the module θ of different pet methods consists of different structures and varying numbers of tunable parameters. in this paper, we investigate the impact of model scaling on different modules, which possess varying numbers of tunable parameters distributed across multiple positions. we find that the larger model scaling can (1) mitigate the effects caused by the difference positions of tunable parameters ( § 5.2) and (2) make pet methods optimize fewer tunable parameters to achieve full-parameter finetuning performance ( § 5.3). to further fathom these phenomena, we will investigate the underlying reasons from an optimization perspective. (3) besides, we also observe that pet methods can optimize almost the similar number of necessarily tuned parameters to exceed random guess performance on the same backbone models ( § 5.3). although phenomenon (3) is not caused by model scaling, we can also explain it from the optimization perspective. next, we together discuss it and the above two findings (1) and (2) in the following paragraphs. why model scaling mitigates the effects caused by the differences in positions of tunable parameters on the pet performance? from the optimal control perspective, a tunable module (θ) of a tuning method can be seen as a controller @xcite to drive plms towards downstream tasks. as the model scale increases, the larger model has higher parameter redundancy @xcite , allowing arbitrary selection of tunable parameters for tuning without greatly degrading performance @xcite @xcite ; thus, controllers (modules) might have higher degrees of freedom. this might explain why the aribitray positions of the tunable parameters have less impact such that all pet methods can achieve the similar performance on the larger models. it is worth noting that even though the distribution of tunable parameters have less impact on the performance, it still affects converge speeds. thus, finding a better parameter distribution to improve the converge speeds for pet methods is a direction worthy of exploring. why model scaling leverages the fewer tunable parameters to achieve full-parameter finetuning performance? tuning θ to steer a plm towards downstream nlp tasks can be seen as adaptations. from the perspective of representation space, the adaptations of pet methods can be re-parameterized into a unified low dimensional subspace @xcite @xcite . @xcite further demonstrate that adaptation on a larger plm can be re-parameterized into the lower dimensional space; this implicitly explains why pet methods can optimize fewer parameters on larger-scale models, e.g., t5 xxl , to meet the full-parameter fine-tuning performance on tasks. why can pet methods optimize the similar numbers of tunable parameters to exceed random guessing? as stated above, the adaptations of the pet methods can be re-parameterized into a unified subspace. @xcite show that this low dimensional subspace is shared among all nlp tasks for the same pet methods. @xcite further suggest that this subspace is also shared among various pet methods. this might implicitly explain why all pet methods can tune the similar numbers of necessary tuned parameters to exceed the random guessing performance on the same models, even for the different tasks ( § 5.3). ## conclusion the realm of model scaling for llms presents important and intriguing directions for the llm community. the increasing of model scale unveils numerous emerging capabilities and advantages. in this work, our primary emphasis is on the impact of model scaling as it pertains to pet methods. through our comprehensive observation studies and in-depth discussions from optimization perspectives, we gain deeper insights into the effects of model scaling on pet and the reasons behind the observed phenomena. we believe that our findings will serve as a catalyst, inspiring further meticulous research and exploration in this area.