A Study of Parameter Efficient Fine-tuning by Learning to Efficiently Fine-Tune (2024)

Abstract: The growing size of large language models (LLMs) requires parameter-efficient fine-tuning (PEFT) methods for their adaptation to new tasks. Existing methods, such as Low-Rank Adaptation (LoRA), typically involve model adaptation by training the PEFT parameters. One open problem required to be solved to effectively employ these methods is the identification of PEFT parameters. More precisely, related works identify PEFT parameters by projecting high dimensional parameters of LLMs onto low dimensional parameter manifolds with predefined projections, or identifying PEFT parameters as projections themselves. To study this problem, we propose a new approach called Learning to Efficiently Fine-tune (LEFT) where we aim to learn spaces of PEFT parameters from data. In order to learn how to generate the PEFT parameters on a learned parameter space while fine-tuning the LLMs, we propose the Parameter Generation (PG) method. In the experimental analyses, we examine the effectiveness of our solutions exploring accuracy of fine-tuned LLMs and characteristics of PEFT parameters on benchmark GLUE tasks.

## introduction most natural language processing applications today rely on pre-trained large language models (llms), which are designed for general use. even so, one can enhance their adaptation to a new specific task through a fine-tuning process in which their parameters are updated with fine-tuning data. however, fine-tuning all the parameters of modern llms is highly computationally complex due to the need for increased computational resources. the problem of efficient fine-tuning of models has been studied @xcite by exploring their objective @xcite , or in general, optimization landscape @xcite . for training a model with d dimensional parameters, @xcite and @xcite suggest performing optimization on a lower dimensional ( @xmath0, where the solutions to the model training problem first appear. @xcite proposed that measuring intrinsic dimension can provide information on the number of free parameters which are required to closely approximate the optimization problem that is solved while fine-tuning bert models. in the context of llms, this approach has been utilized for developing parameter efficient finetuning (peft) methods @xcite @xcite by optimizing only a small subset of parameters while keeping the remaining parameters w 0 of the llms frozen. inspired from @xcite ; @xcite , @xcite hypothesized that the change in parameters during fine-tuning also has a low intrinsic dimension. following this hypothesis, they proposed a method named low-rank adaptation (lora) to compute low-rank approximation of the change ∆w . to employ lora methods effectively, their fine-tuning parameters should be well-designed. the characteristics of fine-tuning parameters: despite the recent efforts @xcite , the characteristics of these parameters (e.g. their roles in adaptation process, impact on outcome of fine-tuning, and their relationship) are still not fully understood. for instance, lora methods constrain fine-tuning of a pre-trained model with a parameter matrix w 0 ∈ r d×k by a lowrank decomposition w 0 + ∆@xmath1. inspired from the intrinsic dimension of manifolds, @xmath2, k) is described as the rank of a and b. in the literature, various methods and configurations have been proposed to define characteristics of a and b under two approaches. these configurations are illustrated in figure 1 . conversely, @xcite explained the role of a through random projection of ∆b when small learning rates are utilized. when the random a matrices are frozen and only b parameters are updated using gradients with upper-bounded norms, the a matrices project ∇ w 0 l onto lower-dimensional manifolds. following these results, @xcite proposed optimizing parameters w 0 on lowdimensional manifolds of rank-deficient gradients as an alternative approach to the lora. in this work, we address the peft problem for llms in computationally constrained settings such as for mobile and edge devices. therefore, we pose the problem of peft for llms using frozen w 0 different from that investigated by the id. similar to the id, we aim to update low-dimensional approximation of original parameters instead of w 0 . more precisely, we aim to learn a low-dimensional representation of newly introduced data during finetuning, which is then integrated to w 0 for inference. following this constraint, we identify a and b parameters by construction as in the ic instead of decomposing w 0 . thereby, we consider their approximation ba as a representation learned during fine-tuning, and we aim to learn these representations efficiently. for this purpose, we first study the following question (q1): q1: how should we identify the structure of a and b for peft of llms? learning to efficiently fine-tune: to elucidate this question, we introduce a new approach named learning to efficiently fine-tune (left). in the left, similar to the id, we consider studying the characteristics of ba on low-dimensional parameter manifolds. however, we aim to learn these manifolds by parameter generation using generative models, such as hypernetworks. in other words, we target learning to generate peft parameters residing on low-dimensional parameter manifolds for learning new representations unlike id and ic as depicted in figure 1 . to this end, we propose a method called parameter generation (pg) to learn how to generate peft parameters in left. @xcite showed that a hypernetwork can learn to generate a distribution of parameters of a deep neural network on a non-trivial manifold, and the hypernetwork can be considered as the coordinate map of this low-dimensional manifold @xcite . therefore, during fine-tuning, we train hypernetworks to learn how to generate the peft parameters on low-dimensional manifolds. similar to the ic, we explore random structures of matrices identifying a. our pg enables us to control how to utilize different random and deterministic structures for identifying a and b by parameter generation. in the experiments, we consider several configurations where the parameters are randomly generated (as in the related work), optimized (as utilized by lora), and generated by conditioning on network properties (e.g. layers and embeddings). however, designing peft parameters according to network properties is another understudied open problem as addressed next. shareability of peft parameters among the layers of llms: existing works applying peft parameters to llms, including the studies focusing on the role of a with random projections @xcite , follow the standard practice of defining separate matrices for each target layer. such applications do not explicitly consider the hierarchy of the layers and their roles, which are investigated in various studies. for instance, @xcite show that basic lexical details such as suffixation or word structure are concentrated at the lower layers of a pre-trained model, whereas non-local dependencies are primarily captured at the higher layers. this may also be connected to why "easy" tasks activate the neurons of llms at shallower layers while "hard" ones at deeper layers, as inspected by @xcite . motivated by these observations, we explore the next follow-up question: q2: how does shareability and dependency of representations learned at different layers guide designing peft parameters and affect their accuracy? in order to investigate this question, we first analyze distribution of parameters trained by lora and that of parameters generated by our pg. then, we examine the accuracy of fine-tuned llms with parameters generated by sharing hypernetworks among different layers of the llms. our contributions can be summarized as follows: organization our paper is organized as follows. we introduce our parameter generation (pg) method in section 2, which is followed by our experimental analyses (section 3). section 4 concludes the paper. figure 2 presents an overview of our method where w l 0 denotes the fixed parameters at the l th layer of a pre-trained llm. the corresponding parameters w l a are obtained by passing the embeddings {e l , p l } ∈ e through the hypernetwork γ. finally, the function f combines these parameters to produce the final fine-tuned parameters w . thereby, the function f learns how to fine-tune the parameters by training the hypernetwork γ which generates fine-tuning parameters, and integrating the generated parameters with w 0 . in this work, we use our pg to generate a separate set of peft parameters for each layer of a pre-trained llm, as shown in figure 3 . unlike lora where the parameters a and b are trained during fine-tuning, we train a generative model to learn how to generate and integrate them by updating the ntf f , which is described next. ## experiments we give a description of the setup used throughout the experiments in section 3.1, which is followed by a motivating example for our approach (section 3.2). we then explore the questions q1 and q2 (sections 3.3 and 3.4, respectively), and present ablation studies to illustrate the properties of our proposed approach (section 3.5). ## conclusion parameter-efficient fine-tuning (peft) methods play a critical role in adapting large language models (llms) to new tasks. existing peft methods typically employ direct optimization on the peft parameters during fine-tuning. in this paper, we propose a new approach called learning to efficiently fine-tune (left) where the focus is on modelling the generative process of these parameters. to this end, we develop the parameter generation (pg) method where we utilize hypernetworks conditioned on the properties of a pre-trained llm. our experimental analyses on the glue benchmark confirm the effectiveness of pg. future work includes its application to text-generation tasks and its extension to data-dependent model adaptation.