From Bottom to Top: Extending the Potential of Parameter Efficient Fine-Tuning (2024)

Abstract: With the proliferation of large language models, Parameter Efficient Fine-Tuning (PEFT) method, which freeze pre-trained parameters and only fine-tune a few task-specific parameters, are playing an increasingly important role. However, previous work primarily applied uniform operations across all layers of the model, overlooking the fact that different layers in a transformer store different information. In the process of exploration, We find that there is a significant differences in fine-tuning strategies between different layers, and fine-tuning only a subset of layers can even achieve comparable performance. Based on this, we propose the Hybrid LoRA-Prefix Tuning(HLPT) method, which uses enhanced LoRA and Prefix-tuning methods with learnable adaptive mechanism separately for the bottom and top layers, and the Half Hybrid LoRA-Prefix Tuning( H^2 LPT) method, which goes a step further, reducing the parameter count to nearly half by omitting fine-tuning in the middle layers. Extensive experiments with large language models on various downstream tasks provide strong evidence for the potential of PEFT focusing on different layers’ interactions and the effectiveness of our methods. Furthermore, we validate the robustness of these methods and their advantages in speeding up training convergence, reducing inference time requirements.

## introduction with the success of large language models like gpt-3 @xcite and ernie3.0 @xcite , large language models (llms) have demonstrated remarkable generative capabilities, enabling them to handle various downstream tasks. moreover, it has spurred the development of numerous large language models such as gpt-j @xcite and llama @xcite . however, this also brings forth a challenge * ping gong is the corresponding author. to address this challenge, the peft @xcite @xcite ) method, as an emerging fine-tuning approach, often adds a subset of parameters to the model to tune while freezing the original pretrained model parameters. significantly reducing the number of parameters needed for fine-tuning while achieving comparable results on downstream tasks, thus greatly reducing resource consumption during training. in previous works, the different information storage between layers was neglected while most of works applied the same method uniformly across all layers of the model. however, prior researches have indicated the distinct effect of different layers. @xcite @xcite . it is generally believed that lower layers contain more specific sentence-level information, middle layers focus on capturing syntactic information and higher layers store more abstract semantic information. there has been a lack of analysis and utilization concerning the distinct roles played by different layers. based on the perspective, we assume that by effectively combining various peft methods according to the distinct roles of different layers, we can achieve a more efficient peft approach, and conduct extensive experiments using llms with zero-shot learning to validate our ideas and the increasingly important generation capability of our methods in contrast to previous methods focusing on understanding tasks. inspired by the work @xcite , we focuses on two efficient methods: lora, which operates in parallel mechanisms and utilizes matrix rank, and prefix, which adds virtual heads before multi-head attention. as shown in figure1, in our figure 1 : this is an analysis of the experimental results regarding the effects of different layers. we divide the llama-7b model into two equal parts based on the number of layers, where "lora+prefix" represents placing lora at the bottom layers and prefix at the top layers, and "prefix+lora" represents the opposite. on the left side is a comparison of the loss during convergence for the two training methods, with the x-axis representing the number of epochs. on the right side is a comparison of the responses generated by the two trained models for the same question. based on the survey mentioned above, we have introduced two methods: hybrid lora-prefix tuning (hlpt), which effectively combines the enhanced lora and prefix methods operating on both the top and bottom layers, and half hybrid lora-prefix tuning (h 2 lpt), which further reduces the number of parameters by half based on hlpt excluding the operations on the middle layers, to help the model adapt more effectively to various downstream tasks and unleash the potential of peft. we conduct lots of experiments on a total of six datasets for mathematical reasoning tasks using large language models, namely llama-7b, llama-13b, and gpt-j (6b) @xcite . the accuracy can be even significantly improved while reducing the number of parameters by up to 71 points compared to the original lora and prefix methods in the main experiment. these experiments highlight the powerful potential of tuning each layer differently based on interactions between different layers to reduce model resource consumption and enhance performance, affirming the effectiveness of the methods presented in this paper. in summary, our main contributions can be outlined as follows: ## background and related works as the model size increases, the significance of peft methods becomes increasingly apparent, leading to the development of several classical and widely-used peft techniques. these methods predominantly involve freezing the original pretrained parameters and fine-tuning only the newly introduced parameters. among these, "adapter tuning" adds adjustable modules to the multi-head attention and feed-forward layers of the transformer model. "prefix tuning" (li and liang, 2021) introduces learnable parameter vectors in front of the input "k" and "v" of each multi-head attention layer in the transformer, expanding the dimensions of "k" and "v." "lora" @xcite incorporates learnable parallel models alongside the multi-head attention layers. on the other hand, many methods have attempted to reduce the number of transformer layers in the model @xcite . subsequently, the adapterdrop @xcite ) method was introduced, which builds upon the peft approach and investigates the impact of removing fine-tuning in the lower layers of the model on both speed and performance. in addition, there are also some studies exploring the prompt tuning methods and the function of neurons in different layers of pretrained transformer models @xcite . building on the excellent peft approach mentioned above, many studies have attempted to integrate existing methods together. the "unipelt" @xcite ) method combines all these methods into a single transformer model by introducing different gating mechanisms for each model, achieving a unified model overall. the "sparse" @xcite explore constructing delta-tuning modules in an automatic manner. "unified view" @xcite employs mathematical analysis to demonstrate the commonalities among adapter, lora, and prefix models using a set of similar mathematical formulas. it finds that multi-head attention achieves better results with fewer parameters, parallel structures outperform serial structures, and adding parallel models to the feed-forward layers yields superior results. ultimately, it proposes a prefix-tuning with parallel adapter structure across the entire model, highlighting the effectiveness of multi-head attention's prefix and parallel lora in the feed-forward layers. this also guides the use of lora and prefix in this paper to adapt to the different roles of various layers. compared to these methods above, we believe our contribution lies in exploring the interactions between different layers aligned with peft methods, clearly delineating the hlpt method and the h 2 lpt method that omits intermediate layer finetuning. our method can improve the effectiveness as reduce resource consumption. at the same time, we primarily verify the increasingly important generation capability of the method in the era of llms. ## analysis of layers to further explore the differences in the combination of the two methods across different layers following the findings in figure 1 , we conduct experimental analysis in this chapter. ## conclusion in this paper, we have explored different layerspecific fine-tuning methods corresponding to the varying information stored at different layers of the model and the function of fine-tuning only a subset of layers, further pushing the boundaries of the peft approach. we conducted extensive experiments to analyze various allocation methods. in the process, we introduced the hlpt method, which involves improved lora in the majority of lower layers and enhanced prefix in the minority of top layers. building upon this, we propose the h 2 lpt method, which omits fine-tuning in the intermediate layers. a large number of experiments have all demonstrated the effectiveness and robustness of our proposed methods on llms. these methods can even achieve significantly better performance in terms of generation capability than the baseline with fewer parameters and inference time while they can smooth training convergence. all experiments highlight the enormous potential of utilizing different fine-tuning methods for layers with varying information storage.