Light- PEFT : Lightening Parameter-Efficient Fine-Tuning via Early Pruning (2024)

Abstract: Parameter-efficient fine-tuning (PEFT) has emerged as the predominant technique for fine-tuning in the era of large language models. However, existing PEFT methods still have inadequate training efficiency. Firstly, the utilization of large-scale foundation models during the training process is excessively redundant for certain fine-tuning tasks. Secondly, as the model size increases, the growth in trainable parameters of empirically added PEFT modules becomes non-negligible and redundant, leading to inefficiency. To achieve task-specific efficient fine-tuning, we propose the Light-PEFT framework, which includes two methods: Masked Early Pruning of the Foundation Model and Multi-Granularity Early Pruning of PEFT. The Light-PEFT framework allows for the simultaneous estimation of redundant parameters in both the foundation model and PEFT modules during the early stage of training. These parameters can then be pruned for more efficient fine-tuning. We validate our approach on GLUE, SuperGLUE, QA tasks, and various models. With Light-PEFT, parameters of the foundation model can be pruned by up to over 40%, while still controlling trainable parameters to be only 25% of the original PEFT method. Compared to utilizing the PEFT method directly, Light-PEFT achieves training and inference speedup, reduces memory usage, and maintains comparable performance and the plug-and-play feature of PEFT.

## introduction large-scale pre-trained language models have demonstrated outstanding performance in various natural language processing domains @xcite @xcite . along with the performance improvements, the scale of model parameters contin-ues to grow, making the cost of fine-tuning models increasingly expensive. moreover, the practice of maintaining a separate copy of the large model for each task in conventional fine-tuning incurs substantial storage costs. to address these challenges, parameter-efficient fine-tuning (peft) has been proposed: freezing most parameters of the foundation model and finetuning only a small number of parameters @xcite @xcite , thereby reducing the computational resource requirements during training and performing nearly full-parameter fine-tuning. in addition, this technique eliminates the need to save an entire model copy for each task. during inference, task-specific models can be obtained by switching directly to the appropriate parameter-efficient module for the given task. however, the training efficiency of existing peft methods still needs improvement. the first problem lies in the excessive redundancy of using a large-scale foundation model during fine-tuning for specific tasks, which results in substantial computational costs. a typical strategy is to integrate peft with quantization @xcite . nonetheless, these methods only quantize parameters to low-bit in memory, without reducing the number of parameters and they still need to be dequantized to high-bit during training, leading to wasted training time. another more direct approach for reducing parameters is model structured pruning @xcite . however, most methods mainly focus on the inference efficiency of the model, which means they may result in higher training costs. the second problem is that as the size of the foundation model increases, the number of parameters in added trainable modules also increases significantly. this introduces a lot of redundancy in trainable parameters, leading to inefficiency in finetuning. for instance, the commonly used methods lora @xcite and qlora @xcite empirically insert the low-rank modules onto fixed weight. however, there is no need to uniformly add trainable modules of the same rank to all weights for fine-tuning each task. an improved approach is the dynamic rank method @xcite @xcite , which adaptively allocates module parameters by progressively calculating the importance of the rank during training. however, these methods require continuous estimation during training and show limited improvement in actual training efficiency. in this paper, we introduce a novel framework named light-peft, which aims to enhance the efficiency of the peft technique during fine-tuning. the framework consists of two methods: masked early pruning of foundation model and multi-granularity early pruning of peft. in the early training stage, light-peft estimates redundant parameters in both the foundation model (heads and intermediate dimensions) and the peft modules (module importance and rank importance) simultaneously. structured pruning is used to eliminate this redundancy, resulting in a lighter foundation model and peft module for more efficient finetuning. to validate the effectiveness of our light-peft framework, we conduct extensive evaluations on various foundation models (roberta, opt-1.3b, opt-6.7b), different peft structures (lora, adapter), and on diverse benchmarks (glue, su-perglue, and question-answering tasks). the empirical results indicate that the proposed light-peft framework outperforms other baseline methods in performance. it significantly improves training efficiency that reduces training memory usage by 39% and accelerates training to 1.6×. additionally, the light-peft framework improves inference efficiency that reduces inference memory by 48% and increases inference speed to 1.6×. ## conclusion this paper introduces light-peft, a novel framework designed to improve the efficiency of the peft technique during fine-tuning. the framework comprises two methods: masked early pruning of foundation model and multi-granularity early pruning of peft. the light-peft framework estimates redundant parameters in both the foundation model and peft modules during the early stage of training and prunes them to achieve more efficient fine-tuning. we validate our approach on glue, superglue, and qa tasks using various models. the experiments demonstrate that light-peft achieves training and inference speedup, reduces memory usage, and maintains comparable performance.