Parameter-efficient Tuning for Large Language Model without Calculating Its Gradients (2023)

Abstract: Fine-tuning all parameters of large language models (LLMs) requires significant computational resources and is time-consuming. Recent parameter-efficient tuning methods such as Adapter tuning, Prefix tuning, and LoRA allow for updating a small subset of parameters in large language models. However, they can only save approximately 30% of the training memory requirements, due to the problem that gradient computation and backpropagation are still necessary for these methods. This paper proposes a novel parameter-efficient tuning method for LLMs without calculating their gradients. Leveraging the discernible similarities between the parameter-efficient modules of the same task learned by both large and small language models, we put forward a strategy for transferring the parameter-efficient modules, originally derived from small language models to much larger ones. To ensure a smooth and effective adaptation process, we further introduce a Bridge model to guarantee dimensional consistency while also stimulating a dynamic interaction between the models. We demonstrate the effectiveness of our method using the T5 and GPT-2 series of language models on the SuperGLUE benchmark. Our method achieves comparable performance to both fine-tuning and parameter-efficient tuning on large language models without needing gradient-based optimization. Additionally, our method achieves up to 5.7x memory reduction compared to parameter-efficient tuning.

## introduction large language models such as gpt3 @xcite , gpt4 (openai, 2023), t5-xxl @xcite , and llama @xcite have demonstrated remarkable capabilities in various natural language processing tasks. however, the sheer number of parameters in these models poses challenges for fine-tuning on common hardware. parameter-efficient tuning methods (e.g., adapter tuning, prefix tuning, or lora) typically involve adding a small number of parameters to the language model and only fine-tuning the added subset of parameters, achieving comparable performance to full fine-tuning. adapter tuning learns the task-specific information @xcite by inserting small taskspecific modules within layers of the transformer. prefix tuning @xcite prepends task-specific trainable prompt tokens to the hidden states within every intermediate transformer layer. lora @xcite merges the low-rank and trainable matrices with the frozen weights at each layer of the transformer. in figure 1 (a), we show the comparison of the gpu memory usage during training for fine-tuning, adapter tuning, lora, and prefix tuning on the recognizing textual entailment (rte) @xcite task. these parameter-efficient tuning methods can save approximately 30% of the gpu memory requirements but still rely on gradient-based optimization, resulting in increased memory demands for llms. to address the above limitations, we propose a novel parameter-efficient tuning method for llms without calculating their gradients. intuitively, both large and small language models (slms) can learn similar task-specific characteristics when applied to downstream tasks. we conduct experiments using existing parameter-efficient tuning methods on the rte task in the superglue benchmark @xcite to validate this hypothesis. as depicted in figure 1 (b), we calculate the similarity between the parameter-efficient tuning modules derived from gpt2-xl and gpt2-base or gpt2-medium foot_0 . specifically, we apply the lora method on gpt2-xl and gpt2-medium to obtain the parameter-efficient modules of the two language models and calculate the cosine similarity of the two parameter-efficient modules. we find that the cosine similarity can reach up to 75%. our observations indicate that these modules exhibit comparable task-specific characteristics throughout the learning process for specific downstream tasks. inspired by these findings, if we can successfully transfer the task-specific characteristics learned by the small language model to the large language model, we can enrich the task-specific capabilities into the llms without needing gradient-based optimization. in this paper, we first utilize existing parameterefficient tuning methods in a small language model to learn the task characteristics of downstream tasks. intuitively, we can directly apply the parameter-efficient module obtained from the small language model to the large language model. however, this would face crucial issues of dimension mismatch and limited interaction with the large language model. to address the issue of dimension mismatch, we employ a projection module to align the dimensions of the parameter-efficient modules between slms and llms. furthermore, to enrich the interaction between the parameterefficient module and the large language model, we introduce a bridge model that can retain the knowledge of the large language model while interacting with the parameter-efficient module, obtaining a parameter-efficient module with dimensions matching the large language model. finally, we seamlessly plug the acquired parameter-efficient module into the large language model for inference. we conduct comprehensive experiments on t5 series @xcite and gpt2 series @xcite of language models to assess the effectiveness of our method using the superglue benchmark, a widely recognized evaluation benchmark for natural language understanding. the results demonstrate that our method performs on par with fine-tuning and parameter-efficient tuning on large language models without needing gradient-based optimization. additionally, our proposed method achieves up to 5.7× memory reduction compared to parameter-efficient tuning. our findings highlight the potential of bridging small and large language models, thereby efficiently leveraging expansive large language models. in summary, our key contributions can be listed as follows: ## method the proposed method utilizes parameter-efficient tuning modules to effectively bridge small and large language models, enriching the task-specific capabilities into the large language model without needing gradient-based optimization. as depicted in figure 2 , the method consists of training and inference. we first employ parameter-efficient tuning methods during the training stage to learn task-specific characteristics in the small language model. then, we fine-tune the bridge model and the acquired parameter-efficient module to enhance the knowledge of the parameter-efficient module. finally, we directly plug the parameter-efficient module into the large language model during the inference stage for efficient predictions. plug-in and bridge model fine-tuning: we first utilize existing parameter-efficient tuning methods such as adapter @xcite , lora (hu et al., 2021), or prefix tuning (li and liang, 2021) in a small language model to learn the task characteristics of downstream tasks. however, directly applying the parameter-efficient modules obtained from the small language model to the large language model would face two issues: dimension mismatch and limited interaction with the large language model. to address the issue of dimension mismatch, we employ a linear projection module w plug as the plug-in model to align the dimensions of the parameter-efficient module with the large language model. furthermore, considering that small language models usually have fewer layers than large language models, we address the layer mismatch by duplicating the layers of the parameter-efficient modules. this duplication enables us to achieve layer alignment with the large language model. intuitively, successfully adapting the parameterefficient modules to the large language models requires a substantial interaction between them and the large language models. to enrich the interaction, we introduce a bridge model that can retain the knowledge of the large language model while interacting with the parameter-efficient module. we employ the pruning method from ladderside-tuning @xcite to obtain such a bridge model, which involves pruning each layer of the large language model f . we use linear projections to downsample the intermediate activa-tions, including word embeddings, from the large language model f to a lower-dimensional bridge model g, with a reduction factor of r, where r can be 8, 16, 32, 64, etc. to retain crucial information from the large language model, we leverage fisher information @xcite to prune the parameters of the large language model and obtain the initial bridge model g. fisher information could effectively evaluate the importance of parameters in the large language model. given the @xmath0, we calculate the importance of each weight vector through subsequently, in order to make the parameterefficient modules (e.g., adapter, lora, or prefix tuning) learned in the small language model interact with the bridge model, we apply the linear ## related work fine-tuning and parameter-efficient tuning: large language models leverage parameterized transformers as a foundational framework and train them on extensive unsupervised language corpora @xcite @xcite . subsequently, specific task objectives are introduced for downstream tasks to perform full fine-tuning on the pretrained language models @xcite @xcite . as the primary method for optimizing pre-trained language models, full fine-tuning involves initializing the model with pretrained weights, updating all model parameters, and storing a separate fully optimized model for each downstream task. however, as the model parameter grows, conducting full fine-tuning on existing computational devices becomes increasingly challenging. in light of this, researchers have started exploring efficient methods for effectively harnessing the power of large language models. as an efficient alternative, parameter-efficient tuning is a promising way of stimulating large language models. compared to vanilla fine-tuning, parameter-efficient tuning methods only tune a small portion of the model parameters while keeping the rest frozen. adapter tuning methods, such as those proposed by @xcite and @xcite , aim to learn taskspecific information by incorporating small-scale task-specific modules into the layers of the transformer. prefix tuning methods, as introduced by @xcite and discussed in the work by @xcite , also introduce additional parameters within the transformer layers. lora, proposed by @xcite , merges low-rank and trainable matrices with the frozen weights at each layer of the transformer. bitfit @xcite is a simple yet effective method that optimizes the bias terms within a model while keeping the other parameters frozen. however, these existing parameter-efficient tuning methods typically rely on gradient-based optimization and still involve substantial memory usage. our proposed method, on the other hand, enables substantial memory savings while maintaining comparable performance. gradient-free optimization: recently, @xcite introduced a method that eliminates the need for gradient updates by directly applying a pruned model to downstream tasks, while the method does not fully exploit the knowledge of the large language model. @xcite propose an efficient transfer learning framework that can adapt large language models to downstream tasks without access to full model parameters, while the method through compute-intensive distillation techniques may be cost-prohibitive for larger models. in contrast, our proposed method enables further utilization of the knowledge contained in the large language model and enables significant memory savings through a simple operation while preserving the performance of large language models. ## conclusion this paper proposes a novel parameter-efficient tuning method for large language models without calculating their gradients. we first learn the parameter-efficient tuning module for small language models. then, the learned parameterefficient tuning module is adapted into large language models with a bridge model that handles the dimensionality mismatch and enables interaction between the parameter-efficient tuning module and the large language model. extensive experiments on the superglue benchmark demonstrate that our method achieves comparable performance to vanilla fine-tuning and parameter-efficient tuning on large language models without needing gradient-based optimization. we believe that our method offers a potential direction to utilize large language models efficiently and economically.