L a MDA : Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation (2024)

Abstract: Low-rank adaptation (LoRA) has become the default approach to fine-tune large language models (LLMs) due to its significant reduction in trainable parameters. However, trainable parameter demand for LoRA increases with increasing model embedding dimensions, leading to high compute costs. Additionally, its backward updates require storing high-dimensional intermediate activations and optimizer states, demanding high peak GPU memory. In this paper, we introduce _LaMDA_, a novel approach to fine-tuning large language models, which leverages low-dimensional adaptation to achieve significant reductions in trainable parameters and peak GPU memory footprint. LaMDA freezes a first projection matrix (PMA) in the adaptation path while introducing a low-dimensional trainable square matrix, resulting in substantial reductions in trainable parameters and peak GPU memory usage. LaMDA gradually freezes a second projection matrix (PMB) during the early fine-tuning stages, reducing the compute cost associated with weight updates to enhance parameter efficiency further.We also present an enhancement, LaMDA++, incorporating a “lite-weight” adaptive rank allocation for the LoRA path via normalized spectrum analysis of pre-trained model weights. We evaluate LaMDA/LaMDA++ across various tasks, including natural language understanding with the GLUE benchmark, text summarization, natural language generation, and complex reasoning on different LLMs.Results show that LaMDA matches or surpasses the performance of existing alternatives while requiring up to **17.7 \times ** fewer parameter updates and up to **1.32 \times ** lower peak GPU memory usage during fine-tuning. Code will be publicly available at https://github.com/ArminAzizi98/LaMDA.

## introduction large language models (llms) have demonstrated remarkable performance in addressing a variety of natural language processing (nlp) tasks due to their generalization ability upon training on large corpus of data @xcite . to fully harness the capabilities of llms, fine-tuning has become the standard approach to serve various downstream tasks. however, full finetuning of llms can be prohibitively costly, making fine-tuning at the edge hardly possible. for example, even the smaller variants of llms with 7b parameters may ask for ∼ 60 gb memory to perform full fine-tuning @xcite . additionally, such approach is prone to causing overfitting and catastrophic forgetting in the over-parameterized regime @xcite . as a solution to these challenges, parameterefficient fine-tuning (peft) techniques were proposed in which either a small portion of model parameters are updated, including the weight adapters @xcite , or taskspecific soft prompts are trained @xcite . among these, low-rank adaptation (lora) @xcite has gained significant popularity. it assumes that the changes in the pre-trained weight reside in a low-rank space and thus adds two trainable low-rank adapters, named the projection matrix a a a (pma) and the projection matrix b b b (pmb) as ba ba ba in parallel to the frozen main path of the model weight w w w (refer to fig. 1(a) ). lora fine-tuning can reduce gpu memory demand and trainable parameter-count since it only fine-tunes the ba ba ba which is much smaller in parameter count compared to w w w . however, the number of trainable-parameters in lora may still be potentially larger than the low intrinsic dimension of a pre-trained llm @xcite . moreover, as evident in fig 1 (a), the input activations x x x that must be stored for backpropagation reside in a d-dimensional space, where d denotes the embedding dimension of the model. subsequently, the activation's gpu memory increases linearly with the embedding size, and lora does not provide any merit in activation memory saving. notably, few contemporary works @xcite have presented solutions of ba ba ba freezing. however, they still suffer from increased activation storage and often demand high rank to compensate for the accuracy drop due to freezing @xcite . to address these issues, in this work we present, large model fine-tuning via spectrally decomposed low-dimensional adaptation (lamda). lamda as demonstrated in fig. 1(c ) employs a trainable low-dimensional adapter (lda), which is a square matrix in the rdimensional space. we keep the pma frozen throughout the fine-tuning, while allow pmb to freeze gradually only after few epochs based on relative magnitude of the singular values. we only keep the lda trainable throughout. this allows the trainable parameters to be independent of d and the activations that are saved for backward pass remain in the r-dimensional space (r ≪ d). thus lamda can significantly reduce the trainable parameter, activation, and optimizer state memory. we summarize our contributions as follows: ## background transformer-based models. each module of an l layer transformer model @xcite usually consists of two sub-blocks: the multi-head selfattention (mhsa) sub-block and the feed-forward network (ffn). each mhsa takes input token embedding x x @xmath0: layernorm is the layer normalization module, and y is the output of the transformer block. low-rank adaptation. lora adds a trainable adaptation path (through the a a a and b b b matrix) parallel to the frozen main path of the respective module (i.e., w w w ). this results in a significant reduction in the number of trainable parameters, the optimizer's state memory, and the required gradient memory: lamda, on the contrary, offers two key benefits: 1) it significantly reduces trainable parameter, activation, and optimizer storage to enhances memory savings compared to lora; 2) it greatly reduces computational cost in the forward pass during finetuning. ## methodology this section provides a detailed explanation of lamda and lamda++ as novel parameterefficient fine-tuning methods. ## experiments this section evaluates lamda and lamda++ on nlu, nlg, and reasoning tasks. ## limitations this study has a few limitations. firstly, the largest model we tested was llama2-7b. due to time constraints associated with the paper's deadline, we could not extend our experiments to larger models, which could provide further insights into the scalability and effectiveness of lamda. our methodology, lamda, has not yet been tested on instruction-following tasks. while the current results are promising, evaluating the performance of lamda in these specific tasks is essential to fully understanding its potential and versatility. we plan to address these limitations in future work by conducting experiments on larger models and a broader range of tasks. we are also eager to test the applicability of our method to vision-language models, which was not explored in this paper.