L o RASC : Expressive and Generalizable Low-rank Adaptation for Large Models via Slow Cascaded Learning (2024)

Abstract: Efficient fine-tuning plays a fundamental role in modern large models, with low-rank adaptation emerging as a particularly promising approach. However, the existing variants of LoRA are hampered by limited expressiveness, a tendency to overfit, and sensitivity to hyperparameter settings. This paper presents LoRA Slow Cascade Learning (LoRASC), an innovative technique designed to enhance LoRA’s expressiveness and generalization capabilities while preserving its training efficiency. Our approach augments expressiveness through a cascaded learning strategy that enables a mixture-of-low-rank adaptation, thereby increasing the model’s ability to capture complex patterns. Additionally, we introduce a slow-fast update mechanism and cascading noisy tuning to bolster generalization. The extensive experiments on various language and vision datasets, as well as robustness benchmarks, demonstrate that the proposed method not only significantly outperforms existing baselines, but also mitigates overfitting, enhances model stability, and improves OOD robustness.

## introduction foundation models, which are large-scale models pre-trained on extensive datasets and subsequently adapted for specific downstream tasks, have become integral to contemporary machine learning frameworks. fine-tuning these models is essential, yet full parameter fine-tuning often encounters significant memory and computational bottlenecks. as a result, parameter-efficient fine-tuning (peft) techniques, which aim to minimize the number of trainable parameters to reduce training costs and improve training stability, have gained increasing prominence. among these techniques, low-rank adaptation (lora) @xcite stands out due to its efficiency in reducing training costs through low-rank approximation for fullparameter updates. however, despite lora's advantages, its limitations in terms of expressiveness and generalization have been noted. some studies suggest that the inherent low-rankness of lora might restrict its expressiveness @xcite @xcite , with a preference for overparameterization, while others indicate a tendency for lora to overfit or exhibit overconfidence @xcite . in this work, we investigate the potential of cascading learning to augment the expressiveness of lora. our approach involves initializing a new lora module at the start of each epoch and integrating this module into the backbone network after the epoch concludes. by employing a mixtureof-low-rank adaptation, we effectively increase the rank of the update matrices, while maintaining low training costs, as each cascading step consumes no more parameters and memory than a single lora model. moreover, this method does not add any inference overhead by remerging each lora module into the backbone network. to improve lora's generalization capabilities, we draw inspiration from optimization techniques. we repurpose certain strategies from optimizers for lora, motivated by the observation that initializing a new lora module for each epoch can represent a descent direction for the dataset. in optimization theory, flat minimizers are preferred, as they are associated with better generalization @xcite . inspired by the fact that the moving average mechanism guides models towards flat minimizers @xcite , we maintain both fast-updating and its moving average version, the slow-updating lora experts. the fast-updating expert is reinitialized regularly to learn from the data over a set number of steps, while the slow-updating expert undergoes updates via a proportional exponential moving average after the fast-updating cycle completes. additionally, mirroring techniques in deep learning optimizers where noise proportional to the gradient scale is used to find flat minima @xcite , we introduce noise at the beginning of each epoch, with the scale tied to the norm of lora's weights. to verify the effectiveness of the proposed method, we conduct extensive experiments on both language and vision tasks. for language tasks, we utilized the llama2 model @xcite on 12 datasets (e.g., superglue, squad, drop, gsm8k, and instructeval), alpaca among other instruct following benchmarks to demonstrate the effectiveness of our design. we can directly apply our approach to lora, lora+ @xcite , dora @xcite , and other members of the lora family, significantly improving their performance in large model transfer learning. for vision tasks, we also validated our approach on the clip pre-trained vit-bigg model with the imagenet dataset, showing a significant performance improvement relative to lora on domain adaptation datasets such as image-r and image-c. the proposed method consistently outperforms the baselines by a large margin. ## experiments we conducted extensive experiments to demonstrate the effectiveness and robustness of lorasc across both nlp and cv domains. for language tasks, we conducted our language experiments using the popular open-source large language model, llama2 foot_0 . we evaluated our approach on several nlu and glu tasks, selecting algorithm 1 pseudo code for lorasc require: pre-trained model weights w 0 , number of epochs t , loss function l, slow update parameter α, noise parameter λ 1: initialize w ← w 0 2: initialize a slow , b slow ▷ initialize slow lora matrices 3: initialize a fast ← a slow , b fast ← b slow ▷ fast lora matrices initialized from slow ones 4: for epoch @xmath0: if @xmath1: reinitialize a fast , b fast ▷ reinitialize fast lora matrices for subsequent epochs 7: end if 8: w ← w + u -λ 2 , λ 2 • std(b slow a slow ) 9: optimizer ← initializeoptimizer(a fast , b fast ) 10: lr_scheduler ← initializelrscheduler(optimizer) 11: for batch in training data do 12: forward pass: l ← l( w + b fast a fast ) 13: backward pass: compute gradients 14: optimizer.step() 15: lr_scheduler.step() 16: end for 17: update slow lora: 18: merge slow lora into main network: w ← w + b slow a slow 21: end for 22: return w both superglue @xcite tasks (including classification and multiple-choice ) and generation tasks. we also tested the model's performance in mathematical reasoning using the gsm8k dataset @xcite . additionally, we performed instruction tuning experiments to verify the transfer learning capability of our method, achieving significant improvements on key metrics such as mmlu @xcite , drop @xcite , bbh @xcite and humaneval @xcite . for visual tasks, we chose the clip vit-bigg/14 @xcite foot_1 as our pretrained model, fine-tuning it on the imagenet-1k @xcite training set and testing it on the validation set. subsequently, we evaluated the trained model on perturbed datasets such as imagenet-a @xcite , imagenet-c (hendrycks and dietterich, 2019), imagenet-r @xcite , imagenet-v2 @xcite , imagenet-sketch @xcite and stylized-imagenet @xcite demonstrating our method's robustness and generalization capabilities. ## conclusion in this paper, we address the limitations of finetuning large pre-trained models, particularly the issue of overfitting and the high computational costs associated with transferring these models to niche tasks. we introduce lorasc, an extension to the lora technique, which incorporates cascading learning, slow-fast updates, and noisy tuning to improve model expressiveness and generalization. we provide a detailed analysis of lorasc and demonstrate its effectiveness through extensive experiments in both the natural language processing (nlp) and computer vision (cv) domains. our method consistently outperforms baseline lora models and their variants (lora+, dora) across multiple datasets and tasks, including superglue, squad, drop, gsm8k, and various instructionfollowing benchmarks. additionally, our method enhances the robustness and transferability of vision models on imagenet and several robustness benchmarks.