L o RA Meets Dropout under a Unified Framework (2024)

Abstract: With the remarkable capabilities, large language models (LLMs) have emergedas essential elements in numerous NLP applications, while parameter-efficientfinetuning, especially LoRA, has gained popularity as a lightweight approachfor model customization. Meanwhile, various dropout methods, initially designedfor full finetuning with all the parameters updated, alleviates overfittingassociated with excessive parameter redundancy. Hence, a possible contradictionarises from negligible trainable parameters of LoRA and the effectiveness ofprevious dropout methods, which has been largely overlooked. To fill this gap,we first confirm that parameter-efficient LoRA is also overfitting-prone. Wethen revisit transformer-specific dropout methods, and establish theirequivalence and distinctions mathematically and empirically. Building upon thiscomparative analysis, we introduce a unified framework for a comprehensiveinvestigation, which instantiates these methods based on dropping position,structural pattern and compensation measure. Through this framework, we revealthe new preferences and performance comparisons of them when involved withlimited trainable parameters. This framework also allows us to amalgamate themost favorable aspects into a novel dropout method named HiddenKey. Extensiveexperiments verify the remarkable superiority and sufficiency of HiddenKeyacross multiple models and tasks, which highlights it as the preferred approachfor high-performance and parameter-efficient finetuning of LLMs.

## introduction recently, transformers @xcite , such as @xcite , palm 2 @xcite and llama 2 @xcite , have been rapidly expanded to billions of parameters, leading to remarkable performance boost. when customizing these models for downstream tasks, * equal contribution. in parallel with this, dropout @xcite ) has been widely adopted to mitigate overfitting, which is generally caused by excessive parameter redundancy. its variants, including dropkey @xcite , dropattention @xcite and hiddencut @xcite , have also demonstrated superiority for transformers. with a specified probability, they randomly deactivate attention logits, weights and hidden representations, respectively. however, the effectiveness of these methods is only verified in full finetuning scenarios, where all the parameters are updated and easily lead to excessive redundancy. when it comes to lora-based peft scenarios, a potential contradiction arises. specifically, since overfitting primarily stems from excessive parameter redundancy, dropout may prove ineffective in lora-based finetuning because of the extremely limited trainable parameters. besides, all the above methods are proposed independently, lacking a clear guideline to unify them systematically, which hinders comprehensive comparative analysis and the development of more effective dropout methods. in this study, we first conduct extensive experiments and confirm that lora also suffers from overfitting easily, which serves as a prerequisite for our following analysis. as shown in figure 5 , as the rank and trainable parameters increase, the model's performance initially improves but gradually deteriorates due to the intensifying overfitting. much more experiments in sec. 4 provide further evidence and affirm that this overfitting susceptibility can be improved with dropout methods. besides, we compare the above transformer-specific dropout methods mathematically and empirically. for the first time, we find that dropkey and dropattention share the equivalent forwarding process, while the gradient stopping operator introduces gradient noise into the backpropagation of dropattention, impairing the training stability and performance. based on the comparative analysis, we identify three key dimensions for a dropout method and derive a unified framework along dropping position, structural pattern and compensation measure. with this framework, empirical experiments firstly reveal the new preferences of these methods in lora scenarios. for example, span-wise hidden-cut is no longer superior to the element-wise one due to the limited tunable parameters. secondly, this framework enables the comprehensive comparisons among different methods. empirically, we find that dropkey performs the best followed by hiddencut, while dropattention exhibits the worst performance due to the gradient noise. as an alternative compensation, bidirectional kullback-leibler (kl) divergence loss consistently achieves performance gains, while jensen-shannon (js) consistency regularization loss becomes ineffective. guided by this framework, we also derive a new dropout method named hiddenkey, which drops attention logits column-wisely and hidden representations element-wisely, respectively, and augment the vanilla loss with kl loss. it consistently exhibits superiority across multiple models in both natural language understanding (nlu) and generation (nlg) tasks, which also fills the largely overlooked gap on the effect of dropout methods on nlg tasks. integrating with input and output dropout does not provide further complementarity, demonstrating the sufficiency of our method. hence, hiddenkey excels as the better method for high-performance and parameter-efficient finetuning of llms on both nlu and nlg tasks. in summary, our contributions are mainly as follows: ## preliminaries in this section, we revisit three transformer-specific dropout methods shown in figure 1 , laying the foundation for the subsequent analysis. dropattention. dropattention @xcite is the first dropout method specially designed for self-attention mechanism. it randomly masks elements or key columns of attention weights, encouraging the utilization of multiple contextualized features instead of overfitting some specific patterns. following eq. 1 and 2, normalized rescaling replaces the traditional one to guarantee the sum of attention weights to be one, and achieves better performance for multiple nlp classification tasks. ## method firstly, we conduct a comparative analysis of the above methods. based on their similarity and differences, we then propose a unified framework along dropping position, structural pattern and compensation measure. finally, this framework guides us to derive a new dropout method named hiddenkey, which exhibits superior performance empirically. ## conclusion we investigate the possible contradiction between the limited trainable parameters of lora and overfitting associated excessive parameter redundancy. after confirming the overfitting-prone property of lora, we analyze existing dropout methods theoretically and empirically, and further introduce a unified framework for thorough comparison. this also guides us to derive a new dropout method, hid-denkey. with its superiority and sufficiency across multiple models and datasets, hiddenkey deserves to be the recommended dropout method to alleviate overfitting in lora-based scenarios.