PR o L o RA : Partial Rotation Empowers More Parameter-Efficient L o RA (2024)

Abstract: With the rapid scaling of large language models (LLMs), serving numerouslow-rank adaptations (LoRAs) concurrently has become increasingly impractical,leading to unaffordable costs and necessitating more parameter-efficientfinetuning methods. In this work, we introduce \text{\textbf{P}artially \textbf{Ro}tation-enhanced \textbf{Lo}w-\textbf{R}ank \textbf{A}daptation (PRoLoRA)} , an intra-layer sharing mechanism comprising fouressential components: broadcast reduction, rotation enhancement,partially-sharing refinement, and rectified initialization strategy. As asuperset of LoRA, PRoLoRA retains its advantages, and effectively circumventthe drawbacks of peer parameter-sharing methods with superior model capacity,practical feasibility, and broad applicability. Empirical experimentsdemonstrate the remarkably higher parameter efficiency of PRoLoRA in bothspecific parameter budget and performance target scenarios, and its scalabilityto larger LLMs. Notably, with one time less trainable parameters, PRoLoRA stilloutperforms LoRA on multiple instruction tuning datasets. Subsequently, anablation study is conducted to validate the necessity of individual componentsand highlight the superiority of PRoLoRA over three potential variants.Hopefully, the conspicuously higher parameter efficiency can establish PRoLoRAas a resource-friendly alternative to LoRA.

## introduction finetuning large language models (llms), such as llama 2 @xcite , gpt-3.5 turbo (openai, 2023), and gemini @xcite , for specific domains and functions (e.g., model alignment @xcite and instruction tuning @xcite ), have become increasingly popular. to alleviate the high costs associated with full finetuning, parameterefficient finetuning (peft), especially lora @xcite , has emerged as a lightweight solution by tuning a minority of parameters and freezing the remaining ones @xcite . however, with the rapid boost in the number of model's parameters, the demand for further enhanced parameter efficiency becomes progressively more imperative, especially when multiple lora are deployed simultaneously. as shown in the to circumvent all the above drawbacks, we introduce a new approach called partially rotationenhanced low-rank adaptation (prolora). it features a parameter-sharing mechanism within the low-rank decomposition matrices, and consists of four essential components: broadcast reduction, rotation enhancement, partially-sharing refinement, and rectified initialization strategy. specifically, we reparameterize the low-rank matrices with multiple chunks along the hidden dimension, and broadcast the first chunks to the others so that trainable parameters can be saved, or equivalently, the rank can be increased multiple times. then a nearly costfree rotation operation along the rank dimension is performed to differentiate the identical chunks for higher expressiveness. besides, a minimal subset of ranks is reserved without sharing for further refined capacity. to ensure the same bounds for initialization as unshared parameters, we also rectify the vanilla kaiming uniform distribution @xcite for shared ones. as a superset of lora, prolora not only pertains the advantages of lora, such as lightweight task switching and optional merging to eliminate extra latency, but also brings about better capacity, practical feasibility, and broader applicability than other parametersharing methods. empirical experiments on multiple instruction tuning datasets validate the higher parameter efficiency of prolora than baselines via two alternative perspectives (i.e., a specific trainable parameter budget and performance target). with half of tunable parameters, prolora achieves 4/6 wins and better average performance over lora. when scaling up to llama2-13b, prolora consistently outperforms lora with the same trainable parameter count. additionally, comprehensive ablation studies demonstrate the necessity of each component and the superiority of prolora over three potential intra-layer sharing variants. overall, prolora achieves significantly higher parameter efficiency, and thereby remarkably alleviates the storage and gpu memory burden in multi-lora scenarios, establishing prolora as a resource-friendly alternative for lora. in summary, our main contributions are as follows: lora series. inspired by the low intrinsic dimensions in over-parameterized models (aghajanyan et al., 2020), hu et al. (2021) proposes lora to reparameterize the weight update with two trainable low-rank matrices, while freezing the pretrained weights. with this lightweight decomposition, lora reduces storage and task-switching overhead by sharing the pretrained models across multiple tasks. theoretically, the linear design of lora enables seamless merging of trainable matrices with frozen weights, thereby avoiding extra inference latency, albeit this operation is typically not performed in multiple lora serving scenarios. subsequently, numerous efforts have been made to further enhance the effectiveness and efficiency of lora. based on singular value decomposition, adalora @xcite achieves the automatic rank allocation by adaptively pruning less important parameters during finetuning, but the varying ranks among layers pose challenges on deploying multiple loras. inspired by random projections @xcite , vera @xcite shares two frozen random matrices across all layers, and updates disentangled combination vectors for each layer. although this approach reduces the number of parameters, it results in performance degradation and several times the additional computation associated with a very high rank. in contrast, tied lora @xcite enhances parameter efficiency by sharing trainable lora matrices across all layers, with the down projection matrix further tied among the query, key, and value modules. additionally, it incorporates scaling vectors to differentiate each module. however, in addition to the high rank similar to vera, this approach also necessitates the shared matrices to have identical shapes, which further restricts its expansion to other linear layers. in contrast, our approach features an intra-layer sharing mechanism to enhance parameter efficiency, thereby circumventing the above drawbacks while exhibiting potential for integration with them. parameter sharing. parameter sharing has been adopted by prior studies to reduce model footprint. universal transformer @xcite proposes to share all layers within a transformer model. systematically, @xcite refines this mechanism with three parameter-sharing strategies across transformer layers, improving the computational and parameter efficiency. similarly, @xcite compares several parameter reduction methods and introduces the subformer model with shared middle layers and embedding factorization, significantly saving the parameters without performance degradation. subsequently, dict-former @xcite reparameterizes the original weights with a shared dictionary, unshared coefficients and indices, resulting in a more compact transformer model and faster computations. targeting on-device deployment, edgeformer @xcite shares the attention and ffn modules, and incorporates peft-based layer adaptation to minimize the number of parameters. more recently, pires et al. (2023) removes the ffn on the decoder layers and shares a single larger ffn across the encoder, achieving substantial gains in both accuracy and latency. differently, our study focuses on multi-lora scenarios and seeks to enhance the parameter efficiency of lora models instead of transformer models. ## method in this section, we introduce partially rotationenhanced low-rank adaptation (prolora), a more parameter-efficient method featuring an intralayer sharing mechanism. briefly, we present four essential components of prolora in section 3.1 namely broadcast reduction, rotation enhancement, partially-sharing refinement, and rectified initialization strategy, followed by its advantage analysis compared to existing peer methods in section 3.2. ## conclusion targeting more lightweight serving in multi-lora scenarios, we introduce prolora, a more efficient method featuring an intra-layer sharing mechanism consisting of broadcast reduction, rotation enhancement, partially-sharing refinement and rectified initialization strategy. empirically, we validate its higher parameter efficiency, scalability, and superiority over other methods, aiming to serve it as a resource-friendly alternative to lora.