Sparse Low-rank Adaptation of Pre-trained Language Models (2023)

Abstract: Fine-tuning pre-trained large language models in a parameter-efficient manner is widely studied for its effectiveness and efficiency. The popular method of low-rank adaptation (LoRA) offers a notable approach, hypothesizing that the adaptation process is intrinsically low-dimensional. Although LoRA has demonstrated commendable performance, it is implemented with a fixed and unalterable intrinsic rank that might not always be the ideal choice. Recognizing the need for more flexible adaptation, we extend the methodology of LoRA to an innovative approach we call sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. We achieve this through the incorporation of a gate unit optimized with proximal gradient method in the training stage, controlling the cardinality of rank under the sparsity of the gate. In the subsequent inference stage, we eliminate the parameter blocks corresponding to the zeroed-out ranks, to reduce each SoRA module back to a concise yet rank-optimal LoRA. Our approach strengthens the representation power of LoRA by initializing it with a higher rank, while efficiently taming a temporarily increased number of parameters via updating in a sparse way. We further introduce a sparsifying scheduler for SoRA, aiming to examine the impact of the number of non-zero parameters on the model’s memorization and generalization. Our experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.

## introduction adapting large-scale pre-trained language models @xcite @xcite @xcite in a parameter-efficient (he * equal contributions † corresponding authors @xcite @xcite manner is increasingly gaining traction within the research community. the methods of this paradigm typically keep most of the parameters of the underlying model unchanged, either insert additional trainable parameters into the model @xcite , or specify a small number of parameters @xcite @xcite to be trainable or reparameterize the adaptation process into a more efficient form @xcite . they have been validated to be effective across various models and tasks, often yielding comparable or even better results than full-parameter fine-tuning. the development potential of parameter-efficient fine-tuning became evident after extensive validation of its performance. these methods offer the opportunity to adapt the base model to fit any data, allowing for enhancements and customization of language models tailored to specific tasks and personalized user characteristics. due to the lightweight nature of the optimized parameters, they can be seamlessly plugged into the model, allowing targeted enhancements to be made. among these methods, low-rank adaptation (lora @xcite ) is considered one of the most efficient methods at present. it assumes that the change of the model's parameters after adaptation is "intrinsically low-dimensional" and performs adaptation by optimizing the matrix obtained from low-rank decomposition. lora avoids forward propagation latency caused by inserting additional neural modules while demonstrating stable performance. although effective, the setup of the intrinsic rank (normally as a hyperparameter) is still unclear. intuitively, a larger rank brings larger optimization space and creates the capacity to handle more challenging tasks. however, in practice, the optimal intrinsic rank would vary according to multiple factors such as the backbone model and the task. given the enormous computational cost of searching hyperparameters on large-scale models (such as @xcite with 175 billion parameters and llama @xcite with 700 million to 65 billion parameters), developing a method based on adaptive ranks is a natural approach. some existing work has attempted to explore this direction @xcite , but they are largely heuristic or introduce additional costs. in this paper, we propose sora, a simple, effective, and automated method for adaptive parameter-efficient fine-tuning. we introduce a gating module with a proximal gradient decent update under l1 regularization to control the sparsity of the updated matrices. after training, the zero entry of the gating vector records the columns of the down-projection matrix and the rows of the up-projection matrix, which can be simply dropped and stored in a more parameterefficient manner. compared to other adaptive approaches, the proximal gradient method has a clear mathematical meaning and does not have to involve other computations and heuristics. for example, adalora @xcite introduces an additional regularizer to ensure that the lower and upper projection matrices strictly adhere to the definition of singular value decomposition (svd), with each matrix being orthogonal. however, this regularization term incurs substantial computational overhead due to the gradient calculations. in contrast, we eliminate this requirement and instead selectively filter low-rank components by controlling the intermediate diagonal matrix. we detailedly compare sora and related methods in section 3. the mechanism of sora also allows us to control the sparsity temporarily and investigate the relationship between the number of non-zero trainable parameters and memorization and generalization capabilities. we propose a sparsifying scheduler and find that the process of model adaptation exhibits a strong "compression capability", and even a tiny portion of parameters (lower than lora rank being 1) could retain considerable performance. extensive experiments are conducted to demonstrate the effectiveness of our method. particularly, our model could consistently outperform parameter-efficient baselines with fewer parameters and 30% shorter training time on a wide range of downstream tasks. the code of this work will be publicly available at https://github.com/ tsinghuac3i/sora . ## a closer look to adaptive rank related work. before introducing our approach, we first briefly recap parameter-efficient tuning and our backbone low-rank adaptation (lora). parameter-efficient tuning is a set of methods that only optimize a small portion of parameters and keep the main model untouched for adaptation. some parameter-efficient methods would insert additional neural modules or parameters to the backbone model, such as adapter @xcite , @xcite . and another line of such methods attempts to specify particular parameters to be trainable or prunable @xcite @xcite . researchers derive a series of variants of parameter-efficient methods to improve the effectiveness or efficiency @xcite @xcite . recently, the applications of parameter-efficient fine-tuning are expanded to multi-modal and instruction-tuning scenarios @xcite . in this paper, we focus more on lora @xcite , which uses low-rank matrices to approximate the change of weights. in lora, pre-trained weights (denoted as w 0 ∈ r p×q ) are frozen, and the trainable lora modules are low-rank decomposition matrices w @xmath0. in this way, the output of the current layer h could be represented as adaptive rank on lora. despite a great step forward in tractability and efficiency, lora is still restricted by its inflexibility in selecting the optimal rank r. unlike continuous hyperparameters such as learning rate and weight decay that can be tuned adaptively online during the training process, lora rank r takes discrete values -the change of which will directly alter the model structures. the optimal choice of rank can vary across different backbone models and downstream tasks. a conservative choice of huge rank r can waste training time and computation resources, while progressively setting r tiny may degrade model performance and r o i 4 l b l x w s a 9 s h 5 j j m 2 1 o j j m k d 4 q y 9 c / c u f d e r x / j z r 8 x b w e h r q c c h 3 p u j e e e i j h c o o t + o 4 w n z a 3 t n e j u a w / / 4 p c o f h z s n n g q g w + x w m a 6 g 1 d d p v c 8 h q i l 7 y a a 0 y i q v b n m b u d + 5 4 l r i 2 l 1 g n o e + x e d k r e k r t f k j / 2 i 4 j g i s / f s u k 6 4 n x c b s k 6 8 n f q g r 3 n q / u o p y 5 z g x c g t 1 j i e 5 y b o z 1 s j y j l p s v 3 u 8 i s y c r 3 x n q w k r t z 4 2 s l x j f x y z u j c w n u n k c z u 3 x s z j y y z r o g d n c c 0 q 9 5 c / m / r p r j e + j l q s y p c s e v h y s o j x m r + p h k k z r n k q s w u a w g z e j a m m j k 0 j z v s c d 7 q y e u k f v n z r m r 1 + 3 q l u c 3 r k m i z n e m v p l i g b t x b e 1 r a q m e z v m k b y 5 w x 5 9 3 5 w i 4 w n h z n f p 7 a + f w b 4 a w q + @xmath1 r o i 4 l b l x w s a 9 s h 5 j j m 2 1 o j j m k d 4 q y 9 c / c u f d e r x / j z r 8 x b w e h r q c c h 3 p u j e e e i j h c o o t + o 4 w n z a 3 t n e j u a w / / 4 p c o f h z s n n g q g w + x w m a 6 g 1 d d p v c 8 h q i l 7 y a a 0 y i q v b n m b u d + 5 4 l r i 2 l 1 g n o e + x e d k r e k r t f k j / 2 i 4 j g i s / f s u k 6 4 n x c b s k 6 8 n f q g r 3 n q / u o p y 5 z g x c g t 1 j i e 5 y b o z 1 s j y j l p s v 3 u 8 i s y c r 3 x n q w k r t z 4 2 s l x j f x y z u j c w n u n k c z u 3 x s z j y y z r o g d n c c 0 q 9 5 c / m / r p r j e + j l q s y p c s e v h y s o j x m r + p h k k z r n k q s w u a w g z e j a m m j k 0 j z v s c d 7 q y e u k f v n z r m r 1 + 3 q l u c 3 r k m i z n e m v p l i g b t x b e 1 r a q m e z v m k b y 5 w x 5 9 3 5 w i 4 w n h z n f p 7 a + f w b 4 a w q + @xmath2 several remedies have been proposed in recent years to enable the flexible tuning of lora rank. for example, rather than setting a fixed rank, valipour et al. @xcite introduce dy-lora in which a pre-defined discrete distribution p b (•) is cast over a range of rank choices. this approach is related to but different from nested dropout @xcite , and can be regarded as optimizing a mixture model with lora modules of different ranks. nevertheless, tuning lora rank straightforwardly and deterministically appears to be a more attractive approach. to devise such an approach, we first gain a crucial hint from the connection between a matrix's rank and its singular value decomposition (svd). let us denote the tunable incremental weight matrix in lora by ∆ := w u w d . we can then formulate its svd as in which u and v are orthogonal respectively, and σ is a (rectangular) diagonal matrix with diagonal elements being the singular values of ∆: in spite of the effectiveness demonstrated through experiments, there are still two problems in adalora that demand rethinking of the methodology and wait for further improvements. first, the sparsity selection criterion in adalora is based on their newly proposed importance score relied on the moving average of weight-gradient product. despite its effectiveness in empirical study, this criterion is largely heuristic, lacking theoretical motivation. second, both the moving average operation of importance scores and the gradients of orthogonality regularization (5) add up to additional computation cost. compared to adalora with the aforementioned limitations, our approach, sora, serves as an amelioration with highly simplified updating rules and is backed up by the theory of sparsity regularization and proximal gradient methods. detailed methodology of sora will be elaborated in the next section. ## our approach the key idea of our approach, sparse low-rank adaptation (sora), is to dynamically adjust the intrinsic rank in the training process with a sparse gating unit trained by proximal gradient method. sora adopts the previously introduced framework of lowrank decomposition because of its widely validated effectiveness and parameter efficiency. ## conclusion our work presents sparse low-rank adaptation (sora), an innovative method for parameterefficient fine-tuning large pre-trained language models. upon the hypothesis that the adaptation process could be intrinsically sparse, we offer a dynamic alternative rank by introducing an optimizable gate with a proximal gradient method to regulate sparsity, thereby expanding the optimization space while enhancing parameter efficiency. the method is simple and theoretically supported with promising performance across various tasks. utilizing sora as a tool, we propose a sparsifying scheduler to analyze the correlation between parameters and memorization and generalization.