R ose L o RA : Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning (2024)

Abstract: Pre-trained language models, trained on large-scale corpora, demonstrate strong generalizability across various NLP tasks. Fine-tuning these models for specific tasks typically involves updating all parameters, which is resource-intensive. Parameter-efficient fine-tuning (PEFT) methods, such as the popular LoRA family, introduce low-rank matrices to learn only a few parameters efficiently. However, during inference, the product of these matrices updates all pre-trained parameters, complicating tasks like knowledge editing that require selective updates. We propose a novel PEFT method, which conducts r ow and c o lumn-wise spar se lo w- r ank a daptation (RoseLoRA), to address this challenge. RoseLoRA identifies and updates only the most important parameters for a specific task, maintaining efficiency while preserving other model knowledge. By adding a sparsity constraint on the product of low-rank matrices and converting it to row and column-wise sparsity, we ensure efficient and precise model updates. Our theoretical analysis guarantees the lower bound of the sparsity with respective to the matrix product. Extensive experiments on five benchmarks across twenty datasets demonstrate that RoseLoRA outperforms baselines in both general fine-tuning and knowledge editing tasks.

## introduction pre-trained language models, trained on extensive and diverse general-domain corpora, exhibit robust generalization capabilities, benefiting various natural language processing (nlp) tasks, such as natural language understanding (kenton and toutanova, 2019; @xcite and generation @xcite . to further adapt these pre-trained models to a specific downstream task, fine-tuning is typically performed. however, these models often comprise numerous parameters, rendering full fine-tuning resource-intensive. to address this challenge, parameter-efficient fine-tuning (peft) methods @xcite are proposed. these method introduce a small number of learnable parameters and update only the lightweight introduced parameters during fine-tuning. among existing methods, lora family @xcite @xcite has gained remarkable popularity because of its high efficiency and good performance. conceptually, these lora methods add new low-rank matrices to model weights for fine-tuning. unlike other peft methods such as adapter @xcite , lora family does not modify the model architecture and is easier to incorporate. lora family has demonstrated notable performance on tasks, such as commonsense reasoning and arithmetic reasoning @xcite , that mainly rely on a language model's ability to understand and generate text without requiring to modify its internal knowledge explicitly. however, some specialized tasks require updating this internal knowledge. for instance, in knowledge editing @xcite , a language model should incorporate new provided knowledge while preserving other existing knowledge simultaneously. on such tasks, the lora family of methods are less-suited due to the coarse-grained control they offer. in particular, the product of the low-rank matrices introduced by lora methods is a dense matrix, which is added to the pre-trained model weights during inference. consequently, all pre-trained parameters are updated, making it challenging to selectively modify specific internal knowledge. this motivates a natural question: is there a peft method that can be effectively employed for tasks that require editing the internal knowledge of language models? to answer this question, we propose a row and column-wise sparse low-rank adaptation method (roselora). the motivation is to identify and update only the most important and influential parameters in the pre-trained model concerning a specific task. in this way, the pre-trained model can be updated effectively with minimal impacts on knowledge that does not require modification. specifically, roselora inherits the structure of lora to enable parameter-efficient fine-tuning. to selectively fine-tune the most important parameters, we introduce a sparsity constraint, i.e., the ℓ 0 norm, on the product of the low-rank matrices. however, this constraint is non-trivial to optimize. while ℓ 0 norm constraint is widely explored in model pruning @xcite @xcite , these methods can only address the sparsity constraint on each low-rank matrix individually. unfortunately, even if each low-rank matrix is sparse, this does not guarantee that their product will be sparse. to overcome this challenge, we propose converting the original sparsity constraint to row and column-wise sparsity constraints on two low-rank matrices (i.e., b and a in lora). we provide a theoretical lower bound of the sparsity of the product of the two low-rank matrices. furthermore, we propose using a sensitivity-based importance score to incrementally solve the row and column-wise sparsity constraints. ## related works in this section we provide a concise overview of related works. ## preliminary in this section, we first briefly introduce the low-rank adaptation (lora) and then introduce importance-aware pruning. ## methodology to efficiently fine-tune a pre-trained language model with selective updating, we propose roselora, a novel lora-style fine-tuning framework with sparse adaptation. the framework is illustrated in figure 1 . we introduce row and column-wise sparsity constraints on the two lowrank matrices, respectively. we theoretically prove that the sparsity lower bound of the product of these low-rank matrices can be guaranteed under these constraints. ## conclusion in this paper, we address the limitations of existing parameter-efficient fine-tuning (peft) methods, particularly the lora family, in handling tasks requiring selective knowledge updates while still being effective for other general nlp tasks. we introduced a novel method, row and column-wise sparse low-rank adaptation (roselora), which selectively updates the most important parameters for specific tasks, maintaining efficiency while minimizing unnecessary changes to the pre-trained model's knowledge. roselora applies a row and column-wise sparsity constraint to the product of low-rank matrices, ensuring efficient updates without modifying the model architecture. our theoretical analysis lower bounds the sparsity of product matrices that affect model's knowledge, and our sensitivity-based importance scoring effectively fulfilled the sparsity constraints. through extensive experiments on five benchmarks encompassing over twenty datasets, roselora demonstrated superior performance on both general-purposed finetuning and knowledge editing tasks compared to existing methods. this highlights its potential as a robust and efficient fine-tuning solution for a wide range of nlp applications.