NL o RA : Nyström-Initiated Low-Rank Adaptation for Large Language Models (2025)

Abstract: Parameter-efficient fine-tuning (PEFT) is essential for adapting large language models (LLMs), with low rank adaptation (LoRA) being the most popular approach. However, LoRA suffers from slow convergence, and some recent LoRA variants, such as PiSSA, primarily rely on Singular Value Decomposition (SVD) for initialization, leading to expensive computation. To mitigate these problems, we resort to Nyström method, which follows a three-matrix manipulation. Therefore, we first introduce StructuredLoRA (SLoRA), investigating to introduce a small intermediate matrix between the low-rank matrices (A) and (B). Secondly, we propose NyströmLoRA (NLoRA), which leverages Nyström-based initialization for SLoRA to improve its effectiveness and efficiency. Finally, we propose IntermediateTune (IntTune) to explore fine-tuning exclusively the intermediate matrix of NLoRA to furthermore boost LLMs’ efficiency. We evaluate our methods on 5 natural language generation (NLG) tasks and 8 natural language understanding (NLU) tasks. On GSM8K, SLoRA and NLoRA achieve accuracies of 56.48% and 57.70%, surpassing LoRA by 33.52% and 36.41% with only 3.67M additional trainable parameters. IntTune boosts average NLG performance over LoRA by 7.45% while using only 1.25% of its parameters. These results demonstrate the efficiency and effectiveness of our approach in enhancing model performance with minimal parameter overhead.

## introduction fine-tuning large language models (llms) has emerged as a fundamental approach to enhancing model capabilities @xcite @xcite and aligning models with specific application requirements @xcite . however, the growing scale of llms introduces significant challenges to llm development, with fine-tuning requiring substantial computational and memory resources @xcite . for example, fine-tuning a llama-65b model requires more than 780 gb of gpu memory @xcite , while training gpt-3 175b requires 1.2 tb of vram @xcite . such resource-intensive processes are infeasible for many researchers and institutions, driving the development of parameterefficient fine-tuning (peft) methods. among these methods, low-rank adaptation (lora) @xcite has received widespread attention due to its ability to achieve competitive performance compared to full parameter fine-tuning, while significantly reducing memory consumption and avoiding additional inference latency. lora enables the indirect training of dense layers in a neural network by optimizing low-rank decomposition matrices that represent changes in the dense layers during adaptation, all while keeping the pre-trained weights fixed. for a pre-trained weight matrix @xmath0, lora introduces a low-rank decomposition ∆@xmath1, where @xmath2, @xmath3, and the rank r ≪ min(m, n). this modifies the forward pass of a layer as follows: despite the above benefits, lora suffers from slow convergence @xcite . to address this issue, some recent lora variants, such as pissa @xcite , choose to conduct initialization of the low rank matrices by using singular value decomposition (svd). however, svdbased initialization is computationally expensive and requires a long time. to mitigate this issue, we investigate using nyström method, which approximates a matrix as a product of three matrices, to approximate svd. to fit the three-matrix structure, we first propose structuredlora (slora), where an additional r × r matrix is inserted between the low-rank matrices a and b, as shown in figure 2 . furthermore, we explore whether an extra matrix can influence the language model's performance, experimental results indicate that slora effectively enhances performance with only a minor increase in the number of parameters, demonstrating the potential of the three-matrix structure for peft. secondly, inspired by nyströmformer @xcite , we proposed nyströmlora (nlora) to leverage nyström method, which con-ducts svd approximation by sampling a subset of rows and columns of the pre-trained parameter matrix to reduce the computational cost, for weight initialization. nlora is supposed to bypass the computational cost of svd's eigenvalue decomposition, reducing time complexity to o(mr +r foot_1 +rn) compared to the o(mn 2 ) complexity of svd-based methods. finally, to explore whether we can further compress the trainable parameters of nlora, we propose intermediatetune (inttune), which exclusively adjusts the intermediate matrix of nlora. this method significantly reduces the number of trainable parameters. specifically, on the evaluation of llama 2-7b across five nlg benchmarks, lora uses 320m parameters, while our inttune method only requires tuning 4m parameters. in the meantime, inttune outperforms lora by 7.45% on average across nlg benchmarks. the comparison of our proposed methods with lora in terms of performance and trainable parameters is illustrated in figure 1 . in summary, our contributions are as follows: ## method the nyström method (baker and taylor, 1979), originating from the field of integral equations, is a approach for discretizing integral equations using a quadrature technique. it is commonly employed for out-of-sample extension problems. specifically, given an eigenfunction problem of the form: the nyström method utilizes a set of s sample points y 1 , y 2 , . . . , y s to approximate f (x) as follows: this approach effectively converts the continuous integral equation into a discrete summation, facilitating numerical computation and enabling out-ofsample extensions. for the pre-trained matrix @xmath4, we assume that it can be decomposed as follows: partition matrix: construct factors: in this way, the matrix w can be approximated as the product of three matrices. based on this finding, we propose an improvement to lora by introducing an intermediate matrix, named as structuredlora (slora). specifically, we introduce an intermediate matrix @xmath5, as illustrated in figure 2 . this modification transforms the weight update into: ## experiments the experiments were performed on nvidia l20 gpus. for these experiments, we follow the experimental setting given by @xcite , we employ the adamw optimizer with a batch size of 4, a learning rate of 2e-4, and a cosine annealing schedule with a warmup ratio of 0.03, all while avoiding weight decay. the parameter lora_alpha is consistently set equal to lora_r, with lora_dropout fixed at 0. adapters are integrated into all linear layers of the base model, and both the base model and adapters utilized float32 precision for computation. we take the convenience to directly cite the baseline performance values from @xcite . in this section, we evaluate the performance of slora and nlora across various benchmark datasets. we compare them with the following baselines: (1) full fine-tune, which updates all model parameters; (2) lora (hu et al., 2021), which approximates weight updates with low-rank matrices while freezing the base model; and (3) pissa @xcite , which initializes adapters using principal singular components and freezes residuals while retain lora's architecture. we evaluate the capabilities of natural language generation (nlg) using the llama 2-7b @xcite and mistral-7b (jiang et al., 2023) models through mathematical reasoning, coding proficiency, and dialogue tasks. additionally, natural language understanding (nlu) tasks were evaluated using the glue dataset @xcite with deberta-v3-base @xcite and robertalarge @xcite . finally, we analyze the empirical effects of exclusively fine-tuning the intermediate matrix on both nlu and nlg tasks. ## conclusion this work advances parameter-efficient fine-tuning strategies for large language models by introducing slora and nlora, along with an exploration of an intermediate matrix fine-tuning method, inttune. slora incorporates a small intermediate matrix, enhancing expressiveness with minimal parameter overhead, while nlora leverages nyström-based initialization to bypass the computational complexity of svd, achieving competitive downstream per-