L o RA - MGPO : Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimization (2025)

Abstract: Parameter-efficient fine-tuning (PEFT), particularly Low-Rank Adaptation (LoRA), adapts large language models (LLMs) by training only a small fraction of parameters. However, as the rank of the low-rank matrices used for adaptation increases, LoRA often exhibits an unstable “double descent” phenomenon, characterized by transient divergence in the training loss, which delays convergence and impairs generalization by causing instability due to the attraction to sharp local minima. To address this, we introduce **LoRA-MGPO**, a framework that incorporates Momentum-Guided Perturbation Optimization (MGPO). MGPO stabilizes training dynamics by mitigating the double descent phenomenon and guiding weight perturbations using momentum vectors from the optimizer’s state, thus avoiding dual gradient computations. Additionally, an adaptive normalization scheme scales the magnitude of perturbations based on an exponential moving average (EMA) of gradient norms, further enhancing stability. While EMA controls the magnitude of the perturbations, MGPO guides their direction, ensuring a more stable optimization trajectory. Experiments on a suite of natural language understanding and generation benchmarks show that LoRA-MGPO consistently achieves superior performance over LoRA and other PEFT methods. The analysis indicates that LoRA-MGPO leads to smoother loss curves, faster convergence, and improved generalization by stabilizing the training process and mitigating the attraction to sharp minima. The code is publicly available at [https://github.com/llm172/LoRA-MGPO](https://github.com/llm172/LoRA-MGPO).

## introduction large language models (llms) have driven significant advancements in natural language processing, establishing new performance benchmarks * corresponding authors on tasks ranging from text generation to semantic understanding @xcite . however, the conventional method of fullparameter fine-tuning (full ft) requires updating billions of parameters, incurring prohibitive memory and computational costs. to overcome this limitation, parameter-efficient fine-tuning (peft) methods have emerged as an effective alternative, enabling efficient adaptation by optimizing only a small subset of model parameters @xcite . among these methods, low-rank adaptation (lora) @xcite is distinguished by its computational efficiency and architectural simplicity. lora approximates the weight update matrix ∆w as a low-rank decomposition, where the original pre-trained weights w 0 remain frozen. the trainable matrices b and a, with rank r ≪ min(m, n), drastically reduce the number of trainable parameters, improving efficiency without altering the model architecture. despite its efficiency, lora's training dynamics can be unstable. as shown in figure 1 , finetuning llama-2-7b @xcite on metamathqa @xcite often exhibits a "double descent" trajectory with initial convergence, transient divergence, and eventual stabilization. this phenomenon worsens with higher ranks and is not unique to lora; full ft can exhibit even more severe double descent, highlighting the general challenge of stabilizing fine-tuning in high-capacity models @xcite . such non-monotonic behavior delays convergence and impairs generalization due to unstable gradients and the attraction to sharp local minima @xcite . addressing these stability issues is crucial. sharpness-aware minimization (sam) @xcite improves generalization by seeking flatter minima. however, its application is hindered by the dual gradient computation require0 1000 2000 3000 4000 5000 @xcite methods with llama-2-7b @xcite on the metamathqa dataset @xcite . for lora, rank (r) and alpha (α) are set to the same values (@xmath0, 64, 128}), with a fixed learning rate of 5e -4. ment, which doubles the training cost @xcite . more efficient variants like momentum-guided sam reuse optimizer states to avoid this overhead but may not guarantee stable convergence. to further enhance stability, complementary techniques such as applying an exponential moving average (ema) to smooth optimization dynamics have been shown to suppress parameter oscillations and improve convergence in certain scenarios @xcite . building on these insights, we propose lora-mgpo, a novel framework that integrates momentum-guided perturbation optimization (mgpo) into lora to mitigate the detrimental effects of double descent. our contributions are twofold: ## method in this section, we first provide a concise overview of the low-rank adaptation (lora) framework. we then introduce lora-mgpo, an extension of lora that integrates momentum-guided perturbation optimization (mgpo) to enhance its stability and efficiency. we describe how mgpo reuses optimizer momentum for guided perturbations of the trainable parameters and incorporates an adaptive normalization scheme to stabilize training. ## related work parameter-efficient fine-tuning (peft) the prohibitive computational and storage costs of fullparameter fine-tuning @xcite have spurred the development of peft techniques for adapting large language models @xcite . by selectively updating a small subset of parameters, peft methods can achieve performance competitive with full fine-tuning while being significantly more efficient @xcite . among the diverse peft strategies, low-rank adaptation (lora) @xcite has gained prominence for its simplicity and effectiveness. recent works have enhanced lora along several directions. one line of work introduces architectural modifications; for instance, dora @xcite integrates learnable magnitude vectors, while adalora @xcite dynamically allocates rank budgets. another direction focuses on improving the training process and initialization, such as adjusting scaling factors in rslora @xcite , using separate learning rates in lora+ @xcite , or refining initialization with pissa @xcite and nlora @xcite . a third direction aims to improve the quality of the parameter updates, for instance by alleviating training biases with ba-lora @xcite or by more closely aligning lora's gradients with those of full fine-tuning, as seen in lora-ga @xcite and lora-pro @xcite . additional work has further explored lora's application in multi-task learning, such as @xcite . distinct from these approaches, our work focuses directly on the underlying optimization dynamics. rather than altering lora's architecture or mimicking full fine-tuning gradients, we introduce a novel training framework to stabilize the optimization process itself. ## conclusion in this work, we addressed the double descent phenomenon in low-rank adaptation (lora), an instability that can affect the fine-tuning of large language models. we proposed lora-mgpo, an optimization framework that integrates momentum-guided perturbation optimization (mgpo). this method aims to find flatter minima by reusing optimizer momentum to guide weight perturbations, combined with an adaptive normalization scheme to improve robustness. our experimental results across a range of natural language understanding (nlu) and natural language generation (nlg) tasks show that lora-mgpo provides improved performance over standard lora and other common peft baselines. this improvement is reflected in more stable convergence trajectories and reduced training instability. lora-mgpo offers a practical approach to overcoming some of the optimization challenges in lora while maintaining its parameter efficiency. future research may explore extending this framework to other parameter-efficient methods or adapting it for different domains, such as vision and speech.