Adaptive L o RA Merge with Parameter Pruning for Low-Resource Generation (2025)

Abstract: This study proposes a simple yet effective LoRA merge method to achieve LLM adaptation for low-resource language generation tasks. The LoRA merge technique, which integrates multiple LoRA modules trained on different tasks, has gained attention as an effective and efficient approach for adapting LLMs to target tasks. However, previous methods are limited in adaptability as they keep the LoRA parameters frozen. Additionally, the low-resource problem has been out of their scope. We propose a LoRA merge method that updates and prunes LoRA parameters through fine-tuning with minimal target task data, which allows finer-grained adjustments of LoRA parameters and enhancement of task adaptability. Extensive experiments have been conducted taking summarization as a benchmark task. Our datasets cover various domains and multiple languages of English and Japanese. The results confirm that the proposed method achieves significant and consistent improvements in task adaptability over the previous methods.

## introduction the rapid advancements in large language models (llms) have significantly enhanced text generation capabilities and performance across tasks such as translation, summarization, question answering, and code generation @xcite @xcite . however, llms often struggle with low-resource tasks, including those involving languages with scarce linguistic resources, specialized programming languages, or tasks in medical and other specialized domains @xcite @xcite . this performance degradation arises from the insufficient adaptation of llms to target tasks, despite their general knowledge obtained during pretraining. fine-tuning is a common method to enhance task-specific performance @xcite , but its effectiveness is often constrained by limited training data in low-resource problems @xcite @xcite . an alternative approach gaining attention is the integration of multiple models, particularly using lora modules @xcite @xcite . for instance, combining a model with general language capabilities and another specialized in a specific task can improve performance on target tasks. such lora merge technique linearly combines lora modules into a single model. existing studies @xcite @xcite typically keep module parameters fixed and only adjust their combination weights, which reduces training costs. however, we assume it limits adaptability to the target task. furthermore, lowresource tasks have been out of their scope. to effectively adapt llms on low-resource language generation tasks, we propose a novel lora merge method that further updates lora modules with minimal target task data while pruning ineffective parameters. previous studies have reported that each decoder layer in llms plays a different role in language generation @xcite . furthermore, analyses of lora modules trained on multiple tasks suggest that these modules learn task-specific representations that vary across layers @xcite . these findings inspired us to hypothesize that lora parameters may require finer-grained adjustments at different layers to better adapt to a target task. based on this hypothesis, our method evaluates the importance of each lora parameter at each layer while pruning away ineffective ones and retraining them in order to enhance task adaptability. we conducted extensive experiments to evaluate and analyze the proposed method taking summarization as a benchmark task. our datasets cover various domains of news, scientific papers, and ra-diology reports in multiple languages of english and japanese. the results confirm that updating lora modules during the merge process improves task adaptability. in addition, pruning ineffective parameters further enhances the performance. the primary contributions of this study are twofold. first, our simple lora merge technique achieves effective llm adaptation to low-resource tasks across various domains and multiple languages with a minimum amount of target-task data. second, we show that lora parameter pruning enhances the task adaptability of llms, which is a novel feature of the pruning technique that often degrades the performance in exchange for the reduction of active parameters. the codes are available at https://github.com/mr0223/ adaptive_lora_merge . ## related work this section discusses the previous lora merge techniques. in addition, we review studies on llm layer analysis that inspired us to conduct parameter pruning during the lora merging process. ## adaptive lora merge with pruning the proposed method achieves effective adaptation to a low-resource target task through training and pruning of lora parameters. figure 1 illustrates the overview of the training procedure in the proposed method. the proposed method applies multiple lora modules trained on related tasks to a frozen llm and further trains them on a target task (section 3.1). during this process, the importance of lora parameters is evaluated at each decoder layer, and the parameters with lower importance are pruned and retrained (section 3.2). we remark that the proposed method does not explicitly 'merge' lora parameters; rather, our merging process is implicit through updates and pruning of all the original lora parameters. ## experiment settings we evaluate the capability of the proposed method for adapting an llm for low-resourced target tasks. intensive experiments are conducted using abstractive summarization as a benchmark task employing datasets of various domains of news, scientific papers, and radiology reports in multiple languages of english and japanese. ## experiment results experiments were conducted independently with three different random seeds, and the results are reported as the average across these runs. ## conclusion we proposed the adaptive merging method for multiple lora modules to improve llms in lowresource tasks. experiments on the five english and japanese summarization tasks show that our method significantly outperforms existing lora merging techniques across domains and languages. future work includes the application of the proposed method to broader tasks and cross-lingual settings. additionally, we plan to evaluate its effectiveness across various llms of different sizes. exploring the merging of more diverse and numerous lora modules is another important direction. currently, the proposed method requires tuning the pruning threshold for each task. automating this process would enhance the practicality of our method.