Low-Rank Adaptation for Multilingual Summarization: An Empirical Study (2024)

Abstract: Although the advancements of pre-trained Large Language Models have significantly accelerated recent progress in NLP, their ever-increasing size poses significant challenges for conventional fine-tuning, especially in memory-intensive tasks. We investigate the potential of Parameter-Efficient Fine-Tuning, focusing on Low-Rank Adaptation (LoRA), in the domain of multilingual summarization, a task that is both challenging (due to typically long inputs), and relatively unexplored. We conduct an extensive study across different data availability scenarios, including high- and low-data settings, and cross-lingual transfer, leveraging models of different sizes. Our findings reveal that LoRA is competitive with full fine-tuning when trained with high quantities of data, and excels in low-data scenarios and cross-lingual transfer. We also study different strategies for few-shot cross-lingual transfer, finding that continued LoRA tuning outperforms full fine-tuning and the dynamic composition of language-specific LoRA modules.

## introduction the emergence of pre-trained large language models (llms), such as palm 2 @xcite , llama 2 @xcite , and the gpt family from openai, has significantly advanced the state of the art in numerous nlp applications. however, the expansion in the size of llms poses significant challenges for traditional fine-tuning, particularly when faced with many downstream tasks or tasks with a large memory footprint, e.g., due to processing long inputs. parameter-efficient fine-tuning (peft) methods have recently shown promise to adapt a pretrained model to different tasks by selectively fine-tuning a small subset of additional parameters. widely-adopted peft techniques include adapters @xcite , low-rank adaptation (lora; @xcite , and prompt-tuning @xcite . among these, lora has become one of the most popular approaches, achieving state-of-the-art performance without introducing latency at inference time. the majority of peft studies have focused on natural language understanding, e.g., classification tasks as exemplified in the glue @xcite and superglue @xcite benchmarks, and monolingual generation, e.g., table-to-text generation or summarization @xcite . in this paper, we empirically investigate the potential of lora in the domain of multilingual summarization, a task that is both challenging and relatively unexplored. multilingual summarization often involves processing lengthy inputs @xcite , providing a natural testbed for the effective use of peft methods. in addition to being able to understand long documents, models are expected to fluently generate sentences in many languages, requiring significant linguistic versatility. multilingual tasks face additional challenges pertaining to the availability of resources (e.g., for training). it is unrealistic to expect that large-scale and highquality data will be available or created for every language @xcite . in scenarios where multilingual data is scarce, peft methods which selectively update a small number of parameters seem more suitable while fine-tuning can lead to overfitting or catastrophic forgetting @xcite . this motivates us to explore the following research questions: (i) can lora be effectively applied to complex multilingual summarization tasks? and (ii) under which conditions does lora exhibit the most potential? to answer these questions, we investigate different data availability scenarios: high-data regime (high quantities of training data are available for all languages), low-data regime (training data is limited but available for all lan-guages), and cross-lingual transfer (zero or only a few examples are available for some languages). in the latter case, a model trained on a high-resource language (e.g., english) is localized to additional languages for which data is scarce or unavailable @xcite . in addition to mimicking real-world conditions, the cross-lingual transfer setting allows us to experiment with the composition of language-specific lora modules, including the recently proposed few-shot lorahub @xcite . our experiments are conducted on two multilingual summarization datasets, xlsum @xcite and xwikis @xcite , using different sizes of the palm 2 model, an llm trained on multilingual text spanning more than 100 languages @xcite . to summarize, our contributions are as follows: (i) we conduct a comprehensive study of the effectiveness of lora for multilingual summarization under different data regimes; (ii) we showcase the benefits of lora in low-data and cross-lingual transfer settings; and (iii) we investigate how to best leverage lora for cross-lingual transfer subject to the availability of target language examples. cross-lingual transfer requires a model to learn a task from labeled data in one language (typically english), and then perform the equivalent task in another language where no or very little labeled data is available @xcite @xcite @xcite . previous studies focusing on peft methods for cross-lingual transfer have explored adapter-based approaches @xcite and composable sparse fine-tuning @xcite , among others. @xcite evaluate prompt-tuning @xcite in a zero-shot setting for cross-lingual summarization, focusing on the wikilingua dataset @xcite . their study does not cover lora, nor does it explore scenarios with more available data (e.g., few-shot settings). model composition and weight merging aim to enable generalization to unseen tasks by combining individually trained models. previous work includes weight composition guided by task similarity @xcite or arithmetic operations such as addition or subtraction @xcite , multi-task prompt pre-training @xcite , and combining models in parameter space by minimizing prediction differences between a merged model and individual models @xcite . for our multilingual summarization task, we also explore the composition of language-specific lora matrices through weight averaging, as well as dynamic weight composition when few-shot samples are available @xcite . ## lora for multilingual summarization we now present the fundamentals of lora @xcite and then discuss how individual lora modules can be combined @xcite for cross-lingual transfer. we also introduce our assumptions regarding the availability of training data for multilingual summarization. ## experimental setup this section introduces the datasets and models used in our study. we further elaborate on the details of our experimental setup, and the metrics used to assess the generated summaries. ## conclusions in this paper, we explored the effectiveness of lora on multilingual summarization across a diverse range of scenarios primarily determined by the availability of training data. we summarize our key findings by comparing the computationally efficient lora against full fine-tuning. lora achieves superior performance to full fine-tuning in zero-shot and few-shot cross-lingual transfer scenarios, and low-data settings (e.g., training data with fewer than 1k samples). this is most pronounced with smaller models (e.g., palm 2-xxs). in the specific case of few-shot learning, lora continued learning outperforms lorahub. lora also achieves overall superior summary faithfulness and conciseness across various scenarios. for larger models like palm 2-s, lora exhibits on-par performance to full fine-tuning. this suggests that model capacity matters. notably, for smaller models like palm 2-xxs, lora displays worse performance in the full fine-tuning (highdata) regime, when said performance is measured via rouge-l, but is consistently superior in terms of faithfulness and conciseness. taken together, our results underscore the utility of peft methods for complex multilingual tasks and cross-lingual transfer. avenues for future work include few-shot transfer and effective ways to combine lora modules, e.g., by learning which modules to activate for different tasks or languages @xcite . it would also be interesting to reproduce our results across varied llms and broader multilingual generation tasks, beyond summarization.