MAPLE : Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models (2024)

Abstract: Parameter efficient finetuning has emerged as a viable solution for improving the performance of Large Language Models without requiring massive resources and compute. Prior work on multilingual evaluation has shown that there is a large gap between the performance of LLMs on English and other languages. Further, there is also a large gap between the performance of smaller open-source models and larger LLMs. Finetuning can be an effective way to bridge this gap and make language models more equitable. In this work, we finetune the Llama-2 and Mistral models on two synthetic multilingual instruction tuning datasets to determine its effect on model performance on six downstream tasks covering forty one languages in all. Additionally, we experiment with various parameters, such as rank for low-rank adaptation and values of quantisation to determine their effects on downstream performance and find that higher rank and higher quantisation values benefit low-resource languages. We find that parameter efficient finetuning of smaller open-source models sometimes bridges the gap between the performance of these models and the larger ones, however, English performance can take a hit. We also find that finetuning sometimes improves performance on low-resource languages, while degrading performance on high-resource languages.

## introduction large language models (llms) show impressive performance on several tasks, sometimes even surpassing human performance. this has been attributed to the vast amounts of training data used during the pretraining phase, as well as various techniques used to align the models during the finetuning phase. several variants of finetuning exist, including supervised finetuning (sft) and instruction tuning @xcite , where the model is finetuned with task specific data, or instructions on how to perform tasks. however, finetuning all the parameters of the model can be expensive and time consuming, due to the increasing size of language models in recent years. parameter efficient finetuning (peft) has emerged as a viable alternative to full finetuning @xcite . most studies on llms focus on training, finetuning and evaluating models in performing tasks in english. recent work on comprehensive evaluation of llm capabilities in non-english settings @xcite have shown that llms perform far worse on languages other than english. studies that compare multilingual performance across different models @xcite , show that there is a large performance gap between large, proprietary and closed models such as gpt-4 and palm2 and smaller open-source models like llama-2-7b and mistral-7b. peft techniques like lora @xcite ) have been shown to strengthen the multilingual capabilities of these open-source llms @xcite . moreover, adapters @xcite have been proposed to boost the capabilities of language models to newer languages. since full finetuning of models is not always feasible due to resource and compute constraints, exploring how far peft techniques can take us in boosting performance on non-english languages is a promising direction. adoption of model quantisation techniques @xcite has also made peft llms more accessible. not much work has been done on analyzing the impact of different choices, configurations and settings for peft on multilingual downstream tasks. in this work we aim to analyse how lora rank and quantisation affects the performance of finetuned models across 6 downstream tasks, covering 40 languages in all. we are interested in knowing whether multilingual peft can lead to reasonable gains in performance, or whether full finetuning of models is required. furthermore, we check if it is better to generate multilingual instructions from larger base models or translate existing english instruction to more languages. lastly, we also study the effect of multilingual finetuning on english performance. prior works have demonstrated that lora finetuning is the most effective peft out of existing techniques so far @xcite . hence, we keep our study limited to analysing different lora and qlora @xcite ) configurations to answer our research questions. our contributions are as follows: parameter efficient finetuning: recently, parameter efficient finetuning has gained significant attention in the nlp research community since full finetuning of llms is prohibitively expensive for most organizations. following early works on adapters @xcite , several finetuning techniques like lora @xcite , (ia) 3 (liu et al., 2022a), p-tuning @xcite and prefix tuning (li and liang, 2021) have been proposed. these techniques make the compute costs manageable by significantly reduce the number of trainable parameters during finetuning. several works have used these techniques for efficient cross lingual transfer @xcite , to tackle catastrophic forgetting @xcite or compose multiple adapters @xcite for multi-task performance. ## conclusion in this paper we perform an extensive analysis of how rank, quantisation, finetuning dataset and base llm effects the performance of the finetuned models on 6 multilingual tasks and alpacaeval when finetuned in a paramater efficient manner. we also beat gpt-4 on question-answering tasks (mlqa and xquad) using just multilingual peft on mistral-7b showing that multilingual finetuning of 7b parameter llms is a promising direction for the future to bridge the gap of performance on multilingual downstream tasks.