Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation (2024)

Abstract: Parameter-efficient fine-tuning (PEFT) methods are increasingly vital in adapting large-scale pre-trained language models for diverse tasks, offering a balance between adaptability and computational efficiency. They are important in Low-Resource Language (LRL) Neural Machine Translation (NMT) to enhance translation accuracy with minimal resources. However, their practical effectiveness varies significantly across different languages. We conducted comprehensive empirical experiments with varying LRL domains and sizes to evaluate the performance of 8 PEFT methods with in total of 15 architectures using the SacreBLEU score. We showed that 6 PEFT architectures outperform the baseline for both in-domain and out-domain tests and the Houlsby+Inversion adapter has the best performance overall, proving the effectiveness of PEFT methods.

## introduction advances in large-scale pre-trained language models have transformed the field for high-resource languages @xcite , but these data and compute-hungry models are not viable for the more-than-7000 low-resource languages (lrls) in the world @xcite @xcite . ideal for the limitations of lrls, parameter-efficient fine-tuning (peft) methods @xcite @xcite are designed to strategically update a small number of parameters within a pre-trained model to be more efficient and adaptable without retraining the entire model. their architecture significantly saves computational resources and storage space while achieving results comparable to full fine-tuning in downstream tasks @xcite . @xcite examined the applicability of 4 peft methods specifically in the context of language translation. moreover, it did not address truly lrls @xcite , nor did it incorporate variation in domains that would allow for an assessment of the models' generalization capabilities. as a result, while the peft methods have shown potential in fine-tuning specific tasks, domains, and languages, the effectiveness of this collection of peft methods for lrl translation has not been systematically examined. in this paper, we explore the performance of different peft architectures in the lrl neural machine translation (nmt) by comparing in-domain and out-of-domain test results, as well as training times. we also investigate the effectiveness of peft methods in translating lrls, focusing specifically on their architectures and performance across various datasets. the contributions of our paper are 1) comprehensive experimentation of peft architectures to reveal the suitability of translating non-latin scripts and lrl pairs; 2) an in-depth assessment of 15 peft architectures using 8 distinct methods to evaluate their effectiveness in lrl translation; and 3) a systematic exploration of experimental settings, including variations in dataset domains and sizes, aimed at enhancing model generalization capabilities. as the field continues to advance rapidly, these peft guidelines provide practical recommendations for improving lrl translations, thus narrowing the language gap. ## the peft methods we focus on the comparative performance of an extensive list of peft methods for lrl nmt under various settings (figure 1 ), offering a broader and distinctive understanding of adapter utility. among all the peft methods, some share the same structure. for example, the bottleneck adapters include bottleneck feed-forward layers in each layer of a transformer model. these layers can be added to various positions within transformer blocks. the houlsby adapter @xcite adds the layers after both the multi-head attention and feed-forward blocks. the pfeiffer adapter @xcite only adds the layers after the feed-forward block. the parallel adapter @xcite deploys the layers parallel to the transformer layers. similarly, the invertible adapters share a similar architecture with bottleneck adapters but with an added invertible adapter layer to the language model embedding layer. the compacter architecture replaces only the linear down-projection and up-projection with a parameterized hypercomplex multiplication layer @xcite . in addition, prefix tuning is a lightweight alternative inspired by prompting (li and liang, 2021) that introduces additional parameters in the multihead attention blocks of each transformer layer. the lora method allows the training of specific dense layers in a neural network indirectly by optimizing the rank-decomposition matrices of specific dense layers during adaptation with the pre-trained weights frozen @xcite . (ia) 3 is built to improve lora with modifications. while lora uses additive composition, (ia) 3 uses element-wise multiplication @xcite . some peft methods combine multiple methods. the mix-and-match (mam) adapter combines lora, prefix tuning, and parallel adapter to form a new adapter @xcite . similarly, unipelt integrates bottleneck adapters, prefix tuning, and lora into a unified setup @xcite . lastly, the language adapter captures languagespecific knowledge for application in various downstream tasks. it is not a distinct adapter architecture; rather, it represents a method of utilizing preexisting architectures. we expected that this ap-proach would enhance the model's performance, given its preexisting familiarity with the language in question. we employed a pre-existing bottleneck adapter for diverse language datasets, training it with masked language modelling on an extensive collection of articles @xcite . ## experimental setup lrls selection we chose sinhala (si), tamil (ta), hindi (hi), and gujarati (gu) as our primary languages to run our translation task (see the number of parameters that are updated and then select the most suitable reduction factor for all the peft architectures. the trainer employed in our study is sourced from the adapter transformers @xcite . each adapter's performance was evaluated using the sacre bilingual evaluation understudy (sacrebleu) score @xcite . training details are given in appendix a.2. we evaluated the performance of our peft architectures using direct fine-tuning with the pretrained model as the baseline. in total, we tested 15 peft architectures supported by the hugging face adapter hub @xcite trained on si-ta 100k nllb language dataset to identify the best methods for further analysis; both the nllb test dataset and the flores test dataset were used to test these models. we then narrowed down the selection to the top two methods with the highest sacrebleu scores from each of the test results, the nllb and the flores dataset. an additional peft architecture was selected based on those that outperformed the baseline for both test datasets and with the shortest training time. extensive experiments were then conducted with these top-selected methods across additional lrl and dataset sizes to determine the optimal configuration. after all experiments were completed, the average performance was calculated to mitigate any variation due to gpu randomness. ## experimental results number of fine-tuned parameters ## discussion result generalization our results demonstrate the robust generalizability of our peft architectures across different training dataset sizes and domains. figure 2 shows that our model consistently outperforms the baseline, on average, in both indomain and out-of-domain testing. specifically for models trained on other domains, the ∆% increase over the baseline is over 50%, demonstrating the ability of peft methods to excel at tasks beyond their training domain. in terms of training dataset sizes, our selected peft architectures show a continuous trend for performance increase compared to the baseline. it is worth noting that our ## conclusion our study delved into a wide range of peft methods to identify the most effective ones for lrl-nmt. particularly focusing on non-latin scripts and lrl-to-lrl translation pairs, our research stands as a valuable guide for lrl-nmt. we found that certain adapters consistently outperformed others, offering enhanced translation accuracy and efficiency in challenging linguistic contexts. furthermore, the adapters' effectiveness was tested and generalized across various dataset domains and sizes, ensuring the applicability of our findings to a broad spectrum of lrl scenarios. looking ahead, these insights pave the way for further advancements in peft methods, aiming to optimize the balance between efficiency and quality in nmt, especially in the challenging context of lrl.