How to Tune a Multilingual Encoder Model for German ic Languages: A Study of PEFT, Full Fine-Tuning, and Language Adapters (2025)

Abstract: This paper investigates the optimal use of the multilingual encoder model mDeBERTa for tasks in three Germanic languages – German, Swedish, and Icelandic – representing varying levels of presence and likely data quality in mDeBERTas pre-training data. We compare full fine-tuning with the parameter-efficient fine-tuning (PEFT) methods LoRA and Pfeiffer bottleneck adapters, finding that PEFT is more effective for the higher-resource language, German. However, results for Swedish and Icelandic are less consistent. We also observe differences between tasks: While PEFT tends to work better for question answering, full fine-tuning is preferable for named entity recognition. Inspired by previous research on modular approaches that combine task and language adapters, we evaluate the impact of adding PEFT modules trained on unstructured text, finding that this approach is not beneficial.

## introduction massively multilingual encoder models like mbert @xcite , xlm-r @xcite and mdeberta @xcite are a workhorse for nlp in many lower-resource languages. however, due to interference between languages @xcite , these models can fall short of reaching their full potential for individual target languages: monolingual models @xcite and models with dedicated language modules @xcite frequently outperform them, raising the question for the best setups for different languages. parameter-efficient fine-tuning (peft) methods, such as bottleneck adapters @xcite , @xcite , and prefix tuning @xcite , have emerged as an alternative to full fine-tuning of pre-trained language models. these methods preserve the model's representations and can lead to better generalisation @xcite . this is especially relevant for multilingual models, trained on diverse data, of which the target language only constitutes a small fraction. fully fine-tuning them on task-specific data risks overwriting some of the multilingual capabilities. language adapters -peft modules trained on unstructured text independently from task finetuning -have shown promise in cross-lingual transfer @xcite . we explore whether language adaptation modules are beneficial even in scenarios where cross-lingual transfer is not required, i.e., where we have inlanguage fine-tuning data. in addition, we use not only bottleneck @xcite adapters but also @xcite , a method that has become popular for llms as its parameters can be merged with the model parameters, adding no inference overhead. in this paper, we investigate strategies for adapting a multilingual encoder model to task data in three languages: german, swedish, and icelandic. for this, we use multilingual deberta @xcite , which is currently the bestperforming model according to the scandeval (nielsen et al., 2024) leaderboard for icelandic, foot_0 the lowest-resourced and thus the most challenging of the three languages. our findings indicate that the effectiveness of full fine-tuning versus peft varies by language. for german, a peft method consistently delivers the best results, although sometimes with marginal gains. for swedish and icelandic, the performance is task-dependent: peft is more beneficial for extractive question-answering (qa), while full finetuning works better for named entity recognition (ner). we hypothesise that in languages with quantitatively more limited or lower-quality representation in the pre-training data, there is less value in preserving the pre-existing representations and more value in increasing the learning capacity. in contrast, for higher-resource languages, capabilities from the pre-training phase are more impactful. similarly, for extractive qa, pre-existing skills weigh higher, while the highly specific nature of ner benefits from full fine-tuning. language adapters do not provide consistent improvements in any of the tasks or languages tested. as the adaptation data we use has likely been used for pre-training the multilingual deberta model, we conclude that the utilisation of this data at pretraining time has already been effective enough. further adaptation, or specialisation, with this same data does not have a clear benefit. ## related work peft methods not only reduce the number of trainable parameters and, consequently, memory usage in comparison to full fine-tuning, but there is also evidence suggesting that they provide better regularisation and help preserve pre-existing model capabilities. for example, @xcite demonstrate that adapter-based fine-tuning outperforms full fine-tuning in cross-lingual transfer setups, likely by avoiding overfitting on the source language. similarly, prefix tuning, another peft method, has been shown to surpass full fine-tuning in extrapolation scenarios @xcite . other works have shown the effectiveness of bottleneck-style adapters in cross-lingual transfer as post-hoc trained language modules in encoder models. @xcite show that bottleneck language adapters in the pfeiffer architecture improve performance in ner, commonsense classification, and extractive qa. even @xcite report that language adapters are effective. other research indicates that language adapters can aid in transferring knowledge to dialectal variants @xcite and that sharing adapters across related languages can be beneficial @xcite . however, the success of language adapters may be task-specific and difficult to measure accurately when using machine-translated evaluation data @xcite . and notably, none of the works used multilingual deberta models, which may explain divergences in results. ## results and discussion all results are presented in ## conclusion we compared the performance of the multilingual encoder model mdeberta across three task adaptation setups: full fine-tuning, bottleneck (pfeiffer) adapters, and lora. based on our evaluations across three tasks and three languages, we found that the choice of the best method is both taskand language-dependent. specifically, extractive qa tasks benefit from peft methods, while ner gets better results with full fine-tuning. for ger-man, a higher-resourced language, peft consistently achieves higher scores. this suggests that the model benefits from fine-grained information learned during pre-training if coverage and (or) quality of the language data in the pre-training corpus are sufficiently high. in contrast, for lowerresourced languages, the increased learning capacity of full fine-tuning proves more advantageous. we also tested language adaptation with pfeiffer adapters and lora on unstructured text data before task adaptation. however, language adapters did not show any benefit. access to target-language task data appears to dispense with the need for them, at least in our experiments where all languages are included in the pre-training data. in future work, we aim to further explore the conditions under which peft methods versus full fine-tuning are most effective. we plan to investigate additional peft methods and tasks and optimise the lora setup, which may not have reached its full potential in our experiments.