The Impact of Language Adapters in Cross-Lingual Transfer for NLU (2024)

Abstract: Modular deep learning has been proposed for the efficient adaption of pre-trained models to new tasks, domains and languages. In particular, combining language adapters with task adapters has shown potential where no supervised data exists for a language. In this paper, we explore the role of language adapters in zero-shot cross-lingual transfer for natural language understanding (NLU) benchmarks. We study the effect of including a target-language adapter in detailed ablation studies with two multilingual models and three multilingual datasets. Our results show that the effect of target-language adapters is highly inconsistent across tasks, languages and models. Retaining the source-language adapter instead often leads to an equivalent, and sometimes to a better, performance. Removing the language adapter after training has only a weak negative effect, indicating that the language adapters do not have a strong impact on the predictions.

## introduction adding smaller components to a large language model (llm) that can be specifically targeted, trained, stacked and exchanged is becoming increasingly common @xcite . particularly adapters @xcite and lora @xcite are widespread for the efficient adaption of llms. they often perform on par or better than fine-tuning the models' parameters while avoiding issues of interference such as catastrophic forgetting @xcite . in this work, we focus on pre-trained targetlanguage adapters for zero-shot cross-lingual transfer. @xcite found that any crosslingual transfer problem can be decomposed in language and task, and introduce a setup that combines task and language adapters, both independently trained on top of a pre-trained multilingual * equal contribution model. this setup is appealing particularly for lowresource and medium-resource languages that lack high-quality data for supervised training as it can be applied to unseen task-language combinations. however, how consistent the effect of the targetlanguage adapter is has not been explored explicitly. in particular, it has not been explored how including target-language adapters compares to keeping the source-language adapter for the cross-lingual transfer. in addition, the detailed ablations by @xcite focus on named entity recognition, while it remains unclear if similar results also hold for higher-level language understanding tasks. therefore, we focus on three multilingual natural language understanding (nlu) benchmarks. we investigate the following questions: rq1. how robust is the positive effect of adding a target-language adapter across languages, models and tasks? to answer this question, we compare the performance with targetlanguage adapters to other setups that keep the source-language adapter or that only include task adapters. ## experimental setup in the following, we introduce the models, adapters, adapter training setups, ablation setups and datasets that we use for our ablation studies of language adapters. a link to our code including hyperparameters used to run our experiments will be published after the anonymity period. the code, in-cluding the hyperparameters used to run our experiments, is available at https://github.com/ oskarholmstrom/lang-adapters-impact . ## results given the large number of combinations of models, tasks and language pairs in our experiments, we summarise them and present individual results of particular interest in this section. the full results can be found in appendix a. ## discussion in section 4 have observed relatively inconsistent results regarding the utility of language adapters, and of target-language adapters in particular. in the following, we discuss the relation of our results to the research questions introduced in section 1, as well as the variance across datasets, limitations of our experiments, and avenues for future work. ## conclusion in this work, we performed extensive ablations on cross-lingual transfer with pre-trained language adapters for nlu benchmarks. we found that the inclusion of target-language adapters appears to have a small benefit on average, but it is slight and varies significantly across languages, models and tasks. as the effect is not robust and we do not observe patterns clear enough to predict it, it remains to be tested for each use case and language individually. keeping the source-language adapter often has a surprisingly good performance, and for one of two models, even leaving out the adapter without substitution is possible without large performance drops. this shows that the model does not rely much on the language adapter, and that language adapters do not appear to be an impactful isolated language module. while this work provides new insights into the utility of language adapters for nlu, many questions remain open. we conclude that there is a need to identify the specific conditions -such as properties of the base model, task, source, and target languages -under which language adapters enhance performance, and thereby unlocking their usefulness in a broader setting. abteen ebrahimi and katharina kann. 2021. how to adapt your pretrained multilingual model to 1600 languages. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 4555-4567, online. association for computational linguistics. fahim faisal and antonios anastasopoulos. 2022. phylogeny-inspired adaptation of multilingual models to new languages. in proceedings of the 2nd conference of the asia-pacific chapter of the association for computational linguistics and the 12th international joint conference on natural language processing (volume 1: long papers), pages 434-452, online only. association for computational linguistics. markus freitag, isaac caswell, and scott roy. 2019. ape at scale and its implications on mt evaluation biases. in proceedings of the fourth conference on machine translation (volume 1: research papers), pages 34-44, florence, italy. association for computational linguistics.