An Efficient Approach for Studying Cross-Lingual Transfer in Multilingual Language Models (2024)

Abstract: The capacity and effectiveness of pre-trained multilingual models (MLMs) for zero-shot cross-lingual transfer is well established. However, phenomena of positive or negative transfer, and the effect of language choice still need to be fully understood, especially in the complex setting of massively multilingual LMs. We propose an efficient method to study transfer language influence in zero-shot performance on another target language. Unlike previous work, our approach disentangles downstream tasks from language , using dedicated adapter units. Our findings suggest that some languages do not largely affect others, while some languages, especially ones unseen during pre-training, can be extremely beneficial or detrimental for different target languages. We find that no transfer language is beneficial for all target languages. We do, curiously, observe languages previously unseen by MLMs consistently benefit from transfer from almost any language. We additionally use our modular approach to quantify negative interference efficiently and categorize languages accordingly. Furthermore, we provide a list of promising transfer-target language configurations that consistently lead to target language performance improvements.

## introduction pretrained multilingual models (mlms) perform surprisingly well in terms of zero-shot cross-lingual transfer even though no explicit cross-lingual signal was present during pretraining. subword fertility @xcite , token sharing @xcite , script @xcite , as well as balanced language representation @xcite contribute to this effectiveness. but, by and large, the most important component seems to be the combination of languages the model is trained and evaluated on. it is important, hence, to figure 1: our approach uses efficient few-step continued tuning (left) and adapter modules (right) to disentangle the effect of task and language to quantify the effect of a transfer language for a given task and model. the left panel depicts the framework for our cross-lingual transfer, while the right panel represents the scenario of multiple language interactions followed by quantifying negative interference. previous attempts at studying cross-lingual transfer fall into two categories. first, the most popular approaches are those which, given a task and a mlm, task-tune the mlm on annotated data from a transfer language and then evaluate on a target language (e.g. @xcite . the problem with such approaches is that (a) they do not disentangle the effect of task and language, since they train directly on the task using annotated data in the transfer language, and (b) it is expensive to task-tune the whole model for all possible transfer languages. second, other approaches tackle the inefficiency problem by relying on bilingual approximations: @xcite for instance train bi-lingual bert @xcite ) models, task-tune them on the transfer language and then evaluate on the target one, and contrast this performance to a monolingual target-language bert. while this approach ignores the fact that language interactions can be different in multilingual and bilingual models @xcite , it does correlate decently with transfer performance on multilingual models. however, it still does not disentangle task from language and is quite expensive, as studying n languages requires training n 2 + n bert models. in this work, we propose an efficient approach to study cross-lingual transfer, outlined in figure 1 , that also disentangles the effect of task-tuning and the effect of language, while operating within the framework of the same mlm. our approach relies on learning a separate task adapter module to perform the downstream task, which needs to only be trained once (hence it is efficient). we then perform unsupervised finetuning on unannotated transfer language data for a minimal number of steps. comparing the performance of the model on the target language with and without the previous step results in a direct assessment of the effect of the transfer language without changing the conditions under which the downstream task was learned. in addition, we extend this framework to quantify the negative interference resulted from the interaction of multiple languages (figure 1 (right)). with the aid of adapter-fusion tuning @xcite , we compare different combinations of language adapters and compute the interference occurring due to increased interactions. we perform extensive analysis using this efficient approach on five downstream tasks using dozens of transfer and target languages (184 in total) and devise a metric (which we dub transfer score) to quantify which languages have/receive positive or adverse effects on/from others. last, we focus our analysis on cross-lingual transfer for languages unseen during the pre-training of the mlm. ## methodology adapters @xcite are light-weight parameter-efficient modules that can be injected between the layers of pretrained models. in their typical usecase, the rest of the model is frozen and only the adapter modules are trained, to adapt a model to a new language, domain, or task. importantly, for our goals, these adapters are also composable: one can stack an independently trained language adapter and task adapter to achieve decent performance for that language on that task. first we use an adapter-based setting to perform our analysis on cross-lingual transfer. furthermore, we extend our study to negative interference and language interaction through another adapter-fusion-based setting. cross-lingual transfer the composable property of adapters allows us to disentangle learning a task from the language representations (the process is also outlined in figure 1 ). in step 1, we first train a task-specific adapter [t] (e.g. named entity recognition), on data from as many languages as possible. this module will be responsible for performing the downstream task independently of input language. we then (step 2) finetune the [base] model (e.g. mbert) on a transfer language α with only a few steps (1, 10, or 100) using masked language modeling, obtaining [base α ]. now the language representations of this finetuned model will be (slightly) biased towards the transfer language. last, in step 3 we reinsert the task adapter in both the finetuned and the original pretrained model, and use both models to test and evaluate on target language data β. the difference in performance between these two models score(β ; [base+t] α )score(β ; @xcite ) will reveal whether transfer language α benefits (if positive) or hurts (if negative) target language β. an obvious caveat of our approach so far is that a single update (or 10 or 100) with a randomly sampled batch in any language does not allow for any robust conclusions. to avoid this issue, we repeat the above process @xmath0. our final transfer score ts(α → β ; base, t) for a given model base and task t turns the difference of the finetuned and original model into a percentage of the original baseline performance, for fairer comparisons at different levels of performance: ts(α → β ; base, t) = 10 1 score(β ;[base+t] α ) n score(β ; [base+t]) score(β ; @xcite ) negative interference the typical definition of negative interference describes it as the phe-nomenon when batches in different languages produce opposite gradients during training. we instead focus on downstream performance, in line with most studies focusing on cross-lingual transfer, assuming that a negative effect on performance implies negative interference. another reason is that, in n dimensional spaces, there extremely high probability of two random vectors being orthogonal; hence any two gradient vectors could certainly be orthogonal without necessarily impacting downstream performance. to quantify negative interference, we follow a modular-based approach depicted in figure 1(right). like before, we separate the task and language, followed by performing interaction among multiple languages. however, we use language adapters at this time instead of continuously finetuning the base model. this strategy allows us to efficiently train multiple language sub-parts only once (step2) followed by mixing those modules through adapter fusion @xcite . in our experiments, we train a set of language adapters and make either monolingual settings or a combination of bilingual/trilingual interactions (step3). then we stack previously trained task adapter while only changing the underlying language combination. finally, we extract the interference score from the difference between already computed multilingual and monolingual counterparts (step4). having these interference scores at hand, we can tell whether a language actually gets benefits or not while influencing the associated languages in a positive/negative manner. for example, consider language a interacting with language b. we can easily quantify the interference of language a by calculating the loss/gain of this bilingual interaction [ab]: a score increase for a compared to its monolingual counterpart (i.e. +@xmath1. we can further extend this to a trilingual setting as well (i.e. +@xmath2. using these scores, we can get different combinations of interference scenarios by counting the co-occurred positive/negative interference. we use | + a, +b| to denote the number of cases where a benefits both itself and b, presenting all possible rules in moreover, we can use these interference combination counts to project languages in an interference representation space. for example, consider a 2-d space of bilingual interaction where the x-axis represents the negative/positive interference a language receives from one such interaction and the y-axis is for the interference it provides to other languages. we can project a language using the dot product of counts (eg. | + a, -b|) with its corresponding quadrant identifier [1, -1]. as a result, the projection coordinates (x a , y a ) for language a in a bilingual interaction could be obtained as follows: using the above-mentioned projections, we visualize a language in a way that represents how much interference it provides as well as receives (see example with each step of the calculation in appendix §f). we can further extend this strategy to the trilingual setting, but now we have to deal with eight axes instead of four. in figure 4 of the result section, we present the language interaction visualizations for bilingual and trilingual scenarios. ## experimental setup we conduct our experiments in two different settings targeted to perform two different analyses: first understanding the language effect on crosslingual transfer and then, extending this to quantify language-language interaction. primarily, we use multilingual bert as our base model and report xlm-r results for comparative model evaluation. we use a total of 38 transfer languages (11 unseen during pretraining) to finetune the mlm using masked language modeling with the process described above. using these transfer languages, we do monolingual finetuning on mbert for either 1, 10, 100, or 1000 steps and each experiment is repeated for 10 times. at the sametime, we trained multilingual task adapters followed by task evaluation on the following tasks: • sentence-level: natural language inference (nli) evaluated on xnli @xcite and americasnli (anli) @xcite datasets. ## results and discussion first, in 4.1, we present a comparative scenario in between continuous training and language interaction in terms of performance improvement over the baseline model. then in 4.2, we discuss the findings of continuous training in the context of cross-lingual transfer. after that, in 4.3, we present the representation of language interactions as well as interference following the strategy discussed in section 2. ## recommendations based on our above findings, we make a number of recommendations in choosing the appropriate transfer language and training scheme for a lowresource setting. 1. there is no universal donor but having multiple transfer languages in the training scheme helps in terms of language interference. 2. for universal recipient languages (eg. typologically diverse unseen ones), including almost any language in the transfer scheme help. 3. low resource unseen languages generally transfer with high variance. a good idea is to include them with other seen languages in the transfer scheme to stabilize the transfer output across a large number of target languages. 4. only some of the unseen low-resource ones show sustained transfer toward other low-resource languages through continuous thousand-step training. usually, the deviation happens during an early stage of training. so just continuing pretraining for longer is not optimal for a scenario with mixed-category languages. 5. the patterns of receiving positive transfer are similar when we use either one language small-(1) bilingual interactions [ab] ## conclusion and future work we devise an efficient approach to study crosslingual transfer in multilingual models for various tasks that disentangles task and language effects. we believe this disentanglement coupled with fewstep fine-tuning has the potential to uncover currently uncharted model behaviors (eg. nli evaluation). our findings suggest languages unseen by mlms clearly exhibit different behavioral pattern compared to other languages in general: they are universal as target, exhibit high variance as transfer language, and their behavior follows similar patterns across tasks. in addition, we do not find a universal donor (a language that benefits all others). last, we find that some languages consistently benefit from settings that resemble "catastrophic forgetting" for other languages, an observation we believe merits a dedicated follow-up study. we hope that our approach will allow for further study of cross-lingual transfer for more languages and mlms, and we plan to extend this in future work, as our findings suggest interesting differences in the behavior of languages used in pre-training and unused ones. eventually, we hope that our study will also lead to guidelines for selecting appropriate transfer languages, as well as more informed methods for the adaptation of mlms to new under-served languages. while our proposed approach being highly efficient to expand the paradigm of cross-lingual transfer evaluation, the findings shed light onto the easy adaptation of mlms for new languages in a low-resource setting.