Investigating the Potential of Task Arithmetic for Cross-Lingual Transfer (2024)

Abstract: Cross-lingual transfer has recently been tackled through modular, parameter-efficient fine-tuning methods which allow arbitrary combinations of language and task modules for transfer of any task to any language. Concurrently, task arithmetic has emerged as a powerful and modular tool for editing pretrained models using multiple full fine-tunings. In this work, we connect the paradigms of task arithmetic and cross-lingual transfer, demonstrating that modularity for cross-lingual transfer can be achieved even with full model fine-tuning. Our approach displays strong performance on a range of multilingual benchmarks encompassing both high-resource and low-resource languages.

## introduction massively multilingual transformer-based models (mmts) @xcite @xcite @xcite have shown impressive performance in cross-lingual transfer due to their ability to learn representations which have a degree of cross-lingual alignment, despite being trained using purely unsupervised objectives (e.g., masked language modeling). this allows an mmt to perform a task in a target language having seen labeled data only in a source language: the so-called zero-shot cross-lingual transfer (zs-xlt). the adaptation of mmts to low-resource languages has been an attractive research area lately, stemming from a need to extend such models to under-represented and unseen languages @xcite @xcite . a particularly popular approach is based on modular and parameter-efficient (peft) adaptation of mmts to particular languages and tasks, generally leading to improved zs-xlt @xcite @xcite @xcite . while the peft methods are typically designed to facilitate modularity and module (re)combination, full fine-tuning appears to exhibit less flexibility in this regard. this has led to the development of techniques for merging multiple fine-tuned models @xcite @xcite . one prominent approach to model merging is based on the so-called task arithmetic: @xcite have proposed editing monolingual and vision models using task vectors derived by subtracting the weights of the pretrained model from those of the fully fine-tuned model. several such vectors can then be applied to the model through arithmetic operations such as addition and subtraction to steer its behaviour in a controlled manner @xcite . in this work, we shed new light on the ability to maintain modularity even for fully fine-tuned mmts in the context of zs-xlt: we delve into the potential of full fine-tuning and task arithmetic for zs-xlt. more precisely, starting from a pretrained mmt, we independently acquire language and task vectors, by fine-tuning the mmt on the language and task data, respectively. these vectors are subsequently combined with mmt through addition or subtraction to obtain the resulting, adapted model tailored for a specific language-task pair in a fully modular fashion. we extensively evaluate several promising variants of combining task and language vectors across a spectrum of multilingual benchmarks, encompassing both highresource and low-resource languages. our findings underscore the potency of task arithmetic for crosslingual transfer and language adaptation, yielding notable performance gains over fully fine-tuned mmts without task arithmetic and other strong zs-xlt baselines, particularly prominent on benchmarks featuring low-resource languages. our code is available at https://github.com/parovicm/ task-arithmetic . ## methodology background and motivation. prior work demonstrated that models which share a portion of the optimization path, typically through a common initialization, can be merged into a single model using weight interpolation while maintaining task accuracy @xcite @xcite . @xcite find that models trained on the same data or on different datasets of the same task tend to cluster together in the weight space. @xcite leverage the task arithmetic to address the challenges of hallucination within dialogue systems. they additionally employ fisher information to weigh the importance of the parameters @xcite participating in the arithmetic. inspired by the previous work on model merging in general and task arithmetic in particular, here we investigate its potential and benefits for modular zs-xlt. task arithmetic: preliminaries. given a pretrained model with the parameters θ 0 ∈ r d and the designated task t , the task-specific parameters θ @xmath0. the task vector of t , denoted by τ @xmath1, is defined as the difference in parameters before and after fine-tuning: τ @xmath2. this vector characterizes the direction in the model's weight space, such that adjusting the parameters in this direction enhances task performance. the acquired task vector can be integrated into the model by a simple addition and an optional scaling factor λ ∈ r governing its influence, yielding a new model with the following parameters: note that when λ = 1, then θ ′ = θ t . adding a task vector (λ > 0) has the effect of promoting a certain 'model behaviour', while subtracting it (λ < 0) 'suppresses' it. in a more general scenario, given n task vectors τ t 1 , ..., τ @xmath3, ..., λ @xmath4, their application to the model yields the following: ## experiments and results tasks and languages. we extensively evaluate our method on two classification tasks and four different datasets: 1) natural language inference (nli) with (a) xnli @xcite covering 14 high-resource and mid-resource languages, and (b) americasnli @xcite spanning 10 low-resource languages from the americas; 2) sentiment classification (sa) with marc @xcite pretrained mmt models. our primary mmt is xlm-r base @xcite , and we also run a subset of experiments with xlm-r large . language vectors are trained on unlabelled data of each language, primarily following the hyperparameters outlined in @xcite . details regarding the used monolingual corpora are provided in appendix a. we train for 50, 000 steps (20,000 steps with xlm-r large ), a batch size is 64, a learning rate is 5 • 10 -5 and a maximum sequence length is set to 256. we select the checkpoint that yields the lowest validation perplexity as the final language vector. task vectors are trained on the corresponding task dataset in the source language (english for xnli, americasnli, and marc; indonesian for nusax). the dataset used for obtaining the task vector for both xnli and americasnli is multinli @xcite . further details about the datasets and tasks are given in appendix b. the hyperparameters are in in the -src+tgt and +src+tgt variants, we use λ @xmath5. following ilharco et al. (2023), we consider scaling factors from the set {0.1, 0.2, ..., 1.0} and choose the one with the highest average performance on the corresponding validation data. the scaling coefficients reaching the best performance are summarized in appendix e. baselines. beyond comparing to the fully finetuned model in all tasks, we compare our models against two strong zs-xlt methods: 1) sparse fine-tuning (sft) for cross-lingual transfer @xcite on americasnli and nusax, and 2) target language-ready (tlr) adapters @xcite on americasnli, which both showed superiority over other established zs-xlt variants with language adaptation such as mad-x (pfeiffer et al., 2020) in those tasks. 3 note that these methods were created with the specific goal of enhancing zs-xlt performance. our primary goal, however, is to gain insight into the interaction between the task arithmetic and cross-lingual transfer. the scores of these baselines are inherited from prior work @xcite . we refrained from conducting experiments with these baselines on the xnli and marc datasets mainly for the following reasons: 1) these methods are tailored to low-resource languages, and exhibit the highest performance in such contexts, while xnli and marc feature high-resource languages; 2) the contributions of this paper do not hinge on direct comparisons with them. instead, we position the task fine-tuned model as our principal baseline, and our goal lies in highlighting the effectiveness of language and task vector compositions relative to a simple task fine-tuning; 3) it is computationally expensive to train language modules for many languages which is necessary in these baselines. ## conclusion we proposed the adoption of task arithmetic in the context of zero-shot cross-lingual transfer, investigating its potential for these transfer scenarios. our approach involves independently creating and combining language and task vectors to attain models customized for specific language-task pairings. we empirically demonstrated the effectiveness of this technique across various multilingual benchmarks.