Cross-Lingual Transfer with Target Language-Ready Task Adapters (2023)

Abstract: Adapters have emerged as a modular and parameter-efficient approach to (zero-shot) cross-lingual transfer. The established MAD-X framework employs separate language and task adapters which can be arbitrarily combined to perform the transfer of any task to any target language. Subsequently, BAD-X, an extension of the MAD-X framework, achieves improved transfer at the cost of MAD-X’s modularity by creating ‘bilingual’ adapters specific to the source-target language pair. In this work, we aim to take the best of both worlds by (i) fine-tuning *task* adapters adapted to the target language(s) (so-called *‘target language-ready’ (TLR)* adapters) to maintain high transfer performance, but (ii) without sacrificing the highly modular design of MAD-X. The main idea of ‘target language-ready’ adapters is to resolve the training-vs-inference discrepancy of MAD-X: the task adapter ‘sees’ the target language adapter for the very first time during inference, and thus might not be fully compatible with it. We address this mismatch by exposing the task adapter to the target language adapter during training, and empirically validate several variants of the idea: in the simplest form, we alternate between using the source and target language adapters during task adapter training, which can be generalized to cycling over any set of language adapters. We evaluate different TLR-based transfer configurations with varying degrees of generality across a suite of standard cross-lingual benchmarks, and find that the most general (and thus most modular) configuration consistently outperforms MAD-X and BAD-X on most tasks and languages.

## introduction and motivation recent progress in multilingual nlp has mainly been driven by massively multilingual transformer models (mmts) such as mbert @xcite , xlm-r @xcite , and mt5 @xcite , which have been trained on the unlabeled data of 100+ languages. their shared multilingual representation spaces enable zero-shot cross-lingual transfer @xcite , that is, performing tasks with a reasonable degree of accuracy in languages that entirely lack training data for those tasks. zero-shot cross-lingual transfer is typically performed by fine-tuning the pretrained mmt on taskspecific data in a high-resource source language (i.e., typically english), and then applying it directly to make task predictions in the target language. in the standard setup, the model's knowledge about the target language is acquired solely during the pretraining stage @xcite . in order to improve the transfer performance, task fine-tuning can be preceded with fine-tuning on unlabeled data in the target language @xcite . nonetheless, the performance on the target languages in such scenarios is lower than that on the source language, and the difference is known as the cross-lingual transfer gap @xcite . crucially, the transfer gap tends to increase for the languages where such transfer is needed the most @xcite : i.e., for low-resource target languages, and languages typologically more distant from the source language (e.g., english) @xcite . adapters @xcite have emerged as a prominent approach for aiding zero-shot cross-lingual transfer @xcite @xcite . they offer several benefits: (i) providing additional representation capacity for target languages; (ii) much more parameterefficient fine-tuning compared to full-model finetuning, as they allow the large mmt's parameters to remain unmodified, and thus preserve the multilingual knowledge the mmt has acquired during pretraining. they also (iii) provide modularity in learning and storing different facets of knowledge @xcite : this property enables them to be combined in favorable ways to achieve better performance, and previously fine-tuned modules (e.g., language adapters) to be reused across different applications. the established adapter-based cross-lingual transfer framework mad-x @xcite trains separate language adapters (las) and task adapters (tas) which can then be arbitrarily combined for the transfer of any task to any language. despite having a highly modular design, stemming primarily from dedicated per-language and per-task adapters, mad-x's tas lack 'adaptivity' to the target language(s) of interest: i.e., its tas are fully target language-agnostic. more precisely, during task fine-tuning, the mad-x ta is exposed only to the source language la, and 'sees' the target language ta and examples from that language for the first time only at inference. this deficiency might result in incompatibility between the ta and the target la, which would emerge only at inference. bad-x @xcite trades off mad-x's high degree of modularity by introducing 'bilingual' language adapters specialized for transfer between the source-target language pair. 1 while such transfer direction specialization results in a better performance, the decrease in modularity results in much larger computational requirements: bad-x requires fine-tuning a dedicated bilingual la for every language pair of interest followed up by fine-tuning a dedicated ta again for each pair. prior work has not explored whether this specialization (i.e., exposing the target language at training time) can be done successfully solely at the level of tas whilst preserving modularity at the la level. such specialization in the most straightforward bilingual setup still requires fine-tuning a dedicated ta for each target language of interest. however, this is already a more pragmatic setup than bad-x since tas are much less computationally expensive to train than las. moreover, as we show in this work, it is possible to also extend ta fine-tuning to more target languages, moving from bilingual specialization to the more universal multilingual 'exposure' and towards multilingual language-universal tas. in this work, we aim to create a modular design inspired by mad-x while seeking to reap the benefits of the exposure to one or more target languages. to this end, we thus introduce target languageready (tlr) task adapters designed to excel at a particular target language or at a larger set of target languages. in the simplest bilingual variant, tlr tas are trained by alternating between source and target las, while the more general version allows cycling over any set of las. creating tlr tas does not require any expensive retraining or alternative training of las. we run experiments with a plethora of standard benchmarks focused on zero-shot cross-lingual transfer and low-resource languages, covering 1) ner on masakhaner; 2) dependency parsing (dp) on universal dependencies; 3) natural language inference (nli) on americasnli and xnli; 4) qa on xquad and tydiqa-goldp. our results show that tlr tas outperform mad-x and bad-x on all tasks on average, and offer consistent gains across a large majority of the individual target languages. importantly, the most general tlr ta, which is shared between all target languages and thus positively impacts modularity and reusability, shows the strongest performance across the majority of tasks and target languages. fine-tuning the ta in such multilingual setups also acts as a multilingual regularization @xcite : while the ta gets exposed to different target languages (i.e., maintaining its tlr property), at the same time it does not overfit to a single target language as it is forced to adapt to more languages, and thus learns more universal cross-language features. our code and models are publicly available at: https: //github.com/parovicm/tlr-adapters . mad-x and bad-x frameworks. mad-x trains dedicated las and tas @xcite . las are trained using unlabeled wikipedia data with a masked language modeling (mlm) objective. tas are trained using task-specific data in the source language. given a source language l s and a target language l t , mad-x trains las for both l s and l t . the ta is trained while stacked on top of the l s la, which is frozen. to make predictions on l t , the l s la is swapped with the l t la. unlike mad-x, which is based on monolingual adapters, bad-x trains bilingual las @xcite . a bilingual la is trained on the unlabeled data of both l s and l t and the ta is then trained on task-specific data in l s , stacked on top of the bilingual la. to perform inference on the task in l t , the same configuration is kept since the bilingual la 'knows' both l s and l t . ## experimental setup evaluation tasks and languages. we comprehensively evaluate our tlr adapter framework on a suite of standard cross-lingual transfer benchmarks. they span four different task families (ner, dp, nli and qa), with a total of six different datasets and 35 different target languages, covering a typologically and geographically diverse language sample of both low-and high-resource languages. for ner, we use the masakhaner dataset (adelani et al., 2021) which contains 10 low-resource languages from the african continent. 3 for dp, we use universal dependencies 2.7 @xcite and inherit the set of 10 typologically diverse low-resource target languages from bad-x @xcite . for nli, we rely on the americasnli dataset @xcite , containing 10 low-resource languages from the americas, as well as a subset of languages from xnli @xcite . finally, for qa we use subsets of languages from xquad @xcite and tydiqa-goldp @xcite . the subsets for xnli, xquad and tydiqa-goldp were selected to combine (i) low-resource languages @xcite , with (ii) higher-resource languages for which dedicated (i.e., 'mad-x') las were readily available. the full overview of all tasks, datasets, and languages with their language codes is provided in language adapters. we train las for the minimum of 100 epochs or 100,000 steps with a batch size of 8, a learning rate of 5 • 10 -5 and a maximum sequence length of 256. 4 we evaluate the las every 1,000 steps for low-resource languages and every 5,000 steps for high-resource ones, and choose the la that yields the lowest perplexity, evaluated on the 5% of the held-out monolingual data (1% for high-resource languages). for the bad-x baseline, we directly use the bilingual las from @xcite . following @xcite , the adapter reduction factor (i.e., the ratio between mmt's hidden size and the adapter's bottleneck size) is 2 for all las. for the mad-x las, we use the efficient pfeiffer adapter configuration @xcite with invertible adapters, whereas bad-x las do not include them. task adapters. we fine-tune tas by stacking them on top of the corresponding las (see figure 1 ). during their fine-tuning, the mmt's parameters and all the las' parameters are frozen. the adapter reduction factor for all tas is 16 as in prior work @xcite (i.e., @xmath0, and, like the las, they use the pfeiffer configuration. the hyperparameters across different tasks, also borrowed from prior work, are listed in transfer setup: details. in all our transfer experiments, the source language l s is fixed to english, and we evaluate different variants described in §2.2. for the mad-x baseline, we rely on its 'mad-x v2.0' variant, which drops the adapters in the last layer of the transformer, which has been found to improve transfer performance across the board @xcite . for the task-multi tlr variant, along with using the english la, we fine-tune tas using the las of all our evaluation languages in that particular dataset. for instance, for dp this spans 10 languages, while for nli, we fine-tune a separate task-multi tlr with the 10 languages from americasnli, and another one for the xnli languages. for the all-multi tlr variant, in addition to english la, we cycle over the las of all our evaluation languages from all the tasks and datasets. ## results and discussion main results. the main results with mbert for all tasks and all languages are shown in breakdown of results across tasks and tlr variants. on ner and dp we observe very similar trends in results. importantly, the most modular all-multi variant offers the highest performance overall: e.g., it reaches the average f1 score of 69.86% in the ner task, while outperforming mad-x by 1.9% on average and on all 9 target languages. pronounced gains with that variant are also indicated in the dp task. the target and bilingual variants also yield gains across the majority of languages, with bilingual being the stronger of the two. however, their overall utility in comparison to all-multi is lower, given their lower performance coupled with lower modularity. on americasnli, all tlr variants display considerable gains over mad-x, achieving 5-6% higher average accuracy. they outperform mad-x on all 10 target languages, except the task-multi variant with only a slight drop on aym. the best variant is once again the most modular all-multi variant, which is better than the baselines and all the other variants on 6/10 target languages. on xnli, which involves some higher-resource languages such as ar, hi and zh, all tlr variants reach higher average accuracy than mad-x. the gains peak around 5-6% on average; however, this is due mainly to sw where mad-x completely fails, achieving the accuracy of random choice. nonetheless, the tlr variants attain better scores on all other languages as well (the only exception is all-multi on ar). besides sw, th also marks a large boost of up to 11.2% with the bilingual variant, while the other languages attain more modest gains of up to 2%. we remark that the bilingual variant now obtains the highest average accuracy: we speculate that this could be a consequence of target languages now being on the higher-resource end compared to masakhaner and americasnli. our final task family, qa, proves yet again the benefits of transfer with tlr adapters. on xquad and tydiqa-goldp, the best tlr variant is now the target adapter. this might be partially due to a good representation of high-resource languages towards language-universal task adapters? strictly speaking, if a new (k + 1)-th target language is introduced to our proposed tlr framework, it would be necessary to train the multilingual tlr ta anew to expose it to the new target language. in practice, massively multilingual tas could still be applied even to languages 'unseen' during ta fine-tuning (e.g., in the same way as the original mad-x framework does). this violates the tlr assumption, as the ta sees the target language only at inference. however, this setup might empirically validate another desirable property of our multilingual tlr framework from figure 1 : exposing the ta at fine-tuning to a multitude of languages (and their corresponding las) might equip the ta with improved transfer capability even to unseen languages. put simply, the ta will not overfit to a single target language or a small set of languages as it must learn to balance across a large and diverse set of languages; see §2. we thus run experiments on masakhaner, ud dp, and americasnli with two subvariants of the most general all-multi variant. first, in the leave-out-task subvariant, we leave out all the las for the languages from the corresponding task dataset when fine-tuning the ta: e.g., for americ-asnli, that subvariant covers the las of all the languages in all the datasets except those appearing in americasnli, so that all americasnli languages are effectively 'unseen' at fine-tuning. the second subvariant, termed leave-out-targ, leaves out only one language at a time from the corresponding dataset: e.g., when evaluating on guarani (gn) in americasnli, the only language 'unseen' by the ta at fine-tuning is gn as the current inference language. the results, summarized in tables 2( a )-(c), reveal that our multilingual ta fine-tuning indeed increases transfer capability also for the 'taunseen' languages, and leads towards languageuniversal tas. the scores with both subvariants offer substantial gains over mad-x for many languages unseen during fine-tuning and in all three tasks. this confirms that (i) mad-x tas tend to overfit to the source language and thus underperform in cross-lingual transfer, and (ii) such overfitting might get mitigated through our proposed 'multilingual regularization' of the tas while keeping the same modularity benefits. additionally, the results also confirm the versatility of the proposed tlr framework, where strong transfer gains are achieved with different sets of languages included in multilingual ta fine-tuning: e.g., the scores with the two leave-out subvariants remain strong and competitive with the full all-multi variant. for the dp task we even observe slight gains with the leave-out-task variant over the original all-multi variant which 'sees' all task languages. we speculate that this might partially occur due to the phenomenon of 'the curse of multilinguality' @xcite kicking in, now at the level of the limited ta budget, but leave this for further exploration in future work. ## conclusion and future work we have presented a novel general framework for adapter-based cross-lingual task transfer, which improves over previous established adapter-based transfer frameworks such as mad-x and bad-x. the main idea is to better equip task adapters (tas) to handle text instances in a variety of target languages. we have demonstrated that this can be achieved via so-called target language-ready (tlr) task adapters, where we expose the ta to the target language as early as the fine-tuning stage. as another major contribution, we have also proposed a multilingual language-universal tlr ta variant which offers the best trade-off between transfer performance and modularity, learning a single universal ta that can be applied over multiple target languages. our experiments across 6 standard cross-lingual benchmarks spanning 4 different tasks and a wide spectrum of languages have validated the considerable benefits of the proposed framework and different transfer variants emerging from it. crucially, the most modular multilingual tlr ta variant offers the strongest performance overall, and it also generalizes well even to target languages 'unseen' during ta fine-tuning. in future work, we plan to further investigate multilingual language-universal task adapters also in multi-task and multi-domain setups, and extend the focus from serial adapters to other adapter architectures, such as parallel adapters @xcite and sparse subnetworks @xcite .