MAD-X : A n A dapter- B ased F ramework for M ulti- T ask C ross- L ingual T ransfer (2020)

Abstract: The main goal behind state-of-the-art pre-trained multilingual models such as multilingual BERT and XLM-R is enabling and bootstrapping NLP applications in low-resource languages through zero-shot or few-shot cross-lingual transfer. However, due to limited model capacity, their transfer performance is the weakest exactly on such low-resource languages and languages unseen during pre-training. We propose MAD-X, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations. In addition, we introduce a novel invertible adapter architecture and a strong baseline method for adapting a pre-trained multilingual model to a new language. MAD-X outperforms the state of the art in cross lingual transfer across a representative set of typologically diverse languages on named entity recognition and causal commonsense reasoning, and achieves competitive results on question answering. Our code and adapters are available at AdapterHub.ml.

## introduction current deep pretrained multilingual models @xcite achieve state-of-the-art results on cross-lingual transfer, but do not have enough capacity to represent all languages. evidence for this is the importance of the vocabulary size @xcite and the curse of multilinguality @xcite , a trade-off between language coverage and model capacity. scaling up a model to cover all of the world's 7,000+ languages is prohibitive. at the same time, limited capacity is an issue even for high-resource languages where state-of-the-art multilingual models underperform their monolingual variants @xcite @xcite , and performance decreases further with lower-resource languages covered by the pretrained models. moreover, the model capacity issue is arguably most severe for languages that were not included in the training data at all, and pretrained models perform poorly on those languages @xcite . in this paper, we propose multiple adapters for cross-lingual transfer (mad-x), a modular framework that leverages a small number of extra parameters to address the fundamental capacity issue that limits pretrained multilingual models. using a state-of-the-art multilingual model as the foundation, we adapt the model to arbitrary tasks and languages by learning modular language-and task-specific representations via adapters @xcite , small bottleneck layers inserted between a model's weights. in particular, using a recent efficient adapter variant @xcite , we train 1) language-specific adapter modules via masked language modelling (mlm) on unlabelled target language data, and 2) task-specific adapter modules via optimising a target task on labelled data in any source language. task and language adapters are stacked as in figure 1 , allowing us to adapt the pretrained multilingual model also to languages that are not covered in the model's (pre)training data by substituting the target language adapter at inference. in order to deal with a mismatch between the shared multilingual vocabulary and target language vocabulary, we propose invertible adapters, a new type of adapter that is well suited to performing mlm in another language. our framework goes beyond prior work on using adapters for crosslingual transfer @xcite by enabling adaptation to languages unseen during pretraining and without learning expensive language-specific token-level embeddings. we compare mad-x against state-of-the-art cross-lingual transfer methods on the standard wikiann ner dataset @xcite and the xcopa dataset @xcite for causal commonsense reasoning, relying on a representative set of typologically diverse languages which includes high-resource, lowresource, as well as languages unseen by the pretrained model. mad-x outperforms the baselines on seen and unseen high-resource and low-resource languages. on the high-resource languages of the challenging xquad qa dataset @xcite , our framework achieves competitive performance while being more parameter-efficient. another contribution of our work is a simple method of adapting a pretrained multilingual model to a new language, which outperforms the standard setting of transferring a model only from labelled source language data. in sum, our contributions are as follows. 1) we propose mad-x, a modular framework that mitigates the curse of multilinguality and adapts a multilingual model to arbitrary tasks and languages. both code and adapter weights are integrated into the adapterhub.ml repository @xcite . 1 2) we propose invertible adapters, a new adapter variant for cross-lingual mlm. 3) we demonstrate strong performance and robustness of mad-x across diverse languages and tasks. 4) we propose a simple and more effective baseline method for adapting a pretrained multilingual model to target languages. 5) we shed light on the behaviour of current methods on languages that are unseen during multilingual pretraining. adapters adapter modules have been originally studied in computer vision tasks where they have been restricted to convolutions and used to adapt a model for multiple domains @xcite . in nlp, adapters have been mainly used for parameter-efficient and quick fine-tuning of a base pretrained transformer model to new tasks @xcite and new domains @xcite , avoiding catastrophic forgetting @xcite . @xcite also use adapters to fine-tune and recover performance of a multilingual nmt model on highresource languages, but their approach cannot be applied to languages that were not seen during pretraining. @xcite employ adapters to transfer a pretrained monolingual model to an unseen language but rely on learning new token-level embeddings, which do not scale to a large number of languages. @xcite combine the information stored in multiple adapters for more robust transfer learning between monolingual tasks. in their contemporaneous work, @xcite generate adapter parameters from language embeddings for multilingual dependency parsing. ## multilingual model adaptation for cross-lingual transfer standard transfer setup the standard way of performing cross-lingual transfer with a state-ofthe-art large multilingual model such as multilingual bert or xlm-r is 1) to fine-tune it on labelled data of a downstream task in a source language and then 2) apply it directly to perform inference in a target language @xcite . a downside of this setting is that the multilingual initialisation balances many languages. it is thus not suited to excel at a specific language at inference time. we propose a simple method to ameliorate this issue by allowing the model to additionally adapt to the particular target language. target language adaptation similar to finetuning monolingual models on the task domain @xcite , we propose to finetune a pretrained multilingual model via mlm on unlabelled data of the target language prior to taskspecific fine-tuning in the source language. a disadvantage of this approach is that it no longer allows us to evaluate the same model on multiple target languages as it biases the model to a specific target language. however, this approach might be preferable if we only care about performance in a specific (i.e., fixed) target language. we find that target language adaptation results in improved cross-lingual transfer performance over the standard setting ( §6). in other words, it does not result in catastrophic forgetting of the multilingual knowledge already available in the pretrained model that enables the model to transfer to other languages. in fact, experimenting with methods that explicitly try to prevent catastrophic forgetting @xcite led to worse performance in our experiments. nevertheless, the proposed simple adaptation method inherits the fundamental limitation of the pretrained multilingual model and the standard transfer setup: the model's limited capacity hinders effective adaptation to low-resource and unseen languages. in addition, fine-tuning the full model does not scale well to many tasks or languages. ## adapters for cross-lingual transfer our mad-x framework addresses these deficiencies and can be used to effectively adapt an existing pretrained multilingual model to other languages. the framework comprises three types of adapters: language, task, and invertible adapters. as in previous work @xcite , adapters are trained while keeping the parameters of the pretrained multilingual model fixed. our framework thus enables learning language and task-specific transformations in a modular and parameter-efficient way. we show the full framework as part of a standard transformer model in figure 1 and describe the three adapter types. ## experiments data we conduct experiments on three tasks: named entity recognition (ner), question answering (qa), and causal commonsense reasoning (ccr). for ner, we use the wikiann @xcite dataset, which was partitioned into train, development, and test portions by @xcite . for qa, we employ the xquad dataset @xcite , a cross-lingual version of squad @xcite . for ccr, we rely on xcopa @xcite , a crosslingual version of copa @xcite . of respective wikipedia sizes); b) their presence in pretrained multilingual models; more precisely, whether data in the particular language was included in the pretraining data of both multilingual bert and xlm-r or not; and c) typological diversity to ensure that different language types and families are covered. in total, we can discern four categories in our language set: 1) high-resource languages and 2) low-resource languages covered by the pretrained sota multilingual models (i.e., by mbert and xlm-r); as well as 3) low-resource languages and 4) truly low-resource languages not covered by the multilingual models. we select four languages from different language families for each category. we highlight characteristics of the 16 languages from 11 language families in we evaluate on all possible language pairs (i.e., on the cartesian product), using each language as a source language with every other language (including itself) as a target language. this subsumes both the standard zero-shot cross-lingual transfer setting @xcite as well as the standard monolingual in-language setting. for ccr and qa, we evaluate on the 12 and 11 languages provided in xcopa and xquad respectively, with english as source language. xcopa contains a typologically diverse selection of languages including two languages (haitian creole and quechua) that are unseen by our main model. xquad comprises slightly less typologically diverse languages that are mainly high-resource. ## results and discussion named entity recognition as our main summary of results, we average the cross-lingual transfer results of each method for each source language across all 16 target languages on the ner dataset. we show the aggregated results in in general, we observe that xlm-r performance is indeed lowest for unseen languages (the right half of the table after the vertical dashed line). xlm-r base mlm-src performs worse than xlm-r, which indicates that source-language finetuning is not useful for cross-lingual transfer in general. 5 on the other hand, xlm-r base mlm-trg is a stronger transfer method than xlm-r on average, yielding gains in 9/16 target languages. however, its gains seem to vanish for low-resource languages. further, there is another disadvantage, outlined in §3: xlm-r base mlm-trg requires fine-tuning the full large pretrained model separately for each target language in consideration, which can be prohibitively expensive. mad-x without language and invertible adapters performs on par with xlm-r for almost all languages present in the pretraining data (left half of the table). this mirrors findings in the monolingual setting where task adapters have been observed to achieve performance similar to regular fine-tuning while being more parameter-efficient @xcite . however, looking at unseen languages, the performance of mad-x that only uses task adapters deteriorates significantly compared to xlm-r. this shows that task adapters alone are not expressive enough to bridge the discrepancy when adapting to an unseen language. adding language adapters to mad-x improves its performance across the board, and their usefulness is especially pronounced for low-resource languages. language adapters help capture the characteristics of the target language and consequently provide boosts for unseen languages. even for high-resource languages, the addition of language-specific parameters yields substantial improvements. finally, invertible adapters provide further gains and generally outperform only using task and language adapters: for instance, we observe gains with mad-x over mad-x -inv on 13/16 target languages. overall, the full mad-x framework improves upon xlm-r by more than 5 f 1 points on average. 5 however, there are some examples (e.g., ja, tk) where it does yield slight gains over the standard xlm-r transfer. to demonstrate that our framework is modelagnostic, we also employ two other strong multilingual models, xlm-r large and mbert as foundation for mad-x and show the results in for a more fine-grained impression of the performance of mad-x in different languages, we show its relative performance against xlm-r in the standard setting in figure 3 . we observe the largest differences in performance when transferring from high-resource to low-resource and unseen languages (top-right quadrant of figure 3 ), which is arguably the most natural setup for crosslingual transfer. in particular, we observe strong gains when transferring from arabic, whose script might not be well represented in xlm-r's vocabulary. we also detect strong performance in the in-language monolingual setting (diagonal) for the subset of low-resource languages. this indicates that mad-x may help bridge the perceived weakness of multilingual versus monolingual models. finally, mad-x performs competitively even when the target language is high-resource. 6 causal commonsense reasoning we show results on transferring from english to each target language on xcopa in ## further analysis impact of invertible adapters we also analyse the relative performance difference of mad-x model + params % model mad-x base 8.25m 3.05 mad-x base -inv 7.96m 2.94 mad-x base -lad -inv 0.88m 0.32 with and without invertible adapters for each source language-target language pair on the ner data set (see section d in the appendix). invertible adapters improve performance for many transfer pairs, and particularly when transferring to lowresource languages. performance is only consistently lower with a single low-resource language as source (maori), likely due to variation in the data. ## conclusion we have proposed mad-x, a general modular framework for transfer across tasks and languages. it leverages a small number of additional parameters to mitigate the capacity issue which fundamentally hinders current multilingual models. mad-x is model-agnostic and can be adapted to any current pre-trained multilingual model as foundation. we have shown that it is particularly useful for adapting to languages not covered by the multilingual model's training model, while also achieving competitive performance on high-resource languages. in future work, we will apply mad-x to other pre-trained models, and employ adapters that are particularly suited for languages with certain properties (e.g. with different scripts). we will also evaluate on additional tasks, and investigate leveraging pre-trained language adapters from related languages for improved transfer to truly low-resource languages with limited monolingual data. lad -inv), 9 (mad-x base -inv), 10 (mad-x base ), 11 (mbert), 12 (mad-x mbert ) , 13 (xlm-r large ), and 14 (mad-x mbert ). each score is averaged over five runs.