MAD - G : M ultilingual Adapter Generation for Efficient Cross-Lingual Transfer (2021)

Abstract: Adapter modules have emerged as a general parameter-efficient means to specialize a pretrained encoder to new domains. Massively multilingual transformers (MMTs) have particularly benefited from additional training of language-specific adapters. However, this approach is not viable for the vast majority of languages, due to limitations in their corpus size or compute budgets. In this work, we propose MAD-G (Multilingual ADapter Generation), which contextually generates language adapters from language representations based on typological features. In contrast to prior work, our time- and space-efficient MAD-G approach enables (1) sharing of linguistic knowledge across languages and (2) zero-shot inference by generating language adapters for unseen languages. We thoroughly evaluate MAD-G in zero-shot cross-lingual transfer on part-of-speech tagging, dependency parsing, and named entity recognition. While offering (1) improved fine-tuning efficiency (by a factor of around 50 in our experiments), (2) a smaller parameter budget, and (3) increased language coverage, MAD-G remains competitive with more expensive methods for language-specific adapter training across the board. Moreover, it offers substantial benefits for low-resource languages, particularly on the NER task in low-resource African languages. Finally, we demonstrate that MAD-G’s transfer performance can be further improved via: (i) multi-source training, i.e., by generating and combining adapters of multiple languages with available task-specific training data; and (ii) by further fine-tuning generated MAD-G adapters for languages with monolingual data.

## introduction multilingual nlp has witnessed large advances, with cross-lingual word embedding spaces @xcite @xcite and, more recently, massively multilingual transformers (mmts) like mbert @xcite , xlm-r @xcite , and mt5 @xcite as main vehicles of cross-lingual transfer. although mmts display impressive (zero-shot) cross-lingual transfer abilities @xcite , their performance has been shown to drop when the target language is typologically distant to the source language, or the size of its pretraining data is limited @xcite . in addition, their coverage of the world's languages-and consequently the range of language technology applications they can support-remains insufficient. 1 adapters @xcite have been proposed as a parameter-efficient means to extend multilingual models to underrepresented languages @xcite . the general practice is to train a language adapter on the unlabeled data for each language @xcite via masked language modeling (mlm). however, this generally requires substantial amounts of monolingual data, which prevents adapters from serving under-resourced languages where such additional language-specific capacity would be most useful. to address this deficiency, we propose multilingual adapter generation (mad-g), a novel paradigm that enables the generation of adapters for low-resource languages by sharing information across languages. instead of learning separate adapters for each language, mad-g leverages contextual parameter generation @xcite @xcite , that is, it learns a single model that can generate a language adapter for an arbitrary target language. at the core of mad-g is a contextual parameter generator which mad-g training: the generator component learns to generate language-specific adapters given uriel vectors of input languages; the parameters of the generator are trained with an mlm objective, where instances of the respective language are passed through the frozen transformer layers and the generated adapter parameters. in the downstream task fine-tuning, both the transformer weights as well as the weights of the generated source-language adapter are frozen; an additional task adapter with randomly initialized weights is placed on top of the generated source language adapter. during target language downstream inference, the generated source language adapters are replaced with the generated target language adapters. takes the typological vector of a language as input and outputs the parameters of the language-specific adapter. the generator's parameters are trained via mlm on the wikipedias of 95 languages, selected to maximize linguistic diversity. unlike prior cpg work @xcite , mad-g generates language adapters that are taskagnostic, thus allowing for an efficient and modular cross-lingual transfer across the board, i.e., the mad-g language adapters can be leveraged in arbitrary downstream tasks @xcite . mad-g shares information across languages (i) at the level of hidden representations by sharing the parameters of the adapter generator as well as (ii) at the typological level by conditioning on features from the uriel database @xcite . the latter additionally enables zero-shot transfer to unseen languages. further, we propose a variant of mad-g in which we generate adapters also conditioned on their transformer layer position (see section 3.2), allowing mad-g to be much more parameter-efficient than adapter-based transfer methods of prior work. in experiments on zero-shot cross-lingual trans-fer on part-of-speech tagging (pos), dependency parsing (dp), and named entity recognition (ner), mad-g demonstrates competitive performance to training more expensive language-specific adapters and shows strong performance in low-resource scenarios, e.g., in the ner task for african languages. what is more, we show that transfer performance can be further improved by (a) multilingual training of task adapters and (b) fine-tuning of generated mad-g adapters, via mlm, on small amounts of monolingual data. finally, we provide a nuanced analysis of transfer performance to unseen languages, highlighting the importance of the diversity of the language sample selected for pretraining. ## background before introducing mad-g in detail in section 3, we recapitulate its key components adopted from previous work. in particular, we discuss language adapters (la) in section 2.1 and contextual parameter generation (cpg) in section 2.2. ## mad-g: methodology mad-g aims to enable resource-efficient adaptation of mmts to a wide range of previously unseen, radically resource-poor languages, 6 and contribute in this manner to more sustainable @xcite and more inclusive nlp @xcite . we couple (i) the computational efficiency of the light-weight adapters (cf. section 2.1) and (ii) knowledge sharing and zero-shot language transfer capabilities of cpg (cf. section 2.2), with (iii) external linguistic (i.e., typological) knowledge @xcite towards supporting arbitrary nlp tasks for (even radically) resource-poor languages. mad-g mitigates important limitations of prior work. unlike @xcite , we generate taskagnostic las, (re)usable across nlp tasks. unlike the mad-x framework @xcite , which trains las independently for each language (requiring sufficient monolingual corpora), mad-g can support unseen and resource-poor languages in downstream tasks by generating las from typological vectors. moreover, mad-g leverages typological relations between languages. we also show that the two approaches can be successfully combined: monolingual mlm fine-tuning of a mad-g-generated la yields further benefits. ## experimental setup tasks and languages. we evaluate on three downstream tasks which provide sufficient evaluation data for low-resource languages: part-ofspeech (pos) tagging, dependency parsing (dp), both on the universal dependencies (ud) 2.7 dataset @xcite , and named entity recognition (ner) on the masakhaner dataset for african languages @xcite . for pos and dp, we evaluate on a substantial subset of all ud languages with available treebanks. 8 we discern between three language groups in evaluation, with some examples in ## results and discussion in what follows, we focus on reporting and analyzing the most important global trends in results with accompanying discussions and side experiments. for completeness, the full results per individual target language are provided in the appendix. single-source transfer. relative to all methods which do not employ language adaptation, we find that the use of mad-g in the primary mad-g and mad-g-ls settings is greatly beneficial on all tasks for mad-g-seen languages in both the single-and multi-source transfer scenarios (see tables 2 and 3), with the very parameter-efficient mad-g-ls being only slightly weaker than the base mad-g variant in general, even slightly outperforming it for some languages and transfer setups. despite having far less capacity per target language, mad-g retains much of the performance gain of mad-x on languages seen during language adapter training, showing that mad-g the use of mad-g is not in general beneficial for mbert-seen languages; this is unsurprising since it is unrealistic to believe that mbert's knowledge of languages observed during its own pretraining can be substantially improved through language adaptation on a much smaller amount of data. at first glance there also does not appear to be any benefit to using mad-g for unseen target languages, except for ner, where gains are substantial. however, averaging the results over all languages in this group does not provide a full picture because it consists of languages whose relationships to those observed during training differ substantially. therefore, we provide a finer-grained analysis below. while the use of typological vectors for generating las allows mad-g to learn features which could generalize well to unseen languages, this assumption should mostly hold for unseen languages whose 'typological relatives' are available during training. to investigate the effect the degree of typological relatedness has on mad-g's generalization ability, we further divide the unseen languages into three subgroups: mbert-genus (the 21 languages whose genus matches that of at least one language seen during mbert pretraining); mad-g-genus (the 4 languages whose genus was not seen during mbert pretraining but was seen during mad-g training); unseen-genus (the 8 languages whose genus is completely unseen). multi-source transfer. when training on 20 languages, while maintaining the overall number of training examples, we observe large gains across all settings and language groups for both pos tagging and dp (see fine-tuning mad-g-initialized adapters. although interesting from a theoretical point of view, the scenario where there is no unannotated data whatsoever available for the target language might be unrealistic. we thus examine a setup where there is a small amount of unannotated data available. in this case, we can still exploit mad-g by generating an initialization of a language-specific adapter for a target language l t , and then fine-tuning its parameters via mlm on the unannotated data. we perform pos tagging and dp experiments when fine-tuning mad-g-initialized languagespecific adapters on the 14 unseen ud languages which have wikipedias. 14 we simulate different degrees of resource-poverty by sampling training datasets with 1,000, 3,000, 10,000, 30,000 and 100,000 words from the full wikipedia. we compare this mad-g-ft setting with the results of finetuning randomly-initialized las on the same data (rand-ft). 15 figure 3 shows that there is a large and consistent improvement on the 14 unseen evaluation languages as their language adapters are fine-tuned on increasingly large amounts of unannotated text. for both tasks, the performance is better when the language adapter is initialized with the weights generated by mad-g than when the weights are randomly initialized. the difference between the two settings is modest for pos tagging, but it is larger for dp and is maintained even when 100,000 training tokens are available. ## conclusion we proposed mad-g, a modular and efficient cross-lingual transfer framework for low-resource languages, that generates task-agnostic adapters for massively multilingual transformers (e.g., mbert) from typological language representations. mad-g performs competitively with a stateof-the-art adapter-based transfer approach mad-x; yet its training is roughly 50 times more efficient per target language. mad-g can also be applied to unseen languages, benefiting those belonging to a genus introduced during its training, and it can be used as a better initialization for "radically lowresource languages"; there, its generated language adapters can be further refined on small amounts of text, improving downstream performance. we further show that cross-lingual performance with adapters can be greatly improved by training on multiple source languages. we release the mad-g code online at: https://github.com/ adapter-hub/adapter-transformers .