Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022)

Abstract: Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We address this issue by introducing language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. In contrast with prior work that learns language-specific components post-hoc, we pre-train the modules of our Cross-lingual Modular (X-Mod) models from the start. Our experiments on natural language inference, named entity recognition and question answering show that our approach not only mitigates the negative interference between languages, but also enables positive transfer, resulting in improved monolingual and cross-lingual performance. Furthermore, our approach enables adding languages post-hoc with no measurable drop in performance, no longer limiting the model usage to the set of pre-trained languages.

## introduction recent work on multilingual nlp has focused on pre-training transformer-based models @xcite on concatenated corpora of a large number of languages @xcite . these multilingual models have been shown to work surprisingly well in cross-lingual settings, despite the fact that they do not rely on direct cross-lingual supervision (e.g., parallel data or translation dictionaries; @xcite @xcite @xcite . however, recent work has uncovered fundamental limitations of multilingual transformers. @xcite observe that pre-training a model with a fixed capacity on an increasing amount of languages only improves its cross-lingual performance up to a certain point, after which performance drops can be measured-a phenomenon known as the curse of multilinguality (figure 2 ). as such, prior work had to find a trade-off between supporting more languages and obtaining better performance on a smaller set of languages. in this work, we address this problem by introducing language-specific, modular components during pre-training (figure 1 ). our cross-lingual, modular (x-mod) language model shares the majority of the transformer parameters between all pretraining languages, while providing each language with individual capacity to learn idiosyncratic information without increasing the total number of trainable parameters per language. while previous adapter-based approaches (figure 3a ) extend pretrained multilingual language models (lms) with modular components after pre-training, we add modular components during pre-training, thereby preparing the model to be extended to new languages post-hoc. our experiments on natural language inference (nli), named entity recognition (ner), and question answering (qa) demonstrate that our modular architecture not only is effective at mitigating interference between languages, but also achieves positive transfer, resulting in improved monolingual and cross-lingual performance. in addition, we show that x-mod can be extended to unseen languages, with no measurable drop in performance, by learning its corresponding modules and leaving the shared parameters frozen. all in all, we propose a multilingual architecture that can scale to a large number of languages without any loss in performance, and can be further extended to new languages after pre-training. 1 2 background and related work we provide a background on multilingual and modular language modelling, as well as approaches that extend lms to new languages. ## proposed approach we propose x-mod, a modular multilingual architecture that combines shared and language-specific parameters. in contrast to prior work, we pretrain modular models from the get-go. our models can be extended to new languages after pretraining, and used for cross-lingual transfer learning in downstream tasks. architecture. as illustrated in figure 1 , we extend the transformer-based architecture from mbert @xcite and xlm-r (conneau et al., 2020) by incorporating languagespecific modules-bottleneck feed-forward layersat every transformer layer. we learn a separate module for each language, whereas the attention and feed-forward components are shared. while the total number of parameters of the model grows linearly with the number of languages, the training and inference cost does not increase (as measured in flops), as only the module in the relevant language is used for each input. inspired by the adapter foot_1 architecture of @xcite we place our 'modules' after the layernorm of the feed-forward transformer block, and the residual connection is placed after the layernorm; 4 the lay-ernorm before and after the modular component is shared. 5 pre-training procedure. similar to @xcite , we pre-train our model on mlm on combined monolingual corpora in multiple languages. examples of each language are passed through the shared embedding matrix as well as the multihead attention and feed-forward components at each layer. as each layer contains a languagespecific modular component, the examples are routed through the respective designated modular bottleneck layer. given that each example only requires access to a single module, modules can be efficiently stored on only a subset of gpus in distributed training. extending to new languages. the modular design of our model allows us to extend it to new languages after pre-training. to that end, we learn new embeddings and adapter modules for the target language through mlm, while the rest of the components are frozen. 6 consequently, we are able to extend the model to a new language by learning a small number of new parameters, without affecting performance in the set of pre-trained languages. following @xcite , we learn a new subword vocabulary for the added languages, and initialize the embeddings of lexically overlapping tokens from the original embedding matrix. fine-tuning on downstream tasks. to transfer the models to cross-lingual downstream tasks, we fine-tune the shared weights only on the source language data, while keeping the modular components and the embedding layer frozen. we follow the standard fine-tuning procedure of adding a prediction head on top of the cls token. we then replace the source language modules (as well as embedding layer for added languages) with the target language parameters, passing the text of the target language through the model. 7 representations of the pre-trained mode; we train these modular components together with the transformer weights, and therefore refer to them as modules. 4 we find that the residual connection proposed by @xcite results in training instabilities when trained together with the transformer weights. ## experimental design we detail the baseline and models ( §4.1), and their training ( §4.2) and evaluation settings ( §4.3). ## results and discussion we present results for pre-trained languages in §5.1 and added languages in §5.2. ## further analysis we further analyze the impact of the number of update steps on x-mod ( §6.1) and compare our method to adapter-based approaches ( §6.2). ## conclusions in this paper, we have evaluated the effectiveness of modular multilingual language modelling across multiple axes. we have demonstrated that by providing additional per-language capacity, while maintaining the total number of trainable parameters per language, we are not only able to mitigate negative interference between languages, but additionally achieve positive transfer. our results suggest that it is sufficient to train our proposed x-mod model only on a subset of languages for which sufficient amounts of textual data is avail17 we follow @xcite and train adapter weights with a learning rate of 0.0001. while they have found that cross-lingual transfer performance of adapters converges at ∼20k update-steps, we would like to stress that our experimental setup is only one of multiple different valid versions. a more thorough investigation to find the optimal number of update steps for pre-training and subsequent adapter training is necessary, which was out of scope for this work. while in this work we have simulated language adding scenarios with a held out set of languages, in future work we aim to evaluate the performance on truly low-resource languages such as masakhaner @xcite and americasnli @xcite . we further aim to evaluate the crosslingual transfer performance from typologically more diverse source languages, besides english.