Larger-Scale Transformers for Multilingual Masked Language Modeling (2021)

Abstract: Recent work has demonstrated the effectiveness of cross-lingual language model pretraining for cross-lingual understanding. In this study, we present the results of two larger multilingual masked language models, with 3.5B and 10.7B parameters. Our two new models dubbed and outperform XLM-R by 1.8% and 2.4% average accuracy on XNLI. Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0.3% on average while handling 99 more languages. This suggests larger capacity models for language understanding may obtain strong performance on high-resource languages while greatly improving low-resource languages. We make our code and models publicly available.

## introduction the goal of this paper is to present a study of the impact of larger capacity models on crosslingual language understanding (xlu). we scale the capacity of xlm-r by almost two orders of magnitude while training on the same cc100 dataset @xcite . our two new multilingual masked language model dubbed xlm-r xl and xlm-r xxl , with 3.5 and 10.7 billion parameters respectively, significantly outperform the previous xlm-r model on cross-lingual understanding benchmarks and obtain competitive performance with the multilingual t5 models @xcite . we show that they can even outperform roberta-large @xcite on the glue benchmark @xcite . recent multilingual masked language models (mlm) like mbert @xcite or xlm @xcite improved crosslingual language understanding by pretraining large transformer models @xcite on mul-1 https://github.com/anonymous tiple languages at once. the xlm-r model @xcite extended that approach by scaling the amount of data by two orders of magnitude, from wikipedia to common-crawl and training longer, similar to roberta @xcite . these models are particularly effective for lowresource languages, where both labeled and unlabeled data is scarce. they enable supervised cross-lingual transfer, where labeled data in one language can be used to solve the same task in other languages, and unsupervised cross-lingual transfer, where low-resource language self-supervised representations are improved using additional unlabeled data from higher-resource languages. furthermore, they reduce the need for training one model per language, and allows the use of a single -potentially much larger -pretrained model that is then fine-tuned on annotated data from many languages. the better performance of self-supervised crosslingual models on low-resource languages comes however at the cost of lower performance on higherresource languages @xcite . when the number of languages becomes large, @xcite even observed an overall decrease of performance on all languages. it was hypothesized that when multilingual models get more capacity, they may showcase strong performance on both high-resource languages and lowresource languages. with only 550m parameters, the xlm-r model is now relatively small compared to new standards. recent work scaled language models to hundreds of billions @xcite or even multiple trillion parameters @xcite , showing consistent gains in doing so. recently, multilingual t5 showed impressive increase in performance by scaling the model capacity to tens of billions of parameters. our study complements these findings by showing the impact of larger capacity models on the important pretraining task of multilingual masked language model-ing. we show promising results for cross-lingual understanding: xlm-r xxl can both obtain a new state of the art on some cross-lingual understanding benchmarks and outperform the roberta-large model on the english glue benchmark @xcite . this suggests that very large-scale multilingual models may be able to benefit from the best of both worlds: obtaining strong performance on high-resource languages while still allowing for zero-shot transfer and low-resource language understanding. we make the following contributions: ## pretraining and evaluation in this section, we describe the model we use and how we scale it, as well as the data and tasks we use for pretraining and evaluation. ## analysis and results in this section, we present our results and compare xlm-r xl and xlm-r xxl performance to other methods from previous work. cross-lingual understanding results. on xnli, we observe in comparison to monolingual english model. for smaller-capacity models like the base and large version of xlm-r, it was shown that the more languages are considered the lower the perfor-mance @xcite , in particular on highresource languages. for instance, xlm-r large was outperformed by roberta large by 1% accuracy on average on several downstream tasks from the glue benchmark, as illustrated in ta-ble2. with larger capacity, we now observe that xlm-r xxl is able to outperform roberta large by 0.3 dev points, going from 92.9 to 93.2 average accuracy, while handling 99 more languages. while a roberta xxl model may outperform xlm-r xxl , we believe it interesting to notice that with more capacity, a multilingual model can get strong high-resource performance while not losing its cross-lingual transfer ability for lower-resource languages. given the compute needed for training such large-scale models, the possibility of training a single very large model on hundreds of languages with state-of-the-art performance on high-resource languages is an encouraging and positive result. discussion and comparison to mt5. both mt5 and xlm-r models obtain strong performance on cross-lingual understanding benchmarks, as well as high performance on english benchmarks (see the score of 91.6 of mt5 xxl on english xnli). many hyperparameters are however different between mt5 and xlm-r models which makes difficult an apple-to-apple comparison. first, as shown in ## conclusion in this study, we scaled the model capacity of the xlm-r model up to 10.7b parameters and obtained stronger performance than previous xlm-r models on cross-lingual understanding benchmarks. we also show that the additional capacity allows a multilingual model to outperform a the roberta large baseline on english benchmarks. our technical study thus suggests that larger capacity multilingual model can obtain state-of-the-art cross-lingual understanding results while maintaining strong performance on high-resource languages. our work provides an alternative to mt5 models, with new state-of-the-art performance on some languages. we release our code and models publicly.