Unsupervised Cross-lingual Representation Learning at Scale (2020)

Abstract: This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6% average accuracy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7% in XNLI accuracy for Swahili and 11.4% for Urdu over previous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code and models publicly available.

## introduction the goal of this paper is to improve cross-lingual language understanding (xlu), by carefully studying the effects of training unsupervised crosslingual representations at a very large scale. we present xlm-r a transformer-based multilingual masked language model pre-trained on text in 100 languages, which obtains state-of-the-art performance on cross-lingual classification, sequence labeling and question answering. multilingual masked language models (mlm) like mbert @xcite and xlm (lample and @xcite have pushed the stateof-the-art on cross-lingual understanding tasks by jointly pretraining large transformer models @xcite on many languages. these models allow for effective cross-lingual transfer, as seen in a number of benchmarks including cross-lingual natural language inference @xcite @xcite , question answering @xcite , and named entity recognition @xcite . however, all of these studies pre-train on wikipedia, which provides a relatively limited scale especially for lower resource languages. in this paper, we first present a comprehensive analysis of the trade-offs and limitations of multilingual language models at scale, inspired by recent monolingual scaling efforts @xcite . we measure the trade-off between high-resource and low-resource languages and the impact of language sampling and vocabulary size. the experiments expose a trade-off as we scale the number of languages for a fixed model capacity: more languages leads to better cross-lingual performance on low-resource languages up until a point, after which the overall performance on monolingual and cross-lingual benchmarks degrades. we refer to this tradeoff as the curse of multilinguality, and show that it can be alleviated by simply increasing model capacity. we argue, however, that this remains an important limitation for future xlu systems which may aim to improve performance with more modest computational budgets. our best model xlm-roberta (xlm-r) outperforms mbert on cross-lingual classification by up to 23% accuracy on low-resource languages. it outperforms the previous state of the art by 5.1% average accuracy on xnli, 2.42% average f1-score on named entity recognition, and 9.1% average f1-score on cross-lingual question answering. we also evaluate monolingual fine tuning on the glue and xnli benchmarks, where xlm-r obtains results competitive with state-of-the-art monolingual models, including roberta @xcite . these results demonstrate, for the first time, that it is possible to have a single large model for all languages, without sacrificing per-language performance. we will make our code, models and data publicly available, with the hope that this will help research in multilingual nlp and low-resource language understanding. ## related work from pretrained word embeddings @xcite to pretrained contextualized representations @xcite and transformer based language models @xcite , unsupervised representation learning has significantly improved the state of the art in natural language understanding. parallel work on cross-lingual understanding @xcite @xcite extends these systems to more languages and to the cross-lingual setting in which a model is learned in one language and applied in other languages. most recently, @xcite and lample and @xcite introduced mbert and xlm -masked language models trained on multiple languages, without any cross-lingual supervision. @xcite propose translation language modeling (tlm) as a way to leverage parallel data and obtain a new state of the art on the cross-lingual natural language inference (xnli) benchmark @xcite . they further show strong improvements on unsupervised machine translation and pretraining for sequence generation. @xcite shows that monolingual bert representations are similar across languages, explaining in part the natural emergence of multilinguality in bottleneck architectures. separately, @xcite demonstrated the effectiveness of multilingual models like mbert on sequence labeling tasks. @xcite showed gains over xlm using cross-lingual multi-task learning, and @xcite demonstrated the efficiency of cross-lingual data augmentation for cross-lingual nli. however, all of this work was at a relatively modest scale, in terms of the amount of training data, as compared to our approach. the benefits of scaling language model pretraining by increasing the size of the model as well as the training data has been extensively studied in the literature. for the monolingual case, @xcite show how large-scale lstm models can obtain much stronger performance on language modeling benchmarks when trained on billions of tokens. gpt @xcite also highlights the importance of scaling the amount of data and roberta @xcite shows that training bert longer on more data leads to significant boost in performance. inspired by roberta, we show that mbert and xlm are undertuned, and that simple improvements in the learning procedure of unsupervised mlm leads to much better performance. we train on cleaned commoncrawls @xcite , which increase the amount of data for low-resource languages by two orders of magnitude on average. similar data has also been shown to be effective for learning high quality word embeddings in multiple languages @xcite . several efforts have trained massively multilingual machine translation models from large parallel corpora. they uncover the high and low resource trade-off and the problem of capacity dilution @xcite . the work most similar to ours is @xcite , which trains a single model in 103 languages on over 25 billion parallel sentences. @xcite further analyze the representations obtained by the encoder of a massively multilingual machine translation system and show that it obtains similar results to mbert on cross-lingual nli. our work, in contrast, focuses on the unsupervised learning of cross-lingual representations and their transfer to discriminative tasks. ## model and data in this section, we present the training objective, languages, and data we use. we follow the xlm approach @xcite as closely as possible, only introducing changes that improve performance at scale. ## evaluation we consider four evaluation benchmarks. for crosslingual understanding, we use cross-lingual natural language inference, named entity recognition, and question answering. we use the glue benchmark to evaluate the english performance of xlm-r and compare it to other state-of-the-art models. ## analysis and results in this section, we perform a comprehensive analysis of multilingual masked language models. we conduct most of the analysis on xnli, which we found to be representative of our findings on other tasks. we then present the results of xlm-r on cross-lingual understanding and glue. finally, we compare multilingual and monolingual models, and present results on low-resource languages. ## conclusion in this work, we introduced xlm-r, our new state of the art multilingual masked language model trained on 2.5 tb of newly created clean com-moncrawl data in 100 languages. we show that it provides strong gains over previous multilingual models like mbert and xlm on classification, sequence labeling and question answering. we exposed the limitations of multilingual mlms, in particular by uncovering the high-resource versus low-resource trade-off, the curse of multilinguality and the importance of key hyperparameters. we also expose the surprising effectiveness of multilingual models over monolingual models, and show strong improvements on low-resource languages.