Language Contamination Helps Explains the Cross-lingual Capabilities of E nglish Pretrained Models (2022)

Abstract: English pretrained language models, which make up the backbone of many modern NLP systems, require huge amounts of unlabeled training data. These models are generally presented as being trained only on English text but have been found to transfer surprisingly well to other languages. We investigate this phenomenon and find that common English pretraining corpora actually contain significant amounts of non-English text: even when less than 1% of data is not English (well within the error rate of strong language classifiers), this leads to hundreds of millions of foreign language tokens in large-scale datasets. We then demonstrate that even these small percentages of non-English data facilitate cross-lingual transfer for models trained on them, with target language performance strongly correlated to the amount of in-language data seen during pretraining. In light of these findings, we argue that no model is truly monolingual when pretrained at scale, which should be considered when evaluating cross-lingual transfer.

## introduction pretrained language models have become an integral part of nlp systems. they come in two flavors: monolingual, where the model is trained on text from a single language, and multilingual, where the model is jointly trained on data from many different languages. monolingual pretrained models are generally applied to tasks in the same language, whereas multilingual ones are used for cross-lingual tasks or transfer. recent work has claimed that monolingual pretrained models are also surprisingly good at transferring between languages, despite ostensibly having never seen the target language before @xcite . however, because of the large scale of pretraining data and because many pretraining corpora are not publicly available, it is currently unknown how much foreign language data exists in monolingual pretraining corpora. in this paper, we show that (1) these data are almost certainly contaminated with very small percentages of text from other languages and that (2) cross-lingual transfer is possible from such data leakage in the pretraining corpus. more specifically, we quantify how multilingual english pretrained models are in two steps. first, we analyze common english pretraining corpora with a large-scale automatic evaluation to estimate their language composition, as well as a smallerscale manual analysis. second, we perform experiments across fifty languages on masked language modeling and part-of-speech (pos) tagging to measure how well the models trained on these pretraining corpora perform outside of english. our analysis finds that these corpora include very small percentages that amount to overall significant amounts of non-english text (figure 1 ), particularly those derived from web-crawled data. furthermore, the models trained on this data perform surprisingly well on other languages; this transfer is strongly correlated with the amount of target language data seen during pretraining. notably, we find that the english t5 outperforms mbert on pos tagging in multiple languages with no finetuning. overall, these results indicate that the considered models are actually multilingual and that their ability to transfer across languages is not zero-shot, despite what has been recently claimed. given the effort required to fully remove all non-english data, we question whether it is practically possible to train truly monolingual models at scale. ## pretraining data composition we first measure how much non-english text exists in commonly used english pretraining corpora with two analyses: an automatic language identification to estimate the amount of foreign language data in these corpora, and a manual qualitative analysis of the text classified as non-english. we consider the following pretraining datasets: english wikipedia (11.8gb); bookcorpus @xcite ; stories (trinh and le 2018, 31gb); openwebtext @xcite , which is an open-source version of webtext @xcite ; cc-news @xcite ; and c4.en @xcite , as provided by @xcite . we use the versions of wikipedia, bookcor-pus, and cc-news used to pretrain roberta. ## cross-lingual transfer of english pretrained models we now ask: how well do models pretrained on these putatively english corpora perform on non-english tasks? while the english data is more multilingual than previously thought, there are many differences between monolingual and multilingual pretraining; non-english data are often tokenized into more subword units foot_1 and are much less frequently observed during monolingual training. we evaluate popular english pretrained models on tasks in more than 50 languages: (masked) language modeling, pos probing, and finetuned pos tagging. we compare the performance of monolingual bert @xcite , roberta @xcite , and t5 @xcite against multilingual mbert @xcite and xlm-r @xcite . we report average performance across five runs with different random seeds for the pos evaluations. the full results and all languages can be found in appendix d. ## discussion in this paper, we demonstrate that english pretrained models are exposed to a considerable amount of non-english data during pretraining, particularly in the case of more recent models that are trained on larger corpora derived from web crawls. we also find that this non-english text acts as a significant source of signal for cross-lingual transfer. other recent work has focused on documenting the composition of pretraining corpora @xcite . @xcite manually audit a variety of multilingual datasets, finding data quality issues that are worse for low-resource languages and, similarly to our work, that texts for many languages are misclassified. in contrast, our focus is on the presence of foreign language data in primarily english corpora. prior work has also shown the ability of monolingual models to transfer to other languages across a wide range of tasks @xcite @xcite @xcite , but these works do not consider the effect of foreign language data leakage as a source of signal. notably, de souza et al. (2021) mention the presence of foreign language data in their corpora but assume the small amounts observed will not affect model performance. however, our findings demonstrate that the amount of foreign language data directly correlates with cross-lingual transfer. an obvious follow-up to our findings would be to retrain the models with text that is verified to only contain english data; this would confirm the effect the leaked non-english data has on the models. we reiterate that the standard method for filtering these datasets, automatic language classifiers, is imperfect. this, and the infeasibility of manual filtering due to the scale of the data, means that controlling for the language the model is pretrained on is nearly impossible. however, the presence of foreign language data in pretraining corpora is not inherently problematic. models trained on these datasets perform exceedingly well on their target languages and generalize to other languages much better than expected. rather, it is important to remember that these models are not performing zero-shot transfer when used in other languages, given the scale and data with which they were pretrained. ## limitations our work has a number of limitations. first, we measure the quantities of non-english data using a language classifier. the amounts of foreign language data we report are estimates for each dataset, as the classifier likely misclassified some examples. we manually audit the types of mistakes made by the language classifier in section 2. additionally, we evaluate downstream performance via pos tagging, and it is possible that the models would exhibit different behavior on other nlp tasks. we also only consider the effect of foreign language contamination for english pretrained models. it is unclear to what extent this phenomenon affects monolingual models for other languages; however, since many of the resources evaluated in this work are also used to pretrain non-english monolingual models (e.g., wikipedia), similar effects would likely be observed. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in proceedings of the ieee international conference on computer vision, pages 19-27.