A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism (2024)

Abstract: We show that content on the web is often translated into many languages, and the low quality of these multi-way translations indicates they were likely created using Machine Translation (MT). Multi-way parallel, machine generated content not only dominates the translations in lower resource languages; it also constitutes a large fraction of the total web content in those languages. We also find evidence of a selection bias in the type of content which is translated into many languages, consistent with low quality English content being translated en masse into many lower resource languages, via MT. Our work raises serious concerns about training models such as multilingual large language models on both monolingual and bilingual data scraped from the web.

## introduction modern ai is enabled by huge amounts of training data, typically several hundred billion tokens to a few trillion tokens @xcite @xcite . training at this scale is only possible with web-scraped data. we explore the effects that the long-term availability of low cost machine translation (mt) has had on the web. 1 we show that content on the web is often translated into many languages, and the quality of these multi-way translations indicates they were primarily created using mt: see figure 1 . machine generated, multi-way parallel translations not only dominate the total amount of translated content on the web in lower resource languages where mt is available, it also constitutes 2 3-4 5-7 8+ multi-way parallelism 67.5 70.0 72.5 75.0 cometqe score ( ) 74.5 72.7 70.4 68.3 figure 1 : the more languages a sentence has been translated into ("multi-way parallelism"), the lower quality the translations are, suggesting a higher prevalence of machine translation. see § 4.4 for more details. a large fraction of the total web content in those languages. we also find evidence of a selection bias in the type of content which is translated into many languages, and therefore over represented in lower resource languages: this content is shorter, more predictable, and has a different topic distribution compared to content translated into a single language. a limited investigation suggests this selection bias is the result of low quality content generated in english (likely produced to generate ad revenue) and translated en masse into many lower resource languages via mt (again, likely to generate ad revenue). our findings raise numerous concerns for multilingual model builders: fluency (especially across sentences) and accuracy are lower for mt data, 2 which could produce less fluent models with more hallucinations, and the selection bias indicates the data may be of lower quality, even before considering mt errors. data quality is crucial in large language model (llm) training, where high quality corpora like books and wikipedia articles are typically upsampled several times @xcite @xcite . our findings also help to explain why lowresource mt @xcite @xcite is challenging, and why filtering noise @xcite ) from webscraped bitext (junczys-dowmunt, 2018; chaudhary et al., 2019) is beneficial for mt training @xcite @xcite . to enable analysis, we create the largest multiway corpus to date, consisting of 6.4b unique sentences in 90 languages. we release code to reproduce our corpus and analysis. 3 2 related work our work is inspired by several recent efforts which seek to understand the characteristics of large scale corpora @xcite @xcite . many works have detected machine translation @xcite @xcite ), but we are not aware of prior work using multi-way parallelism to do so. freitag and firat (2020) explored multi-way parallelism with the goal of improving multilingual mt. exploring multi-way parallelism on the web requires a curated representation of translated content from the web. we build upon ccmatrix @xcite , which is in turn based on common crawl. 4 common crawl is a long running web-scraping project which maintains a free, open source repository of web-scraped data. ccmatrix is created by embedding common crawl sentences into a multilingual space using laser @xcite and then finding bilingual translation pairs using fast approximate nearest neighbor search @xcite . we choose ccmatrix over a corpus from a traditional bitext mining process of document alignment @xcite @xcite followed by sentence alignment @xcite @xcite because it is the largest corpus available at the time of writing (in both number of sentences and language coverage). corpus creation has been optimized to run in about one day on a single i4i.32xlarge aws instance. we create a multi-way parallel representation of the web, consisting of translation tuples containing two or more sentences in different languages which are translations of each other. 5 as a trivial example, ("hello", "hola") in english-spanish and ("hello", "olá") in english-portuguese combine to make (en:"hello", es:"hola", pt:"olá"). we denote this corpus multi-way ccmatrix (mwccmatrix). we iterate through all bitext in ccmatrix, from highest to lowest laser margin score, adding sentence pairs as new tuples in mwccmatrix when neither sentence is already in the new corpus, and expanding tuples already in the new corpus when one sentence or the other (but not both) is already present. this deduplicates the corpus (i.e. adds each unique sentence only once), but allows for more than one sentence in the same language to be added to a given tuple, which tend to differ primarily in punctuation/capitalization (i.e. near duplicates). therefore, we remove all but the first sentence added to each tuple in a given language. deduplication across language pairs brings the total number of sentences down from 21.7b total sentences (10.9b sentence pairs) to 7.9b unique sentences in 2.2b tuples, and near duplicate removal brings it down to 6.4b. pseudocode and a description of the optimizations required to make corpus creation tractable are provided in appendix a. ## discussion & conclusion experiments with qe (see § 4.4) strongly suggest that highly multi-way parallel translations are generated by mt. in lower resource languages, most translations are multi-way parallel (see § 4.2), suggesting that mt content dominates translation content. furthermore, a large fraction of the total sentences in lower resource languages have at least one translation (see § 4.1), implying that a large fraction of the total web in those languages is mt generated. several observations point to a selection bias in the type of data which is translated into many languages, compared to data translated into a single language: it is shorter and more predictable (see § 4.3), and substantially more likely to be from the conversation & opinion topic (see § 4.5). since translations of this data constitute a substantial portion of the total data in low-resource languages, this bias will also appear in low resource languages. an investigation into the increase in conversa-tion & opinion data suggests that this selection bias is the result of low quality content (likely produced to generate ad revenue) being translated via mt en masse into many lower resource languages (again likely for the purpose of generating ad revenue). it also suggests that such data originates in english and is translated into other languages. additional investigation would be required to understand if this finding generalizes to other topics, languages, and levels of multi-way parallelism. our findings also suggest some ways to address the problem of mt output in web-scraped training data: mt detection, which has typically been proposed to filter bitext, could also help to filter monolingual text in lower resource languages. it also suggests that multi-way parallelism is a promising way to detect low quality, machine translated data, especially in lower resource languages, to filter both bilingual and monolingual data.