CCM atrix: Mining Billions of High-Quality Parallel Sentences on the Web (2021)

Abstract: We show that margin-based bitext mining in a multilingual sentence space can be successfully scaled to operate on monolingual corpora of billions of sentences. We use 32 snapshots of a curated common crawl corpus (Wenzel et al, 2019) totaling 71 billion unique sentences. Using one unified approach for 90 languages, we were able to mine 10.8 billion parallel sentences, out of which only 2.9 billions are aligned with English. We illustrate the capability of our scalable mining system to create high quality training sets from one language to any other by training hundreds of different machine translation models and evaluating them on the many-to-many TED benchmark. Further, we evaluate on competitive translation benchmarks such as WMT and WAT. Using only mined bitext, we set a new state of the art for a single system on the WMT’19 test set for English-German/Russian/Chinese. In particular, our English/German and English/Russian systems outperform the best single ones by over 4 BLEU points and are on par with best WMT’19 systems, which train on the WMT training data and augment it with backtranslation. We also achieve excellent results for distant languages pairs like Russian/Japanese, outperforming the best submission at the 2020 WAT workshop. All of the mined bitext will be freely available.

## introduction parallel data, i.e. sentences in two languages which are mutual translations, are a crucial resource for many multilingual natural language processing tasks. traditionally, high quality parallel texts are obtained from the publications of international organizations like the the united nations @xcite or the european parliament @xcite . these are professional human translations, but they are in a more formal language and tend to be limited to political topics. another direction is to rely on volunteers to provide translations for public texts, such as the ted corpus @xcite , news commentary @xcite or opensubtitles @xcite , but this approach lacks scalability. there is also a large body of works which aims in mining bitexts by comparing huge collections of monolingual data. our aim is to mine at massive scale, both in number of possible languages and in quantity of mined parallel sentences. most existing large scale bitext mining techniques use a hierarchical approach. first, a subset of texts that may contain parallel sentences are selected at the document level. subsequently, sentences within these aligned documents are compared to identify parallel ones. this local mining is potentially fast since only a few thousand sentences need to be compared for each document pair. however, sentences not present in these pre-selected documents cannot be aligned, which vastly limits the quantity of mineable bitext. a first system to globally compare all sentences in monolingual collections for many language pairs was presented in @xcite , but was limited to only wikipedia. in this paper, we show that this type of global mining scales to extremely huge corpora: 71 billion sentences, about 120x larger than the work of @xcite . our contributions are: ## related work much previous work has explored the automatic creation of parallel data from monolingual resources. in this section, we detail various approaches and illustrate the differences of our algorithmic approach and the scale of our mining. mining methodology at the start, various approaches used alignment on information beyond text itself, such as with document metadata @xcite . later, work aligned based on text with techniques such as jaccard similarity @xcite @xcite , crosslingual document retrieval @xcite , language models @xcite , translation @xcite , or bag-of-words @xcite . in contrast, we use massively multilingual sentence embeddings trained on almost 100 languages, and then conduct margin-based mining in the multilingual embedding space @xcite @xcite . compared to work such as @xcite , we drastically increase the scale of our mining and produce two orders of magnitude more data -this is possible by the increased efficiency and scalability of our improved mining methods. a few mining approaches were applied to large quantities of language pairs. for example, the paracrawl project foot_0 mined data for all european languages. bitextor @xcite was applied to many languages, but took an approach that required identifying parallel documents first and then extracting aligned sentences. this is similar to the ccaligned project @xcite . in contrast to these, we mine much larger quantities of parallel data due to the global margin-based mining approach that we take. data used to mine many previous methods for data mining focused on wikipedia. @xcite and @xcite , for instance, aligned entire parallel documents. for example, adafre and de @xcite and mohammadi and ghasemaghaee (2010) used machine translation systems to compare dutch and per-sian wikipedias to english, to identify aligned sentences. various other worked used similarities in mentioned entities to align text, such as @xcite and @xcite . work such as @xcite ; @xcite ; @xcite used wikipedia to mine parallel sentences, but focused on fewer languages, often high resource. in contrast, our system mines not in wikipedia but in commoncrawl, a much larger source of data -and is applied to a much larger quantity of languages. work has extended mining beyond wikipedia. for example, paracrawl 1 has been heavily used (e.g. in wmt), which is based on several noisy multilingual crawls @xcite . @xcite focused on mining documents in common crawl rather than sentences. our work continues this line of scalable mining on the web, but pushes to large-scale mining to produce billions of aligned sentences. ## distance-based mining approach we leverage massively multilingual sentence embeddings and a margin-based criterion to mine parallel sentences. the core idea is to learn a multilingual sentence embedding, or an embedding space in which semantically similar sentences are close, independent of the language they are written in. this means that distance in the embedding space can be used to determine if two sentences are mutual translations or not. we use the open source laser @xcite embeddings as they cover over 90 different languages. 2 another recent multilingual sentence embedding is labse @xcite . ## evaluation on translation benchmarks to assess our mined bitext, we train nmt systems only on our mined data and evaluate on several public benchmarks. we do not use any of the training data provided with these corpora, so do not use any available human translated data, and have no guarantee our bitext covers the same domain as the test sets. nevertheless, we show on the many to many ted corpus that our mined data produces high quality translation systems, even through distant language pairs not aligned through english and low resource languages. finally, we demonstrate that models trained on ccmatrix can surpass state of the art systems in wmt'19 and wat'20. ## conclusion we show that margin-based mining in a joint multilingual sentence embedding space can be scaled to monolingual texts of more than 71 billion unique sentences in 90 languages, including several low resource languages. this procedure yields 10.8 billion parallel sentences, out of which only 2.9 billions are aligned with english. we performed an extensive evaluation of the quality of the mined bitexts by training nmt systems for many language pairs. training only on mined data, we outperform the best single nmt systems at wmt'19 for translations between german, russian, and chinese with english, as well as between german and french. we also achieve state-of-the-art bleu scores for translation between russian and japanese at wat'20. all mined data is freely available. 9 we hope this will enable widespread research on multilingual nmt, particularly on languages where training data is not currently available.