W iki M atrix: Mining 135 M Parallel Sentences in 1620 Language Pairs from W ikipedia (2021)

Abstract: We present an approach based on multilingual sentence embeddings to automatically extract parallel sentences from the content of Wikipedia articles in 96 languages, including several dialects or low-resource languages. We do not limit the extraction process to alignments with English, but we systematically consider all possible language pairs. In total, we are able to extract 135M parallel sentences for 16720 different language pairs, out of which only 34M are aligned with English. This corpus is freely available. To get an indication on the quality of the extracted bitexts, we train neural MT baseline systems on the mined data only for 1886 languages pairs, and evaluate them on the TED corpus, achieving strong BLEU scores for many language pairs. The WikiMatrix bitexts seem to be particularly interesting to train MT systems between distant languages without the need to pivot through English.

## introduction most of the current approaches in natural language processing are data-driven. the size of the resources used for training is often the primary concern, but the quality and a large variety of topics may be equally important. monolingual texts are usually available in huge amounts for many topics and languages. however, multilingual resources, i.e. sentences which are mutual translations, are more limited, in particular when the two languages do not involve english. an important source of parallel texts is from international organizations like the european parliament @xcite or the united nations @xcite . several projects rely on volunteers to provide translations for public texts, e.g. news commentary (tiedemann, 2012), opensubtitles @xcite or the ted corpus @xcite wikipedia is probably the largest free multilingual resource on the internet. the content of wikipedia is very diverse and covers many topics. articles exist in more than 300 languages. some content on wikipedia was human translated from an existing article into another language, not necessarily from or into english. eventually, the translated articles have been later independently edited and are not parallel anymore. wikipedia strongly discourages the use of unedited machine translation, foot_1 but the existence of such articles cannot be totally excluded. many articles have been written independently, but may nevertheless contain sentences which are mutual translations. this makes wikipedia a very appropriate resource to mine for parallel texts for a large number of language pairs. to the best of our knowledge, this is the first work to process the entire wikipedia and systematically mine for parallel sentences in all language pairs. in this work, we build on a recent approach to mine parallel texts based on a distance measure in a joint multilingual sentence embedding space @xcite , and a freely available encoder for 93 languages. we approach the computational challenge to mine in almost six hundred million sentences by using fast indexing and similarity search algorithms. the paper is organized as follows. in the next section, we first discuss related work. we then summarize the underlying mining approach. section 4 describes in detail how we applied this approach to extract parallel sentences from wikipedia in 1620 language pairs. in section 5, we assess the quality of the extracted bitexts by training nmt systems for a subset of language pairs and evaluate them on the ted corpus @xcite for 45 languages. the paper concludes with a discussion of future research directions. ## related work there is a large body of research on mining parallel sentences in monolingual texts collections, usually named "comparable coprora". initial approaches to bitext mining have relied on heavily engineered systems often based on metadata information, e.g. @xcite . more recent methods explore the textual content of the comparable documents. for instance, it was proposed to rely on cross-lingual document retrieval, e.g. @xcite or machine translation, e.g. @xcite , typically to obtain an initial alignment that is then further filtered. in the shared task for bilingual document alignment @xcite , many participants used techniques based on n-gram or neural language models, neural translation models and bag-of-words lexical translation probabilities for scoring candidate document pairs. the stacc method uses seed lexical translations induced from ibm alignments, which are combined with set expansion operations to score translation candidates through the jaccard similarity coefficient @xcite @xcite . using multilingual noisy webcrawls such as paracrawl 3 for filtering good quality sentence pairs has been explored in the shared tasks for high resource @xcite and low resource @xcite languages. in this work, we rely on massively multilingual sentence embeddings and margin-based mining in the joint embedding space, as described in @xcite . this approach has also proven to perform best in a low resource scenario @xcite . closest to this approach is the research described in españa-bonet et al. ( 2017 @xcite . however, in all these works, only bilingual sentence representations have been trained. such an approach does not scale to many languages, in particular when considering all possible language pairs in wikipedia. finally, related ideas have been also proposed in @xcite or @xcite . however, in those works, mining is not solely based on multilingual sentence embeddings, but they are part of a larger system. to the best of our knowledge, this work is the first one that applies the same mining approach 3 http://www.paracrawl.eu/ to all combinations of many different languages, written in more than twenty different scripts. in follow up work, the same underlying mining approach was applied to a huge collection of common crawl texts @xcite . hierarchical mining in common crawl texts was performed by el-kishky et al. (2020). 4 wikipedia is arguably the largest comparable corpus. one of the first attempts to exploit this resource was performed by adafre and de @xcite ). an mt system was used to translate dutch sentences into english and to compare them with the english texts, yielding several hundreds of dutch/english bitexts. later, a similar technique was applied to persian/english (mohammadi and ghasemaghaee, 2010). structural information in wikipedia such as the topic categories of documents was used in the alignment of multilingual corpora @xcite . in another work, the mining approach of @xcite was applied to extract large corpora from wikipedia in sixteen languages @xcite . @xcite measured the comparability of wikipedia corpora by the translation equivalents on three languages portuguese, spanish, and english. @xcite came up with a set of features such as wikipedia entities to recognize parallel documents, and their approach was limited to a bilingual setting. @xcite proposed an approach to mine bitexts from wikipedia textual content, but they only considered high-resource languages, namely german, spanish and romanian paired with english. @xcite grounded multilingual mentions to english wikipedia by training cross-lingual embeddings on twelve languages. @xcite searched for parallel text passages in wikipedia by comparing their named entities and time expressions. finally, @xcite propose an approach based on bilingual bil-stm sentence encoders to mine german, french and persian parallel texts with english. parallel data consisting of aligned wikipedia titles have been extracted for twenty-three languages. 5 we are not aware of other attempts to systematically mine for parallel sentences in the textual content of wikipedia for a large number of languages. ## distance-based mining approach the underlying idea of the mining approach used in this work is to first learn a multilingual sentence embedding. the distance in that space can be used as an indicator of whether two sentences are mutual translations or not. using a simple absolute threshold on the cosine distance was shown to achieve competitive results @xcite . however, it has been observed that an absolute threshold on the cosine distance is globally not consistent, e.g. @xcite . this is particularly true when mining bitexts for many different language pairs. ## bitext mining in wikipedia for each wikipedia article, it is possible to get the link to the corresponding article in other languages. this could be used to mine sentences limited to the respective articles. on one hand, this local mining has several advantages: 1) mining is very fast since each article usually has a few hundreds of sentences only; 2) it seems reasonable to assume that a translation of a sentence is more likely to be found in the same article than anywhere in the whole wikipedia. on the other hand, we hypothesize that the margin criterion will be less efficient since one article usually has few sentences which are similar. this may lead to many sentences in the overall mined corpus of the type "name was born on date in city", "building is a monument in city built on date", etc. although those alignments may be correct, we hypothesize that they are of limited use to train an nmt system. the other option is to consider the whole wikipedia for each language: for each sentence in the source language, we mine in all target sentences. this global mining has the advantages that we can try to align two languages even though there are only a few articles in common. a drawback is a potentially increased risk of misalignment. in this work, we chose the global mining option. ## result analysis we run the alignment process for all possible combinations of languages in wikipedia. this yielded 1620 language pairs for which we were able to mine at least ten thousand sentences. remember that mining l 1 → l 2 is identical to l 2 → l 1 , and is counted only once. we propose to analyze and evaluate the extracted bitexts in two ways. first, we discuss the amount of extracted sentences (section 5.1). we then turn to a qualitative assessment by training nmt systems for all language pairs with more than twenty-five thousand mined sentences (section 5.2). ## conclusion we have presented an approach to systematically mine for parallel sentences in the textual content of wikipedia, for all possible language pairs. we use a mining approach based on massively multilingual sentence embeddings @xcite and a margin criterion @xcite . the same approach is used for all language pairs without the need for a language-specific optimization. in total, we make available 135m parallel sentences in 96 languages, out of which only 34m sentences are aligned with english. we were able to mine more than ten thousands sentences for 1620 different language pairs. this corpus of parallel sentences is freely available. 15 we also performed a large scale evaluation of the quality of the mined sentences by training 1886 nmt systems and evaluating them on the 45 languages of the ted corpus @xcite . this approach was recently extended to mine in common crawl texts @xcite .