Majority Voting with Bidirectional Pre-translation For Bitext Retrieval (2021)

Abstract: Obtaining high-quality parallel corpora is of paramount importance for training NMT systems. However, as many language pairs lack adequate gold-standard training data, a popular approach has been to mine so-called “pseudo-parallel” sentences from paired documents in two languages. In this paper, we outline some drawbacks with current methods that rely on an embedding similarity threshold, and propose a heuristic method in its place. Our method involves translating both halves of a paired corpus before mining, and then performing a majority vote on sentence pairs mined in three ways: after translating documents in language x to language y, after translating language y to x, and using the original documents in languages x and y. We demonstrate success with this novel approach on the Tatoeba similarity search benchmark in 64 low-resource languages, and on NMT in Kazakh and Gujarati. We also uncover the effect of resource-related factors (i.e. how much monolingual/bilingual data is available for a given language) on the optimal choice of bitext mining method, demonstrating that there is currently no one-size-fits-all approach for this task. We make the code and data used in our experiments publicly available.

## introduction mining so-called "pseudo-parallel" sentences from sets of similar documents in different languages ("comparable corpora") has gained popularity in recent years as a means of overcoming the dearth of parallel training data for many language pairs. with increasingly powerful computational resources and highly efficient tools such as faiss @xcite at our disposal, the possibility of mining billions of pseudo-parallel bitexts for thousands of language pairs to the end of training a multilingual nmt system has been realized. for example, @xcite perform global mining over billions of sentences in 100 languages, resulting in a massively multilingual nmt system that supports supervised translation in 2200 directions. despite these breakthroughs in high-resource engineering, many questions remain to be answered about bitext mining from a research perspective, with particular attention directed toward the lowresource engineering case, i.e. research settings with limited computational resources. @xcite yield impressive results using hundreds of gpus, aggressive computational optimization, and a global bitext mining procedure (i.e. searching the entire target corpus for a source sentence match), how these results transfer to the low computational resource case is not clear. moreover, the effect of circumstantial (e.g. the resources available for a given language or language pair) or linguistic (e.g. typological) factors on bitext mining performance remains highly understudied. in light of these issues, our contributions are as follows: ## related work mining pseudo-parallel sentences from paired corpora for the purpose of training nmt systems is a decades-old problem, and dozens of solutions have been tried, ranging from statistical or heuristic-based approaches @xcite @xcite @xcite to similarity-based, rule-based, and hybrid approaches @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . benchmarks to measure performance on this task include the bucc 2 '17/18 datasets @xcite , whose task involves spotting gold-standard bitexts within comparable corpora, and the tatoeba dataset @xcite , whose task involves matching gold-standard pairs in truly parallel corpora. relevant to similarity-based mining methods are well-aligned cross-lingual word and sentence embeddings, which are some of the oldest constructs in nlp and have been tackled using hundreds of diverse approaches. even among relatively recent efforts, these approaches range from static, monolingual embeddings @xcite @xcite to static, multilingual ones @xcite @xcite to contextualized, monolingual ones @xcite @xcite @xcite to contextualized, multilingual ones @xcite @xcite @xcite . in this paper, our approach centers around using contextualized, multilingual 2 building and using comparable corpora sentence embeddings for the task of bitext mining. for low-resource languages where parallel training data is little to none, unsupervised nmt can play a crucial role @xcite @xcite @xcite @xcite . however, previous works have only focused on high-resource languages and/or languages that are typologically similar to english. most recently, several works have questioned the universal usefulness of unsupervised nmt and showed its poor results for low-resource languages @xcite . they note the importance of typological similarity between source and target language, in addition to domain proximity and the size and quality of the monolingual corpora involved. they reason that since these conditions can hardly be satisfied in the case of low-resource languages, they result in poor unsupervised performance for these languages. however, recently it has been shown that training a language model on monolingual corpora, followed by training with an unsupervised mt objective, and then training on mined comparable data @xcite can improve mt performance for low-resource languages. in this work, we explore the usefulness of our mined bitext using a similar pipeline. we show an improvement over using only supervised training data for low-resource mt. ## methods an overview of our method for extracting bitexts is given in figure 1 ; the processes are sketched in greater detail in algorithms 1 and 2. the retrieval process begins with a set of english documents and a set of documents in another language xx. both sets of documents are then translated using a pretrained nmt model to obtain xx' documents (english documents translated to xx) and en' documents (xx documents translated to english). we then perform margin-based translation mining (described below in section 4.1 and in algorithm 1) on three sets of documents: the original en-xx documents, the en-en' documents, and the xx-xx' documents. lastly, we perform a majority vote (see algorithm 2, "majority voting") on the resulting sentence pairs, keeping any pair that occurs @xmath0. if mined from a comparable corpus such as wikipedia, these pseudoparallel sentence pairs can then be used to augment the training data of the pretrained nmt models, or (help) train an nmt model from scratch, as in @xcite . alternative methods for filtering an initial set of sentence pairs are also given in algorithm 2 (see comments in blue). empirically, we find our majority voting method to be superior when a pretrained nmt model is available for both languages, while vanilla margin-based mining @xcite performs best in the absence of a pretrained nmt model. results are discussed in greater detail in section 6. ## results & analysis 6.1 tatoeba dataset we mine bitexts on the tatoeba test set in 64 generally low-resource languages (listed in section 5.1) using the primary mining procedure described in algorithm 1 with intersection retrieval, in addition to seven different secondary mining procedures, namely: the results are broken down across languages by resource availability (as in "high-resource" or "low- procedure average gain over baseline (best results only) average gain over baseline (all results) average gain over baseline (langs with transl. support) best results by resource capacity* average gain over baseline (by resource capacity) margin scoring only (artetxe and schwenk, 2019a) +6.9 +5.2 +3.6 level 0: 6 lang. level 1: 18 lang. level 2: 2 lang. level 3: 2 lang. 2 †, 6 ‡ level 0: +7.2 level 1: +5.2 level 2: +1.8 level 3: +3.4 level 4: +1.0 xx-to-en @xmath1.2 +3.3 +3.3 level 0: 1 lang. level 1: 7 lang. level 2: 2 lang. level 3: 7 lang. level 4: 1 lang. level 0: +3.9 level 1: +2.8 level 2: +0.1 level 3: +4.3 level 4: +1.8 bidirectional @xmath2.6 +4.0 +4.0 level 0: 2 lang. level 1: 3 lang. level 2: 2 lang. level 3: 1 lang. level 0: +7.3 level 1: +3.9 level 2: +2.6 level 3: +4.0 level 4: +1.0 * using resource categorizations from joshi et al. (2020) † extinct languages ‡ constructed (artificial) languages ## conclusion we propose a novel method of mining sentence pairs from both comparable and parallel corpora, and demonstrate success on both the tatoeba goldstandard similarity search task and on mining pseudo-parallel sentences for downstream nmt training. we uncover the problematic nature of setting a similarity score threshold for this task, particularly in the context of document-level min-ing. we introduce a heuristic algorithm that filters translations from non-translations by voting on sentence pairs mined in three different ways, which avoids having to laboriously train and re-train nmt systems to tune a similarity score threshold. descriptions of the various mining methods are found in section 4.