Boosting Unsupervised Machine Translation with Pseudo-Parallel Data (2023)

Abstract: Even with the latest developments in deep learning and large-scale language modeling, the task of machine translation (MT) of low-resource languages remains a challenge. Neural MT systems can be trained in an unsupervised way without any translation resources but the quality lags behind, especially in truly low-resource conditions. We propose a training strategy that relies on pseudo-parallel sentence pairs mined from monolingual corpora in addition to synthetic sentence pairs back-translated from monolingual corpora. We experiment with different training schedules and reach an improvement of up to 14.5 BLEU points (English to Ukrainian) over a baseline trained on back-translated data only.

## introduction after the great advancements in machine translation (mt) quality brought by neural mt (nmt; @xcite trained on millions of pre-translated sentence pairs, there came a realization that parallel data is expensive and surely not available for most language pairs in the world. researchers started focusing their attention on methods leveraging monolingual data for machine translation @xcite and even explored the extreme scenario of training a translation system in a completely unsupervised way with no parallel data at all @xcite . the recent impressive progress in language modeling did not leave the area of machine translation intact. however, the translation capabilities of large language models such as the latest gpt models @xcite are weak for underrepresented languages @xcite and unsupervised mt aimed at low-resource languages still deserves special attention. there are two ways to approach machine translation trained exclusively on monolingual data. in the absence of parallel texts, the monolingual training sentences can either be coupled with their synthetic counterparts which are automatically generated through back-translation @xcite , or with authentic counterparts which are automatically selected from existing monolingual texts to be as close translations as possible @xcite . researchers have successfully explored both of these avenues with the conclusion that it is indeed possible to train a functional mt system on monolingual texts only. however, little attention has been paid to combining the two approaches together. in this paper, we work with the standard framework for training unsupervised mt but we incorporate an additional training step where sentence pairs mined from monolingual corpora are used to train the model with a standard supervised mt objective. we consider the mined sentence pairs as pseudo-parallel as they should ideally be identical in meaning but in practice only share a certain degree of similarity. we show that they improve the translation quality nonetheless. we experiment with different training schedules to determine when to incorporate the pseudo-parallel data and when to remove it from the training. in section 2, we summarize the related work on the topics of unsupervised mt and parallel corpus mining. in section 3, we introduce our method, focusing on how we obtain the pseudoparallel sentences and how we incorporate them into the unsupervised mt training. section 4 gives the results of our experiments which are discussed in section 5. ## related work we separate two lines of work in the area of low-resource mt: unsupervised training on monolingual data where the research focuses on the training techniques (unsupervised mt) and supervised training on mined parallel sentences where the research focuses on how to create the training corpus (parallel corpus mining). ## unsupervised mt with pseudo-parallel data it was demonstrated by @xcite al. (2018b) and lample et al. (2018a) that the key elements of an unsupervised neural mt are shared model parameters, good initialization, and iterative learning on back-translated data. we build upon the existing work in unsupervised mt and extend the training procedure with a training step leveraging pseudo-parallel sentence pairs obtained from monolingual training corpora. ## conclusion we have demonstrated the benefits of mt training on pseudo-parallel data in situations when true parallel data is not available. while the pseudo-parallel corpus alone does not reach sufficient quality for standard supervised mt training, it works well in combination with iterative back-translation. it is optimal to train the model until convergence on both pseudo-parallel and synthetic sentence pairs, remove the pseudo-parallel corpus and continue training with iterative back-translation only. incorporating similar sentence pairs into the standard unsupervised mt training increases translation quality across all evaluated language pairs with an improvement of up to 14.5 bleu over the baseline trained without pseudo-parallel data and 8.5 bleu over a hybrid unsupervised system (en-→uk). furthermore, we observed that in some situations (en← →kk), the iterative back-translation becomes trapped in a suboptimal state where no learning occurs. introducing pseudo-parallel data can rescue the model from this state and trigger the learning process. after evaluating our approach on a legal test set, we believe that training on pseudo-parallel sentences could be particularly useful for domain-specific unsupervised mt. if we have two indomain monolingual corpora at hand, parallel corpus mining is an efficient strategy to retrieve translation information. the pseudo-parallel corpus helps the training despite being noisy. we hypothesize that while exact translations help the model find correct correspondences, also the noise can introduce new information and prevent the model from memorizing some of the artifacts of backtranslated sentences. we leave it up to future research to evaluate whether a cleaner but smaller corpus would bring even larger gains.