Quick Back-Translation for Unsupervised Machine Translation (2023)

Abstract: The field of unsupervised machine translation has seen significant advancement from the marriage of the Transformer and the back-translation algorithm. The Transformer is a powerful generative model, and back-translation leverages Transformer’s high-quality translations for iterative self-improvement. However, the Transformer is encumbered by the run-time of autoregressive inference during back-translation, and back-translation is limited by a lack of synthetic data efficiency. We propose a two-for-one improvement to Transformer back-translation: Quick Back-Translation (QBT). QBT re-purposes the encoder as a generative model, and uses encoder-generated sequences to train the decoder in conjunction with the original autoregressive back-translation step, improving data throughput and utilization. Experiments on various WMT benchmarks demonstrate that a relatively small number of refining steps of QBT improve current unsupervised machine translation models, and that QBT dramatically outperforms standard back-translation only method in terms of training efficiency for comparable translation qualities.

## introduction the capabilities of neural machine translation (mt) models have seen considerable advancement in recent years @xcite @xcite . however, paired translation data for supervision is often difficult to procure without specialized expertise, especially for low-resource languages. proposed originally in the statistical machine translation regime @xcite , unsupervised machine translation (umt) aligns languages without parallel data by leveraging vocabularylevel statistics or abstract language representations. modern umt approaches combine selfsupervised learning and statistical methods @xcite @xcite , and more recently pair generative language modeling with iterative back-translation @xcite @xcite . iterative backtranslation (bt) creates synthetic translation pairs from monolingual corpora, and uses them to train neural mt models with a reconstruction loss @xcite . models trained with iterative back-translation hold the state-of-the-art (sota) performance on a variety of umt scenarios @xcite @xcite . recent work suggests that increased data diversity during back-translation improves translation performance, and further increasing diversity remains an open challenge @xcite . back-translation involves massive amounts of sequence generation, which usually follow an autoregressive (ar) inference procedure. in ar inference, tokens are generated one step at a time, which can not be parallelized @xcite . non-autoregressive (nar) transformer models have been proposed as a method to improve inference speed for translation models, in which sequence transduction is done in parallel @xcite @xcite . in this paper, we address both autoregressive inference and data diversification limitations with a novel, framework for umt, called quick back-translation (qbt). with qbt, the transformer encoder is trained as a bidirectional generative translation model. in what we call the encoder back-translation (ebt) step, the encoder back-translates its own synthetic outputs, training as an independent umt model. the encoder-synthesized translations are then leveraged as synthetic signal for the decoder, providing increased data diversity in what we call an encoder back-translated distillation (ebtd) step. the transformer architecture is kept intact under quick back-translation, allowing the new optimization procedures to supplement or enhance the original back-translation objective. the qbt methods add more diverse training signals for umt by dual usage of the encoder both as a module inside the transformer and as a standalone nar translation model, which offer a significant speedup on top of standard bt. we conduct experiments on multiple translation benchmarks of different resources, and qbt is able to achieve umt qualities on par with previous sota but with high efficiency in run-time, especially for long sequences. ## background unsupervised machine translation for many language pairs there do not exist sufficient examples to train a supervised translation model. it has been proposed to leverage monolingual data only to learn a map between languages, called unsupervised machine translation (umt). originally proposed as a probabilistic decipherment problem @xcite , modern umt now leverages representation learning @xcite and pre-trained generative models @xcite @xcite . generativemodel-based umt techniques rely on language model pre-training and self-supervised learning @xcite . common pre-training objectives include masked language modeling (mlm), causal language modeling (clm), denoising autoencoding (dae), and the masked sequence to sequence (mass) objective @xcite @xcite , which are used for initializing umt models. back translation back-translation (bt) refers to a model or set of models that recursively train using reverse synthetic translations. originally proposed to supplement parallel data during supervised mt training @xcite , bt has been extended to multi-agent scenarios @xcite , and fully unsupervised translation in the statistical machine translation regime @xcite and recently with language modeling @xcite @xcite . most recently, the crossmodel back-translated distillation (cbd) method @xcite used back-translation between models pre-trained with independent back-translation, all in the fully unsupervised scenario. they find that more synthetic data diversity improves even high quality umt models. ## preliminaries let d s and d t be the monolingual corpus of sentences in a source domain s and a target domain t, respectively, and they are not aligned. given a source sentence @xmath0, x 2 , . . . , x m ) ∼ d s with length m, umt aims to learn a conditional generation model p θ (y|x) with parameters θ so that @xmath1, y 2 , . . . , y n ) ∼ d t with length n is closest to the true translation for x after decoding. transformer model the transformer @xcite is a sequence-to-sequence model with an encoder-decoder architecture built with attention mechanisms. given the source sentence x, it models the conditional target distribution via there is a major difference between the selfattention computation of the encoder and decoder in sequence generation. the encoder self-attention allows each token representation to attend to all the other tokens before and after it, thus is bidirectional. for the decoder, as the future tokens are unknown before being generated sequentially, each token representation at t can only attend to the previous tokens @xmath2, which is autoregressive. during conditional generation such as mt, the encoder only needs to be run once with full parallelization on x, and the decoder needs to be run n steps for every t, making it a bottleneck for generation speed. back-translation by translating in the backward direction from target to source, bt @xcite provides synthetic parallel corpus for data augmentation used by many umt approaches. in particular, bt generates x ∼ d s from a reverse model p ϕ (x|y) based on a target sample y ∼ d t , and then uses (x, y) as pseudo pairs for supervision for learning the source to target model p θ (y|x). for umt, bt is mostly used in an iterative fashion @xcite , switching between the source language s and the target language t to jointly improve the language alignments and translation qualities @xcite . in our setup we also tie both translation directions with the same model, essentially making ϕ = θ. ## quick back-translation iterative bt with transformer relies on sequential generation of translations from the decoder, which may not be parallelizable due to its autoregressive design. we propose quick back-translation (qbt), a non-autoregressive (nar) generation approach to provide increased synthetic data and reduced inference time during bt. in contrast to common nar transformer designs, we directly repurpose the transformer encoder as an nar generation model without any modifications. in particular, the back-translation from target y to source x is obtained via and the argmax of the probability distribution is taken for each position independently in the sequence. here the only new parameter is the output transformation matrix w eo , which we tie with the encoder input embedding w @xmath3. bidirectional encoder this essentially shortcuts the computation from input to output by ignoring the decoder. it makes the translation nar using only the bidirectional encoder whose computation is fully parallelizable with linear time complexity, with constant generation steps. note however that output length is now confined to equal input length. an illustration of qbt in comparison with bt is presented in figure 1 . as an additional benefit to the fast run-time of encoder synthesis, qbt provides increased synthetic data diversity to the bt procedure, improving machine translation performance. in the following we describe different components of applying qbt for improving umt models. ## experimental setup data and evaluation we evaluate our proposed methods on wmt news crawl datasets. 5 we prepare all of the monolingual data for english, french, german, and romanian, up to year 2017. the datasets have sentence counts of 190m, 62m and 270m, 2.9m respectively. we encode data with byte-pair encoding (bpe) @xcite with a dictionary of 60k sub-words provided by @xcite . each language pair: english and german, english and french, and en3: update encoder and its embeddings {θ e , w e } with one ebt step starting from z (with algorithm 2); 4: sample a new mini-batch b of either z ∼ d s or z ∼ d t 5: update decoder and its embeddings {θ d , w d } with one ebtd step starting from z (with algorithm 3); 6: sample a new mini-batch b of either z ∼ d s or z ∼ d t 7: update full model θ with a bt step starting from z; 8: until convergence glish and romanian, have distinct bilingual bpe dictionaries. following bpe, sentences with length 175 or greater are removed. models are scored on the wmt'14 english-french, wmt'16 english-german and wmt'16 english-romanian parallel test sets @xcite . we evaluate the models with bleu scores @xcite following the same setup of previous works @xcite . ## limitations our qbt methods serve as an augmentation of the standard bt algorithm for learning unsupervised machine translation models, mainly with efficiency considerations in low computational resource scenarios. they still can not fully replace the standard bt procedures, as qbt still needs bt inside the algorithms to achieve competitive translation qualities. even so, our performance gain compared to previous state-of-the-art umt models are modest, whereas we mainly gain in training speed. due to the limited resource setups, the models that we experimented are considered relatively small in sizes compared with the large language models nowadays. it would be interesting to explore further on the larger scales. our methods currently apply to encoder-decoder transformer architectures, which may limit the applications. for different model architectures, we may hope to apply similar ideas of using a fast (and potentially separate) generation model to generate synthetic data for bt-like algorithms. furthermore, the use of encoder for back-translation (ebt) with nar generation is also limited in that the output sequence length can not go beyond the input sequence length, which we hope to relax with more sophisticated nar strategies in the future.