Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation (2022)

Abstract: Back-translation is a critical component of Unsupervised Neural Machine Translation (UNMT), which generates pseudo parallel data from target monolingual data. A UNMT model is trained on the pseudo parallel data with \text{\bf translated source} , and translates \text{\bf natural source} sentences in inference. The source discrepancy between training and inference hinders the translation performance of UNMT models. By carefully designing experiments, we identify two representative characteristics of the data gap in source: (1) \text{\textit{style gap}} (i.e., translated vs. natural text style) that leads to poor generalization capability; (2) \text{\textit{content gap}} that induces the model to produce hallucination content biased towards the target language. To narrow the data gap, we propose an online self-training approach, which simultaneously uses the pseudo parallel data \{ natural source, translated target \} to mimic the inference scenario. Experimental results on several widely-used language pairs show that our approach outperforms two strong baselines (XLM and MASS) by remedying the style and content gaps.

## introduction in recent years, there has been a growing interest in unsupervised neural machine translation (unmt), which requires only monolingual corpora to accomplish the translation task @xcite @xcite . the key idea of unmt is to use backtranslation (bt) @xcite to construct * work was done when zhiwei he was interning at tencent ai lab. ## data gap between training and inference in this section, we identity two representative data gaps between training and inference data for inference input ppl natural 242 translated 219 unmt: style gap and content data. we divide the test sets into two portions: the natural input portion with source sentences originally written in the source language and the translated input portion with source sentences translated from the target language. due to the limited space, we conduct the experiments with pre-trained xlm initialization and perform analysis with different kinds of inputs (i.e., natural and translated inputs) on de⇒en newstest2013-2018 unless otherwise stated. ## online self-training for ummt to bridge the data gap between training and inference of unmt, we propose a simple and effective method through self-training. for the translation task @xmath0, we generate the source-original training samples from the source corpus x to improve the model's translation performance on natural inputs. for each batch, we apply the forward model mt @xmath1: in this way, we build a sample {x, y * } with natural input, on which the model can be trained by minimizing: under the framework of unmt training, the final objective function can be formulated as: ## analysis 6.1 translationese output since the self-training samples are translated sentences on the target side, there is concern that the improvement achieved by self-training only comes from making the model outputs better match the translated references, rather than enhancing the model's ability on natural inputs. to dispel the concern, we conducted the following experiments: (1) evaluate the fluency of model outputs in terms of language model ppl and (2) evaluate the translation performance on google paraphrased wmt19 en⇒de test sets @xcite . output fluency we exploit the monolingual corpora of target languages to train the 4-gram language models. translation performance on paraphrased references @xcite collected additional human translations for newstest2019 with the ultimate aim of generating a natural-to-natural test set. we adopt the hq(r) and hq(all 4), which have higher human adequacy rating scores, to re- approach en-fr en-de en-ro avg. ⇒ ⇐ ⇒ ⇐ ⇒ ⇐ xlm unmt 101 147 250 145 152 126 154 +st 101 144 253 147 156 138 157 mass unmt 100 145 256 144 143 119 151 +st 103 146 263 142 156 133 157 data augmentation back-translation @xcite @xcite and self-training @xcite @xcite have been well studied in the supervised nmt. in the unsupervised scenario, @xcite have shown that multilingual pretrained language models can be used to retrieve the pseudo parallel data from the large monolingual data. han et al. (2021) use generative pre-training language models, e.g., gpt-3, to perform zero-shot translations and use the translations as few-shot prompts to sample a larger synthetic translations dataset. the most related work to ours is that offline self-training technology used to enhance low-resource unmt @xcite . in this paper, the proposed online self-training method for unmt can be applied to both high-resource and low-resource scenarios without extra computation to generate the pseudo parallel data. translationese problem translationese problem has been investigated in machine translation evaluation @xcite @xcite . these works aim to analyze the effect of translationese in bidirectional test sets. in this work, we revisit the translationese problem in unmt and find it causes the inaccuracy evaluation of unmt performance since the training data entirely comes from the translated pseudo-parallel data. ## conclusion pseudo parallel corpus generated by backtranslation is the foundation of unmt. however, it also causes the problem of translationese and results in inaccuracy evaluation on unmt performance. we attribute the problem to the data gap between training and inference and identify two data gaps, i.e., style gap and content gap. we conduct the experiments to evaluate the impact of the data gap on translation performance and propose the online self-training method to alleviate the data gap problems. our experimental results on multiple language pairs show that the proposed method achieves consistent and significant improvement over the strong baseline xlm and mass models on the test sets with natural input. source mindestens ein bayern-fan wurde verletzt aus dem stadion transportiert . reference at least one bayern fan was taken injured from the stadium . unmt at least one scotland fan was transported injured from the stadium . source übrigens : münchen liegt hier ausnahmsweise mal nicht an der spitze . reference incidentally , for once munich is not in the lead . unmt remember , edinburgh is not at the top of the list here for once . source justin bieber in der hauptstadt : auf bieber-expedition in berlin reference justin bieber in the capital city : on a bieber expedition in berlin unmt justin bieber in the capital : on bieber-inspired expedition in nyc source zum vergleich : in diesem jahr werden in deutschland 260.000 einheiten fertig . reference in comparison , 260,000 units were completed in this year in germany. unmt to date , 260,000 units are expected to be finished in the uk this year . source deutschland schiebe ein wohnungsdefizit vor sich her , das von jahr zu jahr größer wird . reference germany has a housing deficit which increases every year .