Self-Training for Unsupervised Neural Machine Translation in Unbalanced Training Data Scenarios (2021)

Abstract: Unsupervised neural machine translation (UNMT) that relies solely on massive monolingual corpora has achieved remarkable results in several translation tasks. However, in real-world scenarios, massive monolingual corpora do not exist for some extremely low-resource languages such as Estonian, and UNMT systems usually perform poorly when there is not adequate training corpus for one language. In this paper, we first define and analyze the unbalanced training data scenario for UNMT. Based on this scenario, we propose UNMT self-training mechanisms to train a robust UNMT system and improve its performance in this case. Experimental results on several language pairs show that the proposed methods substantially outperform conventional UNMT systems.

## introduction recently, unsupervised neural machine translation (unmt) that relies solely on massive monolingual corpora has attracted a high level of interest in the machine translation community @xcite @xcite @xcite @xcite . with the help of cross-lingual language model pretraining @xcite @xcite , the denoising auto-encoder @xcite , and backtranslation @xcite , unmt has achieved remarkable results in several translation tasks. however, in real-world scenarios, in contrast to the many large corpora available for high-resource languages such as english and french, massive monolingual corpora do not exist for some extremely low-resource languages such as estonian. the unmt system usually performs poorly in a low-resource scenario when there is not an adequate training corpus for one language. in this paper, we first define and analyze the unbalanced training data scenario for unmt. based on this scenario, we propose a self-training mechanism for unmt. in detail, we propose selftraining with unsupervised training (st-ut) and self-training with pseudo-supervised training (st-pt) strategies to train a robust unmt system that performs better in this scenario. to the best of our knowledge, this paper is the first work to explore the unbalanced training data scenario problem in unmt. experimental results on several language pairs show that the proposed strategies substantially outperform conventional unmt systems. ## unbalanced training data scenario in this section, we first define the unbalanced training data scenario according to training data size. consider one monolingual corpus {x} in highresource language l 1 and another monolingual corpus {y } in low-resource language l 2 . the data size of {x} and {y } are denoted by |x| and |y |, respectively. in an unbalanced training data scenario, |x| is generally much larger than |y | so that training data {x} is not fully utilized. to investigate unmt performance in an unbalanced training data scenario, we empirically chose english (en) -french (fr) as the language pair. the detailed experimental settings for unmt are given in section 5. we used a transformer based xlm toolkit and followed the settings of @xcite . we randomly extracted 2 million sentences for each language from all 50 million sentences in the en and fr training corpora to create small corpora and simulate unbalanced training data scenarios. ## background we first briefly describe the three components of the unmt model @xcite : cross-lingual language model pre-training, the denoising auto-encoder @xcite , and back-translation @xcite . crosslingual language model pre-training provides a naive bilingual signal that enables the backtranslation to generate pseudo-parallel corpora at the beginning of the training. the denoising autoencoder acts as a language model to improve translation quality by randomly performing local substitutions and word reorderings. generally, back-translation plays an important role in achieving unsupervised translation across two languages. the pseudo-parallel sentence pairs produced by the model at the previous iteration are used to train the new translation model. the general back-translation probability is optimized by maximizing ## self-training mechanism for unmt based on these previous empirical findings and analyses, we propose a self-training mechanism to generate synthetic training data for unmt to alleviate poor performance in the unbalanced training data scenario. the synthetic data increases the diversity of low-resource language data, further enhancing the performance of the translation, even though the synthetic data may be noisy. as the unmt model is trained, the quality of synthetic data becomes better, causing less and less noise. compared with the original unmt model that the synthetic data is just used as the source part, we also use the synthetic data as the target part in our proposed methods. newly generated synthetic data, together with original monolingual data, are fully utilized to train a robust unmt system in this scenario. according to the usage of the generated synthetic training data, our approach can be divided into two strategies: st-ut (algorithm 1) and st-pt (algorithm 2). ## conclusion unmt has achieved remarkable results on massive monolingual corpora. however, a unmt system usually does not perform well in a scenario where there is not an adequate training corpus for one language. based on this unbalanced training data scenario, we proposed two self-training strategies for unmt. experimental results on several language pairs show that our proposed strategies substantially outperform unmt baseline.