Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation (2021)

Abstract: Self-training has proven effective for improving NMT performance by augmenting model training with synthetic parallel data. The common practice is to construct synthetic data based on a randomly sampled subset of large-scale monolingual data, which we empirically show is sub-optimal. In this work, we propose to improve the sampling procedure by selecting the most informative monolingual sentences to complement the parallel data. To this end, we compute the uncertainty of monolingual sentences using the bilingual dictionary extracted from the parallel data. Intuitively, monolingual sentences with lower uncertainty generally correspond to easy-to-translate patterns which may not provide additional gains. Accordingly, we design an uncertainty-based sampling strategy to efficiently exploit the monolingual data for self-training, in which monolingual sentences with higher uncertainty would be sampled with higher probability. Experimental results on large-scale WMT English⇒German and English⇒Chinese datasets demonstrate the effectiveness of the proposed approach. Extensive analyses suggest that emphasizing the learning on uncertain monolingual sentences by our approach does improve the translation quality of high-uncertainty sentences and also benefits the prediction of low-frequency words at the target side.

## introduction leveraging large-scale unlabeled data has become an effective approach for improving the performance of natural language processing (nlp) models @xcite @xcite . as for neural machine translation (nmt), compared to the parallel data, the monolingual data is available in large quantities for many languages. several approaches on boosting the nmt performance with the monolingual data have been proposed, e.g., data augmentation @xcite , semisupervised training @xcite @xcite , pre-training @xcite . among them, data augmentation with the synthetic parallel data @xcite is the most widely used approach due to its simple and effective implementation. it has been a de-facto standard in developing the large-scale nmt systems @xcite @xcite . self-training @xcite is one of the most commonly used approaches for data augmentation. generally, self-training is performed in three steps: (1) randomly sample a subset from the large-scale monolingual data; (2) use a "teacher" nmt model to translate the subset data into the target language to construct the synthetic parallel data; (3) combine the synthetic and authentic parallel data to train a "student" nmt model. recent studies have shown that synthetic data manipulation @xcite and training strategy optimization @xcite in the last two steps can boost the self-training performance significantly. however, how to efficiently and effectively sample the subset from the large-scale monolingual data in the first step has not been well studied. intuitively, self-training simplifies the complexity of generated target sentences @xcite @xcite , and easy patterns in monolingual sentences with deterministic translations may not provide additional gains over the self-training "teacher" model @xcite . related work on computer vision also reveals that easy patterns in unlabeled data with the deterministic prediction may not provide additional gains @xcite . in this work, we investigate and identify the uncertain monolingual sentences which implicitly hold difficult patterns and exploit them to boost the self-training performance. specifically, we measure the uncertainty of the monolingual sentences by using a bilingual dictionary extracted from the authentic parallel data ( §2.1). experimental results show that nmt models benefit more from the monolingual sentences with higher uncertainty, except on those with excessively high uncertainty ( §2.3). by conducting the linguistic property analysis, we find that extremely uncertain sentences contain relatively poor translation outputs, which may hinder the training of nmt models ( §2.4). inspired by the above finding, we propose an uncertainty-based sampling strategy for selftraining, in which monolingual sentences with higher uncertainty would be selected with higher probability ( §3.1). large-scale experiments on wmt english⇒german and english⇒chinese datasets show that self-training with the proposed uncertainty-based sampling strategy significantly outperforms that with random sampling ( §3.3). extensive analyses on the generated outputs confirm our claim by showing that our approach improves the translation of uncertain sentences and the prediction of low-frequency target words ( §3.4). ## observing monolingual uncertainty in this section, we aimed to understand the effect of uncertain monolingual data on self-training. we first introduced the metric for identifying uncertain monolingual sentences, then the experimental setup and at last our preliminary results. notations. let x and y denote the source and target languages, and let x and y represent the sentence domains of corresponding languages. let @xmath0, y i )} n @xmath1, where x @xmath2, y @xmath3. let m @xmath4, where x @xmath5. our objective is to obtain a translation model f : @xmath6, that can translate sentences from language x to language y . ## exploiting monolingual uncertainty by analyzing the effect of monolingual data uncertainty on self-training in section 2, we understood that monolingual sentences with relatively high uncertainty are more informative while also with high quality, which motivates us to emphasize the training on these sentences. in this section, we introduced the uncertainty-based sampling strategy for self-training and the overall framework. ## conclusion in this work, we demonstrate the necessity of distinguishing monolingual sentences for self-training in nmt, and propose an uncertainty-based sampling strategy to sample monolingual data. by sampling monolingual data with relatively high uncertainty, our method outperforms random sampling significantly on the large-scale wmt english⇒german and english⇒chinese datasets. further analyses demonstrate that our uncertainty-based sampling approach does improve the translation quality of high uncertainty sentences and also benefits the prediction of low-frequency words at the target side. the proposed technology has been applied to transmart foot_5 @xcite , an interactive machine translation system in tencent, to improve the performance of its core translation engine. future work includes the investigation on the confirmation bias issue of self-training and the effect of decoding strategies on self-training sampling.