Mixup Decoding for Diverse Machine Translation (2021)

Abstract: Diverse machine translation aims at generating various target language translations for a given source language sentence. To leverage the linear relationship in the sentence latent space introduced by the mixup training, we propose a novel method, MixDiversity, to generate different translations for the input sentence by linearly interpolating it with different sentence pairs sampled from the training corpus during decoding. To further improve the faithfulness and diversity of the translations, we propose two simple but effective approaches to select diverse sentence pairs in the training corpus and adjust the interpolation weight for each pair correspondingly. Moreover, by controlling the interpolation weight, our method can achieve the trade-off between faithfulness and diversity without any additional training, which is required in most of the previous methods. Experiments on WMT’16 en-ro, WMT’14 en-de, and WMT’17 zh-en are conducted to show that our method substantially outperforms all previous diverse machine translation methods.

## introduction neural machine translation (nmt) @xcite @xcite @xcite has achieved significant success in improving the quality of machine translation. despite these successes, nmt still faces problems in translation diversity @xcite . due to the existence of lexical diversity, syntactic diversity and synonymous words in the target language, one source language sentence usually corresponds to multiple proper translations. however, existing nmt models mostly consider the one-to-one mapping but neglects the one-to-many mapping between the source and target languages. many studies have been proposed to tackle such issues by exploiting the diversity in the model space, such as using different experts @xcite , applying different multi-head attentions @xcite , and utilizing different models @xcite . although the model-oriented methods have been well studied, the data-oriented method still lacks exploration. in this work, we focus on improving the translation diversity by exploiting the diversity in the sentence space. since different translations of one source sentence share the same semantics, their sentence-level embeddings will gather in the same region in the target sentence space. in other words, each sentence in this region is a translation of the source sentence. by sampling different sentences from this region, we can obtain various translations. to sample different translations from this region, we propose a simple but effective method, mixdiversity. as aforementioned, the nmt model learns a one-to-one mapping between the source and target languages. given the source sentence and the generated tokens in the decoder, the nmt model can map the source sentence into a corresponding target sentence. therefore, to obtain various translations on the target side, we need to find the corresponding inputs for the nmt model. by mixing the source sentence with the sampled sentence pairs in the training corpus via linear interpolation, we can obtain mixed sentences as inputs for the nmt model and map them into a corresponding sentence in the target sentence space. by assigning a larger interpolation weight for the source sentence, the mixed sentence then has similar semantics, and the corresponding translation has higher faithfulness to the source sentence. in this way, by mixing the source sentence with different sentence pairs during decoding, we can obtain diverse mixed sentences as inputs for the nmt model and map them to different translations for the source sentence. given that nmt models are non-linear functions, the interpolation weight of the input sentences could decline, and the semantic of the output could shift to the randomly sampled sentence pairs. to guarantee the consistency of the interpolation weight during decoding, we force the nmt model to learn to maintain the proportion between the mixed sentences with the mixup training strategy @xcite which linearly interpolates two randomly sampled sentence pairs in both encoder and decoder during training. the main idea of our approach is illustrated in figure 1 , where we mix one source sentence with four different sentence pairs sampled from the training corpus to obtain four variant mixed samples as inputs for the nmt model and map the mixed sentences to four diverse sentences in the target space. ## conclusion in this work, we propose a novel method, mixdiversity, for the diverse machine translation. compared with the previous model-oriented methods, mixdiversity is a data-oriented method that generates different translations of the input sentence by utilizing the diversity in the sentence latent space. we also propose two simple but effective methods to select the mixup samples and adjust the mixup weights for each sample. to evaluate the overall performance synthetically, we design a new evaluation metric, eda. experimental results show that mixdiversity outperforms all previous methods in the field of diverse machine translation. reference therefore , the human beings hope that one day on the mars to establish a base of residence , and ultimately to mars immigration , it turned into a second home of mankind .