Bridging the Gap between Training and Inference: Multi-Candidate Optimization for Diverse Neural Machine Translation (2022)

Abstract: Diverse NMT aims at generating multiple diverse yet faithful translations given a source sentence. In this paper, we investigate a common shortcoming in existing diverse NMT studies: the model is usually trained with single reference, while expected to generate multiple candidate translations in inference. The discrepancy between training and inference enlarges the confidence variance and quality gap among candidate translations and thus hinders model performance. To deal with this defect, we propose a multi-candidate optimization framework for diverse NMT. Specifically, we define assessments to score the diversity and the quality of candidate translations during training, and optimize the diverse NMT model with two strategies based on reinforcement learning, namely hard constrained training and soft constrained training. We conduct experiments on NIST Chinese-English and WMT14 English-German translation tasks. The results illustrate that our framework is transparent to basic diverse NMT models, and universally makes better trade-off between diversity and quality. Our source codeis available at https://github.com/DeepLearnXMU/MultiCanOptim .

## introduction recently, neural machine translation (nmt) has achieved impressive progress in improving translation quality @xcite @xcite . despite the remarkable success, nmt models still suffer from lacking translation diversity, which is essential due to the following reasons. first, similar to natural language, variability and expressiveness are the core features of translations. second, only focusing on increasing translation accuracy during training will bias the nmt model to common phrases, exacerbating data sparsity @xcite . in conclusion, improving translation diversity is a promising direction in nmt community. to achieve diverse nmt, several studies have explored various training or decoding strategies, including: 1) constraining decoding with diversity regularization @xcite , 2) sampling from the mixture of models @xcite , and 3) conditioning decoding with diverse signals @xcite . however, all these approaches train models on single-reference corpus, while expecting them to generate multiple candidate translations during inference. we argue that such discrepancy between training and inference prevents the models from learning one-to-many relations efficiently. firstly, since the predictions of nmt models are encouraged to fit the one-hot distribution of single-reference corpus, the model confidence of generating top1 candidate translations will be much larger than that of the rest candidates, limiting translation diversity. secondly, only one reference is used to get optimization signal at the training time, resulting in significant quality drops of top2-topk translations. one direct way addressing these issues is to train the models using multi-reference training data. nevertheless, its construction is quite expensive and thus impractical. to overcome the above issues, in this paper, we propose a novel multi-candidate optimization framework for diverse nmt. the basic idea is to guide an nmt model to learn diverse translation from its candidate translations based on reinforcement learning (rl). during training, the model generates multiple candidate translations, of which rewards are quantified according to their diversity and quality. since directly optimizing model parameters with the above two rewards involves backpropagating through discrete decoding decision, we explore two specific methods to train the diverse nmt model: 1) hard constrained training. we transform the rewards to discrete scalars, prevent-ing the model from learning those candidate translations with low rewards. 2) soft constrained training. we introduce minimum risk training (mrt) to minimize the risks of obtaining diversity and quality rewards. compared with previous works, our proposed framework reduces the confidence variance among candidate translations and improves the quality of top2-topk translations during inference, achieving better performance in terms of both diversity and quality. overall, the major contributions of our work are three-fold: further, @xcite replace the syntactic codes with latent domain variables derived from target sentences, which is more computationally efficient. @xcite sample the encoderdecoder attention heads of transformer to affect source word selection. despite their successes, an obstacle of these approaches lies in the discrepancy between training and inference, that is, learning diverse translations from a single-reference corpus. this enlarges the confidence and quality gaps among candidate translations, limiting the potential of diverse nmt models. multi-candidate optimization in natural language generation. since single-reference corpus is insufficient to model one-to-many relations in natural language generation (nlg), researchers have introduced multi-candidate optimization to nlg tasks such as image captioning and paraphrasing. most of representative works among them generate pseudo training references and focus on improving diversity @xcite @xcite . conversely, in nmt community, previous studies on multicandidate optimization mainly aim at improving low-resource translation quality rather than diversity, which is similar to other data augmentation methods in nmt, such as back-translation and forward-translation @xcite @xcite . for example, khayrallah et al. (2020) improve the translation quality of low-resource language pairs by sampling paraphrases of the reference sentence. different from these tasks, diverse nmt is more challenging since it requires the generation results to be accurate as well as diverse. for better balancing quality and diversity, we propose a novel multi-candidate optimization framework with rl. reinforcement learning in nmt. reinforcement learning (rl) has become an appealing path for advancement in nmt, as it firstly allows to optimize non-differentiable objectives, and secondly reduces exposure bias in auto-regressive sequence generators. to this end, various methods have been proposed. in ranzato et al. (2016), wu et al. (2017) and edunov et al. (2018b), the authors employ the reinforce algorithm to optimize models with metric-based reward (i.e. senetnce-level bleu). different from them, he et al. (2016) propose to train two reverse nmt models through a duallearning mechanism. and bahdanau et al. (2017) diversity & quality rewards { " 𝑦 ! (#) , … , " 𝑦 % # , … , " 𝑦 & # } reward computation diverse nmt model 𝑥 (#) figure 1: our multi-candidate optimization framework: the diverse nmt model generates k candidate translations to receive individual diversity and quality rewards, which are then used for model optimization. use actor-critic method that predicts the reward by a critic network. in this work, we follow shen et al. (2016), wieting et al. (2019) and wang and sennrich (2020) that adopt minimum risk training (mrt) to minimize the reward during training. to the best of our knowledge, our work is the first attempt employing rl to model one-to-many relations for diverse nmt. ## multi-candidate optimization framework as a significant extension of conventional nmt, given a source sentence, a diverse nmt model aims at producing a set of different candidate translations. similar to conventional nmt, the most commonly-used training strategy of diverse nmt is to minimize the training objective based on maximum likelihood estimation (mle): to deal with the above issue, we propose a multicandidate optimization framework based on reinforcement learning (rl). as shown in figure 1 , a diverse nmt model generates k candidate translations using its original method as additional references during training. particularly, given a source sentence x (n) , the model picks an action each time it generates a candidate translation y (n) k . diversity and quality rewards of y (n) k are observed once it is completed, which are then used to optimize model parameters. please note that our framework is model-irrelevant and thus can be compatible with any diverse nmt model. next, we will introduce the reward computation and training procedure in following subsections. ## case study from figure 6 , we can see that there are only some simple substitutions (highlighted in blue) in transformer's results. tree2code generates more diverse translations, while containing more mis-translation and under-translation problems (highlighted in red). after applying our framework, tree2code+sct generates better translations in terms of both diversity and quality. ## conclusion in this paper, we first point out that the widely used single-reference training is not the preferred solution for diverse nmt. it causes discrepancy between training and inference, and prevents the model from learning one-to-many mapping relationships. consequently, we propose a novel multicandidate optimization framework which is modelirrelevant and can be compatible with any diverse nmt model. empirical results suggest that: 1) multi-candidate optimization is an universally effective manner on boosting the performance of diverse nmt; 2) model-based metrics can better reflect the translation quality than its n-gram-based counterpart under diverse nmt context; 3) multicandidate optimization offers nmt abilities to reduce the confidence variance and improve the translation quality of candidate translations.