Uncertainty-Aware Semantic Augmentation for Neural Machine Translation (2020)

Abstract: As a sequence-to-sequence generation task, neural machine translation (NMT) naturally contains intrinsic uncertainty, where a single sentence in one language has multiple valid counterparts in the other. However, the dominant methods for NMT only observe one of them from the parallel corpora for the model training but have to deal with adequate variations under the same meaning at inference. This leads to a discrepancy of the data distribution between the training and the inference phases. To address this problem, we propose uncertainty-aware semantic augmentation, which explicitly captures the universal semantic information among multiple semantically-equivalent source sentences and enhances the hidden representations with this information for better translations. Extensive experiments on various translation tasks reveal that our approach significantly outperforms the strong baselines and the existing methods.

## introduction in recent years neural machine translation (nmt) has demonstrated state-of-the-art performance on many language pairs with advanced architectures and large scale data @xcite @xcite . at training time the parallel data only contains one source sentence as the input and the rest reasonable ones are ignored, while at inference the resulting model has to deal with adequate variations under the same meaning. this discrepancy of the data distribution poses a formidable learning challenge of the inherent uncertainty in machine translation. since typically there are several semantically-equivalent source sentences that can be translated to the same target sentence, but the model only observes one at training time. thus it is natural to enable an nmt model trained with the token-level cross-entropy (ce) to capture such a rich distribution, which exactly motivates our work. intuitively, the nmt model should be trained under the guidance of the same latent semantics that it will access at inference time. in their seminal work, the variational models @xcite @xcite introduce a continuous latent variable to serve as a global semantic signal to guide the generation of target translations. @xcite consider an universal topic representation for each sentence pair as global semantics for enhancing representations learnt by nmt models. @xcite minimize the difference between the representation of source and target sentences. although their methods yield notable results, they are still limited to one-to-one parallel sentence pairs. to address these problems, we present a novel uncertainty-aware semantic augmentation method, which takes account of the intrinsic uncertainty sourced from the one-to-many nature of machine translation @xcite . specifically, we first synthesize multiple reasonable source sentences to play the role of inherent uncertainty 1 for each target sentence. to achieve this, we introduce a controllable sampling strategy to cover adequate variations for inputs, by quantifying the sharpness of the word distribution in each decoding step and taking the proper word (the one with the maximum probability if sharp enough or determined by multinomial sampling) as the output. then a semantic constrained network (scn) is developed to summarize multiple source sentences that share the same meaning into a closed semantic region, augmented by which the model generates translations finally. by integrating such soft correspondences into the translation process, the model can intuitively work well when fed with an unfamiliar literal expression that can be supported by its underlying semantics. in addition, given the effectiveness of leveraging monolingual data in improving translation quality @xcite , we further propose to combine the strength of both semantic augmentation and massive monolingual data distributed in the target language. we conduct extensive experiments in both a supervised setup with bilingual data only, and a semi-supervised setup where both bilingual and target monolingual data are available. we evaluate the proposed approach on the widely used wmt14 @xmath0, wmt16 @xmath1, nist @xmath2. experimental results show that the proposed approach consistently improves translation performance on multiple language pairs. as another bonus, by adding monolingual data in german, our approach yields an additional gain of +1.5∼+3.3 bleu points on wmt16 @xmath3. extensive analyses reveal that: ## preliminary neural machine translation @xcite directly models the translation probability of a target sentence @xmath4, ..., y ty given its corresponding source sentence @xmath5, ..., x tx : related work on data augmentation. da has been used to improve the diversity of training signals for nmt models, like randomly shuffle (swap) or drop some words in a sentence @xcite @xcite , randomly replace one word in the original sentences with another word @xcite @xcite @xcite , syntax-aware methods @xcite , as well as using target monolingual data @xcite @xcite @xcite @xcite @xcite . more recently, @xcite introduce several variations of sampling strategies targeting difficult-to-predict words. @xcite have studied that what benefits from data augmentation across different methods and tasks. @xcite propose to improve the robustness of nmt models towards perturbations and minor errors by introducing adversarial inputs into training process. in contrast, we aim at bridging the discrepancy of the data distribution between the training and the inference phases, through augmenting each training instance with multiple semantically-equivalent source inputs. related work on uncertainty in nmt. recently, there are increasing number of studies investigating the effects of quantifying uncertainties in different applications @xcite @xcite @xcite . however, most work in nmt has focused on improving accuracy without much consideration for the intrinsic uncertainty of the translation task itself. in their seminal work, the latent variable models @xcite introduce a (set of) continuous latent variable(s) to model underlying semantics of source sentences and to guide the generation of target translations. @xcite propose a forest-to-sequence nmt model to make use of exponentially many parse trees of the source sentence. @xcite have focused on analyzing the uncertainty in nmt that demonstrate how uncertainty is captured by the model distribution and how it affects search strategies. @xcite propose to quantify the confidence of nmt model predictions based on model uncertainty. our work significantly differs from theirs. we model the inherent uncertainty by representing multiple source sentences into a closed semantic region, and use this semantic information to enhance nmt models where diverse literal expressions intuitively be supported by their underlying semantics. ## uncertainty-aware semantic augmentation for nmt here, we present the uncertainty-aware semantic augmentation (as shown in figure 1 ), which takes account of the intrinsic uncertainty of machine translation and enhances the latent representation semantically. for each sentence-pair (x, y), supposing x (y) is a set of correct source sentences for y, in which each sentence x is assumed to have the same meaning as x. given a training corpus d, we introduce the objective function as: to encourage the scn to extract the core semantics (z and z) for x and x respectively, while constraining them into a closed semantic region. it is formulated as the negative kullback-leibler (kl) divergence between the semantic distributions p φ (z|x) and p φ (z|x), where φ denotes the combined parameters of the encoder and the scn. intuitively, our new objective is exactly a regularized version of the widely used maximum likelihood estimation (mle) in conventional nmt. the models are trained to optimize both the translation loss and the semantic agreement between x and x. in the following sections, we will first describe how to summarize multiple source sentences into a closed semantic region by developing a semantic constrained network (scn) in section 3.1. and then introduce the proposed controllable sampling strategy in section 3.2 to construct adequate and reasonable variations for source inputs. ## experiments we examine our method upon advanced trans-former @xcite and conduct experiments on four widely used translation tasks, including wmt14 @xmath6, wmt16 @xmath7, nist @xmath8. ## conclusion and future work we present an uncertainty-aware semantic augmentation method to bridge the discrepancy of the data distribution between the training and the inference phases for dominant nmt models. in particular, we first synthesize a proper number of source sentences to play the role of intrinsic uncertainties via the controllable sampling for each target sentence. then, we develop a semantic constrained network to summarize multiple source inputs into a closed semantic region which is then utilized to augment latent representations. experiments on wmt14 @xmath9, wmt16 @xmath10, nist @xmath11. while we showed that uncertainty-aware semantic augmentation with gaussian priors is effective, more work is required to investigate if such an approach will also be successful for more sophisticated priors. in addition, learning universal representations among semantically-equivalent source and target sentences @xcite ) can complete the proposed method.