Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation (2022)

Abstract: Most dominant neural machine translation (NMT) models are restricted to make predictions only according to the local context of preceding words in a left-to-right manner. Although many previous studies try to incorporate global information into NMT models, there still exist limitations on how to effectively exploit bidirectional global context. In this paper, we propose a Confidence Based Bidirectional Global Context Aware (CBBGCA) training framework for NMT, where the NMT model is jointly trained with an auxiliary conditional masked language model (CMLM). The training consists of two stages: (1) multi-task joint training; (2) confidence based knowledge distillation. At the first stage, by sharing encoder parameters, the NMT model is additionally supervised by the signal from the CMLM decoder that contains bidirectional global contexts. Moreover, at the second stage, using the CMLM as teacher, we further pertinently incorporate bidirectional global context to the NMT model on its unconfidently-predicted target words via knowledge distillation. Experimental results show that our proposed CBBGCA training framework significantly improves the NMT model by +1.02, +1.30 and +0.57 BLEU scores on three large-scale translation datasets, namely WMT’14 English-to-German, WMT’19 Chinese-to-English and WMT’14 English-to-French, respectively.

## introduction in recent years, neural machine translation (nmt) has achieved great progress and attracted more attention. most dominant nmt models mainly adopt an encoder-decoder framework @xcite @xcite @xcite with the teacher-forcing strategy @xcite for training. despite its success, the unidirectional property of teacherforcing strategy restricts nmt models to only focus on the local context, i.e., the preceding words of the to-be-predicted target word at each decoder step. apparently, this strategy tends to be limited because word dependencies are always bidirectional involving both preceding and succeeding words on the target side. to address this issue, many previous researches attempt to exploit global information on the target side @xcite @xcite @xcite @xcite @xcite . typically, they introduce the modelling of target-side global context in the reverse direction by pairing the conventional left-to-right (l2r) nmt model with a right-to-left (r2l) auxiliary model. however, in these methods, the modelling of reverse global context is separate from the local context of preceding words. thus, they cannot sufficiently encourage the nmt model to exploit bidirectional global context @xcite . meanwhile, some of them adopt bidirectional decoding, which often relies on multi-pass decoding or specially customized decoding algorithms @xcite @xcite . another series of studies @xcite @xcite @xcite resort to leveraging target-side bidirectional global context contained in large-scale pre-trained language models (plm), such as elmo @xcite and bert @xcite . these plms are normally not bilingual-aware for translation and trained independently of the nmt model. as a special case, @xcite design a conditional masked language modelling objective to make bert aware of source input during the fine- figure 1: the distribution of the nmt-predicted probabilities to the corresponding ground-truth words on the training set of wmt'14 english-to-german translation task, which is output by a fully-trained transformer model using teacher-forcing strategy. for instance, the model predicts 25.67% target ground-truth words with probabilities between 0.0∼0.1 given totally correct preceding words at each time step. as for how to effectively incorporate global information into nmt models, another notable deficiency of previous work is that they do not pertinently enhance the nmt model according to its word-level prediction confidence. ideally, under the teacher-forcing strategy, a well-trained nmt model should assign high probabilities to the target ground-truth words based on correct previous words, which, however, is not the case. figure 1 depicts the predicted word-level probabilistic distribution of a fully-trained transformer model. we find that, even based on totally correct preceding words, there is a considerable portion of target groundtruth words that the model predicts with relatively low probabilities. the reasonable cause of this phenomenon is that the nmt model cannot confidently predict these target words according to only the local context of preceding words @xcite . hence, we should especially refine the nmt model on these unconfidently-predicted target words. in this paper, we propose a confidence based bidirectional global context aware (cbbgca) training framework for nmt. under our framework, the nmt model is jointly trained with a conditional masked language model (cmlm) which is essentially bilingual-aware and contains bidirectional global context on the target side. specifically, the cbbgca training consists of two stages. at the first stage, we jointly train the nmt model and cmlm in a multi-task learning manner by sharing the encoders of the two models. this preliminarily enhances the nmt model because the encoder is additionally supervised by the signal from the cmlm decoder that contains bidirectional global context. at the second stage, we employ the cmlm to pertinently refine the training of the nmt model on those unconfidently-predicted target words via confidence based knowledge distillation. by doing so, our model can be further encouraged to effectively leverage the bilingualaware bidirectional global context contained in the cmlm. to sum up, the major contributions of our paper are as follows: ## cbbgca training framework in this section, we will introduce our proposed cb-bgca training framework that employs a cmlm to enhance the nmt model according to its prediction confidence. in the following subsections, we first describe the basic architectures of our nmt model and cmlm. then, we introduce the training procedures of our cbbgca framework, involving two stages. ## analysis 4.1 ablation study to fully investigate each part of our proposed training framework, we conduct ablation studies on wmt'14 @xmath0. for the feasible options in two-stage strategy, "w/o. shareenc" represents not sharing encoders at the first training stage and its performance decreases by 0.56 bleu score. this shows that the nmt encoder is enhanced by the joint training with the cmlm. as for "w/o. cbkd" 4 , which means not performing kd on any target words at the second training stage (i.e., α=0), its performance also decreases by 0.40 bleu score. this demonstrates the effect of pertinently incorporating bidirectional global context into the nmt model on its unconfidently-predicted target words. ## conclusion in this paper, we propose a cbbgca training framework for the nmt model to effectively exploit target-side bidirectional global context with an auxiliary cmlm. the training consists of two stages. at the first stage, we introduce multi-task learning to benefit the nmt model by sharing its encoder with an auxiliary cmlm. then, at the second stage, through confidence based knowledge distillation, we use the cmlm as teacher to especially refine the nmt model on unconfidentlypredicted target words. experimental results show that our framework can significantly improve the nmt model. compared with previous work, neither external nor synthetic data are needed and only the nmt model is involved during inference. mechanism. we can see that the incorporation of external roberta brings some improvements to the nmt model under different settings (w/ roberta vs. w/o roberta). besides, it is notable that "cbbgca" still achieves consistent improvements over "transformer" and "multi-300k" regardless of whether using roberta or not, which demonstrates the compatibility and orthogonality of our proposed training framework to those approaches @xcite @xcite @xcite of integrating external large-scale plms.