Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation (2022)

Abstract: Token-level adaptive training approaches can alleviate the token imbalance problem and thus improve neural machine translation, through re-weighting the losses of different target tokens based on specific statistical metrics (e.g., token frequency or mutual information). Given that standard translation models make predictions on the condition of previous target contexts, we argue that the above statistical metrics ignore target context information and may assign inappropriate weights to target tokens. While one possible solution is to directly take target contexts into these statistical metrics, the target-context-aware statistical computing is extremely expensive, and the corresponding storage overhead is unrealistic. To solve the above issues, we propose a target-context-aware metric, named conditional bilingual mutual information (CBMI), which makes it feasible to supplement target context information for statistical metrics. Particularly, our CBMI can be formalized as the log quotient of the translation model probability and language model probability by decomposing the conditional joint distribution. Thus CBMI can be efficiently calculated during model training without any pre-specific statistical calculations and large storage overhead. Furthermore, we propose an effective adaptive training approach based on both the token- and sentence-level CBMI. Experimental results on WMT14 English-German and WMT19 Chinese-English tasks show our approach can significantly outperform the Transformer baseline and other related methods.

## introduction neural machine translation (nmt) @xcite @xcite @xcite source: target: píng wěn jiā sù huí fù zhèng cháng xíng shǐ sù dù , róng rù nǐ zhōu wéi de chē liú 。 return to normal traffic speed by accelerating smoothly to blend with the traffic around you . despite the different mappings from the source sentence, existing target-context-free metrics (i.e., frequency and bmi) equally assess the two 'traffic' tokens, while our cbmi can distinguish the different dependencies of the two tokens on the source sentence with the guidance of target contexts. recently, various adaptive training approaches @xcite have been proposed to alleviate the above problem for nmt. generally, these approaches re-weight the losses of different target tokens based on specific statistical metrics. for example, @xcite take the token frequency as an indicator and encourage the nmt model to focus more on low-frequency tokens. @xcite further propose the bilingual mutual information (bmi) to measure the word mapping diversity between bilinguals, and down-weight the tokens with relatively lower bmi values. despite their achievements, there are still limita-tions in these adaptive training approaches. given that the standard translation model autoregressively makes predictions on the condition of previous target contexts, we argue that the statistical metrics used in the above approaches ignore target context information and may assign inaccurate weights for target tokens. specifically, although existing statistical metrics can reflect complex characteristics of target tokens (e.g., mapping diversity), they fail to model how these properties vary across different target contexts. secondly, for the identical target tokens in different positions of a target sentence (e.g., two 'traffic' tokens in the figure 1 ), they may be mapped from different source-side tokens, but such target-context-free metrics cannot distinguish the above different mappings. in summary, it is necessary to incorporate target context information into the above statistical metrics. one possible solution is to directly take target context information into account and conduct target-context-aware statistical calculations. but in this way, the calculation cost and storage overhead will become huge and unrealistic 1 . therefore, it is non-trivial to design a suitable target-context-aware statistical metric for adaptive training in the field of nmt. in this paper, we aim to address the above issues in adaptive training methods. firstly, we propose a novel target-context-aware metric, named conditional bilingual mutual information (cbmi), to measure the importance of different target tokens by their dependence on the source sentence. specifically, we calculate cbmi by the mutual information between a target token and its source sentence on the condition of its target contexts. with the aid of target-context-aware calculations, cbmi can easily model the various characteristics of target tokens under different target contexts, and of course can distinguish identical target tokens with different source mappings. regarding the computational efficiency, through decomposing the conditional joint distribution in the aforementioned mutual information, our cbmi can be formalized as the log quotient of the translation model probability and language model probability 2 . therefore, cbmi can be efficiently calculated dur-1 take the vanilla bmi @xcite as an example, to process the raw wmt14 en-de training data (about 1.5gb), it takes about 12 cpu hours and 2gb disk storage to save the bmi values. to make matters worse, the cost will increase dozens of times in target-context-aware statistical calculations. 2 the detailed derivation process is shown in equation ( 7 ). please note that the language model is only used during training and thus does not affect the inference speed. we evaluate our approach on the wmt14 english-german and wmt19 chinese-english translation tasks. experimental results on both datasets demonstrate that our approach can significantly outperform the transformer baseline and other adaptive training methods. further analyses reveal that cbmi can also reflect the adequacy of translation, and our cbmi-based adaptive training can improve translation adequacy meanwhile maintain fluency. the main contributions of this paper can be summarized as follows: ## approaches in this section, we first introduce the definition of cbmi (section 3.1). then, we illustrate how to adjust the weights for the training losses of target tokens based on the token-and the sentence-level cbmi (section 3.2). figure 2 shows the overall training process of our approach. ## analysis in this section, we provide in-depth analyses on the effectiveness of our cbmi and conduct experiments on the validation set of wmt14 en-de with the transformer base model. ## conclusion in this paper, we propose a target-context-aware metric for target tokens, named conditional bilingual mutual information (cbmi). compared with previous statistical metrics, our cbmi only increases limited computational costs to incorporate the target context and provides a more suitable assessment for tokens. furthermore, based on the token-and sentence-level cbmi, we design a cbmi-based adaptive training strategy to amply the contributions of the important tokens. experimental results on two wmt tasks demonstrate the effectiveness of our proposed approach. further analyses show that cbmi can improve translation adequacy and serve as an appropriate indicator for the translation prior selection. the cbmi prior on the tokens in the three intervals. formally, the prior distribution q(y j ) for target token y j can be represented as follows: subsequently, we calculate the cross-entropy loss between the selected prior distribution and the model predicted distribution as an additional term and incorporate it with the original cross-entropy loss in equation ( 2 ) to make up the new training objective: to verify the reasonablility of the prior selection strategy, we compare the effects of the three priors on each single cbmi intervals in figure 5 . as we expected, the bleu results also conform with the accuracy results in figure 4 , indicating that the most helpful prior distribution can be highly related to the cbmi values of tokens.