Prevent the Language Model from being Overconfident in Neural Machine Translation (2021)

Abstract: The Neural Machine Translation (NMT) model is essentially a joint language model conditioned on both the source sentence and partial translation. Therefore, the NMT model naturally involves the mechanism of the Language Model (LM) that predicts the next token only based on partial translation. Despite its success, NMT still suffers from the hallucination problem, generating fluent but inadequate translations. The main reason is that NMT pays excessive attention to the partial translation while neglecting the source sentence to some extent, namely overconfidence of the LM. Accordingly, we define the Margin between the NMT and the LM, calculated by subtracting the predicted probability of the LM from that of the NMT model for each token. The Margin is negatively correlated to the overconfidence degree of the LM. Based on the property, we propose a Margin-based Token-level Objective (MTO) and a Margin-based Sentence-level Objective (MSO) to maximize the Margin for preventing the LM from being overconfident. Experiments on WMT14 English-to-German, WMT19 Chinese-to-English, and WMT14 English-to-French translation tasks demonstrate the effectiveness of our approach, with 1.36, 1.50, and 0.63 BLEU improvements, respectively, compared to the Transformer baseline. The human evaluation further verifies that our approaches improve translation adequacy as well as fluency.

## introduction neural machine translation (nmt) has achieved great success in recent years @xcite @xcite @xcite @xcite , which generates accurate and fluent translation through modeling the next word conditioned on both the source sentence and partial translation. however, nmt faces the hallucination problem, i.e., translations are fluent but inadequate to the source sentences. one important reason is that the nmt model pays excessive attention to the partial translation to ensure fluency while failing to translate some segments of the source sentence @xcite , which is actually the overconfidence of the language model (lm). in the rest of this paper, the lm mentioned refers to the lm mechanism involved in nmt. many recent studies attempt to deal with the inadequacy problem of nmt from two main aspects. one is to improve the architecture of nmt, such as adding a coverage vector to track the attention history @xcite , enhancing the crossattention module @xcite @xcite , and dividing the source sentence into past and future parts @xcite . the other aims to propose a heuristic adequacy metric or objective based on the output of nmt. @xcite and @xcite enhance the model's reconstruction ability and increase the coverage ratio of the source sentences by translations, respectively. although some researches @xcite @xcite point out that the lack of adequacy is due to the overconfidence of the lm, unfortunately, they do not propose effective solutions to the overconfidence problem. from the perspective of preventing the overconfidence of the lm, we first define an indicator of the overconfidence degree of the lm, called the margin between the nmt and the lm, by subtracting the predicted probability of the lm from that of the nmt model for each token. a small mar-gin implies that the nmt might concentrate on the partial translation and degrade into the lm, i.e., the lm is overconfident. accordingly, we propose a margin-based token-level objective (mto) to maximize the margin. furthermore, we observe a phenomenon that if target sentences in the training data contain many words with negative margin, they always do not correspond to the source sentences. these data are harmful to model performance. therefore, based on the mto, we further propose a margin-based sentence-level objective (mso) by adding a dynamic weight function to alleviate the negative effect of these "dirty data". we validate the effectiveness and superiority of our approaches on the transformer @xcite , and conduct experiments on large-scale wmt14 english-to-german, wmt19 chinese-to-english, and wmt14 english-to-french translation tasks. our contributions are: ## background given a source sentence @xmath0, x 2 , ..., x n }, the nmt model predicts the probability of a target sentence @xmath1, y 2 , ..., y t } word by word: based on the encoder-decoder architecture, the encoder of nmt maps the input sentence x to hidden states. at time step t, the decoder of nmt employs the output of the encoder and @xmath2. the training objective of nmt is to minimize the negative log-likelihood, which is also known as the cross entropy loss function: the lm measures the probability of a target sentence similar to nmt but without knowledge of the source sentence x: the lm can be regarded as the part of nmt decoder that is responsible for fluency, only takes @xmath3. the training objective of the lm is almost the same as nmt except for the source sentence x: the nmt model predicts the next word y t according to the source sentence x and meanwhile ensures that y t is fluent with the partial translation @xmath4. however, when nmt pays excessive attention to translation fluency, some source segments may be neglected, leading to inadequacy problem. this is exactly what we aim to address in this paper. ## the approach in this section, we firstly define the margin between the nmt and the lm (section 3.1), which reflects the overconfidence degree of the lm. then we put forward the token-level (section 3.2) and sentencelevel (section 3.3) optimization objectives to maximize the margin. finally, we elaborate our twostage training strategy (section 3.4). ## experimental settings we conduct experiments on three large-scale nmt tasks, i.e., wmt14 english-to-german (@xmath5, wmt14 english-to-french (@xmath6, and wmt19 chinese-to-english (@xmath7. datasets. for @xmath8, we use 4.5m training data. following the same setting in @xcite , we use newstest2013 as validation set and newstest2014 as test set, which contain 3000 and 3003 sentences, respectively. for @xmath9, the training dataset contains about 36m sentence pairs, and we use newstest2013 with 3000 sentences as validation set and newstest2014 with 3003 sentences as test set. for @xmath10, we use 20.5m training data and use newstest2018 as validation set and new-stest2019 as test set, which contain 3981 and 2000 sentences, respectively. for @xmath11, the number of merge operations in byte pair encoding (bpe) @xcite ) is set to 32k for both source and target languages. for @xmath12, we use a shared vocabulary generated by 32k bpes. evaluation. we measure the case-sensitive bleu scores using multi-bleu.perl foot_4 for @xmath13. for @xmath14, case-sensitive bleu scores are calculated by moses mteval-v13a.pl script foot_5 . moreover, we use the paired bootstrap resampling @xcite for significance test. we select the model which performs the best on the validation sets and report its performance on the test sets for evaluation. model and hyperparameters. we conduct experiments based on the transformer @xcite and implement our approaches with the opensource tooklit opennmt-py @xcite . following the transformer-base setting in @xcite , we set the hidden size to 512 and the encoder/decoder layers to 6. all three tasks are trained with 8 nvidia v100 gpus, and the batch size for each gpu is 4096 tokens. the beam size is 5 and the length penalty is 0.6. adam optimizer (kingma and ba, 2014) is used in all the models. the lm architecture is the decoder of the transformer excluding the cross-attention layers, sharing the embedding layer and the pre-softmax linear transformation with the nmt model. for @xmath15, @xmath16, and @xmath17, the number of training steps is 150k for jointly pretraining stage and 150k for finetuning foot_6 . during pretraining, we set λ lm to 0.01 for all three tasks foot_7 . experimental results shown in appendix a indicate that the lm has converged after pretraining for all the three tasks. during finetuning, the margin function m(∆) in section 3.2 is set to quintic, and we will analyze the four m(∆) in section 5.4. λ m in eq. 7 is set to 5, 8, and 8 on @xmath18, @xmath19, respectively. for mso, the threshold k in eq. 10 is set to 30% for @xmath20, 40% for @xmath21. the two hyperparameters (i.e., λ m and k) are searched on validation sets, and the selection details are shown in appendix b. the baseline model (i.e., vanilla transformer) is trained for 300k steps for @xmath22, @xmath23. moreover, we use a joint training model as our secondary baseline, namely nmt+lm, by jointly training the nmt model and the lm throughout the training stage with 300k steps. the training steps of all the models are consistent, thus the experiment results are strictly comparable. ## results and analysis we first evaluate the main performance of our approaches (section 5.1 and 5.2). then, the human evaluation further confirms the improvements of translation adequacy and fluency (section 5.3). finally, we analyze the positive impact of our models on the distribution of margin and explore how each fragment of our method works (section 5.4). ## related work translation adequacy of nmt. nmt suffers from the hallucination and inadequacy problem for a long time @xcite @xcite . many studies improve the architecture of nmt to alleviate the inadequacy issue, including tracking translation adequacy by coverage vectors @xcite , modeling a global representation of source side @xcite , dividing the source sentence into past and future parts @xcite , and multi-task learning to improve encoder and cross-attention modules in decoder @xcite @xcite . they inductively increase the translation adequacy, while our approaches directly maximize the margin between the nmt and the lm to prevent the lm from being overconfident. other studies enhance the translation adequacy by adequacy metrics or additional optimization objectives. @xcite minimize the difference between the original source sentence and the reconstruction source sentence of nmt. @xcite pro-pose a coverage ratio of the source sentence by the model translation. @xcite evaluate the fluency and adequacy of translations with an evaluation module. however, the metrics or objectives in the above approaches may not wholly represent adequacy. on the contrary, our approaches are derived from the criteria of the nmt model and the lm, thus credible. language model augmented nmt. language models are always used to provide more information to improve nmt. for low-resource tasks, the lm trained on extra monolingual data can rerank the translations by fusion @xcite @xcite , enhance nmt's representations @xcite , and provide prior knowledge for nmt @xcite . for data augmentation, lms are used to replace words in sentences @xcite @xcite . differently, we mainly focus on the margin between the nmt and the lm, and no additional data is required. @xcite propose the simple fusion approach to model the difference between nmt and lm. differently, it is trained to optimize the residual probability, positively correlated to p n m t /p lm which is hard to optimize and the lm is still required in inference, slowing down the inference speed largely. ## conclusion we alleviate the problem of inadequacy translation from the perspective of preventing the lm from being overconfident. specifically, we firstly propose an indicator of the overconfidence degree of the lm in nmt, i.e., margin between the nmt and the lm. then we propose margin-based token-level and sentence-level objectives to maximize the margin. experimental results on three large-scale translation tasks demonstrate the effectiveness and superiority of our approaches. the human evaluation further verifies that our methods can improve translation adequacy and fluency. future for neural machine translation. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 931-941, hong kong, china. association for computational linguistics. jinhua zhu, yingce xia, lijun wu, di he, tao qin, wengang zhou, houqiang li, and tieyan liu. 2020. incorporating bert into neural machine translation. in international conference on learning representations.