Understanding and Addressing the Under-Translation Problem from the Perspective of Decoding Objective (2024)

Abstract: Neural Machine Translation (NMT) has made remarkable progress over the past years. However, under-translation and over-translation remain two challenging problems in state-of-the-art NMT systems. In this work, we conduct an in-depth analysis on the underlying cause of under-translation in NMT, providing an explanation from the perspective of decoding objective. To optimize the beam search objective, the model tends to overlook words it is less confident about, leading to the under-translation phenomenon. Correspondingly, the model’s confidence in predicting the End Of Sentence (EOS) diminishes when under-translation occurs, serving as a mild penalty for under-translated candidates. Building upon this analysis, we propose employing the confidence of predicting EOS as a detector for under-translation, and strengthening the confidence-based penalty to penalize candidates with a high risk of under-translation.Experiments on both synthetic and real-world data show that our method can accurately detect and rectify under-translated outputs, with minor impact on other correct translations.

## introduction neural machine translation (nmt) has made remarkable progress over the past years @xcite @xcite . under-translation and over-translation are two typical problems in nmt, where under-translation means some words are mistakenly untranslated, and over-translation means some words are unnecessarily translated for multiple times @xcite . despite the rapid progress of nmt, the mechanisms behind the occurrence of under-translation and over-translation remain unclear, and these problems continue to be challenging obstacles faced by nmt systems @xcite . prior efforts have primarily focused on modeling the explicit coverage of source content @xcite @xcite or enforcing the attention-based source coverage during decoding @xcite , without advancing the understanding of why nmt systems tend to overlook or repeat certain source words. @xcite empirically discovered that source words with a large translation entropy or those requiring reordering during translation are more likely to be ignored. however, this observation lacks a comprehensive explanation. in this work, we conduct an in-depth analysis on the underlying cause of under-translation in nmt, providing an explanation from the perspective of decoding objective. specifically, to gain deeper insights into the characteristics of under-translation and over-translation, we first generate synthetic data to simulate sentence-level and document-level translation scenarios, which allows us to automatically identify translation errors through predefined rules. the findings of sentence-level translation analysis indicate that words with high translation entropy, i.e., source words with a wide range of possible translations, are at a higher risk of being under-translated. in contrast, low-entropy words, whose translations are more predictable, are more likely to be over-translated, which aligns with the findings of @xcite . from the document-level translation experiments, we further observe the phenomenon of sentence-level under-translation, where the last sentence containing many high-entropy words is typically omitted. additionally, we notice that when under-translation occurs, the probability of predicting the end of sentence (eos) is generally lower, suggesting that the model is unwilling to stop generation when the translation is incomplete. although nmt imposes a penalty on predicting eos, under-translation still occurs frequently, which can be explained from the perspective of decoding objective. nmt typically employs beam search for decoding, with an objective of finding the most probable sentence, or maximizing the log-probability normalized by sentence length @xcite . consequently, nmt has a strong incentive to ignore high-entropy words, as they have low translation probabilities that contradict the decoding objective. moreover, we theoretically reveal that the lower eos probability serves as a penalty for under-translated candidates, but the penalty often underweighs the benefits of dropping multiple high-entropy words, leading to the occurrence of under-translation. building upon this analysis, we propose enhancing the eos penalty on under-translated candidates to prevent under-translation. since the model's confidence in predicting eos diminishes when undertranslation occurs, we employ the prediction probability of eos as a detector for under-translation. for beam search candidates with high risk of undertranslation, we take the eos probability as penalty and scale it to be proportional to the translation length. the detection ensures minimal interference with correct translations, and the scaling balances the impact of penalty with the benefits of dropping high-entropy source words, thereby more effectively preventing under-translations. in summary, our contributions are: ## experiments on synthetic data under-translation and over-translation are two challenging problems in neural machine translation, with one primary difficulty being the lack of automatic metrics for them. previous work has mainly relied on human annotators to identify translation errors, which is time-consuming. in this section, we introduce a method for constructing synthetic data to simulate sentence-level and document-level translation scenarios, which allows us to automatically identify under-translation and over-translation specifically, we create synthetic data containing only three words: a, b, and c, representing low-entropy, mid-entropy, and high-entropy words, respectively. we establish a fixed translation probability from source words to target words, as shown in ## approach as observed in the previous section, undertranslation poses a significant challenge in nmt, particularly when high-entropy words and difficult sentences are involved. to provide a comprehensive understanding of this phenomenon, we offers an intuitive explanation and theoretical analysis from the perspective of the decoding objective in section 3.1. following this, we investigate the relationship between the eos probability and undertranslation in section 3.2, and then design the eos penalty to prevent under-translation in section 3.3. ## related work research on the under-translation problem in nmt can be roughly divided into three categories: modeling source coverage, exploring translation entropy, and integrating penalties during the decoding. source coverage. @xcite pointed out the problems of under-translation and over-translation in neural machine translation and proposed a method for tracking source coverage vector to adjust the decoding of model. similarly, @xcite ; @xcite ; @xcite focused on modeling the coverage of source content. @xcite introduced a coverage embedding vector to track the coverage of source words. @xcite proposed to reconstruct the source sentence, thereby ensuring that all source information is included in the target side. @xcite addressed under-translation by making the model aware of translated and untranslated contents. translation entropy. @xcite discovered that source words with a large translation entropy or those requiring reordering during translation are more likely to be ignored, and proposed pre-ordering the source sentence or converting high-entropy words into special tokens to prevent under-translation. @xcite ; @xcite further employed entropy to assess translation difficulty, using this information to create test sets or enhance evaluation metrics. decoding penalty. @xcite proposed the length penalty and coverage penalty to penalize under-translation. the length penalty adjusts the log-probability score based on sentence length, reducing bias towards short sentences. the coverage penalty encourages source-target attention to cover all source words to prevent under-translation. ## conclusion in this paper, we conduct an in-depth study of the under-translation problem in nmt, explaining and addressing this problem from the perspective of the decoding objective. our analysis reveals that under-translation is more likely to occur on challenging words or sentences. we explain it from the perspective of the decoding objective and propose enhancing the eos penalty on under-translated candidates to prevent under-translation. experimental results show that our method effectively mitigates the occurrence of under-translation, with only a minor impact on other correct translations.