Predicting and Using Target Length in Neural Machine Translation (2020)

Abstract: Attention-based encoder-decoder models have achieved great success in neural machine translation tasks. However, the lengths of the target sequences are not explicitly predicted in these models. This work proposes length prediction as an auxiliary task and set up a sub-network to obtain the length information from the encoder. Experimental results show that the length prediction sub-network brings improvements over the strong baseline system and that the predicted length can be used as an alternative to length normalization during decoding.

## introduction in recent years, neural network (nn) models have achieved great improvements in machine translation (mt) tasks. @xcite introduced the encoder-decoder network, @xcite developed the attention-based architecture, and @xcite proposed the transformer model with self-attentions, which delivers state-of-the-art performances. despite the success achieved in neural machine translation (nmt), current nmt systems do not model the length of the output explicitly, and thus various length normalization approaches are often used in decoding. length normalization is a common technique used in the beam search of nmt systems to enable a fair comparison of partial hypotheses with different lengths. without any form of length normalization, regular beam searches will prefer shorter hypotheses to longer ones on average, as a negative logarithmic probability is added at each step, resulting in lower (more negative) scores for longer sentences. the simplest way is to normalize the score of the current partial hypothesis (e i 1 ) by its length (|i|): in addition to investigating various types of length normalization, their rationality is rarely explored. although length normalization appears to be simple and effective, it is still an additional technique to help a "weak" machine translation model that cannot handle the hypothesis length properly. in this work it is proposed to model the target length using the neural network itself in a multi-task learning way. the estimated length information can either be implicitly included in the network to "guide" translation, or it can be used explicitly as an alternative to length normalization during decoding. the experimental results on various datasets show that the proposed system achieves improvements compared to the baseline model and the predicted length can easily be used to replace the length normalization. ## related work multi-task learning is an important training strategy that aims to improve the generalization performance of the main task with some other related tasks @xcite . with regard to deep learning, multitask learning is applied successfully in many areas, such as natural language processing @xcite , and speech processing @xcite . in this work, the prediction of the target length while generating translation hypotheses can be seen as a multi-task learning application. @xcite and @xcite attribute the fact that beam search prefers shorter candidates due to the local normalization of nmt. to address this problem, in addition to the standard length normalization technique, @xcite propose a more complicated correction with a hyperparameter that can be adjusted for different language pairs. in @xcite , a word reward function is proposed that simulates the coverage vector in statistical machine translation so that the decoder prefers a long translation. huang et al. (2017) and yang et al. (2018) suggest variations of this reward that provide better guarantees during search. there are also works on target vocabulary prediction in the encoder-decoder model that implicitly predicts the target length @xcite . in our work, the target length is explicitly modeled by the neural network itself, which indicates that the entire system relies more on statistics rather than heuristics. ## neural length model to predict the target length based on the standard transformer architecture @xcite , we build a multi-layer sub-network that only requires information from the source sequence (or the encoder). in this work the length prediction task is considered as a classification task for different lengths. other methods, such as directly generating a real number, binarizing the length, or performing multiple binary classification tasks, are also being tested, but the classification method performs best. ## conclusion in this paper, we propose a length prediction subnetwork based on the transformer architecture, and a method of using the length prediction information on the decoder side, namely cross-concat. in decoding, we use the predicted length to calculate a logarithmic linear penalty in the beam search in order to replace the length normalization. experimental results show that the sub-network can predict target length well and further improve translation quality. in addition, the predicted length can be used to replace the length normalization with a better and more mathematically explainable control of the output length. for future work, the use of length prediction in positional encoding @xcite and nonautoregressive (or partially autoregressive) nmt @xcite @xcite could be further investigated.