Incorporating Noisy Length Constraints into Transformer with Length-aware Positional Encodings (2020)

Abstract: Neural Machine Translation often suffers from an under-translation problem due to its limited modeling of output sequence lengths. In this work, we propose a novel approach to training a Transformer model using length constraints based on length-aware positional encoding (PE). Since length constraints with exact target sentence lengths degrade translation performance, we add random noise within a certain window size to the length constraints in the PE during the training. In the inference step, we predict the output lengths using input sequences and a BERT-based length prediction model. Experimental results in an ASPEC English-to-Japanese translation showed the proposed method produced translations with lengths close to the reference ones and outperformed a vanilla Transformer (especially in short sentences) by 3.22 points in BLEU. The average translation results using our length prediction model were also better than another baseline method using input lengths for the length constraints. The proposed noise injection improved robustness for length prediction errors, especially within the window size.

## introduction in autoregressive neural machine translation (nmt), a decoder generates one token at a time, and each output token depends on the output tokens generated so far. the decoder's prediction of the end of the sentence determines the length of the output sentence. this prediction is sometimes made too earlybefore all of the input information is translated-causing a so-called under-translation. transformer has sinusoidal positional encoding to incorporate the token position information in the sequence into its encoder and decoder @xcite . there are some previous studies for controlling an output length in transformer. @xcite proposed two variants of length-aware positional encodings called length-ratio positional encoding (lrpe) and length-difference positional encoding (ldpe) to control the output length based on the given length constraints in automatic summarization. @xcite applied ldpe and lrpe to nmt. they trained an nmt model using output length constraints based on ldpe and lrpe along with special tokens representing length ratio classes between input and output sentences, while they used the input sentence length at the inference time. however, the length of an input sentence is not a reliable estimator of the output length, because the actual output length varies with the content of the input. using length constraints in the decoder is a promising approach to the under-translation problem. we propose an nmt method based on lrpe and ldpe with a bert-based output length prediction. the proposed method adds noise to the output length constraints in training to improve its robustness against the possible length variances in the translation. in our experiments with an english-to-japanese dataset, the bert-based output length prediction outperformed the use of the input length, and the proposed method, including noise injection into the training-time length constraints, improved the translation performance in bleu for short sentences. ## positional encoding for control output length the following is the sinusoidal positional encoding (pe) proposed by @xcite : ## proposed method the transformer-based model with lrpe and ldpe generates a sequence that almost matches the given length. this characteristic is not always appropriate in the problem of machine translation because some translation variants have different lengths. in this paper, we incorporate random noise to the length constraints used by lrpe and ldpe during training to improve the robustness for such length variants. we also propose using an output length prediction based on bert. noise injection is expected to improve the robustness against possible length prediction errors. ## experiments to investigate the performance of our proposed method, we conducted english-to-japanese translation experiments between a vanilla transformer and its variants with lrpe and ldpe, implemented using opennmt @xcite . ## conclusion we proposed an nmt method using length-aware positional encodings with a training-time length noise injection and a bert-based inference-time length prediction. robustness for translation length variations, including length prediction errors, especially for those within the noise window size. the experimental results show the effectiveness of the proposed method in short sentences. our future work will pursue a more effective output length prediction to suppress overtranslations because we also found some over-translations in the test set, possibly caused by the length constraints.