How Length Prediction Influence the Performance of Non-Autoregressive Translation? (2021)

Abstract: Length prediction is a special task in a series of NAT models where target length has to be determined before generation. However, the performance of length prediction and its influence on translation quality has seldom been discussed. In this paper, we present comprehensive analyses on length prediction task of NAT, aiming to find the factors that influence performance, as well as how it associates with translation quality. We mainly perform experiments based on Conditional Masked Language Model (CMLM) (Ghazvininejad et al., 2019), a representative NAT model, and evaluate it on two language pairs, En-De and En-Ro. We draw two conclusions: 1) The performance of length prediction is mainly influenced by properties of language pairs such as alignment pattern, word order or intrinsic length ratio, and is also affected by the usage of knowledge distilled data. 2) There is a positive correlation between the performance of the length prediction and the BLEU score.

## introduction though transformer @xcite has promoted conventional autoregressive generation (ar) by leveraging multi-head self-attention to avoid recurrence at training, decoders that generate each token conditioned on previously generated tokens still make it impossible to take full advantage of parallelism during inference: p(y |x) = t i p(y i |@xmath0. non-autoregressive machine translation (nat) @xcite was proposed to parallelize the generation by allowing the prediction of each token independently, known as the conditional independence assumption i.e. p(y |x) = t i p(y i |x). however, it always results in significant performance degradation. therefore, how to improve nat performance remains an open question. there are basically two directions of nat models. one is insertion-based models where multi-ple tokens can be inserted into an existing unfinished sentence, and the sentence length dynamically changes during insertion @xcite @xcite . another predetermines the target side length n and generates tokens in a fixed length space conditioned on wellpredicted tokens and optionally copied source embedding within n iterations @xcite @xcite . in this paper, we mainly focus on the critical component in the second direction -length prediction which has been rarely discussed. specifically, we investigate rq1: what influences accuracy of length prediction, and rq2: how accuracy of target sentence length prediction affects the translation quality. to answer these questions, we first analyze intrinsic properties of sentence lengths for different language pairs. then we evaluate various length prediction strategies and dig into potential influential factors. finally the correlation between accuracy of length prediction and translation quality is investigated. instead of predicting aligned length token by token mentioned above, naive linear projection has been applied extensively due to its simplicity @xcite @xcite , that is to learn a ratio α from the training set, mapping source length m to the length of the target side: @xmath1. obviously the frequentist statistical notion of α neglects the uniqueness of each individual instance. thus, classifiers based on the source representation are used to predict target length or length offset @xcite , but the fixed length determined in advance still limits the flexibility during inference. aiming to alleviate this issue, insertion-based models are proposed, they implicitly learn the latent variable length by dynamic insertion. @xcite generates text by inserting multiple tokens to readily existing unfinished sentence. levenshtein transformer (levt) @xcite extends the model by incorporating the deletion operation and decomposing the insertion operation into two steps -first predicting insertion positions and then specific tokens given the position. though various approaches are proposed to address the length prediction either explicitly or implicitly, comprehensive comparison, analysis on potential factors affecting their accuracy and the impact on final translation quality are not fully explored. in this work, we shall fill this gap. ## model and dataset in this section, we first introduce conditional masked language model (cmlm) framework @xcite , on which we perform all experiments throughout this work, followed by datasets details and the experimental setup. ## length prediction method modelling target length can be implemented in either simple or complex manners depending on learnable parameters, but it's mathematically defined as below: as such, it can be solved by any maximum likelihood estimation (mle) algorithms. ## error analysis in this section, we analyse the predicted length of nat models trained on distilled (denoted as d) and raw (denoted as r) data, as well as the length of the hypothesis from the teacher model (denoted as kd). we define the following metrics to quantify the error for each sentence and average them over the corpus: where the is the length error, + is the absolute error regardless of the skew, % is the error ratio that excludes the influence of the sentence length (e.g. 3 tokens error in a 5 tokens target and a 50 tokens target is different), and %+ is the absolute error ratio. all errors are measured in the subtokens level. ## influence on translations two additional factors are also discussed including the sentence length and the number of refinements. ## conclusion from the analysis of length prediction as well as related factors, we have a deeper understanding of it and draw following conclusions: specifically, even if 100% accuracy is achieved, it still can't peak the best bleu when only judged by reference translation which is not the unique ground truth, because it ignores the linguistic diversity. we find significant improvement can be obtained by a flexible length beam search by over 3 bleu. this reveals that, with regard to improving the nat translation, flexible decoding strategies are more effective than exhausting towards accurate length prediction, since essentially no ground truth of length exists owing to the complexity and diversity of languages. elegant parallel decoding methods are more promising, such as dynamically changing the length as we evaluated, and meanwhile retaining the simplicity of mask predicting of nat.