Non-Autoregressive Machine Translation: It’s Not as Fast as it Seems (2022)

Abstract: Efficient machine translation models are commercially important as they can increase inference speeds, and reduce costs and carbon emissions. Recently, there has been much interest in non-autoregressive (NAR) models, which promise faster translation. In parallel to the research on NAR models, there have been successful attempts to create optimized autoregressive models as part of the WMT shared task on efficient translation. In this paper, we point out flaws in the evaluation methodology present in the literature on NAR models and we provide a fair comparison between a state-of-the-art NAR model and the autoregressive submissions to the shared task. We make the case for consistent evaluation of NAR models, and also for the importance of comparing NAR models with other widely used methods for improving efficiency. We run experiments with a connectionist-temporal-classification-based (CTC) NAR model implemented in C++ and compare it with AR models using wall clock times. Our results show that, although NAR models are faster on GPUs, with small batch sizes, they are almost always slower under more realistic usage conditions. We call for more realistic and extensive evaluation of NAR models in future work.

## introduction non-autoregressive neural machine translation (nar nmt, or nat; @xcite is an emerging subfield of nmt which focuses on increasing the translation speed by changing the model architecture. the defining feature of non-autoregressive models is the conditional independence assumption on the output probability distributions; this is in contrast to autoregressive models, where the output distributions are conditioned on the previous outputs. this conditional independence allows one to decode the target tokens in parallel. this can substantially reduce the decoding time, especially for longer target sentences. the decoding speed is assessed by translating a test set and measuring the overall time the process takes. this may sound simple, but there are various aspects to be considered that can affect decoding speed, such as batching, number of hypotheses in beam search or hardware used (i.e., using cpu or gpu). decoding speed evaluation is a challenging task, especially when it comes to comparability across different approaches. unlike translation quality, decoding speed can be measured exactly. however, also unlike translation quality, different results are obtained from the same system under different evaluation environments. the wmt efficient translation shared task aims to evaluate efficiency research and encourages the reporting of a range of speed and translation quality values to better understand the trade-off across different model configurations @xcite . in this paper, we follow the emerging best practices developed in the efficiency shared task and directly compare with the submitted systems. in the development of nar models, modeling error and its subsequent negative effect on translation quality remains the biggest issue. therefore, the goal of contemporary research is to close the performance gap between the ar models and their nar counterparts, while maintaining high decoding speed. considering these stated research goals, the evaluation should comprise of assessing translation quality as well as decoding speed. translation quality is usually evaluated by scoring translations of an unseen test set either using automatic metrics, such as bleu @xcite , chrf @xcite or comet @xcite , or using human evaluation. to prevent methods from eventually overfitting to a single test set, new test sets are published each year as part of the wmt news translation shared task. in contrast, translation quality evaluation in nar research is measured almost exclusively on the wmt 14 english-german test set, using only bleu scores. automatic evaluation of translation quality remains an open research problem, but current research advises against relying on a single metric, and especially against relying on only bleu @xcite . in our experiments, we follow the recent best practices by using multiple metrics and recent test sets. in this paper, we examine the evaluation methodology generally accepted in literature on nar methods, and we identify a number of flaws. first, the results are reported on different hardware architectures, which makes them incomparable, even when comparing only relative speedups. second, most of the methods only report latency (decoding with a single sentence per batch) using a gpu; we show that this is the only setup favors nar models. third, the reported baseline performance is usually questionable, both in terms of speed and translation quality. finally, despite the fact that the main motivation for using nar models is the lower time complexity, the findings of the efficiency task are ignored in most of the nar papers. we try to connect the separate worlds of nar and efficient translation research. we train nonautoregressive models based on connectionist temporal classification (ctc), an approach previously shown to be effective @xcite @xcite . we employ a number of techniques for improving the translation quality, including data cleaning and sequence-level knowledge distillation @xcite . we evaluate our models following a unified evaluation methodology: in order to compare the translation quality with the rest of the nar literature, we report bleu scores measured on the wmt 14 test set, on which we achieve state-ofthe-art performance among (both single-step and iterative) nar methods; we also evaluate the translation quality and decoding speed of our models in the same conditions as the efficiency task. we find that despite achieving very good results among the nat models on the wmt 14 test set, our models fall behind in translation quality when measured on the recent wmt 21 test set using three different automatic evaluation metrics. moreover, we show that gpu decoding latency is the only scenario in which non-autoregressive models outperform autoregressive models. this paper contributes to the research commu-nity in the following aspects: first, we point out weaknesses in standard evaluation methodology of non-autoregressive models. second, we link the worlds of non-autoregressive translation and optimization of autoregressive models to provide a better understanding of the results achieved in the related work. ## non-autoregressive nmt the current state-of-the-art nmt models are autoregressive -the output distributions are conditioned on the previously generated tokens @xcite . the decoding process is sequential in its nature, limiting the opportunities for parallelization. non-autoregressive models use output distributions which are conditionally independent of each other, which opens up the possibility of parallelization. formally, the probability of a sequence y given the input x in a non-autoregressive model with parameters θ is modeled as unsurprisingly, the independence assumption in nar models has a negative impact on the translation quality. the culprit for this behavior is the multimodality problem -the inability of the model to differentiate between different modes of the joint probability distribution over output sequences inside the distributions corresponding to individual time steps. a classic example of this issue is the sentence "thank you" with its two equally probable german translations "danke schön" and "vielen dank" @xcite . because of the independence assumption, a non-autoregressive model cannot assign high probabilities to these two translations without also allowing for the incorrect sentences "vielen schön" and "danke dank". knowledge distillation @xcite ) has been successfully employed to reduce the negative influence of the multimodality problem in nar models @xcite . synthetic data tends to be less diverse than authentic texts, therefore the number of equally likely translation candidates gets smaller @xcite . a number of techniques have been proposed for training nar models, including iterative methods @xcite , auxiliary training objectives @xcite , or latent variables @xcite @xcite . in some form, all of the aforementioned approaches use explicit target length estimation, and rely on one-to-one correspondence between the output distributions and the reference sentence. a group of methods that relax the requirement of the strict one-to-one alignment between the model outputs and the ground-truth target sequence include aligned cross-entropy @xcite and connectionist temporal classification @xcite . the schema of the ctc-based model, as proposed by @xcite , is shown in figure 1 . the model extends the transformer architecture @xcite . it consists of an encoder, a state-splitting layer, and a nonautoregressive decoder. the encoder has the same architecture as in the transformer model. the state-splitting layer, applied on the encoder output, linearly projects and splits each state into k states with the same dimension. the decoder consists of a stack of transformer layers. unlike the transformer model, the self-attention in the non-autoregressive decoder does not use the causal mask, so the model is not prevented from attending to future states. since the output length is fixed to k-times the length of the source sentence, the model is permitted to output blank tokens. different positions of the blank tokens in the output sequence represent different alignments between the outputs and the ground-truth sequence. connectionist temporal classification @xcite is a dynamic algorithm that efficiently computes the standard cross-entropy loss summed over all possible alignments. we choose the ctc-based architecture for our models because it has been previously shown to be effective for nar nmt @xcite and performs well in the context of non-autoregressive research. it is also one of the fastest nar architectures since it is not iterative. ## evaluation methodology the research goal of the non-autoregressive methods is to improve translation quality while maintaining the speedup brought by the conditional independence assumption. this means that careful thought should be given to both quantifying the speed gains and the translation quality evaluation. the speed-vs-quality trade-off can be characterized by the pareto frontier. in this section we discuss the evaluation from both perspectives. translation quality. in the world of nonautoregressive nmt, the experimental settings are not very diverse. the primary language pair for translation experiments is english-german, sometimes accompanied by english-romanian to simulate the low-resource scenario. these language pairs, along with the widely used test sets -wmt 14 @xcite for en-de and wmt 16 @xcite for en-ro -became the de facto standard benchmark for nar model evaluation. a common weakness seen in the literature is the use of weak baseline models. the base variant of the transformer model is used almost exclusively @xcite @xcite @xcite . we argue that using weaker baselines might lead to overrating the positive effects brought by proposed improvements. since the baseline autoregressive models are used to generate the synthetic parallel data for knowledge distillation, the weakness is potentially further amplified in this step. evaluation is normally with automatic metrics only, and often only bleu is reported. in light of recent research casting further doubt on the reliability of bleu as a measure of translation quality @xcite , we argue that this is insufficient. decoding speed. the current standard in evaluation of nar models is to measure translation latency using a gpu, i.e., the average time to translate a single sentence without batching. since the time depends on the hardware, relative speedup is usually reported along with latency. this is a reasonable approach but we need to keep in mind the associated difficulties. first, the results achieved on different hardware architectures are not easily comparable even when considering the relative speedups. we also note that the relative speedup values should always be accompanied by the corresponding decoding times in absolute numbers. sometimes, this information is missing from the published results @xcite . we argue that measuring only gpu latency disregards other use-cases. in the wmt efficiency shared task, the decoding speed is measured in five scenarios. the speed is reported using a gpu with and without batching, using all 36 cpu cores (also, with and without batching), and using a single cpu core without batching. in batched decoding, the shared task participants could choose the optimal batch size. our results in section 5 show that measuring latency is the only one that favors nar models, and as the batch size increases, ar models quickly reach higher translation speeds. ## experiments we experiment with non-autoregressive models for english-german translation. we used the data provided by the wmt 21 news translation shared task organizers @xcite . as our baseline model, we use the ctc-based nar model as described by @xcite . we use stack of 6 encoder and 6 decoder layers, separated by the state splitting layer which extends the state sequence 3 times. we implement our models foot_0 in the marian toolkit @xcite . for the ctc loss computation, we use the warp-ctc library @xcite . ## results in this section, we try to view the results of the nar and efficiency research in a shared perspective. we evaluate our models and present results in terms of translation quality and decoding speed. we compare the results to the related work on both non-autoregressive translation and model optimization. translation quality. the research on nonautoregressive models uses the bleu score @xcite measured on the wmt 14 test set @xcite as a standard benchmark for evaluating translation quality. we use sacrebleu @xcite in we see that there is a considerable difference in the translation quality between the nar models and the ar large teacher model. also note that when we train an ar base model on the distilled data, it outperforms the nar large model by a considerable margin. another thing we notice is the enormous difference in the comet scores between the ar and nar models. the ar base models achieve comparable bleu scores to the nar large models, but differ substantially in the comet score. from a look at the system outputs, we hypothesize that the nar systems produce unusual errors which bleu does not penalise as heavily as comet. this might suggest that nar models would rank poorly in human evaluation relative to their autoregressive counterparts, despite the reasonable bleu score values. another reason might be that the different errors of nar models are causing a domain mismatch between the comet training data and the data being evaluated. decoding speed. we follow the decoding time evaluation methodology of the wmt 21 efficient translation shared task @xcite . we recreate the hardware conditions that were used in the task. for the gpu decoding measurements, we use a single nvidia ampere a100 gpu. the cpu evaluation was performed on a 36-core cpu intel xeon gold 6354 server from oracle cloud. to amortize the various computation overheads, the models submitted to the shared task are evaluated on a million sentence benchmark dataset. we measure the overall wall time to translate the model latency (ms) shared task dataset with different batching settings on both the gpu and the 36-core cpu. the decoding times are shown in figures 2 and 4 for the gpu and cpu times, respectively. we do not report the single-core cpu latencies as the decoding speed of the nar models falls far behind the efficient ar models in this setup and the translation of the dataset takes too long. we can see that in case of gpu decoding that all models benefit from having larger batch sizes. however, the non-autoregressive models are much faster when the batch size is small. we also ran the evaluation on an nvidia pascal p100 gpu, which showed that when the batch size is large enough, autoregressive models eventually match the speed of non-autoregressive models. we show the decoding times on the pascal gpu in figure 3 . in comparison with efficient ar models. in ## conclusions in this paper, we challenge the evaluation methodology adopted by the research on non-autoregressive models for nmt. we argue that in terms of translation quality, the evaluation should include newer test sets and metrics other than bleu (particularly comet and chrf). this will provide more insight and put the results into the context of the recent research. from the decoding speed perspective, we should bear in mind various use-cases for the model deployment, such as the hardware environment or batching conditions. preferably, the research should evaluate the speed gains across a range of scenarios. finally, given that the latency condition -translation of one sentence at a time on a gpu -already translates too fast to be perceived by human users of mt, there is currently no compelling scenario that warrants the deployment of nar models.