Non-Autoregressive Neural Machine Translation: A Call for Clarity (2022)

Abstract: Non-autoregressive approaches aim to improve the inference speed of translation models by only requiring a single forward pass to generate the output sequence instead of iteratively producing each predicted token. Consequently, their translation quality still tends to be inferior to their autoregressive counterparts due to several issues involving output token interdependence. In this work, we take a step back and revisit several techniques that have been proposed for improving non-autoregressive translation models and compare their combined translation quality and speed implications under third-party testing environments. We provide novel insights for establishing strong baselines using length prediction or CTC-based architecture variants and contribute standardized BLEU, chrF++, and TER scores using sacreBLEU on four translation tasks, which crucially have been missing as inconsistencies in the use of tokenized BLEU lead to deviations of up to 1.7 BLEU points. Our open-sourced code is integrated into fairseq for reproducibility.

## introduction traditional sequence-to-sequence models aim to predict a target sequence e i 1 = e 1 , . . . , e i , . . . , e i of length i given an input sequence f j 1 = f 1 , . . . , f j , . . . , f j of length j. in the autoregressive case, this is done token by token, and the probability distribution for the output at timestep i is conditioned on the source sentence f j 1 but also on the preceding outputs of the model e i-1 1 , and parameterized by θ: even though these types of models are widely deployed, one of the major drawbacks is the inherent 1 https://github.com/facebookresearch/fairseq/ pull/4431 left-to-right factorization that requires iterative generation of output tokens, which is not efficiently parallelizable on modern hardware such as gpus or tpus. non-autoregressive translation, on the other hand, assumes conditional independence between output tokens, allowing all tokens to be generated in parallel. effectively, it removes the dependence on the decoding history for generation: this modeling assumption, despite its computational advantages, tends to be more restrictive and introduces the multimodality problem @xcite where consecutive output tokens are repeated or fail to correctly incorporate the preceeding information to form a meaningful translation. overcoming this problem is one of the key challenges to achieving parity with autoregressive translation on a wide variety of tasks and architectures. closest to the presented work is a study by @xcite that also analyzes recent works regarding their effectiveness and successfully combines them to achieve remarkable translation quality. in this work, we additionally address several shortcomings in the literature: 1) standardized bleu, chrf++, and ter scores using sacrebleu accompanied by open source fairseq code; 2) more realistic non-autoregressive speed-up expectations, by comparing against faster baselines; 3) a call for clarity in the community regarding data pre-processing and evaluation settings. ## experimental setup datasets & knowledge distillation we perform our experiments on two datasets: wmt'14 english↔german (en↔de, 4m sentence pairs), and wmt'16 english↔romanian (en↔ro, 610k sentence pairs), allowing us to evaluate on 4 translation directions. as is common practice in the non-autoregressive literature, we train our models with sequence-level knowledge distillation (kd, @xcite , obtained from transformer base teacher models @xcite with a beam size of 5. we tokenize all data using the moses tokenizer and also apply the moses scripts @xcite for punctuation normalization. byte-pair encoding @xcite is used with 40, 000 merge operations. for en↔ro, we use wmt'16 provided scripts to normalize the ro side, and to remove diacritics for @xmath0. @xmath1, who besides computing bleu on tokenized text, compared on diacritic-free text. more details are summarized in appendix d.1. models in this work, we focused our efforts on four models/techniques for training nonautoregressive transformer (nat) models: vanilla nat @xcite , glancing transformer (glat, @xcite , connectionist temporal classification (ctc, @xcite , and deep supervision (ds, @xcite , see appendices a and b for a short overview. for all nat models, we use learned encoder and decoder positional embeddings, shared word embeddings (press and wolf, 2017), no labelsmoothing, keep adam betas and epsilon as defaults with (0.9, 0.98) and 1e-8 respectively, train for 200k (en↔de) / 30k (en↔ro) update steps with 10k / 3k warmup steps using an inverse square root schedule @xcite where the 5 best checkpoints are averaged based on validation bleu. for ctc-based models, the source upsampling factor is 2. more details can be found in our open-sourced training procedures. we intentionally did not fine-tune hyperparameters for each translation direction and instead opted for choices that will most likely transfer across datasets. for autoregressive transformer (at) models, we focus on the base @xcite architecture, as well as a deep encoder (11 layers) & shallow decoder (2 layers) variant @xcite . ## a call for clarity from our experiments with different algorithms, we noticed a few problems that are currently not properly addressed in the literature. these issues make it harder to do a rigorous comparison, so here, we explain how we addressed them, and give recommendations for future work. ## conclusion in this work, we provide multiple standardized sacrebleu metrics for non-autoregressive machine translation models, accompanied by reproducible and modular fairseq code. we hope that our work solves comparison problems in the literature, provides more realistic speed-up expectations, and accelerates research in this area to achieve parity with autoregressive models in the future.