What Have We Achieved on Non-autoregressive Translation? (2024)

Abstract: Recent advances have made non-autoregressive (NAT) translation comparable to autoregressive methods (AT). However, their evaluation using BLEU has been shown to weakly correlate with human annotations. Limited research compares non-autoregressive translation and autoregressive translation comprehensively, leaving uncertainty about the true proximity of NAT to AT. To address this gap, we systematically evaluate four representative NAT methods across various dimensions, including human evaluation. Our empirical results demonstrate that despite narrowing the performance gap, state-of-the-art NAT still underperforms AT under more reliable evaluation metrics. Furthermore, we discover that explicitly modeling dependencies is crucial for generating natural language and generalizing to out-of-distribution sequences.

## introduction non-autoregressive translation, where the model generates translations in parallel, demonstrates notable decoding speed advantages compared with traditional autoregressive translation @xcite and large language models for translation @xcite . however, it suffers from performance degradation compared to autoregressive counterparts @xcite . the degradation stems from the independence assumption, which ignores the inter-token language dependency on the target side. various methods are proposed to mitigate the performance gap @xcite @xcite @xcite . although representative methods @xcite @xcite have reported comparable translation performance to at, almost all nat methods are evaluated under belu scores @xcite . although * equal contribution. bleu has been long adopted, recent work @xcite argues that it is not a reasonable metric, considerably underperforming alternative metrics such as comet @xcite or large language model evaluation @xcite . limited work has been devoted to a systematic evaluation of advanced nat against at, leaving a significant gap in the research literature. to address this gap, we conduct a comprehensive evaluation of representative nat methods, aiming to reveal existing limitations and provide insights for future research. our primary focus is on fully non-autoregressive methods which generate translations in a one-shot manner, achieving the most decoding efficiency advantage. we consider mgmo @xcite for advanced optimization, ctc @xcite for modeling latent alignment, and dat @xcite for explicit target-side dependency modeling. cmlm @xcite is adopted as the representative iterative nat method. all models are tested on representative benchmark datasets under a comprehensive evaluation , including rule-based metrics, model-based metrics and gpt4-based metrics @xcite . moreover, we conduct human evaluation under the mqm framework @xcite to gain further insights into the performance of nat models that may be overshadowed by global automatic evaluations. automatic evaluation demonstrates varying degrees of advantage for at over nat models. in general, dat achieves the most competitive performance, followed by mgmo and ctc. under rule-based evaluation metrics such as belu and chrf @xcite , dat can achieve comparable or even superior performance compared to at. however, this competitiveness diminishes when using model-based metrics such as comet @xcite or gpt4-based evaluation, under which at significantly outperforms all nat mod-els. fine-grained human evaluation indicates that nat models incorporating explicit dependency modeling (e.g., dat and cmlm) achieve similar levels of translation fluency with at, yet suffering various translation accuracy errors. compared with at, nat tends to produce more grammar or punctuation errors. models without explicit dependency modeling (mgmo and ctc) suffer the most mistranslation and omission errors. on the other hand, models with latent alignments (ctc and dat) are more prone to spelling and addition errors. most of these errors are due to nat's inadequate dependency modeling. specifically, dat's addition errors occur when it generates repeated translations, known as n-gram repetition. this can be easily overlooked by bleu evaluation, which measures n-gram precision, explaining why dat performs well in terms of bleu but not comet. the n-gram repetition mainly stems from the weak, though explicit, dependency modeling. dat limits inter-token dependency within one step using a one-linear-layer attention module for decoding efficiency. in contrast, at can depend on the entire generation history and encode it with powerful transformer blocks. to validate our assumption, we train an asymmetric at with a one-layer decoder and observe similar n-gram repetitions. furthermore, adding an additional linear layer to the transition attention in dat effectively reduces the repetition, corroborating our hypothesis. apart from translation quality, we compare at with nat from the perspective of generalization and robustness. empirical findings demonstrate that explicit dependency modeling is crucial for generating human-like languages and generalizing to out-of-distribution samples, which nat methods lack or are still weak at. on the other hand, weak dependency exhibits stronger robustness to input perturbations, as it is less affected by exposure bias @xcite . future research on nat should focus on how to consolidate explicit language dependency while maintaining decoding efficiency. we release our resources at https://github.com/hjznlp/nat_ vs_at . ## method we begin with a brief introduction to autoregressive and non-autoregressive machine translation, before introducing four representative nat methods. ## experiment and setup datasets and models. we conduct experiments on wmt16 en⇒ro and wmt21 de⇒en with 4 representative nat methods apart from the vanilla nat and at. for knowledge distillation, we train an autoregressive model on the raw data as the teacher model to generate the distilled dataset. to achieve peak performance, we employ glancing training @xcite for ctc and dat training. based on empirical results, we adopt standard beam search for all models during inference. further details are provided in appendix c.