State-of-the-art neural machine translation models generate outputs autoregressively, where every step conditions on the previously generated tokens. This sequentialnature causes inherent decoding latency. Non-autoregressive translation techniques, on the other hand, parallelize generation across positions and speed up inference at the expense of translation quality. Much recent effort has been devoted to non-autoregressive methods, aiming for a better balance between speed and quality . In this work, we re-examine the trade-off and argue that transformer-based autoregressive models can be substantially sped up without loss in accuracy. Specifically, we study autoregressive models with encoders and decoders of varied depths. Our extensive experiments show that given a sufficiently deep encoder, a one-layer autoregressive decoder yields state-of-the-art accuracy with comparable latency to strong non-autoregressive models . Our findings suggest that the latency disadvantage for autoregressive translation has been overestimated due to a suboptimal choice of layer allocation, and we provide a new speed-quality baseline for future research toward fast, accurate translation.
During the recent years, much effort has been invested in non-autoregressive neural machine translation , which appears to be an efficient alternative to state-of-the-art autoregressive machine translation on modern GPUs. In contrast to the latter where generation is sequential, the former allows generation to be parallelized across target token positions. Non-autoregressive machine translation provides a tradeoff between translation quality and inference speed, but some of the latest models have achieved impressive tradeoffs compared to autoregressive baselines . In this work, we reexamine this tradeoff and argue that autoregressive baselines can be substantially sped up without loss in accuracy. Specifically, we study autoregressive models with encoders and decoders of varied depths. Our extensive experiments show that given a sufficiently deep encoder, a single-layer autoregressive decoder can substantially outperform strong non-autoregressive models with comparable inference speed. We show that the speed disadvantage for autoregressive baselines compared to non-autoregressive methods has been overestimated in three aspects: suboptimal layer allocation, insufficient speed measurement, and lack of knowledge distillation. Our results establish a new protocol for future research toward fast, accurate machine translation. Our code is available at URL
R
State-of-the-art
During the recent years, much effort has been invested in non-autoregressive
R
models generate outputs autoregressively, where every step conditions on the previously generated tokens. This sequentialnature causes inherent decoding latency.
, which appears to be an efficient alternative to state-of-the-art autoregressive machine translation on modern GPUs. In contrast to the latter where generation is sequential, the former allows generation to be parallelized across target token positions.
R
translation techniques, on the other hand, parallelize generation across positions and speed up inference at the expense of translation quality. Much recent effort has been devoted to non-autoregressive methods, aiming for a better balance between speed and quality
machine translation provides a tradeoff between translation quality and inference speed, but some of the latest models have achieved impressive tradeoffs compared to autoregressive baselines
R
re-examine the trade-off
reexamine this tradeoff
R
transformer-based autoregressive models
autoregressive baselines
R
one-layer autoregressive decoder yields state-of-the-art accuracy with comparable latency to
single-layer autoregressive decoder can substantially outperform
R
. Our findings suggest that the latency
with comparable inference speed. We show that the speed
R
translation
baselines compared to non-autoregressive methods
R
due to a suboptimal choice of
in three aspects: suboptimal
R
and we provide a new speed-quality baseline
insufficient speed measurement, and lack of knowledge distillation. Our results establish a new protocol
R
translation.
machine translation. Our code is available at URL