Non-autoregressive neural machine translation achieves remarkable inference acceleration compared to autoregressive models. However, current non-autoregressive models still fall behind their autoregressive counterparts in prediction accuracy. We attribute the accuracy gaps to two disadvantages of non-autoregressive models : a) learning simultaneous generation under the overly strong conditional independence assumption; b) lacking explicit target language modeling. In this paper , we propose Glancing Transformer (GLAT) to address the above disadvantages, which reduces the difficulty of learning simultaneous generation and introduces explicit target language modeling in the non-autoregressive setting at the same time . Experiments on several benchmarks demonstrate that our approach significantly improves the accuracy of non-autoregressive models without sacrificing any inference efficiency . In particular, GLAT achieves 30.91 BLEU on WMT 2014 German-English, which narrows the gap between autoregressive models and non-autoregressive models to less than 0.5 BLEU score .
Although non-autoregressive models with one-iteration generation achieve remarkable inference speed-up, they still fall behind their autoregressive counterparts in prediction accuracy. The non-autoregressive models with the best accuracy currently rely on multiple decoding iterations, which largely sacrifice the inference speed of non-autoregressive models . Inspired by the way of learning word dependencies in autoregressive and iterative-decoding models , we propose Glancing Transformer (GLAT) with a glancing language model (GLM), which learns to capture the word dependency gradually . Experiments on three benchmarks demonstrate that our approach can significantly improve the accuracy of non-autoregressive models without multiple decoding iterations . In particular, GLAT achieves state-of-the-art results among non-iterative models and even outperforms top iterative counterparts in some specific benchmarks .
R
Non-autoregressive neural machine translation achieves remarkable inference acceleration compared to autoregressive models. However, current
Although
R
models
models with one-iteration generation achieve remarkable inference speed-up, they
R
We attribute the accuracy gaps to two disadvantages
The non-autoregressive models with the best accuracy currently rely on multiple decoding iterations, which largely sacrifice the inference speed
R
: a) learning simultaneous generation under the overly strong conditional independence assumption; b) lacking explicit target language modeling. In this paper
. Inspired by the way of learning word dependencies in autoregressive and iterative-decoding models
R
to address the above disadvantages, which reduces the difficulty of learning simultaneous generation and introduces explicit target language modeling in the non-autoregressive setting at the same time
with a glancing language model (GLM), which learns to capture the word dependency gradually
R
several
three
R
significantly improves
can significantly improve
R
sacrificing any inference efficiency
multiple decoding iterations
R
30.91 BLEU on WMT 2014 German-English, which narrows the gap between autoregressive models and non-autoregressive models to less than 0.5 BLEU score
state-of-the-art results among non-iterative models and even outperforms top iterative counterparts in some specific benchmarks