Improving Autoregressive NMT with Non-Autoregressive Model (2020)

Abstract: Autoregressive neural machine translation (NMT) models are often used to teach non-autoregressive models via knowledge distillation. However, there are few studies on improving the quality of autoregressive translation (AT) using non-autoregressive translation (NAT). In this work, we propose a novel Encoder-NAD-AD framework for NMT, aiming at boosting AT with global information produced by NAT model. Specifically, under the semantic guidance of source-side context captured by the encoder, the non-autoregressive decoder (NAD) first learns to generate target-side hidden state sequence in parallel. Then the autoregressive decoder (AD) performs translation from left to right, conditioned on source-side and target-side hidden states. Since AD has global information generated by low-latency NAD, it is more likely to produce a better translation with less time delay. Experiments on WMT14 En-De, WMT16 En-Ro, and IWSLT14 De-En translation tasks demonstrate that our framework achieves significant improvements with only 8% speed degeneration over the autoregressive NMT.

## introduction neural machine translation (nmt) based on encoder-decoder framework has gained rapid progress over recent years @xcite @xcite @xcite . all these high-performance nmt models generate target languages from left to right in an autoregressive manner. an obvious limitation of autoregressive translation (at) is that the inference process can hardly be parallelized, and the inference time is linear with respect to the length of the target sequence. to speed up the inference of machine translation, non-autoregressive translation (nat) models have been proposed, which generate all target tokens independently and simultaneously @xcite @xcite . although nat is successfully trained with the help from an at model as its teacher via knowledge distillation @xcite , there is no work focusing on improving the quality of at using nat. therefore, a natural question arises, can we boost at with nat? in this paper, we propose a novel and effective encoder-nad-ad framework for nmt, in which the newly added non-autoregressive decoder (nad) can provide target-side global information when autoregressive decoder (ad) translates, as illustrated in figure 1 . briefly speaking, the encoder is first used to encode the source sequence into a sequence of vector representations. nad then reads the encoder representations and generates a coarse target sequence in parallel. given the source-side and target-side contexts separately captured by the encoder and nad, ad learns to generate final translation token by token. our proposed model can fully combine two major advantages compared to previous work @xcite . on the one hand, due to the lower latency during inference of nat, the decoding efficiency of our proposed framework is only slightly lower than the standard nmt models, as shown in figure 1 . on the other hand, since ad can asses the global target-side context provided by nad, it has the potential to generate a better translation by fully exploiting source-side and target-side contexts. we conduct massive experiments on wmt14 en⇒de, wmt16 en⇒ro and iwslt14 de⇒en translations tasks. experimental results demonstrate that our proposed model achieves substantial improvements with only 8% degradation in decoding efficiency compared to the standard nmt. ## the framework our goal in this work is to improve autoregressive nmt using the non-autoregressive model with lower latency during inference. figure 2 shows the model architecture of the proposed framework. next, we will detail individual components and introduce an algorithm for training and inference. ## experiments we use 4-gram nist bleu @xcite as the evaluation metric, and sign-test @xcite to test for statistical significance. ## related work there are many design choices in the encoderdecoder framework based on different types of layers, such as rnn-based @xcite , cnn-based @xcite , and selfattention based @xcite approaches. particularly, relying entirely on the attention mechanism, the transformer introduced by @xcite can improve the training speed as well as model performance. in term of speeding up the decoding of the neural transformer, @xcite modified the autoregressive architecture to directly generate target words in parallel. in past two years, non-autoregressive and semi-autoregressive models have been extensively studied @xcite @xcite @xcite @xcite . previous work shows that nat can be improved via knowledge distillation from at models. in contrast, the idea of improving at with nat is not well explored. the most relevant to our proposed framework is deliberation network @xcite , which leverages the global information by observing both back and forward information in sequence decoding through a deliberation process. recently, @xcite proposed asynchronous bidirectional decoding for nmt (abd-nmt), which extended the conventional encoder-decoder framework by introducing a backward decoder. different from abd-nmt, synchronous bidirectional sequence generation model perform left-to-right decoding and right-to-left decoding simultaneously and interactively @xcite . @xcite introduced a adaptive multi-pass decoder to standard nmt models. however, the above models improve translation quality while greatly reducing inference efficiency. ## conclusion in this work, we propose a novel encoder-nad-ad framework for nmt, aiming at improving the quality of autoregressive decoder with global information produced by the newly added nonautoregressive decoder. we extensively evaluate the proposed model on three machine translation tasks (en⇒de, en⇒ro, and de⇒en). compared to existing deliberation network @xcite which suffers from serious decoding speed degradation, our proposed model achieves a significant improvement in translation quality with little degradation of decoding efficiency compared to the stateof-the-art autoregressive nmt.