Glancing Transformer for Non-Autoregressive Neural Machine Translation (2021)

Abstract: Recent work on non-autoregressive neural machine translation (NAT) aims at improving the efficiency by parallel decoding without sacrificing the quality. However, existing NAT methods are either inferior to Transformer or require multiple decoding passes, leading to reduced speedup. We propose the Glancing Language Model (GLM) for single-pass parallel generation models. With GLM, we develop Glancing Transformer (GLAT) for machine translation. With only single-pass parallel decoding, GLAT is able to generate high-quality translation with 8×-15× speedup. Note that GLAT does not modify the network architecture, which is a training method to learn word interdependency. Experiments on multiple WMT language directions show that GLAT outperforms all previous single pass non-autoregressive methods, and is nearly comparable to Transformer, reducing the gap to 0.25-0.9 BLEU points.

## introduction transformer has been the most widely used architecture for machine translation @xcite . despite its strong performance, the decoding of transformer is inefficient as it adopts the sequential auto-regressive factorization for its probability model (figure 1a ). recent work such as the non-autoregressive transformer (nat), aims to decode target tokens in parallel to speed up the generation @xcite . however, the vanilla nat still lags behind the transformer in translation quality -with a gap of about 7.0 bleu points. nat assumes the conditional independence of the target tokens given the source sentence. we suspect that nat's conditional independence assumption prevents learning word interdependency in the target ## probability models of machine translation we state and compare different probability models for machine translation. a machine translation task can be formally defined as a sequence to sequence generation problem: given the source sentence @xmath0, x 2 , ..., x n }, to generate the target sentence @xmath1, y 2 , ..., y t } according to the conditional probability p (y |x; θ), where θ denotes the parameter set of a network. different methods factorize the conditional probability differently. the transformer uses the autoregressive factorization to maximize the following likelihood: the vanilla nat consists of the same encoder as the transformer and a parallel decoder with layers of multi-head attention @xcite . during training, it uses the conditional independent factorization for the target sentence: notice that, nat's log-likelihood is an approximation to the full log-likelihood log p (y |x; θ). during inference, the encoder representation is copied as the input to the decoder, therefore all tokens on the target side can be generated in parallel. such a conditional independence assumption does not hold in general, which explains the inferior performance of nat. multi-pass iterative decoding approaches such as mask-predict @xcite extends the vanilla nat. it still uses the conditional independent factorization, together with the random masking scheme: the number of masked tokens distributes uniformly from 1 to the total number of tokens in the target sequence. such training objective is used to learn a refinement model θ that can predict the masked tokens given the source sentence x and words generated in the previous iteration. the vanilla nat breaks word interdependency, while mlm requires multiple passes of decoding to re-establish the word interdependency. our goal in this work is to design a better probability model and a training objective to enable word interdependency learning for single-pass parallel generation. ## glancing transformer in this section, we present glat in detail. glat uses the same encoder-decoder architecture as the vanilla nat @xcite . glat differs from the vanilla nat in that it explicitly encourages word interdependency via training with glancing language model (glm). it differs from the iterative nat with mlm in that it is trained to produce single pass parallel decoding while mlm is used for prediction refinement. ## experiments in this section, we first introduce the settings of our experiments, then report the main results compared with several strong baselines. ablation studies and further analysis are also included to verify the effects of different components used in glat. ## conclusion in this paper, we propose glancing transformer with a glancing language model to improve the performance of single-pass parallel generation models. with the glancing language model, the model starts from learning the generation of sequence fragments and gradually moving to whole sequences. experimental results show that our approach significantly improves the performance of non-autoregressive machine translation with single-pass parallel generation. as glat achieves competitive performance compared with autoregressive models, applying our approach to other generation tasks is a promising direction for future work.