Improving Non-Autoregressive Neural Machine Translation via Modeling Localness (2022)

Abstract: Non-autoregressive translation (NAT) models, which eliminate the sequential dependencies within the target sentence, have achieved remarkable inference speed, but suffer from inferior translation quality. Towards exploring the underlying causes, we carry out a thorough preliminary study on the attention mechanism, which demonstrates the serious weakness in capturing localness compared with conventional autoregressive translation (AT). In response to this problem, we propose to improve the localness of NAT models by explicitly introducing the information about surrounding words. Specifically, temporal convolutions are incorporated into both encoder and decoder sides to obtain localness-aware representations. Extensive experiments on several typical translation datasets show that the proposed method can achieve consistent and significant improvements over strong NAT baselines. Further analyses on the WMT14 En-De translation task reveal that compared with baselines, our approach accelerates the convergence in training and can achieve equivalent performance with a reduction of 70% training steps.

## introduction based on the encoder-decoder framework @xcite , neural machine translation (nmt) @xcite @xcite has achieved tremendous success in the past several years due to its excellent performance. currently, state-ofthe-art nmt systems are built in an autoregressive manner, which generates target tokens one by one from continuous representations summarized by the encoder. however, with the constraint of output tokens conditioned on previously generated ones, autoregressive translation (at) inevitably suffers from serious latency during decoding, which becomes a bottleneck of inference speed. towards accelerating the inference process, non-autoregressive neural machine translation (nat) @xcite @xcite has been proposed to break the above bottleneck. instead of sequential generation in autoregressive translation, nat models output the entire target sentence at once. unfortunately, removing sequential dependencies within the target sentence brings nat models with serious weakness in capturing the highly multimodal distribution of target sentences @xcite . accordingly, nat models suffer from two kinds of incoherent translations, including repetitive translations and incomplete translations @xcite , which leads to inferior translation performance. since both types of translation errors do not commonly appear in at models, several works @xcite have proposed to leverage a well-trained at model to enhance the training of nat models. inspired by this consideration, we empirically carry out a thorough study to present the weakness of nat by investigating the distinction between nat and at models. specifically, we inspect the attention mechanism on two translation tasks and reveal that, in contrast to autoregressive models, nat models lack the ability of either modeling the localness @xcite or producing localness-aware representations. motivated by this observation, we propose to improve non-autoregressive machine translation via explicitly modeling localness. specifically, we incorporate multi-layer temporal convolutions (mtc) into both encoder and decoder sides to enhance the ability to model localness-aware representations in nat models. to validate the effectiveness of our approach, we implement this method on two advanced nat models, namely conditional masked language model (cmlm) @xcite and vanilla-nat @xcite . extensive experiments on typical translation benchmarks demonstrate that our proposed approach can significantly and consistently improve the translation quality by up to 1.0 bleu points over a series of strong nat baselines. further analyses reveal that our approach enhances the ability to generate localness-aware representations. in addition, the analysis on the wmt14 en⇒de task shows that compared with the original cmlm model, our approach accelerates the convergence in training and achieves comparable performance with a reduction of 70% training steps. in summary, the contributions of this work are detailed as follows: ## background autoregressive neural machine translation in recent years, autoregressive models, which predict the target sentence sequentially conditioned on translation history, have achieved extraordinary success on machine translation. specifically, given a source sentence @xmath0, . . . , x i }, a standard encoder-decoder autoregressive framework @xcite optimizes the conditional probability of a target sentence @xmath1, . . . , y j }, namely: in the inference, the conditional dependency on translation history produces the autoregressive property, which predicts the token sequentially based on previous output tokens in the target. this nature of sequential processing results in high latency in translation. non-autoregressive neural machine translation in contrast, @xcite proposed to accelerate the decoding process by generating target tokens in parallel. in practice, by breaking the probabilistic factorization, the prediction is modeled as a product of the probability, which is independent for each token: ## preliminary study in this section, we conduct a thorough empirical study on the attention mechanism to observe the distinction between at and nat models. in practice, a good probabilistic distribution of attention weights suggests a good alignment between source and target words, and usually leads to more accurate translation @xcite @xcite . inspired by this observation, we systematically investigate the cross-attention weights on both wmt14 en⇒de and wmt16 en⇒ro translation tasks. specifically, given a source sentence @xmath2, . . . , x i }, for the j-th token in the target sentence @xmath3, . . . , y j }, the attention probability particularly noted that the number of surrounding indexes involved is less than k when the window exceeds the bounds of the sequence. for a target sentence, lap is computed as γ @xmath4. we average γ m on all translated sentences to obtain mean local attention probability (mlap): γ = 1 m m γ m . as seen, mlap evaluates attention weights on a fixed window within source words, which is considered as local information. we conduct this study on top of the advanced cmlm model and compare mlap with the last layer and all heads of the encoder on three models: 1) transformer (at model), 2) nat, and 3) nat-mtc. as shown in figure 1 , the results reveal two key points. first of all, as the window size increases, the computation of mlap includes more words. the decrease of mlap indicates that attentions between target and source words usually focus on neighbor words within the source sentence. secondly, compared with transformer, the nat model holds a smaller mlap, which demonstrates that it lacks the ability of modeling localness effectively. however, our approach corrects this phenomenon effectively (shown by the bar with color blue vs. the bar with color orange). ## approach in this section, we elaborate the proposed framework of improving nat models via explicitly modeling localness. first of all, convolutional neural networks benefit from effectively capturing local information and have achieved remarkable success in computer vision @xcite . inspired by this observation, we augment nat models with temporal convolutions to enhance localness-aware representations. as shown in figure 2 , we stack temporal convolutions on top of the representations of word embedding. @xmath5. the input sentence incorporating into transformer as a feature extractor, the encoder in nat models is dealing with a more sophisticated task and takes a more important role than the decoder regarding the translation quality @xcite . therefore, it is natural to incorporate multi-layer temporal convolutions into the encoder. besides, compared with the conventional decoder in autoregressive models, nat models avoid the causal mask @xcite in self-attention networks (san). therefore, an identical temporal convolutional structure can be introduced into the decoder accordingly. in our implementation, we use multi-layer stacked temporal convolutions to model localness-aware representations. the presented approach is model-agnostic and can be applied in different nat models. in this paper, we mainly implement the proposed method on top of advanced cmlm @xcite and vanilla-nat @xcite ) models. ## analysis in this section, we conduct extensive analyses on the wmt14 en-de translation task to better understand our model in terms of: 1) effects of different strategies, 2) effects of the sentence length, 3) convergence speed, 4) effects of the number of decoding iterations, 5) effects of predicted length candidates, and 6) case study. ## conclusion in this paper, we propose to improve nat models via explicitly modeling localness. first of all, we conduct a thorough empirical study on the attention mechanism and reveal that compared with autoregressive models, existing nat models lack the ability to effectively modeling local information. furthermore, we incorporate temporal con-volutions into both encoder and decoder sides to enhance localness-aware representations in nat models. empirical results on a variety of language pairs and two advanced nat models demonstrate the effectiveness and universality of our approach. further analyses confirm that the proposed method benefits translations for long sentences and accelerates convergence during training.