Non-Autoregressive Neural Machine Translation with Consistency Regularization Optimized Variational Framework (2022)

Abstract: Variational Autoencoder (VAE) is an effective framework to model the interdependency for non-autoregressive neural machine translation (NAT). One of the prominent VAE-based NAT frameworks, LaNMT, achieves great improvements to vanilla models, but still suffers from two main issues which lower down the translation quality: (1) mismatch between training and inference circumstances and (2) inadequacy of latent representations. In this work, we target on addressing these issues by proposing posterior consistency regularization. Specifically, we first perform stochastic data augmentation on the input samples to better adapt the model for inference circumstance, and then conduct consistency training on posterior latent variables to construct a more robust latent representations without any expansion on latent size. Experiments on En<->De and En<->Ro benchmarks confirm the effectiveness of our methods with about 1.5/0.7 and 0.8/0.3 BLEU points improvement to the baseline model with about 12.6\times faster than autoregressive Transformer.

## introduction neural machine translation (nmt) achieves great success in recent years, and typical sequence-tosequence frameworks like transformer @xcite achieved state-of-the-art performance on the task of nmt. in this framework, source sentences are translated in an autoregressive (at) manner where each token is generated depending on previously generated tokens. inevitably, such sequential decoding strategy results in a high inference latency. to alleviate this issue, nonautoregressive translation (nat; @xcite was proposed to speed-up decoding procedure by generating target tokens in parallel. however, the translation quality of vanilla nat is compromised, one of the most significant problems is multi-modality and it usually results in multiple translation results, duplicate or missing words in target sentences of nat models @xcite . this situation results from the conditional independence proposed by nat, since models are trained to maximize the log-probability of target tokens at each position while the interdependency is omitted. the key to alleviate the multi-modality issue is to model the dependency information of targets implicitly or explicitly so decoder can easily learn and capture the information between target tokens and generate more accurate translations. for example, @xcite and (2020b) model the target dependency by providing observed target tokens in training and performing iterative inference. @xcite generates intermediate representations by permuting the source sentences in the target order. @xcite , @xcite and @xcite model the target dependency by introducing objective functions with alignment. @xcite and @xcite improve the translation quality by proposing additional regularizations. zhou and keung (2020) and @xcite utilize the external information like monolingual data or semantic structure to help the training. previous studies have validated the effectiveness of applying vae on at @xcite @xcite and nat @xcite ) frameworks. a prominent nat model is lanmt foot_0 @xcite which encodes the source and target tokens into intermediate gaussian distribution latent variables and outperforms vanilla nat with about 5.0 bleu points on wmt14 en-de task with 12.5× speedup to base transformer. however, there exists a slight lag behind the state-of-the-art fully nat models. it may be attributed to two reasons: (1) the inadequate representations of latent variables. figure 1 shows the effect of latent size to translation qualities. obviously, the optimal capacity of latent variables is significantly lower than the model's hidden size (512) while high-capacity latent variables conversely deteriorate the performance because the minimization between prior and posterior becomes difficult @xcite . (2) the mismatch between training and inference circumstances that the posterior module receives the gold sentence as inputs during training but imperfect initial translation instead during inference. thus, in this paper, we aim to improve the robustness of the latent representation and move the training circumstance close to inference circumstance. to this end, we propose a consistency regularized posterior network for better latent representations and closer training-inference circumstance with no extended latent size. specifically, it can be split into two main steps: we first apply stochastic data augmentation methods to inject stochastic noises in posterior inputs x and y to get two different views (x 1 , y 1 ) and (x 2 , y 2 ), which are then transformed to corresponding latent variables z 1 , z 2 by the posterior network. secondly, the consistency regularization step tries to minimise the gap between z 1 and z 2 since they are derived from the same pair of input x and y. these two steps enable the sequences better represented by the sizelimited latent variables which is trained to be more robust to the noises in input samples. meanwhile, posterior module receives noisy views instead of gold samples during training, it is more adaptive to the inference time which receives imperfect initial translations as inputs. we verified the performance and effectiveness of our methods on wmt14 @xmath0. our methods outperform the latent variable baseline with about 1.5/0.7 and 0.8/0.3 bleu points improvement on four benchmarks. with these improvements, we achieve the comparable performance to the stateof-the-art fully nat approaches: 25.65/30.23 and 31.56/31.20 of bleu scores with similar decoding speed, and it can be improved further with latent search. the contributions of our work can be summarized as follows: ## approach the posterior module is considered to train with consistency regularization and data augmentation for better translation quality. in this section, we will introduce the details of our method, including the overall network architecture, the objective and procedure of training with consistency regularization, four data augmentation methods and three decoding strategies applied for inference.