Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism (2020)

Abstract: Non-autoregressive models generate target words in a parallel way, which achieve a faster decoding speed but at the sacrifice of translation accuracy. To remedy a flawed translation by non-autoregressive models, a promising approach is to train a conditional masked translation model (CMTM), and refine the generated results within several iterations. Unfortunately, such approach hardly considers the sequential dependency among target words, which inevitably results in a translation degradation. Hence, instead of solely training a Transformer-based CMTM, we propose a Self-Review Mechanism to infuse sequential information into it. Concretely, we insert a left-to-right mask to the same decoder of CMTM, and then induce it to autoregressively review whether each generated word from CMTM is supposed to be replaced or kept. The experimental results (WMT14 En ↔ De and WMT16 En ↔ Ro) demonstrate that our model uses dramatically less training computations than the typical CMTM, as well as outperforms several state-of-the-art non-autoregressive models by over 1 BLEU. Through knowledge distillation, our model even surpasses a typical left-to-right Transformer model, while significantly speeding up decoding.

## introduction neural machine translation (nmt) models have achieved a great success in recent years @xcite @xcite @xcite . typically, nmts use autoregressive decoders, where the words are generated one-by-one. however, due to the left-to-right dependency, this computationally-intensive decoding process cannot be easily parallelized, and therefore causes a large latency @xcite . to break the bottleneck of autoregression, several non-autoregressive models have been proposed to induce the decoder to generate all target words simultaneously @xcite @xcite . despite the acceleration of computation efficiency, these models usually suffers from the cost of translation accuracy. even worse, they decode a target only in one shot, and thus miss a chance to remedy a flawed translation. against them, a promising research line is to perform refinement on the generated result within several iterations @xcite . along this line, @xcite propose a mask-predict decoding strategy, which iteratively refines the generated translation given the most confident target words predicted from the previous iteration. this model is trained using an objective of conditional masked translation modeling (cmtm), by predicting the masked words conditioned on the rest of observed words. however, cmtm just learns from a subset of words instead of the entire target in terms of a training step. as a result, it will iterate more times over the training dataset to explore the contextual relationship within a sentence, and thus will struggle with a huge cost of the whole training time @xcite . most importantly, cmtm extensively bases upon the assumption of conditional independence, making it hard to capture the strong correlation across the adjacent words @xcite . inevitably, this issue will still degrade the translation performance, such as outputting repetitive words @xcite . to address the issues, our idea is to infuse sequential information into cmtm. accordingly, we propose a self-review mechanism, a discriminative task in which the model learns to autoregressively distinguish the ground truth target from the non-autoregressively generated output of itself. as shown by ardecoder (short for autoregressive decoder) in figure 1 , we firstly switch on the autoregressive mode of the same decoder with cmtm by inserting a left-to-right mask. more importantly, we then require this ardecoder to recurrently review whether each generated word from cmtm is supposed to be a replacement or just an original. in this way, this mechanism constrains our model to review each predicted word only based on previous ones, which is able to not only correct the prediction errors, but also facilitates the learning of conditional dependence of the target words. moreover, this mechanism could also help speed up the whole training by learning from all target words rather than a small masked subset. we extensively validate our model on the datasets of wmt14 en↔de and wmt16 en↔ro. the experimental results demonstrate that our model outperforms several state-of-the-art non-autoregressive models by over 1 bleu. through knowledge distillation, our model even achieves competitive performance compared with the typical left-to-right transformer, while significantly reducing the cost of time during inference. meanwhile, we also prove that the training speed of our model is much faster than the typical cmlm. ## ablation study and analysis upon cmlm, we additionally introduce a self-review mechanism during training, whereas mask-predict @xcite is optimized with only the first two terms in equation 10 . during inference, we abandon ardecoder, and our model performs decoding as same as mask-predict. in this section, we will compare closely to mask-predict to validate the contribution of our proposed self-review mechanism. ## conclusion in this paper, we identify the drawback of cmtm that it is insufficient to capture the sequential correlations among target words. to tackle it, we propose a self-review mechanism that is able to infuse sequential information into cmtm. on several benchmark datasets, we demonstrate that our approach achieves a huge improvement against previous non-autoregressive models and a competitive result to the state-of-theart transformer model. through ablation study, our proposed mechanism is also proved to speed up the training of a cmtm model.