Jointly Masked Sequence-to-Sequence Model for Non-Autoregressive Neural Machine Translation (2020)

Abstract: The masked language model has received remarkable attention due to its effectiveness on various natural language processing tasks. However, few works have adopted this technique in the sequence-to-sequence models. In this work, we introduce a jointly masked sequence-to-sequence model and explore its application on non-autoregressive neural machine translation~(NAT). Specifically, we first empirically study the functionalities of the encoder and the decoder in NAT models, and find that the encoder takes a more important role than the decoder regarding the translation quality. Therefore, we propose to train the encoder more rigorously by masking the encoder input while training. As for the decoder, we propose to train it based on the consecutive masking of the decoder input with an n -gram loss function to alleviate the problem of translating duplicate words. The two types of masks are applied to the model jointly at the training stage. We conduct experiments on five benchmark machine translation tasks, and our model can achieve 27.69/32.24 BLEU scores on WMT14 English-German/German-English tasks with 5+ times speed up compared with an autoregressive model.

## introduction the encoder-decoder based sequence-to-sequence framework @xcite has achieved great success on the task of neural machine translation (nmt) @xcite @xcite @xcite . in this framework, the encoder takes the source sentence as input and extracts its hidden representation, based on which the decoder generates the target sentence word by word and from left to right, i.e., * corresponding author. in an autoregressive manner, which is a natural bottleneck for the inference speed due to the sequential conditional dependence. as the performance of nmt models have been substantially promoted, the translation efficiency is becoming a new research hotspot. nonautoregressive neural machine translation (nat) models are proposed to reduce the translation latency while inference, by removing the conditional dependence between target tokens and predicting all tokens in parallel @xcite . as the context dependency cannot be utilized while decoding, the inference speedup of nat models comes at the cost of the degradation in performance. as studied by previous works @xcite , the inferior accuracy of nat models mainly occurs from two aspects: 1) the source-side information is not adequately encoded which results in incomplete translation; 2) the decoder cannot handle the task well which leads to repeated translations and poor performance on long sentences. to tackle these problems and promote the performance of nat models, in this paper, we empirically conduct a thorough study on the functionalities of the encoder and decoder in nat models, and conclude that the encoder has a more direct influence on the final translation performance, and is harder to train than the decoder. therefore, we propose a jointly masked sequence-to-sequence model which is inspired by the idea of masked language modeling @xcite . specifically, for the encoder, we follow the masking strategy of bert @xcite and randomly mask a number of tokens of the source sentence. this strategy trains the encoder more rigorously by forcing it to encode the complete information with residual input. for the decoder, we mask the consecutive fragment of the target sentence to make the decoder concentrate more on predicting adjacent tokens, and propose an n-gram based loss function to learn the consecutive tokens as a whole objective. in this way, we can alleviate the problem of repeated translations of nat models. during inference, we adopt a mask-and-predict @xcite strategy to iteratively generate the translation result, which masks and predicts a subset of the current translation candidates in each iteration. we verify the effectiveness of our model on five benchmark translation tasks including wmt14 english ↔ german, wmt16 english ↔ romanian and iwslt14 @xmath0. our model outperforms all the nat models in comparison, and can achieve comparative performance with its autoregressive counterpart while enhanced with 5+ times speedup on inference (27.69/32.24 bleu scores and 5.73 times speedup on the wmt14 en-de/de-en tasks with an autoregressive teacher of 28.04/32.69 bleu scores). our main contributions can be summarized as follows: ## preliminary study to explore the functionalities of the encoder and decoder in nat models, we conduct a thorough empirical study. we mainly follow the settings in @xcite . we train a basic nat model proposed by @xcite , except that we remove the fertility predictor and keep the decoder input as a hard copy of the source sentence in a similar way with @xcite . we conduct the following experiments on the iwslt14 german to english dataset and train the model with the same number of training steps for each setting. we study the importance of the encoder and decoder from three aspects. firstly, we vary the number of encoder and decoder layers respectively to see which will bring more performance gain. specifically, on a basic model with a 5-layer encoder and a 5-layer decoder, we increase the number of layers to the encoder and decoder separately. results are illustrated in thirdly, we further conduct an investigation on the encoder input, encoder output and decoder input to evaluate their importance in the inference stage. during inference, we add random noise to the three types of inputs respectively, by randomly replacing the embeddings of some tokens with random noise. this experiment is conducted on a basic 5-layer encoder and decoder nat model, and the results are illustrated in figure 1 @xcite . obviously, the encoder input and encoder output both largely influence the translation quality, which implies that the encoder plays an important role in the inference of nat models, while the decoder input is the least important due to its conditional independence in nature. in a word, the performance of nat models rely more on the encoder rather than the decoder. ## methodology while most existing nat works only focus on refining the decoder to obtain better performance, we have explored and shown the significance of the encoder in the previous section. therefore, we propose to improve the translation performance by further manipulating the encoder, and we will introduce the proposed framework to tackle the problems discussed above in this section. we start with the problem definition. problem definition given a pair of source and target sentence (x, y) ∈ (x , y) from the parallel training dataset x and y, the negative loglikelihood objective function of an nmt model can be written as: ## conclusion in this paper, we propose a jointly masked sequence-to-sequence model for nonautoregressive neural machine translation. we first empirically investigate the functionalities of the non-autoregressive translation model, and source: target: what if you could make the genetic changes to eliminate diabetes or alzheimer &apos;s or reduce the risk of cancer or eliminate stroke ? nat-ft: what if you could make the genetic changes in order to eliminate diabetes diabetes or alzheimer disease or reduce reduce the cancer of cancer or strostroke ? cm-nat: what if you could make the genetic changes to eliminate diabetes or alzheimer alzheimer &apos;s , or reduce cancer risk risk or stro stro dents ? jm-nat: what if you could make the genetic changes to eliminate diabetes or alzheimer &apos;s disease or the reduce cancer risk or eliminate stroke ? in the future, we will extend the investigation on the functionalities of the encoder and decoder to other sequence-to-sequence tasks such as text summarization and text style transfer to explore more applications of our model.