What Works and Doesn’t Work, A Deep Decoder for Neural Machine Translation (2022)

Abstract: Deep learning has demonstrated performance advantages in a wide range of natural language processing tasks, including neural machine translation (NMT). Transformer NMT models are typically strengthened by deeper encoder layers, but deepening their decoder layers usually results in failure. In this paper, we first identify the cause of the failure of the deep decoder in the Transformer model. Inspired by this discovery, we then propose approaches to improving it, with respect to model structure and model training, to make the deep decoder practical in NMT. Specifically, with respect to model structure, we propose a cross-attention drop mechanism to allow the decoder layers to perform their own different roles, to reduce the difficulty of deep-decoder learning. For model training, we propose a collapse reducing training approach to improve the stability and effectiveness of deep-decoder training. We experimentally evaluated our proposed Transformer NMT model structure modification and novel training methods on several popular machine translation benchmarks. The results showed that deepening the NMT model by increasing the number of decoder layers successfully prevented the deepened decoder from degrading to an unconditional language model. In contrast to prior work on deepening an NMT model on the encoder, our method can deepen the model on both the encoder and decoder at the same time, resulting in a deeper model and improved performance.

## introduction with the help of the deep neural network, the feature extraction capability of models has been substantially enhanced @xcite . deep neural network models are also popular for natural language processing (nlp) tasks. the most typical deep neural network model in nlp is based on the convolutional neural network (cnn) @xcite and transformer @xcite structures, and the deep pretrained transformer language model has begun to dominate nlp. the deep neural network model has also attracted substantial interest in neural machine translation (nmt), for both theoretical research @xcite @xcite and competition evaluation @xcite @xcite . because it has been demonstrated that deep neural network models can benefit from an enriched representation, deep nmt models also show advantages with respect to translation performance @xcite . although deep models have been extensively studied in machine translation and are frequently used to improve translation performance, almost all work on deepening models has focused on increasing the number of encoder layers; there has been very little research on deepening the decoder. through preliminary experiments on varying the number of decoder layers in the transformer nmt model, we observed that, when the decoder is deepened beyond a certain number of layers, the translation performance of the overall model fails to improve; moreover, it declines rapidly to near zero. this demonstrates that there are flaws in the current structure or training method, and the deep-decoder nmt model cannot be trained. by analyzing the training process of the deepdecoder model, we found that the training perplexity of the model was relatively low, but the translation performance of the obtained model was much worse than that of a shallow model. inspired by this phenomenon, we hypothesize that, as the decoder deepens, the model may increasingly ignore the source inputs and degenerate to an unconditional language model, even though a low perplexity can be obtained on the training set. in this case, the purpose of translation learning is not achieved, and thus the model training fails. according to our hypotheses, preventing the decoder from degenerating to an unconditional language model is the key to overcoming the failure of deep-decoder nmt model training. consequently, we propose two aspects of model improvement: model structure and model training. in model structure, the only difference between the decoder of the nmt model and that of the unconditional language model is cross-attention; therefore, we focus mainly on this structure. in model training, we aim to make the decoder output distant from the output of the unconditional language model to avoid fitting the target sentences while ignoring the source inputs in the training dataset. specifically, we propose a cross-attention drop (cad) mechanism for the deep-decoder layer structure. the original intention of this mechanism is that we suspected that the degeneration of the deep decoder to an unconditional language model was caused by the training difficulties resulting from too many cross-attentions. because the purpose of cross-attention is to force the decoder layer to obtain features from the source representation, the different layers in the deep decoder should perform distinct roles. however, the conventional deep decoder requires each layer to extract source features similarly, thus increasing the training difficulty. as a result, to minimize training loss, the model chooses to memorize the training target sentences directly and ignore the source inputs. in this mechanism, we drop the cross-attention in some decoder layers to lower the overall training difficulty, thereby preventing the failure of deep-decoder training. in addition to structural changes, we also propose a decoder dropout regularization (ddr) loss and anti-lm-degradation (ald) loss for joint model optimization, based on contrastive learning; these increase the stability of deep-decoder nmt model training and avoid degeneration to an unconditional language model. our experiments were conducted mainly on two popular machine translation benchmarks: wmt14 english-to-german and english-to-french. the results of the experimental exploration of decoders with different depths show that a successfully trained depth decoder greatly benefits the overall translation performance and can work with the deep encoder to achieve higher translation performance. moreover, the novel training approaches that we propose both increase the stability of the training of the deep-decoder model and enable additional improvements. ## our method given bilingual parallel sentences ⟨x, y⟩, the nmt model learns a set of parameters θ by maximizing the likelihood j (y|x, θ), which is represented as the product of the conditional probabilities of all target words: ## experiment 4.1 setup dataset to compare with previous work, we conducted experiments on two classical machine translation datasets: wmt14 english-to-german (@xmath0. the corpus sizes are 4.5m and 36m for the @xmath1, respectively. following common practice, we concatenated newstest2012 and newstest2013 as the validation set and used newstest2014 as the test set. we employed tokenizer.pl in moses @xcite to tokenize en, de, and fr sentences, and then used bpe @xcite to split the words into subwords. a joint bpe strategy with 40k merge operations between source and target languages was adopted to construct the vocabulary. configuration we adopted the most widely used transformer @xcite network as our research basis foot_0 . two typical parameter settings are often used to fulfill various needs: transformer base and transformer big. both settings employ a six-layer encoder and a sixlayer decoder. the differences between them are the embedding width, feedforward network size, and number of attention heads, which are 512/2048/8 for base and 1024/4096/16 for big. we used multi-bleu.perl and detokenized sacrebleu foot_1 to evaluate the translation performance on test sets, for fair comparison with previous work. other hyperparameter settings for model training were consistent with @xcite . the number of training steps was 200k for @xmath2, the batch size was 4096 tokens per gpu, and the models were trained on eight nvidia v100 gpus. ## ablation study we conducted ablation studies on the modifications that we made to both the model structure and training to investigate their respective effects on the translation performance. the ablation research was conducted on the wmt14 @xmath3, as before, and the model employed was the base, deep-30l-full model. we began by adding extra r-drop, ddr, ald, and cad techniques to its baseline model (base, deep-30l). the results in we also conducted ablation evaluation of the model structure and training method on the entire model. according to the results, cad had the greatest influence on the translation performance, which is consistent with the conclusion stated above, based on the results in ## conclusion in this paper, we investigated the problem of deepdecoder collapse in nmt when the decoder is deepened. we introduced a cad mechanism, ddr loss, and ald loss to solve this problem. using this model, we demonstrated that a deep model with balanced numbers of encoder and decoder layers outperforms either encoder deepen only or decoder deepen only nmt models. our model outperformed previous similar models on the wmt14 @xmath4, confirming the effectiveness of our approach. for future work, we intend to incorporate methods from related work on deep nmt to further improve the performance of our translation model.