Multiscale Collaborative Deep Models for Neural Machine Translation (2020)

Abstract: Recent evidence reveals that Neural Machine Translation (NMT) models with deeper neural networks can be more effective but are difficult to train. In this paper, we present a MultiScale Collaborative (MSC) framework to ease the training of NMT models that are substantially deeper than those used previously. We explicitly boost the gradient back-propagation from top to bottom levels by introducing a block-scale collaboration mechanism into deep NMT models. Then, instead of forcing the whole encoder stack directly learns a desired representation, we let each encoder block learns a fine-grained representation and enhance it by encoding spatial dependencies using a context-scale collaboration. We provide empirical evidence showing that the MSC nets are easy to optimize and can obtain improvements of translation quality from considerably increased depth. On IWSLT translation tasks with three translation directions, our extremely deep models (with 72-layer encoders) surpass strong baselines by +2.2 +3.1 BLEU points. In addition, our deep MSC achieves a BLEU score of 30.56 on WMT14 English-to-German task that significantly outperforms state-of-the-art deep NMT models. We have included the source code in supplementary materials.

## introduction neural machine translation (nmt) directly models the entire translation process using a large neural network and has gained rapid progress in recent years @xcite . the structure of nmt models has evolved quickly, such as rnn-based @xcite , cnn-based @xcite and attentionbased @xcite systems. all of these models follow the encoder-decoder framework with attention @xcite @xcite paradigm. deep neural networks have revolutionized the state-of-the-art in various communities, from computer vision to natural language processing. however, training deep neural networks has been always a challenging problem. to encourage gradient flow and error propagation, researchers in the field of computer vision have proposed various approaches, such as residual connections @xcite , densely connected networks @xcite and deep layer aggregation @xcite . in natural language processing, constructing deep architectures has shown effectiveness in language modeling, question answering, text classification and natural language inference @xcite @xcite . however, among existing nmt models, most of them are generally equipped with 4-8 encoder and decoder layers @xcite . deep neural network has been explored relatively little in nmt. recent evidence @xcite shows that model depth is indeed of importance to nmt, but a degradation problem has been exposed: by simply stacking more layers, the translation quality gets saturated and then degrades rapidly. to address this problem, @xcite proposed a transparent attention mechanism to ease the optimization of the models with deeper encoders. @xcite continued this line of research but construct a much deeper encoder for transformer by adopting the pre-norm method that establishes a direct way to propagate error gradients from the top layer to bottom levels, and passing the combination of previous layers to the next. while notable gains have been reported over shallow models, the improvements of translation quality are limited when the model depth is beyond 20. in addition, degeneration of translation quality is still observed when the depth is beyond 30. as a result, two questions arise naturally: how to break the limitation of depth in nmt models? and how to fully utilize the deeper structure to further improve the translation quality? in this paper, we address the degradation problem by proposing a multiscale collaborative (msc) framework for constructing nmt models with very deep encoders. 1 in particular, the encoder and decoder of our model have the same number of blocks, each consisting of one or several stacked layers. instead of relying on the whole encoder stack directly learns a desired representation, we let each encoder block learn a fine-grained representation and enhance it by encoding spatial dependences using a bottom-up network. for coordination, we attend each block of the decoder to both the corresponding representation of the encoder and the contextual representation with spatial dependences. this not only shortens the path of error propagation, but also helps to prevent the lower level information from being forgotten or diluted. we conduct extensive experiments on wmt and iwslt translation tasks, covering three translation directions with varying data conditions. on iwslt translation tasks, we show that: on the wmt14 @xmath0, we obtain improved results by deep msc networks with a depth of 48 layers, outperforming strong baselines by +2.5 bleu points, and also defeat state-of-theart deep nmt models @xcite with identical or less parameters. 2 ## background given a bilingual sentence pair (x, y), an nmt model learns a set of parameters θ by maximizing the log-likelihood p (y|x; θ), which is typically 1 in our scenario, we mainly study the depth of encoders. the reason is similar in @xcite : 1) encoders have a greater impact on performance than decoders; 2) increasing the depth of the decoder will significantly increase the complexity of inference. we take the state-of-the-art transformer as our baseline model. specifically, the encoder consists of a stack of l identical layers, each of which comprises two subcomponents: a self-attention mechanism followed by a feed-forward network. layer normalization @xcite is applied to the input of each subcomponent (i.e., pre-norm) and a residual skip connection @xcite adds each subcomponent's input to its output. formally, block 1 layer block 2 layer source inputs block n layer block 1 layer block 2 b layer target inputs block n b layer output probabilities 1 m 2 m n m 0 c n c 2 c 1 c (a) overall framework. following sections, we simplify the equations as for the encoder and decoder, respectively. as discussed by @xcite , applying layer normalization to the input of each subcomponent is the key to learning deep encoders, as it establishes a direct way to pass gradient from the top-most layer to bottom layers: to solve this problem, we propose a novel approach to shorten the path of error propagation from l to bottom layers of the encoder. ## multiscale collaborative deep model in this section, we introduce the details of the proposed approach, a multiscale collaborative (msc) framework for constructing extremely deep nmt models. the framework of our method consists of two main components shown in figure 1(a) . first, a block-scale collaboration mechanism establishes shortcut connections from the lower levels of the encoder to the decoder (as described in 3.1), which is the key to training very deep nmt models. we give explanation by seeing the gradient propagation process. second, we further enhance source representations with spatial dependencies by contextual collaboration, which is discussed in section 3.2. ## experiments we first evaluate the proposed method on iwslt14 english↔german (en↔de) and iwslt17 @xmath1. to make the results more convincing, we also experiment on a larger wmt14 @xmath2. ## related work researchers have constructed deep nmt models that use linear connections to reduce the gradient propagation length inside the topology @xcite @xcite or read-write operations on stacked layers of memories @xcite . such work has been conducted on the basis of the conventional rnn architectures and may not be fully applicable to the advanced transformer. recently, @xcite introduced a transparent network into nmt models to ease the optimization of models with deeper encoders. to improve gradient flow they let each decoder layer find an unique weighted combination of all encoder layer outputs, instead of just the top encoder layer. @xcite found that adopting the proper use of layer normalization helps to learn deep encoders. a method was further proposed to combine layers and encourage gradient flow by simple shortcut connections. @xcite introduced a depth-scaled initialization to improve norm preservation and proposed a merged attention sublayer to avoid the computational overhead for deep models. researchers have also explored growing nmt models in two stages @xcite , in which shallow encoders and decoders are trained in the first stage and subsequently held constant, when another set of shallow layers are stacked on the top. in concurrent work, @xcite studied the effect of the computation order of residual connection and layer normalization, and proposed an parameter initialization method with lipschitz restrictions to ensure the convergence of deep transformers. our method significantly differs from these methods, solving the problem by associating the decoder with the encoder with multi-granular dependencies in different space-scales. exploiting deep representations have been studied to strengthen feature propagation and encourage feature reuse in nmt @xcite @xcite . all of these works mainly attend the decoder to the final output of the encoder stack, we instead coordinate the encoder and the decoder at earlier stage. ## conclusion and future work in this paper, we propose a multisacle collaborative framework to ease the training of extremely deep nmt models. specifically, instead of the top-most representation of the encoder stack, we attend the decoder to multi-granular source information with different space-scales. we have shown that the proposed approach boosts the training of very deep models and can bring improvements on translation quality from greatly increased depth. experiments on various language pairs show that the msc achieves prominent improvements over strong baselines as well as previous deep models. in