Lipschitz Constrained Parameter Initialization for Deep Transformers (2020)

Abstract: The Transformer translation model employs residual connection and layer normalization to ease the optimization difficulties caused by its multi-layer encoder/decoder structure. Previous research shows that even with residual connection and layer normalization, deep Transformers still have difficulty in training, and particularly Transformer models with more than 12 encoder/decoder layers fail to converge. In this paper, we first empirically demonstrate that a simple modification made in the official implementation, which changes the computation order of residual connection and layer normalization, can significantly ease the optimization of deep Transformers. We then compare the subtle differences in computation order in considerable detail, and present a parameter initialization method that leverages the Lipschitz constraint on the initialization of Transformer parameters that effectively ensures training convergence. In contrast to findings in previous research we further demonstrate that with Lipschitz parameter initialization, deep Transformers with the original computation order can converge, and obtain significant BLEU improvements with up to 24 layers. In contrast to previous research which focuses on deep encoders, our approach additionally enables Transformers to also benefit from deep decoders.

## introduction neural machine translation has achieved great success in the last few years @xcite @xcite . the transformer @xcite , which has outperformed previous rnn/cnn based translation models @xcite , is based on multi-layer self-attention networks and can be trained very efficiently. the multi-layer structure allows the transformer to model complicated functions. increasing the depth of models can increase their capacity but may also cause optimization difficulties @xcite @xcite @xcite . in order to ease optimization, the transformer employs residual connection and layer normalization techniques which have been proven useful in reducing optimization difficulties of deep neural networks for various tasks @xcite . however, even with residual connections and layer normalization, deep transformers are still hard to train: the original transformer @xcite only contains 6 encoder/decoder layers. @xcite show that transformer models with more than 12 encoder layers fail to converge, and propose the transparent attention (ta) mechanism which combines outputs of all encoder layers into a weighted encoded representation. @xcite find that deep transformers with proper use of layer normalization are able to converge and propose to aggregate previous layers' outputs for each layer. @xcite explore incrementally increasing the depth of the transformer big by freezing pre-trained shallow layers. concurrent work closest to ours is @xcite . they address the same issue, but propose a different layer-wise initialization approach to reduce the standard deviation. our contributions are as follows: process dropout + norm input/norm process dropout + norm input (a) (b) process dropout + norm output process dropout + norm output in model in res in model in res out ln out res out ln/res out ln/res in res in model out ln/res in res in model out res out res deep transformers, and propose to initialize deep transformers under the lipschitz constraint; ## convergence of different computation orders in this paper we focus on the convergence of the training of deep transformers. to alleviate the training problem for the standard transformer model, layer normalization @xcite and residual connection @xcite ) are adopted. ## lipschitz constrained parameter initialization since the diminished residual connections ( in which case w σ will be greater than or at least equal to 1.0, and the residual connection of v1 will not be shrunk anymore. to achieve this goal, we can constrain elements of in model + in res to be in @xcite and ensure that their standard deviation is smaller than 1.0. from equation 4 we obtain: after applying equation 2 in equation 5, we find that: thus, as long as b-@xmath0, the requirements for the corresponding σ described in equation 1 can be satisfied. to achieve this goal, we can simply constrain the range of elements of x to be smaller than 1 and initialize the sub-model before layer normalization to be a k-lipschitz function, where ## experiments we use the training data described in section 2 to examine the effectiveness of the proposed lipschitz constrained parameter initialization approach. in practice, we initialize embedding matrices and weights of linear transformations with uniform distributions of [-e, +e] and [-l, +l] respectively. we use 2 esize+vsize as e and 1 isize as l where esize, vsize and isize stand for the size of embedding, vocabulary size and the input dimension of the linear transformation respectively. 4 results for two computation orders with the new parameter initialization method are shown in while previous approaches @xcite only increase the depth of the encoder, we suggest that deep decoders should also be helpful. we analyzed the influence of deep encoders and decoders separately and results are shown in ## conclusion in contrast to previous works @xcite @xcite which show that deep transformers with the computation order as in @xcite have difficulty in convergence, we show that deep transformers with the original computation order can converge as long as proper parameter initialization is performed. we first investigate convergence differences between the published transformer @xcite and its official implementation @xcite , and compare the differences of computation orders between them. we conjecture that the convergence issue of deep transformers is because layer normalization sometimes shrinks residual connections, we support our conjecture with a theoretical analysis ( our experiments show the effectiveness of our simple approach on the convergence of deep transformers, which achieves significant improvements on the wmt 14 english to german and the wmt 15 czech to english news translation tasks. we also study the effects of deep decoders in addition to deep encoders extending previous works.