Deep Neural Machine Translation with Weakly-Recurrent Units (2018)

Abstract: Recurrent neural networks (RNNs) have represented for years the state of the art in neural machine translation. Recently, new architectures have been proposed, which can leverage parallel computation on GPUs better than classical RNNs. Faster training and inference combined with different sequence-to-sequence modeling also lead to performance improvements. While the new models completely depart from the original recurrent architecture, we decided to investigate how to make RNNs more efficient. In this work, we propose a new recurrent NMT architecture, called Simple Recurrent NMT, built on a class of fast and weakly-recurrent units that use layer normalization and multiple attentions. Our experiments on the WMT14 English-to-German and WMT16 English-Romanian benchmarks show that our model represents a valid alternative to LSTMs, as it can achieve better results at a significantly lower computational cost.

## introduction neural machine translation (nmt) @xcite is a sequence-tosequence problem that requires generating a sentence in a target language from a corresponding sentence in a source language. similarly to other language processing task, nmt has mostly employed recurrent neural networks (rnns) @xcite @xcite the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. and manning, 2015), in both their lstm @xcite and gru @xcite variants, which can model long-range dependencies. besides their simplicity, the choice of rnns is also due to their expressive power, which has been proven equivalent to turing machines @xcite . rnns have represented so far the state of the art of machine translation, and have constantly been enhanced to improve their performance. nonetheless, their explicit time dependencies make training of deep rnns computationally very expensive @xcite . recent works have proposed new nmt architectures, not based on rnns, that obtained significant improvements both in training speed and translation quality: the so-called convolutional sequence-to-sequence @xcite and the self-attentive or transformer @xcite models. speed improvements by these models mainly come from the possibility of parallelizing computations over word sequences, as both models do not have time dependencies. on the other hand, performance improvements appear to be due to the path lengths needed by the networks to connect distant words in a sentence: linear for rnns, logarithmic for convolutional models, and constant for the transformer. in this paper we propose a neural architecture that shares some properties with the abovementioned models, while maintaining a recurrent design. our hypothesis is that current rnns for nmt have not been designed to take full advantage of deep structures and that better design could lead to improved performance and efficiency. contemporary to this work, @xcite have shown that rnn can still outperform the transformer model when using better hyper-parameters. we start by discussing previous efforts that proposed simplified and theoretically grounded versions of the lstm rnn, which very recently lead to the so-called simple recurrent unit (sru). then, we introduce our nmt architecture based on weakly-recurrent units, which we name simple recurrent nmt (sr-nmt). we present machine translation results on two public benchmark, wmt14 english-german and wmt16 english-romanian, and compare the results of our architecture against lstm and sru based nmt, using similar settings for all of them. results show that sr-nmt trains faster than lstm nmt and outperforms both lstm and sru nmt. in particular, sr-nmt with 8-layers even outperforms google's nmt 8-layer lstm architecture @xcite . moreover, training our model took the equivalent of 12 days on a single k80 gpu against the 6 days on 96 k80 gpus reported by @xcite . finally, the nmt architecture presented in this paper was developed in opennmt-py @xcite and the code is publicly available on github 1 . ## related works rnns are an important tool for nmt, and have ranked at the top of the wmt news translation shared tasks @xcite in the last three years @xcite @xcite . recurrent nmt is also the first approach that outperformed phrasebased statistical mt @xcite . despite the important results, training of rnns remains inefficient because of an intrinsic lack of parallelism and the necessity of redundant parameters in its lstms and grus @xcite variants. @xcite reduce training time in two different ways: by reducing the network size with tied embeddings @xcite and by adding layer normalization to their architecture @xcite . in fact, the reduction of the covariate shift produced by this mechanism shows to significantly speed up convergence of the training algorithm. of course, it does not alleviate the lack of parallelism. @xcite studied rnns and found that the classical stacked rnn architecture does not have a clear notion of depth. in fact, when performing back-propagation through time, the gradient is sent backward in both the horizontal and 1 https://github.com/mattiadg/sr-nmt vertical dimensions, thus having a double notion of depth, which also hurts the optimization procedure. they propose as a solution the notions of deep transition, from one hidden state to the following hidden state, and the notion of deep output, from the last rnn layer to the network output layer. the winning model in wmt17 actually implemented both of them @xcite . @xcite proposed stronglytyped rnns, which are variants of vanilla rnn, gru and lstm that respect some constraints and are theoretically grounded on the concept of strongly-typed quasi-linear algebra. a stronglytyped quasi-linear algebra imposes constraints on the allowed operations for an rnn. in particular, in this framework there is a constraint inspired from the type system from physics, and one inspired by functional programming. the idea of types forbids the sum of vectors generated from different branches of computation. in the case of rnns, this means that it is not possible to sum among them the previous hidden state and the current input, as they are produced by different computation branches. the second constraint aims to simulate the distinction among pure functions and functions with side effects, typical of functional programming. in fact, as rnns own a state, they can approximate algorithms and also produce "side effects". according to the authors, side effects manifest when the horizontal (time-dimension) connections are altered, and are the reason behind the poor behavior of techniques such as dropout @xcite or batch normalization @xcite when they are applied to the horizontal direction straightforwardly @xcite . thus, the side effects should be confined to a part of the network that cannot hinder the learning process. the solution they propose consists in using learnable parameters only in stateless equations (learnware), while the states are combined in parameterless equations (firmware). the combination is achieved through the use of dynamic average pooling (or peephole connections), which allows the network to use equations with parameters to compute the states and the gates, and then use the gate vectors to propagate forward horizontally the hidden state. the authors show theoretically that strongly-typed rnns have generalization capabilities similar to their classical coun-terparts, and confirm it with an empirical investigation over several tasks, where the strongly-typed rnns achieve results not worse than their classical counterparts while training for less time. in addition, the absence of parameters in the state combination cancels the problem of depth introduced by pascanu and colleagues, as these models need only the classical back-propagation and not backpropagation through time. quasi-recurrent neural networks @xcite are an extension of the previous work that use gated convolutions in order to not compute functions of isolated input tokens, but always consider the context given by a convolutional window. srus @xcite , are a development of the units proposed by balduzzi and ghifary designed for training speed efficiency. the equations can be easily cuda optimized, while a good task performance is obtained by stacking many layers in a deep network. srus use highway connections @xcite to enable the training of deep networks. moreover, srus can parallelize the computation over the time steps also in the decoder. in fact during training the words of the whole sequence are known and there is no dependency on the output of the previous time step. as for strongly-typed rnns, the information from the context is propagated with dynamic average pooling, which is much faster to compute than matrix multiplications. srus were tested on a number of tasks, including machine translation, and showed performance similar to lstms, but with significantly lower training time. however to obtain results comparable to a weak lstm-based nmt, srus require many more layers of computation. the results show that a single sru has a significantly lower representation capability than a single lstm. in addition, every layer adds little overhead in terms of training time per epoch, but also the results show little improvement. in this work we further develop the idea of srus, and propose an nmt architecture that can outperform lstm-based nmt. ## simple recurrent nmt we propose a sequence-to-sequence architecture that uses an enhanced version of srus (see figure 1 ) to improve the training process, in particular with many layers, and increase the representation capability. in fact, although @xcite show that they can train networks with up to 10 layers of srus, both in encoder and decoder, without overfitting, their results are far from the state of the art of recurrent nmt. our design goals are addressed in a way similar to @xcite . we add an attention layer within every decoder unit, and make the training more stable by adding a layer normalization layer @xcite after every matrix multiplication with a parameter matrix. the layer normalization reduces the covariate shift @xcite , thus it makes easier the training of deep networks. in addition to layer normalization, our units use highway connections @xcite , which enable the training of deep networks. our sr-nmt architecture is shown in figure 2 our weakly-recurrent units used in the encoder and decoder both separate learnware and firmware, although not being strongly typed (balduzzi and ghifary, 2016) as they include highway connections summing vectors of different types. in the following, we introduce in detail the encoder and decoder networks of our simple recurrent nmt architecture. ## experiments we implemented our architecture in py-torch @xcite inside the opennmtpy toolkit @xcite . all the tested models have been trained with the adam @xcite optimizer until convergence, using the typical initial learning rate of 0.0003, and default values for β 1 and β 2 . at convergence, the models were further trained until new convergence with learning rate 0.00015 @xcite . the model used to restart the training is selected according to the perplexity on the validation set. we applied dropout of 0.1 before every multiplication by a parameter matrix, and in the case of lstm it is applied only to vertical connections in order to use the lstm version optimized in cuda. the batch size is 64 for all the experiments and all the layers for all the models have an output size of 500. ## evaluation in this section, we describe the evaluation of our models with the two benchmarks. as our main goal is to prove that sr-nmt represent a valid alternative to lstms, we have put more effort on wmt14 en-de, which is widely used as a benchmark dataset. the experiments on wmt16 en-ro are aimed to verify the effectiveness of our models in a different language pair with a different data size. ## results in this section, we discuss the performance in terms of training speed and translation quality of our architecture. ## ablation experiments in this section, we evaluate the importance of our enhancements to the original sru unit, namely multi-attention and layer normalization, and of the highway connections, which were already present in the original formulation of srus. we take our sr-nmt model with 4 layers and remove from it one component or a set of components. all the combinations are reported. results refer to the wmt14 en-de task after performing only one training stage. in other words, we did not restart training after convergence as we did for the systems reported in from ## conclusions in this paper we have presented a simple recurrent nmt architecture that enhances previous srus @xcite by adding elements of other architectures, namely layer normalization and multiple attentions. our goal was to explore the possibility to make weakly-recurrent units competitive with lstms for nmt. we have shown that our sr-nmt architecture is able to outperform more complex lstm nmt models on two public benchmarks. in particular, sr-nmt performed even better than the gnmt system, while using a simpler optimization policy, a vanilla beam search and a fraction of its computational resources for training. future work will be in the direction of further enhancing sr-nmt by integrating core components that seem to particularly boost performance of the best non recurrent nmt architectures.