Paper:
MULTIPLICATIVE LSTM FOR SEQUENCE MODELLING
We introduce multiplicative LSTM (mLSTM), a novel recurrent neural network architecture for sequence modelling that combines the long short-term memory (LSTM) and multiplicative recurrent neural network architectures. mLSTM is characterised by its ability to have different recurrent transition functions for each possible input, which we argue makes it more expressive for autoregressive density estimation. We demonstrate empirically that mLSTM outperforms standard LSTM and its deep variants for a range of character level modelling tasks, and that this improvement increases with the complexity of the task. This model achieves a test error of 1.19 bits/character on the last 4 million characters of the Hutter prize dataset when combined with dynamic evaluation.
1 INTRODUCTION
Recurrent neural networks (RNNs) are powerful sequence density estimators that can use long contexts to make predictions. They have achieved tremendous success in (conditional) sequence modelling tasks such as language modelling, machine translation and speech recognition. Generative models of sequences can apply factorization via the product rule to perform density estimation of the sequence x1:T = {x1, . . . , xT },
P (x1, . . . , xT ) = P (x1)P (x2|x1)P (x3|x2, x1) · · ·P (xT |x1 . . . xT−1). (1)
RNNs can model sequences with the above factorization by using a hidden state to summarize past inputs. The hidden state vector ht is updated recursively using the previous hidden state vector ht−1 and the current input xt as ht = F(ht−1, xt), (2) where F is a differentiable function with learnable parameters. In a vanilla RNN, F multiplies its inputs by a matrix and squashes the result with a non-linear function such as a hyperbolic tangent (tanh). The updated hidden state vector is then used to predict a probability distribution over the next sequence element, using function G. In the case where x1:T consists of mutually exclusive discrete outcomes, G may apply a matrix multiplication followed by a softmax function:
P (xt+1) = G(ht). (3)
Generative RNNs can evaluate log-likelihoods of sequences exactly, and are differentiable with respect to these log-likelihoods. RNNs can be difficult to train due to the vanishing gradient problem (Bengio et al., 1994), but advances such as the long short-term memory architecture (LSTM) (Hochreiter & Schmidhuber, 1997) have allowed RNNs to be successful. Despite their success, generative RNNs (as well as other conditional generative models) are known to have problems with recovering from mistakes (Graves, 2013). Each time the recursive function of the RNN is applied and the hidden state is updated, the RNN must decide which information from the previous hidden state to store, due to its limited capacity. If the RNN’s hidden representation remembers the wrong information and reaches a bad numerical state for predicting future sequence elements, for instance as a result of an unexpected input, it may take many time-steps to recover.
We argue that RNN architectures with hidden-to-hidden transition functions that are input-dependent are better suited to recover from surprising inputs. Our approach to generative RNNs combines LSTM units with multiplicative RNN (mRNN) factorized hidden weights, allowing flexible input-dependent
transitions that are easier to control due to the gating units of LSTM . We compare this multiplicative LSTM hybrid architecture with other variants of LSTM on a range of character level language modelling tasks. Multiplicative LSTM is most appropriate when it can learn parameters specifically for each possible input at a given timestep. Therefore, its main application is to sequences of discrete mutually exclusive elements, such as language modelling and related problems.
1.1 INPUT-DEPENDENT TRANSITION FUNCTIONS
RNNs learn a mapping from previous hidden state ht−1 and input xt to hidden state ht. Let ĥt denote the input to the next hidden state before any non-linear operation:
ĥ(t) =Whhht−1 +Whxxt, (4)
where Whh is the hidden-to-hidden weight matrix, and Whx is the input-to-hidden weight matrix. For problems such as language modelling, xt is a one-hot vector, meaning that the output of Whxxt is a column in Whx, corresponding to the unit element in xt.
The possible future hidden states in an RNN can be viewed as a tree structure, as shown in Figure 1. For an alphabet of N inputs and a fixed ht−1, there will be N possible transition functions between ht−1 and ĥt. The relative magnitude of Whhht−1 to Whxxt will need to be large for the RNN to be able to use long range dependencies, and the resulting possible hidden state vectors will therefore be highly correlated across the possible inputs, limiting the width of the tree and making it harder for the RNN to form distinct hidden representations for different sequences of inputs. However, if the RNN has flexible input-dependent transition functions, the tree will be able to grow wider more quickly, giving the RNN the flexibility to represent more probability distributions.
In a vanilla RNN, it is difficult to allow inputs to greatly affect the hidden state vector without erasing information from the past hidden state. However, an RNN with a transition function mapping ĥt ← ht−1 dependent on the input would allow the relative values of ht to vary with each possible input xt, without overwriting the contribution from the previous hidden state, allowing for more long term information to be stored. This ability to adjust to new inputs quickly while limiting the overwriting of information should make an RNN more robust to mistakes when it encounters surprising inputs, as the hidden vector is less likely to get trapped in a bad numerical state for making future predictions.
1.2 MULTIPLICATIVE RNN
The multiplicative RNN (mRNN) (Sutskever et al., 2011) is an architecture designed specifically to allow flexible input-dependent transitions. Its formulation was inspired by the tensor RNN, an RNN architecture that allows for a different transition matrix for each possible input. The tensor RNN
features a 3-way tensor W 1:Nhh , which contains a separately learned transition matrix Whh for each input dimension. The 3-way tensor can be stored as an array of matrices
W (1:N) hh = {W (1) hh , ...,W (N) hh }, (5)
where superscript is used to denote the index in the array, and N is the dimensionality of xt. The specific hidden-to-hidden weight matrix W (xt)hh used for a given input xt is then
W (xt) hh = N∑ n=1 W (n) hh x (n) t . (6)
For language modelling problems, only one unit of xt will be on, and W (xt) hh will be the matrix in W (1:N) hh corresponding to that unit. Hidden-to-hidden propagation in the tensor RNN is then given by
ĥ(t) =W (xt) hh ht−1 +Whxxt. (7)
The large number of parameters in the tensor RNN make it impractical for most problems. mRNNs can be thought of as a tied-parameter approximation to the tensor RNN that use a factorized hiddento-hidden transition matrix in place of the normal RNN hidden-to-hidden matrix Whh, with an input-dependent intermediate diagonal matrix diag(Wmxxt). The input-dependent hidden-to-hidden weight matrix, W (xt)hh is then
W (xt) hh =Whmdiag(Wmxxt)Wmh. (8)
An mRNN is thus equivalent to a tensor RNN using the above form for W (xt)hh . For readability, an mRNN can also be described using intermediate state mt as follows:
mt = (Wmxxt) (Wmhht−1) (9) ĥt =Whmmt +Whxxt. (10)
mRNNs have improved on vanilla RNNs at character level language modelling tasks (Sutskever et al., 2011; Mikolov et al., 2012), but have fallen short of the more popular LSTM architecture, for instance as shown with LSTM baselines from (Cooijmans et al., 2016). The standard RNN units in an mRNN do not provide an easy way for information to bypass its complex transitions, resulting in the potential for difficulty in retaining long term information.
1.3 LONG SHORT-TERM MEMORY
LSTM is a commonly used RNN architecture that uses a series of multiplicative gates to control how information flows in and out of internal states of the network (Hochreiter & Schmidhuber, 1997). There are several slightly different variants of LSTM, and we present the variant used in our experiments.
The LSTM hidden state receives inputs from the input layer xt and the previous hidden state ht−1:
ĥt =Whxxt +Whhht−1. (11)
The LSTM network also has 3 gating units – input gate i, output gate o, and forget gate f – that have both recurrent and feed-forward connections:
it = σ(Wixxt +Wihht−1) (12) ot = σ(Woxxt +Wohht−1) (13) ft = σ(Wfxxt +Wfhht−1), (14)
where σ is the logistic sigmoid function. The input gate controls how much of the input to each hidden unit is written to the internal state vector ct, and the forget gate determines how much of the previous internal state ct−1 is preserved. This combination of write and forget gates allows the network to control what information should be stored and overwritten across each time-step. The internal state is updated by
ct = ft ct−1 + it ĥt. (15)
The output gate controls how much of each unit’s activation is preserved. It allows the LSTM cell to keep information that is not relevant to the current output, but may be relevant later. The final output of the hidden state is given by ht = tanh(ct ot). (16) This is slightly different from the typical LSTM variant, where the output gate is applied after the tanh. LSTM’s ability to control how information is stored in each unit has proven generally useful.
1.4 COMPARING LSTM WITH MRNN
The LSTM and mRNN architectures both feature multiplicative units, but these units serve different purposes. LSTM’s gates are designed to control the flow of information through the network, whereas mRNN’s gates are designed to allow transition functions to vary across inputs. LSTM gates receive input from both the input units and hidden units, allowing multiplicative interactions between hidden units, but also potentially limiting the extent of input-hidden multiplicative interaction. LSTM gates are also squashed with a sigmoid, forcing them to take values between 0 and 1, which makes them easier to control, but less expressive than mRNN’s linear gates. For language modelling problems, mRNN’s linear gates do not need to be controlled by the network because they are explicitly learned for each input. They are also placed in between a product of 2 dense matrices, giving more flexibility to the possible values of the final product of matrices.
2 MULTIPLICATIVE LSTM
Since the LSTM and mRNN architectures are complimentary, we propose the multiplicative LSTM (mLSTM), a hybrid architecture that combines the factorized hidden-to-hidden transition of mRNNs with the gating framework from LSTMs. The mRNN and LSTM architectures can be combined by adding connections from the mRNN’s intermediate state mt (which is redefined below for convenience) to each gating units in the LSTM, resulting in the following system:
mt = (Wmxxt) (Wmhht−1) (17) ĥt =Whxxt +Whmmt (18) it = σ(Wixxt +Wimmt) (19) ot = σ(Woxxt +Wommt) (20) ft = σ(Wfxxt +Wfmmt). (21)
We set the dimensionality of mt and ht equal for all our experiments. We also chose to share mt across all LSTM unit types, resulting in a model with 1.25 times the number of recurrent weights as LSTM for the same number of hidden units.
The goal of this architecture is to combine the flexible input-dependent transitions of mRNNs with the long time lag and information control of LSTMs. The gated units of LSTMs could make it easier to control (or bypass) the complex transitions in that result from the factorized hidden weight matrix. The additional sigmoid input and forget gates featured in LSTM units allow even more flexible input-dependent transition functions than in regular mRNNs.
3 RELATED APPROACHES
Many recently proposed RNN architectures use recurrent depth, which is depth between recurrent steps. Recurrent depth allows more non-linearity in the combination of inputs and previous hidden states from every time step, which in turn allows for more flexible input-dependent transitions. Recurrent depth has been found to perform better than other kinds of non-recurrent depth for sequence modelling (Zhang et al., 2016). Recurrent highway networks (Zilly et al., 2016) use a more sophisticated recurrent depth that carefully controls propagation through layers using gating units. The gating units also allow for a greater deal of multiplicative interaction between the inputs and hidden units. While adding recurrent depth could improve our model, we believe that maximizing the input-dependent flexibility of the transition function is more important for expressive sequence modelling. Recurrent depth can do this through non-linear layers combining hidden and input contributions, but our method can do this independently of depth.
Another approach, multiplicative integration RNNs (Wu et al., 2016), use Hadamard products instead of addition when combining contributions from input and hidden units. When applying this to LSTM, this architecture achieves impressive sequence modelling results. The main difference between multiplicative integration LSTM and mLSTM is that mLSTM applies the Hadamard product between the multiplication of two matrices. In the case of LSTM, this allows for the potential for greater expressiveness, without significantly increasing the size of the model.
4 EXPERIMENTS
4.1 SYSTEM SETUP
Our experiments compared the performance of mLSTM with regular LSTM for different characterlevel language modelling tasks of varying complexity1. Gradient computation in these experiments used truncated backpropagation through time on sequences of length 100, only resetting the hidden state every 10 000 timesteps to allow networks access to information far in the past. All experiments used a variant of RMSprop, (Tieleman & Hinton, 2012), with normalized updates in place of a learning rate. All unnormalized update directions v∗, computed by RMSprop, were normalized to have length `, where ` was decayed exponentially over training:
v ← `√ vT∗ v∗ v∗. (22)
We found that this allowed for fast convergence with larger batch sizes, allowing for greater parallelization during training without hurting performance.
We compared mLSTM to previously reported regular LSTM, stacked LSTM, and RNN character-level language models. The stacked LSTMs were all 2-layer, and both LSTM layers contained direct connections from the inputs and to the outputs. We used the Penn Treebank dataset (Marcus et al., 1993) to test small scale language modelling, the processed and raw versions of the Wikipedia text8 dataset (Hutter, 2012) to test large scale language modelling and byte level language modelling respectively, and the European parliament dataset (Koehn, 2005) to investigate multilingual fitting.
4.2 PENN TREEBANK
The Penn treebank dataset is relatively small, and consists of only case insensitive English characters, with no punctuation. It is one of the most widely used language modelling bench mark tasks. Due to its small size, the main bottleneck for performance is overfitting.
We fitted an mLSTM with 700 hidden units to the Penn Treebank dataset, with no regularization other than early stopping. We used a slightly different version of this dataset, where the frequently occurring token <unk> was replaced by a single character, shortening the file by about 4%. To make our results comparable to other results on this dataset, we computed the total cross entropy of the test set file and divided this by the number of characters in the original test set. The results are shown in
1Code to replicative our large scale experiments on the Hutter prize dataset is available at https:// github.com/benkrause/mLSTM.
Table 1, where it can be seen that mLSTM achieved 1.35 bits/char test set error, compared with 1.38 bits/char for an unregularized LSTM (Cooijmans et al., 2016).
4.3 TEXT8 DATASET
Text8 contains 100 million characters of English text taken from Wikipedia in 2006, consisting of just the 26 characters of the English alphabet plus spaces. This dataset can be found at http: //mattmahoney.net/dc/textdata. This corpus has been widely used to benchmark RNN character level language models, with the first 90 million characters used for training, the next 5 million used for validation, and the final 5 million used for testing. The results of these experiments are shown in Table 2.
The first set of experiments we performed were designed to be comparable to those of Zhang et al. (2016), who benchmarked several deep LSTMs against shallow LSTMs on this dataset. The shallow LSTM had a hidden state dimensionality of 512, and the deep versions had reduced dimensionality to give them roughly the same number of parameters. Our experiment used an mLSTM with a hidden dimensionality of 450, giving it slightly fewer parameters than the past work, and our own LSTM baseline with hidden dimensionality 512. mLSTM showed an improvement over our baseline and the previously reported best deep LSTM variant.
We ran additional experiments to compare a large mLSTM with other reported experiments. We trained an mLSTM with hidden dimensionality of 1900 on the text8 dataset. mLSTM was able to fit the training data well and achieved a competitive performance; however it was outperformed by other architectures that are less prone to over-fitting.
4.4 HUTTER PRIZE DATASET
We performed experiments using the raw version of the Wikipedia dataset, originally used for the Hutter Prize compression benchmark (Hutter, 2012). This dataset consists mostly of English language text and mark-up language text, but also contains text in other languages, including non-Latin languages. The dataset is modelled using a UTF-8 encoding, and contains 205 unique bytes.
We compared mLSTMs and 2-layer stacked LSTMs for varying network sizes, ranging from about 3–20 million parameters. These results all used RMS prop with normalized updates, stopping after 4 epochs, with test performance measured on the last 5 million bytes. Hyperparameters for each mLSTM and stacked LSTM were kept constant across all sizes, and were reused from the previous experiment using the text8 dataset. The results, shown in Figure 2, show that mLSTM gives a modest improvement across all network sizes.
We hypothesized that mLSTM’s superior performance over stacked LSTM was in part due to its ability to recover from surprising inputs. To test this we looked at each network’s performance after viewing surprising inputs that occurred naturally in the test set by creating a set of the 10% characters
with the largest average loss taken by mLSTM and stacked LSTM. Both networks perform roughly equally on this set of surprising characters, with mLSTM and stacked LSTM taking losses of 6.27 bits/character and 6.29 bits/character respectively. However, stacked LSTM tended to take much larger losses than mLSTM in the timesteps immediately following surprising inputs. One to four time-steps after a surprising input occurred, mLSTM and stacked LSTM took average losses of (2.26, 2.04, 1.61, 1.51) and (2.48, 2.25, 1.79, 1.67) bits per character respectively, as shown in Figure 3. mLSTM’s overall advantage over stacked LSTM was 1.42 bits/char to 1.53 bits/char; mLSTM’s advantage over stacked LSTM was greater after a surprising input than it is in general.
We also tested our largest mLSTM and stacked LSTM models using dynamic evaluation, where the network’s weights are adapted to fit recent sequences (Graves, 2013). Dynamic evaluation can be thought of as a type of fast weights memory structure (Ba et al., 2016) that draws the network towards regularities that recently occurred in the sequence, causing the network to assign higher probabilities to these regularities occurring again soon. Unlike other approaches to fast weights where the network learns to control the weights, dynamic evaluation uses the error signal and gradients to update the weights, which potentially increases its effectiveness, but also limits its scope to conditional generative modelling, when the outputs can be observed after they are predicted. Rather than performing fully online dynamic evaluation, we adapted incrementally to short sequences, allowing for gradients to be passed back over longer time scales. Thus, we divided the test set into sequences of length 50 in order of occurrence. After predicting a sequence and incurring a loss, we trained the RNN for a single iteration on that sequence, using RMSprop (with a learning rate, as normalized updates no longer make sense in a stochastic setting), and weight decay. After updating the RNN, we recomputed the forward pass through this sequence to update the final hidden state. The updated RNN was then used to predict the next sequence of 50 elements, and this process was repeated. Dynamic evaluation was applied to a test set of the last 4 million bytes (instead of last 5 million) to be comparable with the only other dynamic evaluation result (Graves, 2013).
The best results for stacked LSTM and mLSTM are given in Table 3, alongside results from the literature. mLSTM performs at near state-of-the-art level when evaluated statically, and greatly outperforms the best static models and other dynamic models when evaluated dynamically.
4.5 MULTILINGUAL LEARNING
Character-level RNNs are able to simultaneously model multiple languages using shared parameters. These experiments compared the relative ability of LSTM and mLSTM at fitting a dataset in a single language and a combined dataset with two separate languages. We used the first 100 million characters of the English and Spanish translations of the European parliament dataset to make an English dataset and a Spanish dataset. Each dataset was split 90-5-5 for training, validation, and testing. We created a third Spanish-English hybrid dataset by combining the Spanish and English datasets, resulting in a dataset twice as large. These datasets were left in their raw form, containing punctuation and both
upper-case and lower-case letters. The LSTMs in these experiments had a hidden dimensionality of 2200, and the mLSTMs had a hidden dimensionality of 1900. All experiments were run for 4 epochs.
mLSTM generally outperformed LSTM at this task, as shown in Table 4. However, there also seemed to be an interaction effect with the number of languages. Increasing the complexity of the task by forcing the RNN to learn 2 languages instead of 1 presented a larger fitting difficulty for LSTM than for mLSTM.
5 DISCUSSION
This work combined the mRNN’s factorized hidden weights with the LSTM’s hidden units for generative modelling of discrete multinomial sequences. This mLSTM architecture was motivated by its ability to have both controlled and flexible input-dependent transitions, to allow for fast changes to the distributed hidden representation without erasing information. In a series of character-level language modelling experiments, mLSTM showed improvements over LSTM and its deep variants. This relative improvement increased with the complexity of the task, and provides evidence that mLSTM has a more powerful fitting ability for character-level language modelling than regular LSTM and its common deep variants. mLSTM performed competitively at large scale characterlevel language modelling, and achieved dramatic improvement over the state of the art with 1.19 bits/character on the Hutter prize dataset when combined with dynamic evaluation, motivating further investigation of dynamic evaluation for RNN sequence modelling.
While these results are promising, it remains to be seen how mLSTM performs at word-level language modelling and other discrete multinomial generative modelling tasks, and whether mLSTM can be formulated to apply more broadly to tasks with continuous or non-sparse input units. We also hope this work will motivate further exploration in generative RNN architectures with flexible input-dependent transition functions.

Review:
This paper proposes an extension of the multiplicative RNN [1] where the authors apply the same reparametrization trick to the weight matrices of the LSTM. 
The paper proposes some interesting tricks, but none of them seems to be very crucial. For instance, in Eq. (16), the authors propose to multiply the output gate inside the activation function in order to alleviate the saturation problem in logistic sigmoid or hyperbolic tangent. Also, the authors share m_t across the inference of different gating units and cell-state candidates, at the end this brings only 1.25 times increase on the number of model parameters. Lastly, the authors use a variant of RMSProp where they add an additional hyper-parameter $\ell$ and schedule it across the training time. It would be nicer to apply the same tricks to other baseline models and show the improvement with regard to each trick.
With the new architectural modification to the LSTM and all the tricks combined, the performance is not as great as we would expect. Why didn’t the authors apply batch normalization, layer normalization or zoneout to their models? Was there any issue with applying one of those regularization or optimization techniques? 
At the fourth paragraph of Section 4.4, where the authors connect dynamic evaluation with fast weights is misleading. I find it a bit hard to connect dynamic evaluation as a variant of fast weights. Fast weights do not use test error signal. In the paper, the authors claim that “dynamic evaluation uses the error signal and gradients to update the weights, which potentially increases its effectiveness, but also limits its scope to conditional generative modelling, when the outputs can be observed after they are predicted”, and I am afraid to tell that this assumption is very misleading. We should never assume that test label information is given at the inference time. The test label information is there to evaluate the generalization performance of the model. In some applications, we may get the label information at test time, e.g., stock prediction, weather forecasting, however, in many other applications, we don’t. For instance, in machine translation, we don't know what's the best translation at the end, unlike weather forecasting. Also, it would be fair to apply dynamic evaluation to all the other baseline models as well to compare with the BPC score 1.19 achieved by the proposed mLSTM.
The quality of the work is not that bad, but the novelty of the paper is not that good either. The performance of the proposed model is oftentime worse than other methods, and it is only better when dynamic evaluation is coupled together. However, dynamic evaluation can improve the other methods as well.
[1] Ilya et al., “Generating Text with Recurrent Neural Networks”, ICML’11