Original: M OGRIFIER LSTM
Revised: M OGRIFIER LSTM

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many advances in Natural Language Processing have been based upon more expressive models for how inputs interact with the context in which they occur.
Revised: Many advances in Natural Language Processing have been based upon more expressive models for how inputs interact with the context in which they occur.

Original: Recurrent networks, which have enjoyed a modicum of success, still lack the generalization and systematicity ultimately required for modelling language.
Revised: Recurrent networks, which have enjoyed a modicum of success, still lack the generalization and systematicity ultimately required for modelling language.

Original: In this work, we propose an extension to the venerable Long Short-Term Memory in the form of mutual gating of the current input and the previous output.
Revised: In this work, we propose an extension to the venerable Long Short-Term Memory in the form of mutual gating of the current input and the previous output.

Original: This mechanism affords the modelling of a richer space of interactions between inputs and their context.
Revised: This mechanism affords the modelling of a richer space of interactions between inputs and their context.

Original: Equivalently, our model can be viewed as making the transition function given by the LSTM context-dependent.
Revised: Equivalently, our model can be viewed as making the transition function given by the LSTM context-dependent.

Original: Experiments demonstrate markedly improved generalization on language modelling in the range of 3–4 perplexity points on Penn
Revised: Experiments demonstrate markedly improved generalization on language modelling in the range of 3–4 perplexity points on Penn

Original: Treebank and Wikitext-2, and 0.01–0.05 bpc on four character-based datasets.
Revised: Treebank and Wikitext-2, and 0.01–0.05 bpc on four character-based datasets.

Original: We establish a new state of the art on all datasets with the exception of Enwik8, where we close a large gap between the LSTM and Transformer models.
Revised: We establish a new state of the art on all datasets with the exception of Enwik8, where we close a large gap between the LSTM and Transformer models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The domination of Natural Language Processing by neural models is hampered only by their limited ability to generalize and questionable sample complexity (Belinkov and Bisk 2017; Jia and Liang 2017; Iyyer et al. 2018; Moosavi and Strube 2017; Agrawal et al. 2016), their poor grasp of grammar (Linzen et al. 2016;
Revised: The domination of Natural Language Processing by neural models is hampered only by their limited ability to generalize and questionable sample complexity (Belinkov and Bisk 2017; Jia and Liang 2017; Iyyer et al. 2018; Moosavi and Strube 2017; Agrawal et al. 2016), their poor grasp of grammar (Linzen et al. 2016;

Original: Kuncoro et al. 2018), and their inability to chunk input sequences into meaningful units (Wang et al. 2017).
Revised: Kuncoro et al. 2018), and their inability to chunk input sequences into meaningful units (Wang et al. 2017).

Original: While direct attacks on the latter are possible, in this paper, we take a language-agnostic approach to improving Recurrent Neural Networks (RNN, Rumelhart et al.
Revised: While direct attacks on the latter are possible, in this paper, we take a language-agnostic approach to improving Recurrent Neural Networks (RNN, Rumelhart et al.

Original: that brought about many advances in tasks such as language modelling, semantic parsing, machine translation, with no shortage of non-NLP applications either (Bakker 2002; Mayer et al. 2008).
Revised: (1988)), which brought about many advances in tasks such as language modelling, semantic parsing, machine translation, with no shortage of non-NLP applications either (Bakker 2002; Mayer et al. 2008).

Original: Many neural models are built from RNNs including the sequence-to-sequence family (Sutskever et al. 2014) and its attention-based branch (Bahdanau et al. 2014).
Revised: Many neural models are built from RNNs including the sequence-to-sequence family (Sutskever et al. 2014) and its attention-based branch (Bahdanau et al. 2014).

Original: Thus, innovations in RNN architecture tend to have a trickle-down effect from language modelling, where evaluation is often the easiest and data the most readily available, to many other tasks, a trend greatly strengthened by ULMFiT (Howard and Ruder 2018), ELMo (Peters et al.
Revised: Thus, innovations in RNN architecture tend to have a trickle-down effect from language modelling, where evaluation is often the easiest and data the most readily available, to many other tasks, a trend greatly strengthened by ULMFiT (Howard and Ruder 2018), ELMo (Peters et al.

Original: and BERT (Devlin et al.
Revised: and BERT (Devlin et al.

Original: 2018) which promote language models from architectural blueprints to pretrained building blocks.
Revised: 2018), which promote language models from architectural blueprints to pretrained building blocks.

Original: To improve the generalization ability of language models, we propose an extension to the LSTM (Hochreiter and Schmidhuber 1997), where the LSTM’s input x is gated conditioned on the output of the previous step h prev .
Revised: To improve the generalization ability of language models, we propose an extension to the LSTM (Hochreiter and Schmidhuber 1997), where the LSTM’s input x is gated conditioned on the output of the previous step h prev .

Original: Next, the gated input is used in a similar manner to gate the output of the previous time step.
Revised: Next, the gated input is used in a similar manner to gate the output of the previous time step.

Original: After a couple of rounds of this mutual gating the last updated x and h prev are fed to an LSTM.
Revised: After a couple of rounds of this mutual gating, the last updated x and h prev are fed to an LSTM.

Original: By introducing these additional of gating operations, in one sense, our model joins the long list of recurrent architectures with gating structures of varying complexity which followed the invention of Elman Networks (Elman 1990).
Revised: By introducing these additional of gating operations, in one sense, our model joins the long list of recurrent architectures with gating structures of varying complexity which followed the invention of Elman Networks (Elman 1990).

Original: Examples include the LSTM, the GRU (Chung et al.
Revised: Examples include the LSTM, the GRU (Chung et al.

Original: , and even designs by Neural Architecture Search (Zoph and Le 2016).
Revised: , and even designs by Neural Architecture Search (Zoph and Le 2016).

Original: Intuitively, in the lowermost layer, the ﬁrst gating step scales the input embedding (itself a representation of the average context in which the token occurs) depending on the actual context, resulting in a contextualized representation of the input.
Revised: Intuitively, in the lowermost layer, the ﬁrst gating step scales the input embedding (itself a representation of the average context in which the token occurs) depending on the actual context, resulting in a contextualized representation of the input.

Original: While intuitive, as Section 4 shows, this interpretation cannot account for all the observed phenomena.
Revised: While intuitive, as Section 4 shows, this interpretation cannot account for all the observed phenomena.

Original: In a more encompassing view, our model can be seen as enriching the mostly additive dynamics of recurrent transitions placing it in the company of the Input Switched Afﬁne Network (Foerster et al.
Revised: In a more encompassing view, our model can be seen as enriching the mostly additive dynamics of recurrent transitions placing it in the company of the Input Switched Afﬁne Network (Foerster et al.

Original: 2017) with a separate transition matrix for each possible input, and the Multiplicative RNN (Sutskever et al. 2011) which factorizes the three-way tensor of stacked transition matrices. Also following x
Revised: x

Original: [Figure]
Revised: [Figure]

Original: hx -
Revised: hxx

Original: [Figure]
Revised: [Figure]

Original: x
Revised: x

Original: [Figure]
Revised: [Figure]

Original: x
Revised: hh

Original: [Figure] h [Figure] h
Revised: L S

Original: [Figure]
Revised: [Figure]

Original: M
Revised: M

Original: [Figure]
Revised: [Figure]

Revised: 2017) with a separate transition matrix for each possible input, and the Multiplicative RNN (Sutskever et al. 2011), which factorizes the three-way tensor of stacked transition matrices.

Original:  this line of research are the Multiplicative Integration LSTM (Wu et al. 2016) and – closest to our model in the literature – the Multiplicative LSTM (Krause et al. 2016).
Revised: Also following this line of research are the Multiplicative Integration LSTM (Wu et al. 2016) and – closest to our model in the literature – the Multiplicative LSTM (Krause et al. 2016).

Original: The results in Section 3. demonstrate the utility of our approach which consistently improves on the LSTM and establishes a new state of the art on all but the largest dataset, Enwik8, where we match similarly sized transformer models.
Revised: The results in Section 3. demonstrate the utility of our approach, which consistently improves on the LSTM and establishes a new state of the art on all but the largest dataset, Enwik8, where we match similarly sized transformer models.

Original:  M ODEL
Revised:  M ODEL

Original: To allow for ease of subsequent extension, we present the standard LSTM update (Sak et al. 2014) with input and state of size m and n respectively as the following function:
Revised: To allow for ease of subsequent extension, we present the standard LSTM update (Sak et al. 2014) with input and state of size m and n respectively as the following function:

Original: [Equation]
Revised: [Equation]

Original: The updated state c and the output h are computed as follows:
Revised: The updated state c and the output h are computed as follows:

Original: [Equation]
Revised: [Equation]

Original: where σ is the logistic sigmoid function, (cid:12) is the elementwise product, W ∗∗ and b ∗ are weight matrices and biases.
Revised: where σ is the logistic sigmoid function, (cid:12) is the elementwise product, W ∗∗ and b ∗ are weight matrices and biases.

Original: While the LSTM is typically presented as a solution to the vanishing gradients problem, its gate i can also be interpreted as scaling the rows of weight matrices W j ∗ (ignoring the non-linearity in j ).
Revised: While the LSTM is typically presented as a solution to the vanishing gradients problem, its gate i can also be interpreted as scaling the rows of weight matrices W j ∗ (ignoring the non-linearity in j ).

Original: In this sense, the LSTM nudges Elman Networks towards context-dependent transitions and the extreme case of Input Switched Afﬁne Networks.
Revised: In this sense, the LSTM nudges Elman Networks towards context-dependent transitions and the extreme case of Input Switched Afﬁne Networks.

Original: If we took another, larger step towards that extreme we could end up with Hypernetworks (Ha et al. 2016).
Revised: If we took another, larger step towards that extreme, we could end up with Hypernetworks (Ha et al. 2016).

Original: Here, instead, we take a more cautious step and equip the LSTM with gates that scale the columns of all its weight matrices W ∗∗ in a context-dependent manner.
Revised: Here, instead, we take a more cautious step, and equip the LSTM with gates that scale the columns of all its weight matrices W ∗∗ in a context-dependent manner.

Original: The scaling of the matrices W ∗ x
Revised: The scaling of the matrices W ∗ x

Original: (those that transform the cell input) makes the input embeddings dependent on the cell state, while the scaling of W ∗ h does the reverse.
Revised: (those that transform the cell input) makes the input embeddings dependent on the cell state, while the scaling of W ∗ h does the reverse.

Original: The Mogriﬁer 1 LSTM is an LSTM where two inputs x and h prev modulate one another in an alternating fashion before the usual LSTM computation takes place (see Fig. 1).
Revised: The Mogriﬁer 1 LSTM is an LSTM where two inputs x and h prev modulate one another in an alternating fashion before the usual LSTM computation takes place (see Fig. 1).

Original: That is,
Revised: That is,

Original: Mogrify( x , c prev , h prev ) =
Revised: Mogrify( x , c prev , h prev ) =

Original: LSTM( x ↑ , c prev , h ↑ prev ) where the modulated inputs x ↑ and h ↑ prev are deﬁned as the highest indexed x i and h i prev , respectively, from the interleaved sequences:
Revised: LSTM( x ↑ , c prev , h ↑ prev ) where the modulated inputs x ↑ and h ↑ prev are deﬁned as the highest indexed x i and h i prev , respectively, from the interleaved sequences

Original: [Equation]
Revised: [Equation]

Revised: for odd

Original: [Equation] 
Revised: [Equation] [Equation]

Original: for eveni ∈ [
Revised: for even

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: with x − 1 = x and h 0 prev = h prev .
Revised: with x − 1 = x and h 0 prev = h prev .

Original: The number of “rounds”, r ∈ N , is a hyperparameter. r =recovers the LSTM.
Revised: The number of “rounds”, r ∈ N , is a hyperparameter; r =recovers the LSTM.

Original: Multiplication with the constant 2 ensures that randomly initialized Q
Revised: Multiplication with the constant 2 ensures that randomly initialized Q

Original: i , R i matrices result in transformations close to identity.
Revised: i , R i matrices result in transformations close to identity.

Original: To reduce the number of additional model parameters, we typically factorize the Q i , R i matrices as products of low-rank matrices: Q i =
Revised: To reduce the number of additional model parameters, we typically factorize the Q i , R i matrices as products of low-rank matrices: Q i =

Original: Q i left Q i right with Q i ∈ R m × n , Q i left ∈ R m × k , Q i right ∈ R k × n  where k < min ( m, n ) is the rank.
Revised: Q i left Q i right with Q i ∈ R m × n , Q i left ∈ R m × k , Q i right ∈ R k × n , where k < min ( m, n ) is the rank.

Original:  E XPERIMENTS 3. T HE C ASE FOR S MALL -S CALE
Revised:  E XPERIMENTS 3. T HE C ASE FOR S MALL -S CALE

Original: Before describing the details of the data, the experimental setup and the results, we take a short detour to motivate work on smaller-scale datasets.
Revised: Before describing the details of the data, the experimental setup and the results, we take a short detour to motivate work on smaller-scale datasets.

Original: A recurring theme in the history of sequence models is that the problem of model design is intermingled with optimizability and scalability.
Revised: A recurring theme in the history of sequence models is that the problem of model design is intermingled with optimizability and scalability.

Original: Elman Networks are notoriously difﬁcult to optimize, a property that ultimately gave birth to the idea of the LSTM, but also to more recent models such as the Unitary Evolution RNN (Arjovsky et al. 2016) and ﬁxes like gradient clipping (Pascanu et al. 2013).
Revised: Elman Networks are notoriously difﬁcult to optimize, a property that ultimately gave birth to the idea of the LSTM, but also to more recent models such as the Unitary Evolution RNN (Arjovsky et al. 2016) and ﬁxes like gradient clipping (Pascanu et al. 2013).

Original: Still, it is far from clear – if we could optimize these models well – how different their biases would turn out to be.
Revised: Still, it is far from clear – if we could optimize these models well – how different their biases would turn out to be.

Original: The non-separability of model and optimization is fairly evident in these cases.
Revised: The non-separability of model and optimization is fairly evident in these cases.

Original: Scalability, on the other hand, is often optimized for indirectly.
Revised: Scalability, on the other hand, is often optimized for indirectly.

Original: Given the limited ability of current models to generalize, we often compensate by throwing more data at the problem.
Revised: Given the limited ability of current models to generalize, we often compensate by throwing more data at the problem.

Original: To ﬁt a larger dataset, model size must be increased.
Revised: To ﬁt a larger dataset, model size must be increased.

Original: Thus the best performing models are evaluated based on their scalability 3 .
Revised: Thus the best performing models are evaluated based on their scalability 3 .

Original: Today, scaling up still yields tangible gains on down-stream tasks, and language modelling data is abundant.
Revised: Today, scaling up still yields tangible gains on down-stream tasks, and language modelling data is abundant.

Original: However, we believe that simply scaling up will not solve the generalization problem and better models will be needed.
Revised: However, we believe that simply scaling up will not solve the generalization problem and better models will be needed.

Original: Our hope is that by choosing small enough datasets so that model size is no longer the limiting factor, we get a number of practical advantages:
Revised: Our hope is that by choosing small enough datasets, so that model size is no longer the limiting factor, we get a number of practical advantages:

Original: (cid:63) Generalization ability will be more clearly reﬂected in evaluations even without domain adaptation.
Revised: (cid:63) Generalization ability will be more clearly reﬂected in evaluations even without domain adaptation.

Original: (cid:63) Turnaround time in experiments will be reduced, and the freed up computational budget can be put to good use by controlling for nuisance factors.
Revised: (cid:63) Turnaround time in experiments will be reduced, and the freed up computational budget can be put to good use by controlling for nuisance factors.

Original: (cid:63) The transient effects of changing hardware performance characteristics are somewhat lessened.
Revised: (cid:63) The transient effects of changing hardware performance characteristics are somewhat lessened.

Original: Thus, we develop, analyse and evaluate models primarily on small datasets.
Revised: Thus, we develop, analyse and evaluate models primarily on small datasets.

Original: Evaluation on larger datasets is included to learn more about the models’ scaling behaviour and because of its relevance for applications, but it is to be understood that these evaluations come with much larger error bars and provide more limited guidance for further research on better models.
Revised: Evaluation on larger datasets is included to learn more about the models’ scaling behaviour and because of its relevance for applications, but it is to be understood that these evaluations come with much larger error bars and provide more limited guidance for further research on better models.

Original: 3. D ATASETS
Revised: 3. D ATASETS

Original: We compare models on both word and character-level language modelling datasets.
Revised: We compare models on both word and character-level language modelling datasets.

Original: The two wordlevel datasets we picked are the Penn Treebank (PTB) corpus by Marcus et al.
Revised: The two wordlevel datasets we picked are the Penn Treebank (PTB) corpus by Marcus et al.

Original: with preprocessing from Mikolov et al.
Revised: with preprocessing from Mikolov et al.

Original: (2010) and Wikitext-2 by Merity et al.
Revised: (2010) and Wikitext-2 by Merity et al.

Original: (2016) which is about twice the size of PTB with a larger vocabulary and lighter preprocessing.
Revised: (2016), which is about twice the size of PTB with a larger vocabulary and lighter preprocessing.

Original: These datasets are deﬁnitely on the small side, but – and because of this – they are suitable for exploring different model biases.
Revised: These datasets are deﬁnitely on the small side, but – and because of this – they are suitable for exploring different model biases.

Original: Their main shortcoming is the small vocabulary size, only in the tens of thousands, which makes them inappropriate for exploring the behaviour of the long tail.
Revised: Their main shortcoming is the small vocabulary size, only in the tens of thousands, which makes them inappropriate for exploring the behaviour of the long tail.

Original: For that, open vocabulary language modelling and byte pair encoding (Sennrich et al. 2015) would be an obvious choice.
Revised: For that, open vocabulary language modelling and byte pair encoding (Sennrich et al. 2015) would be an obvious choice.

Original: Still, our primary goal here is the comparison of the LSTM and Mogriﬁer architectures, thus we instead opt for character-based language modelling tasks where vocabulary size is not an issue, the long tail is not truncated, and there are no additional hyperparameters as in byte pair encoding that make fair comparison harder.
Revised: Still, our primary goal here is the comparison of the LSTM and Mogriﬁer architectures, thus we instead opt for character-based language modelling tasks, where vocabulary size is not an issue, the long tail is not truncated, and there are no additional hyperparameters as in byte pair encoding that make fair comparison harder.

Original: The ﬁrst character-based corpus is Enwik8 from the Hutter Prize dataset (Hutter 2012).
Revised: The ﬁrst character-based corpus is Enwik8 from the Hutter Prize dataset (Hutter 2012).

Original: Following common practice, we use the ﬁrst 90 million characters for training, and the remaining 10 million evenly split between validation and test.
Revised: Following common practice, we use the ﬁrst 90 million characters for training and the remaining 10 million evenly split between validation and test.

Original: The character-level task on the
Revised: The character-level task on the

Original: [Table]
Revised: [Table]

Original: Mikolov preprocessed PTB corpus (Merity et al. 2018) is unique in that it has the disadvantages of closed vocabulary without the advantages of word-level modelling, but we include it for comparison to previous work.
Revised: Mikolov preprocessed PTB corpus (Merity et al. 2018) is unique in that it has the disadvantages of closed vocabulary without the advantages of word-level modelling, but we include it for comparison to previous work.

Original: The ﬁnal character-level dataset is the Multilingual Wikipedia Corpus (MWC,
Revised: The ﬁnal character-level dataset is the Multilingual Wikipedia Corpus (MWC,

Original: Kawakami et al.
Revised: Kawakami et al.

Original:  from which we focus on the English and Finnish language subdatasets in the single text, large setting.
Revised: (2017)), from which we focus on the English and Finnish language subdatasets in the single text, large setting.

Original: 3. S ETUP
Revised: 3. S ETUP

Original: We tune hyperparameters following the experimental setup of Anonymised (2020), using a black-box hyperparameter tuner based on batched Gaussian Process Bandits (Golovin et al. 2017)
Revised: We tune hyperparameters following the experimental setup of Melis et al. (2018) using a black-box hyperparameter tuner based on batched Gaussian Process Bandits (Golovin et al. 2017).

Original: For the
Revised: For the

Original: LSTM, the tuned hyperparameters are the same: input_embedding_ratio , learning_rate , l2_penalty , input_dropout , inter_layer_dropout , state_dropout , output_dropout .
Revised: LSTM, the tuned hyperparameters are the same: input_embedding_ratio , learning_rate , l2_penalty , input_dropout , inter_layer_dropout , state_dropout , output_dropout .

Original: For the Mogriﬁer, the number of rounds r and the rank k of the low-rank approximation is also tuned (allowing for full rank, too).
Revised: For the Mogriﬁer, the number of rounds r and the rank k of the low-rank approximation is also tuned (allowing for full rank, too).

Original: For word-level tasks, BPTT (Werbos et al.
Revised: For word-level tasks, BPTT (Werbos et al.

Original: 1990) window size is set to 70 and batch size to 64.
Revised: 1990) window size is set to 70 and batch size to 64.

Original: For character-level tasks, BPTT window size is set to 150 and batch size to 128 except for Enwik8 where the window size is 500.
Revised: For character-level tasks, BPTT window size is set to 150 and batch size to 128 except for Enwik8 where the window size is 500.

Original: Input and output embeddings are tied for word-level tasks following Inan et al.
Revised: Input and output embeddings are tied for word-level tasks following Inan et al.

Original: (2016) and Press and Wolf (2016).
Revised: (2016) and Press and Wolf (2016).

Original: Optimization is performed with Adam (Kingma and Ba 2014) with β 1 = 0 , a setting that resembles RMSProp without momentum.
Revised: Optimization is performed with Adam (Kingma and Ba 2014) with β 1 = 0 , a setting that resembles RMSProp without momentum.

Original: Gradients are clipped (Pascanu et al. 2013) to norm 10.
Revised: Gradients are clipped (Pascanu et al. 2013) to norm 10.

Original: We switch to averaging weights similarly to Merity et al.
Revised: We switch to averaging weights similarly to Merity et al.

Original: (2017) after a certain number of checkpoints with no improvement in validation cross-entropy, or at 80% of the training time at the latest.
Revised: (2017) after a certain number of checkpoints with no improvement in validation cross-entropy or at 80% of the training time at the latest.

Original: We found no beneﬁt to using two-step ﬁnetuning.
Revised: We found no beneﬁt to using two-step ﬁnetuning.

Original: Model evaluation is performed with the standard, deterministic dropout approximation or MonteCarlo averaging (Gal and Ghahramani 2016) where explicitly noted (MC).
Revised: Model evaluation is performed with the standard, deterministic dropout approximation or MonteCarlo averaging (Gal and Ghahramani 2016) where explicitly noted (MC).

Original: In standard dropout evaluation, dropout is turned off while in MC dropout predictions are averaged over randomly sampled dropout masks (200 in our experiments).
Revised: In standard dropout evaluation, dropout is turned off while in MC dropout predictions are averaged over randomly sampled dropout masks (200 in our experiments).

Original: Optimal softmax temperature is determined on the validation set, and in the MC case dropout rates are scaled (Anonymised 2020).
Revised: Optimal softmax temperature is determined on the validation set, and in the MC case dropout rates are scaled (Melis et al. 2018).

Original: Finally, we report results with and without dynamic evaluation (Krause et al. 2017).
Revised: Finally, we report results with and without dynamic evaluation (Krause et al. 2017).

Original: Hyperparameters for dynamic evaluation are tuned using the same method (see Appendix A for details).
Revised: Hyperparameters for dynamic evaluation are tuned using the same method (see Appendix A for details).

Original: We make the code and the tuner output available at <anonymised> .
Revised: We make the code and the tuner output available at https://github.com/deepmind/lamb.

Original: 3. R ESULTS
Revised: 3. R ESULTS

Original: Table 1 lists our results on word-level datasets.
Revised: Table 1 lists our results on word-level datasets.

Original: On the PTB and Wikitext-2 datasets, the Mogriﬁer has lower perplexity than the LSTM by 3–4 perplexity points regardless of whether or not dynamic evaluation (Krause et al. 2017) and Monte-Carlo averaging are used.
Revised: On the PTB and Wikitext-2 datasets, the Mogriﬁer has lower perplexity than the LSTM by 3–4 perplexity points regardless of whether or not dynamic evaluation (Krause et al. 2017) and Monte-Carlo averaging are used.

Original: On both datasets, the state of the art is held by the AWD LSTM (Merity et al. 2017) extended with Mixture of Softmaxes (Yang
Revised: On both datasets, the state of the art is held by the AWD LSTM (Merity et al. 2017) extended with Mixture of Softmaxes (Yang

Original: [Table]
Revised: [Table]

Original: et al. 2017) and FRAGE (Gong et al. 2018).
Revised: et al. 2017) and FRAGE (Gong et al. 2018).

Original: The Mogriﬁer improves the state of the art without either of these methods on PTB, and without FRAGE on Wikitext-2.
Revised: The Mogriﬁer improves the state of the art without either of these methods on PTB, and without FRAGE on Wikitext-2.

Original: Table 2 lists the character-level modelling results.
Revised: Table 2 lists the character-level modelling results.

Original: On all datasets, our baseline LSTM results are much better than those previously reported for LSTMs highlighting the issue of scalability and experimental controls.
Revised: On all datasets, our baseline LSTM results are much better than those previously reported for LSTMs, highlighting the issue of scalability and experimental controls.

Original: In some cases, these unexpectedly large gaps may be down to lack of hyperparameter tuning as in the case of Merity et al.
Revised: In some cases, these unexpectedly large gaps may be down to lack of hyperparameter tuning as in the case of Merity et al.

Original: (2017), or in others,
Revised: (2017), or in others,

Original: to using a BPTT window size (50) that is too small for character-level modelling (Melis et al. 2017) in order to ﬁt the model into memory.
Revised: to using a BPTT window size (50) that is too small for character-level modelling (Melis et al. 2017) in order to ﬁt the model into memory.

Original: The Mogriﬁer further improves on these baselines by a considerable margin.
Revised: The Mogriﬁer further improves on these baselines by a considerable margin.

Original: Even the smallest improvement of 0.012 bpc on the highly idiosyncratic, character-based, Mikolov preprocessed PTB task is equivalent to gaining about 3 perplexity points on word-level PTB.
Revised: Even the smallest improvement of 0.012 bpc on the highly idiosyncratic, character-based, Mikolov preprocessed PTB task is equivalent to gaining about 3 perplexity points on word-level PTB.

Original: MWC, which was built for open-vocabulary language modelling, is a much better smaller-scale character-level dataset.
Revised: MWC, which was built for open-vocabulary language modelling, is a much better smaller-scale character-level dataset.

Original: On the English and the
Revised: On the English and the

Original: Finnish corpora in MWC, the Mogriﬁer enjoys a gap of 0.033-0.046 bpc.
Revised: Finnish corpora in MWC, the Mogriﬁer enjoys a gap of 0.033-0.046 bpc.

Original: Finally, on the Enwik dataset, the gap is 0.029-0.039 bpc in favour of the Mogriﬁer.
Revised: Finally, on the Enwik dataset, the gap is 0.029-0.039 bpc in favour of the Mogriﬁer.

Original: xhxxxhh
Revised: xhxxxhh

Original: L S T
Revised: L S T

Original: M
Revised: M

Original: [Figure]
Revised: [Figure]

Original: ⦁ ⦁
Revised: ⦁ ⦁

Original: Mogriﬁer
Revised: Mogriﬁer

Original: [Table]
Revised: [Table]

Original: Of particular note is the comparison to Transformer-XL (Dai et al. 2019), a state-of-the-art model on larger datasets such as Wikitext-103 and Enwik8.
Revised: Of particular note is the comparison to Transformer-XL (Dai et al. 2019), a state-of-the-art model on larger datasets such as Wikitext-103 and Enwik8.

Original: On PTB, without dynamic evaluation, the
Revised: On PTB, without dynamic evaluation, the

Original: Transformer-XL is on par with our LSTM baseline which puts it about 3.5 perplexity points behind the Mogriﬁer.
Revised: Transformer-XL is on par with our LSTM baseline which puts it about 3.5 perplexity points behind the Mogriﬁer.

Original: On Enwik8, also without dynamic evaluation, the Transformer-XL has a large, 0.09 bpc advantage at similar parameter budgets, but with dynamic evaluation this gap disappears.
Revised: On Enwik8, also without dynamic evaluation, the Transformer-XL has a large, 0.09 bpc advantage at similar parameter budgets, but with dynamic evaluation this gap disappears.

Original: However, we did not test the Transformer-XL ourselves, so fair comparison is not possible due to differing experimental setups and the rather sparse result matrix for the Transformer-XL.
Revised: However, we did not test the Transformer-XL ourselves, so fair comparison is not possible due to differing experimental setups and the rather sparse result matrix for the Transformer-XL.

Original:  A NALYSIS 4. A BLATION S TUDY
Revised:  A NALYSIS 4. A BLATION S TUDY

Original: The Mogriﬁer consistently outperformed the LSTM in our experiments.
Revised: The Mogriﬁer consistently outperformed the LSTM in our experiments.

Original: The optimal settings were similar across all datasets, with r ∈ { 5 , 6 } and k ∈ [40 . . .
Revised: The optimal settings were similar across all datasets, with r ∈ { 5 , 6 } and k ∈ [40 . . .

Original: (see Appendix B for a discussion of hyperparameter sensitivity).
Revised: (see Appendix B for a discussion of hyperparameter sensitivity).

Original: In this section, we explore the effect of these hyperparameters and show that the proposed model is not unnecessarily complicated.
Revised: In this section, we explore the effect of these hyperparameters and show that the proposed model is not unnecessarily complicated.

Original: To save computation, we tune all models using a shortened schedule with only 145 epochs instead of 964 and a truncated BPTT window size of 35 on the word-level PTB dataset, and evaluate using the standard, deterministic dropout approximation with a tuned softmax temperature.
Revised: To save computation, we tune all models using a shortened schedule with only 145 epochs instead of 964 and a truncated BPTT window size of 35 on the word-level PTB dataset, and evaluate using the standard, deterministic dropout approximation with a tuned softmax temperature.

Original: Fig.
Revised: Fig.

Original: 3 shows that the number of rounds r greatly inﬂuences the results.
Revised: 3 shows that the number of rounds r greatly inﬂuences the results.

Original: Second, we found the low-rank factorization of Q i and R i to help a bit, but the full-rank variant is close behind which is what we observed on other datasets, as well.
Revised: Second, we found the low-rank factorization of Q i and R i to help a bit, but the full-rank variant is close behind which is what we observed on other datasets, as well.

Original: Finally, to verify that the alternating gating scheme is not overly complicated, we condition all newly introduced gates on the original inputs x and h prev (see Fig. 2).
Revised: Finally, to verify that the alternating gating scheme is not overly complicated, we condition all newly introduced gates on the original inputs x and h prev (see Fig. 2).

Original: That is, instead of Eq.
Revised: That is, instead of Eq.

Original: 1 and Eq.
Revised: 1 and Eq.

Original: 2 the no-zigzag updates are:
Revised: 2 the no-zigzag updates are

Original: [Equation]
Revised: [Equation]

Original: for odd i [Equation] [Equation] [Equation]
Revised: for odd i ∈ [1 . . .

Original: r ] 
Revised: r ] ,

Original: [Equation]
Revised: [Equation]

Original: for even i ∈ [1 . . .
Revised: for even i ∈ [1 . . .

Original: r ] 
Revised: r ] .

Original: In our experiments, the no-zigzag variant underperformed the baseline Mogriﬁer by a small but signiﬁcant margin, and was on par with the r = 2 model in Fig.
Revised: In our experiments, the no-zigzag variant underperformed the baseline Mogriﬁer by a small but signiﬁcant margin, and was on par with the r = 2 model in Fig.

Original: 3 suggesting that the Mogriﬁer’s iterative reﬁnement scheme does more than simply widen the range of possible gating values of x and h prev to (0 , 2 (cid:100) r/ 2 (cid:101) ) and (0 , 2 (cid:98) r/ 2 (cid:99) ) , respectively.
Revised: 3 suggesting that the Mogriﬁer’s iterative reﬁnement scheme does more than simply widen the range of possible gating values of x and h prev to (0 , 2 (cid:100) r/ 2 (cid:101) ) and (0 , 2 (cid:98) r/ 2 (cid:99) ) , respectively.

Original: 4. C OMPARISON TO THE M LSTM
Revised: 4. C OMPARISON TO THE M LSTM

Original: The Multiplicative LSTM (Krause et al. 2016), or mLSTM for short, is closest to our model in the literature.
Revised: The Multiplicative LSTM (Krause et al. 2016), or mLSTM for short, is closest to our model in the literature.

Original: It is deﬁned as mLSTM( x , c prev , h prev ) =
Revised: It is deﬁned as mLSTM( x , c prev , h prev ) =

Original: LSTM( x , c prev , h m prev )  where h m prev =
Revised: LSTM( x , c prev , h m prev ) , where h m prev =

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: LSTM
Revised: LSTM

Original: Mogrifier
Revised: Mogrifier

Original: [Figure]
Revised: [Figure]

Original: LSTM
Revised: LSTM

Original: Mogrifier
Revised: Mogrifier

Original: [Figure]
Revised: [Figure]

Original: ( W mx x ) (cid:12) ( W mh h prev ) .
Revised: ( W mx x ) (cid:12) ( W mh h prev ) .

Original: In this formulation, the differences are readily apparent.
Revised: In this formulation, the differences are readily apparent.

Original: First, the mLSTM allows for multiplicative interaction between x and h prev , but it only overrides h prev  while in the
Revised: First, the mLSTM allows for multiplicative interaction between x and h prev , but it only overrides h prev , while in the

Original: Mogriﬁer the interaction is two-way which, as the ablation study showed, is important.
Revised: Mogriﬁer the interaction is two-way, which – as the ablation study showed – is important.

Original: Second, the mLSTM can change not only the magnitude but also the sign of values in h prev , something with which we experimented in the Mogriﬁer, but could not get to work.
Revised: Second, the mLSTM can change not only the magnitude but also the sign of values in h prev , something with which we experimented in the Mogriﬁer, but could not get to work.

Original: Furthermore, in the deﬁnition of h m prev  the unsquashed linearities and their elementwise product make the mLSTM more sensitive to initialization and unstable during optimization.
Revised: Furthermore, in the deﬁnition of h m prev , the unsquashed linearities and their elementwise product make the mLSTM more sensitive to initialization and unstable during optimization.

Original: On the Enwik8 dataset, we greatly improved on the published results of the mLSTM (Krause et al.
Revised: On the Enwik8 dataset, we greatly improved on the published results of the mLSTM (Krause et al.

Original: In fact, even our LSTM baseline outperformed the mLSTM by 0.03 bpc.
Revised: In fact, even our LSTM baseline outperformed the mLSTM by 0.03 bpc.

Original: We also conducted experiments on PTB based on our reimplementation of the mLSTM following the same methodology as the ablation study and found that the mLSTM did not improve on the LSTM (see Table 3).
Revised: We also conducted experiments on PTB based on our reimplementation of the mLSTM following the same methodology as the ablation study and found that the mLSTM did not improve on the LSTM (see Table 3).

Original: Krause et al.
Revised: Krause et al.

Original: (2016) posit and verify the recovery hypothesis which says that having just suffered a large loss, the loss on the next time step will be smaller on average for the mLSTM than for the
Revised: (2016) posit and verify the recovery hypothesis which says that having just suffered a large loss, the loss on the next time step will be smaller on average for the mLSTM than for the

Original: LSTM.
Revised: LSTM.

Original: This was found not to be the case for the Mogriﬁer.
Revised: This was found not to be the case for the Mogriﬁer.

Original: Neither did we observe a signiﬁcant change in the gap between the LSTM and the Mogriﬁer in the tied and untied embeddings settings which would be expected if recovery was affected by x and h prev being in different domains.
Revised: Neither did we observe a signiﬁcant change in the gap between the LSTM and the Mogriﬁer in the tied and untied embeddings settings, which would be expected if recovery was affected by x and h prev being in different domains.

Original: 4. T HE R EVERSE C OPY T ASK
Revised: 4. T HE R EVERSE C OPY T ASK

Original: Our original motivation for the Mogriﬁer was to allow the context to amplify salient and attenuate nuisance features in the input embeddings.
Revised: Our original motivation for the Mogriﬁer was to allow the context to amplify salient and attenuate nuisance features in the input embeddings.

Original: We conduct a simple experiment to support this point of view.
Revised: We conduct a simple experiment to support this point of view.

Original: Consider the reverse copy task where the network reads an input sequence of tokens and a marker token after which it has to repeat the input in reverse order.
Revised: Consider the reverse copy task where the network reads an input sequence of tokens and a marker token after which it has to repeat the input in reverse order.

Original: In this simple sequence-tosequence learning (Sutskever et al. 2014) setup, the reversal is intended to avoid the minimal time lag problem (Hochreiter and Schmidhuber 1997) which is not our focus here.
Revised: In this simple sequence-tosequence learning (Sutskever et al. 2014) setup, the reversal is intended to avoid the minimal time lag problem (Hochreiter and Schmidhuber 1997), which is not our focus here.

Original: The experimental setup is as follows.
Revised: The experimental setup is as follows.

Original: For the training set, we generate 500 000 examples by uniformly sampling a given number of tokens from a vocabulary of size 1000.
Revised: For the training set, we generate 500 000 examples by uniformly sampling a given number of tokens from a vocabulary of size 1000.

Original: The validation and test sets are constructed similarly and contain 10 000 examples.
Revised: The validation and test sets are constructed similarly, and contain 10 000 examples.

Original: The model consists of an independent, unidirectional encoder and a decoder whose total number of parameters is 10 million.
Revised: The model consists of an independent, unidirectional encoder and a decoder, whose total number of parameters is 10 million.

Original: The decoder is initialized from the last state of the encoder.
Revised: The decoder is initialized from the last state of the encoder.

Original: Since overﬁtting is not an issue here, no dropout is necessary and we only tune the learning rate, the l2 penalty, and the embedding size for the LSTM.
Revised: Since overﬁtting is not an issue here, no dropout is necessary, and we only tune the learning rate, the l2 penalty, and the embedding size for the LSTM.

Original: For the Mogriﬁer, the number of rounds r and the rank k of the low-rank approximation are also tuned.
Revised: For the Mogriﬁer, the number of rounds r and the rank k of the low-rank approximation are also tuned.

Original: We compare the case where both the encoder and decoder are LSTMs to where both are Mogriﬁers.
Revised: We compare the case where both the encoder and decoder are LSTMs to where both are Mogriﬁers.

Original: Fig.
Revised: Fig.

Original: 4a shows that for sequences of length 50 and 100 both models can solve the task perfectly.
Revised: 4a shows that, for sequences of length 50 and 100, both models can solve the task perfectly.

Original: At higher lengths though, the Mogriﬁer has a considerable advantage.
Revised: At higher lengths though, the Mogriﬁer has a considerable advantage.

Original: Examining the best hyperparameter settings found, the embedding/hidden sizes for the LSTM and Mogriﬁer are 498/787 vs 41/1054 at 150 steps, and 493/790 vs 181/961 at 200 steps.
Revised: Examining the best hyperparameter settings found, the embedding/hidden sizes for the LSTM and Mogriﬁer are 498/787 vs 41/1054 at 150 steps, and 493/790 vs 181/961 at 200 steps.

Original: Clearly, the Mogriﬁer was able to work with a much smaller embedding size than the LSTM which is in line with our expectations for a model with a more ﬂexible interaction between the input and recurrent state.
Revised: Clearly, the Mogriﬁer was able to work with a much smaller embedding size than the LSTM, which is in line with our expectations for a model with a more ﬂexible interaction between the input and recurrent state.

Original: We also conducted experiments with a larger model and vocabulary size and found the effect even more pronounced (see Fig. 4b).
Revised: We also conducted experiments with a larger model and vocabulary size, and found the effect even more pronounced (see Fig. 4b).

Original: 4. W HAT THE M OGRIFIER IS NOT
Revised: 4. W HAT THE M OGRIFIER IS NOT

Original: The results on the reverse copy task support our hypothesis that input embeddings are enriched by the Mogriﬁer architecture, but that cannot be the full explanation as the results of the ablation study indicate.
Revised: The results on the reverse copy task support our hypothesis that input embeddings are enriched by the Mogriﬁer architecture, but that cannot be the full explanation as the results of the ablation study indicate.

Original: In the following, we consider a number of hypotheses about where the advantage of the
Revised: In the following, we consider a number of hypotheses about where the advantage of the

Original: Mogriﬁer lies and the experiments that provide evidence against them.
Revised: Mogriﬁer lies and the experiments that provide evidence against them.

Original: (cid:69) Hypothesis: the beneﬁt is in scaling x and h prev .
Revised: (cid:69) Hypothesis: the beneﬁt is in scaling x and h prev .

Original: We veriﬁed that data dependency is a crucial feature by adding a learnable scaling factor to the LSTM inputs.
Revised: We veriﬁed that data dependency is a crucial feature by adding a learnable scaling factor to the LSTM inputs.

Original: We observed no improvement.
Revised: We observed no improvement.

Original: Also, at extremely low-rank (less than 5) settings where the amount of information in its gating is small, the Mogriﬁer loses its advantage.
Revised: Also, at extremely low-rank (less than 5) settings where the amount of information in its gating is small, the Mogriﬁer loses its advantage.

Original: (cid:69) Hypothesis: the beneﬁt is in making optimization easier.
Revised: (cid:69) Hypothesis: the beneﬁt is in making optimization easier.

Original: We performed experiments with different optimizers (SGD, RMSProp), with intra-layer batch normalization and layer normalization on the LSTM gates.
Revised: We performed experiments with different optimizers (SGD, RMSProp), with intra-layer batch normalization and layer normalization on the LSTM gates.

Original: While we cannot rule out an effect on optimization difﬁculty, in all of these experiments the gap between the LSTM and the Mogriﬁer was the same.
Revised: While we cannot rule out an effect on optimization difﬁculty, in all of these experiments the gap between the LSTM and the Mogriﬁer was the same.

Original: (cid:69) Hypothesis: exact tying of embeddings is too constraining, the beneﬁt is in making this relationship less strict.
Revised: (cid:69) Hypothesis: exact tying of embeddings is too constraining, the beneﬁt is in making this relationship less strict.

Original: Experiments conducted with untied embeddings and character-based models demonstrate improvements of similar magnitude.
Revised: Experiments conducted with untied embeddings and character-based models demonstrate improvements of similar magnitude.

Original: (cid:69) Hypothesis: the beneﬁt is in the low-rank factorization of Q i , R i implicitly imposing structure on the LSTM weight matrices.
Revised: (cid:69) Hypothesis: the beneﬁt is in the low-rank factorization of Q i , R i implicitly imposing structure on the LSTM weight matrices.

Original: We observed that the full-rank Mogriﬁer also performed better than the plain LSTM.
Revised: We observed that the full-rank Mogriﬁer also performed better than the plain LSTM.

Original: We conducted additional experiments where the LSTM’s gate matrices were factorized and observed no improvement.
Revised: We conducted additional experiments where the LSTM’s gate matrices were factorized and observed no improvement.

Original: (cid:69) Hypothesis: the beneﬁt comes from better performance on rare words.
Revised: (cid:69) Hypothesis: the beneﬁt comes from better performance on rare words.

Original: The observed advantage on character-based modelling is harder to explain based on frequency.
Revised: The observed advantage on character-based modelling is harder to explain based on frequency.

Original: Also, in the reverse copy experiments, a large number of tokens were sampled uniformly, so there were no rare words at all.
Revised: Also, in the reverse copy experiments, a large number of tokens were sampled uniformly, so there were no rare words at all.

Original: (cid:69) Hypothesis: the beneﬁt is speciﬁc to the English language.
Revised: (cid:69) Hypothesis: the beneﬁt is speciﬁc to the English language.

Original: This is directly contradicted by the
Revised: This is directly contradicted by the

Original: Finnish MWC and the reverse copy experiments.
Revised: Finnish MWC and the reverse copy experiments.

Original: (cid:69) Hypothesis: the beneﬁt is in handling long-range dependencies better.
Revised: (cid:69) Hypothesis: the beneﬁt is in handling long-range dependencies better.

Original: Experiments in the episodic setting (i.e. sentence-level language modelling) exhibited the same gap as the non-episodic ones.
Revised: Experiments in the episodic setting (i.e. sentence-level language modelling) exhibited the same gap as the non-episodic ones.

Original: (cid:69) Hypothesis: the scaling up of inputs saturates the downstream LSTM gates.
Revised: (cid:69) Hypothesis: the scaling up of inputs saturates the downstream LSTM gates.

Original: The idea here is that saturated gates may make states more stable over time.
Revised: The idea here is that saturated gates may make states more stable over time.

Original: We observed the opposite: the means of the standard LSTM gates in the Mogriﬁer were very close between the two models but their variance was smaller in the Mogriﬁer.
Revised: We observed the opposite: the means of the standard LSTM gates in the Mogriﬁer were very close between the two models, but their variance was smaller in the Mogriﬁer.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: We presented the Mogriﬁer LSTM, an extension to the LSTM, with state-of-the-art results on several language modelling tasks.
Revised: We presented the Mogriﬁer LSTM, an extension to the LSTM, with state-of-the-art results on several language modelling tasks.

Original: Our original motivation for this work was that the context-free representation of input tokens may be a bottleneck in language models and by conditioning the input embedding on the recurrent state some beneﬁt was indeed derived.
Revised: Our original motivation for this work was that the context-free representation of input tokens may be a bottleneck in language models and by conditioning the input embedding on the recurrent state some beneﬁt was indeed derived.

Original: While it may be part of the explanation, this interpretation clearly does not account for the improvements brought by conditioning the recurrent state on the input and especially the applicability to character-level datasets.
Revised: While it may be part of the explanation, this interpretation clearly does not account for the improvements brought by conditioning the recurrent state on the input and especially the applicability to character-level datasets.

Original: Positioning our work on the Multiplicative RNN line of research offers a more compelling perspective.
Revised: Positioning our work on the Multiplicative RNN line of research offers a more compelling perspective.

Original: To give more credence to this interpretation, in the analysis we highlighted a number of possible alternative explanations and ruled them all out to varying degrees.
Revised: To give more credence to this interpretation, in the analysis we highlighted a number of possible alternative explanations, and ruled them all out to varying degrees.

Original: In particular, the connection to the mLSTM is weaker than expected as the Mogriﬁer does not exhibit improved recovery (see
Revised: In particular, the connection to the mLSTM is weaker than expected as the Mogriﬁer does not exhibit improved recovery (see

Original: Section 4.2), and on PTB the mLSTM works only as well as the LSTM.
Revised: Section 4.2), and on PTB the mLSTM works only as well as the LSTM.

Original: At the same time, the evidence against easier optimization is weak, and the Mogriﬁer establishing some kind of sharing between otherwise independent LSTM weight matrices is a distinct possibility.
Revised: At the same time, the evidence against easier optimization is weak, and the Mogriﬁer establishing some kind of sharing between otherwise independent LSTM weight matrices is a distinct possibility.

Original: Finally, note that as shown by Fig.
Revised: Finally, note that as shown by Fig.

Original: 1 and Eq.
Revised: 1 and Eq.

Original: 1-2, the Mogriﬁer is a series of preprocessing steps composed with the LSTM function, but other architectures, such as Mogriﬁer GRU or Mogriﬁer
Revised: 1-2, the Mogriﬁer is a series of preprocessing steps composed with the LSTM function, but other architectures, such as Mogriﬁer GRU or Mogriﬁer

Original: Elman Network are possible.
Revised: Elman Network are possible.

Original: We also leave investigations into other forms of parameterization of context-dependent transitions for future work.
Revised: We also leave investigations into other forms of parameterization of context-dependent transitions for future work.

Revised: A CKNOWLEDGMENTS

Revised: We would like to thank Ben Krause for the Transformer-XL dynamic evaluation results, Laura

Revised: Rimell, Aida Nematzadeh, Angeliki Lazaridou, Karl Moritz Hermann, Daniel Fried for helping with experiments, Chris Dyer, Sebastian Ruder and Jack Rae for their valuable feedback

Original: A PPENDIX A H YPERPARAMETER T UNING R ANGES
Revised: A PPENDIX A H YPERPARAMETER T UNING R ANGES

Original: In all experiments
Revised: In all experiments

Original: [Table]
Revised: [Table]

Original: we tuned hyperparameters using Google Vizier (Golovin et al. 2017).
Revised: we tuned hyperparameters using Google Vizier (Golovin et al. 2017).

Original: The tuning ranges are listed in Table 4.
Revised: The tuning ranges are listed in Table 4.

Original: Obviously, mogriﬁer_rounds and mogriﬁer_rank are tuned only for the
Revised: Obviously, mogriﬁer_rounds and mogriﬁer_rank are tuned only for the

Original: Mogriﬁer.
Revised: Mogriﬁer.

Original: If input _ embedding _ ratio (cid:62) 1 , then the input/output embedding sizes and the hidden sizes are set to equal and the linear projection from the cell output into the output embeddings space is omitted.
Revised: If input _ embedding _ ratio (cid:62) 1 , then the input/output embedding sizes and the hidden sizes are set to equal and the linear projection from the cell output into the output embeddings space is omitted.

Original: Similarly, mogrifier _ rank (cid:54) 0 is taken to mean full rank Q ∗ , R ∗ without factorization.
Revised: Similarly, mogrifier _ rank (cid:54) 0 is taken to mean full rank Q ∗ , R ∗ without factorization.

Original: Since Enwik8 is a much larger dataset
Revised: Since Enwik8 is a much larger dataset

Original: [Table]
Revised: [Table]

Original: we don’t tune input_embedding_ratio and specify tighter tuning ranges for dropout based on preliminary experiments (see Table 5)
Revised: we don’t tune input_embedding_ratio and specify tighter tuning ranges for dropout based on preliminary experiments (see Table 5)

Original: [Table]
Revised: [Table]

Original: Dynamic evaluation hyperparameters were tuned according to Table 6.
Revised: Dynamic evaluation hyperparameters were tuned according to Table 6.

Original: The highest possible value for max_time_steps , the BPTT window size, was 20 for word [Table] and 50 for character-level tasks.
Revised: The highest possible value for max_time_steps , the BPTT window size, was 20 for word, and 50 for character-level tasks.

Original: The batch size for estimating the mean squared gradients over the training data was set to 1024, gradient clipping was turned off
Revised: The batch size for estimating the mean squared gradients over the training data was set to 1024, gradient clipping was turned off

Original: [Table]
Revised: [Table]

Original: and the l2 penalty was set to zero.
Revised: and the l2 penalty was set to zero.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: A PPENDIX B H YPERPARAMETER S ENSITIVITY
Revised: A PPENDIX B H YPERPARAMETER S ENSITIVITY

Original: The parallel coordinate plots in Fig.
Revised: The parallel coordinate plots in Fig.

Original: 5 and 6, give a rough idea about hyperparameter sensitivity.
Revised: 5 and 6, give a rough idea about hyperparameter sensitivity.

Original: The red lines correspond to hyperparameter combinations closest to the best solution found.
Revised: The red lines correspond to hyperparameter combinations closest to the best solution found.

Original: To ﬁnd the closest combinations, we restricted the range for each hyperparameter separately to about 15% of its entire tuning range.
Revised: To ﬁnd the closest combinations, we restricted the range for each hyperparameter separately to about 15% of its entire tuning range.

Original: For both the LSTM and the Mogriﬁer, the results are at most 1.2 perplexity points off the best result, so our results are somewhat insensitive to jitter in the hyperparameters.
Revised: For both the LSTM and the Mogriﬁer, the results are at most 1.2 perplexity points off the best result, so our results are somewhat insensitive to jitter in the hyperparameters.

Original: Still, in this setup, grid search would require orders of magnitude more trials to ﬁnd comparable solutions.
Revised: Still, in this setup, grid search would require orders of magnitude more trials to ﬁnd comparable solutions.

Original: On the other hand, the tuner does take advantage of the stochasticity of training and repeated runs with the same parameters may be give slightly worse results.
Revised: On the other hand, the tuner does take advantage of the stochasticity of training, and repeated runs with the same parameters may be give slightly worse results.

Original: To estimate the extent of this effect, on
Revised: To gauge the extent of this effect, on

Original: PTB we estimated the standard deviation in reruns of the LSTM with the best hyperparameters to be about 0.2 perplexity points, but the mean was about 0.7 perplexity points off the result produced with the weights saved in best tuning run.
Revised: PTB we estimated the standard deviation in reruns of the LSTM with the best hyperparameters to be about 0.2 perplexity points, but the mean was about 0.7 perplexity points off the result produced with the weights saved in best tuning run.