Original: O N THE S TATE OF THE A RT OF E VALUATION IN N EURAL L ANGUAGE M ODELS
Revised: O N THE S TATE OF THE A RT OF E VALUATION IN N EURAL L ANGUAGE M ODELS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Ongoing innovations in recurrent neural network architectures have provided a steady inﬂux of apparently state- of the-art results on language modelling bench- marks
Revised: Ongoing innovations in recurrent neural network architectures have provided a steady inﬂux of apparently state-of-the-art results on language modelling benchmarks.

Original: However these have been evaluated using differing codebases and limited computational resources which represent uncontrolled sources of experimental variation
Revised: However, these have been evaluated using differing codebases and limited computational resources, which represent uncontrolled sources of experimental variation.

Original: We reevaluate several popular architectures and regularisation methods with large-scale automatic black box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised , outperform more recent models.
Revised: We reevaluate several popular architectures and regularisation methods with large-scale automatic black-box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models.

Original: We establish a new state of the art on the Penn Treebank and Wikitext- corpora, as well as strong baselines on the
Revised: We establish a new state of the art on the Penn Treebank and Wikitext-2 corpora, as well as strong baselines on the

Original: Hutter Prize dataset.
Revised: Hutter Prize dataset

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The scientiﬁc process by which the deep learning research community operates is guided by empirical studies that evaluate the relative quality of models.
Revised: The scientiﬁc process by which the deep learning research community operates is guided by empirical studies that evaluate the relative quality of models.

Original: Complicating matters, the measured performance of a model depends not only on its architecture (and data), but it can strongly depend on hyperparameter values that affect learning, regularisation, and capacity.
Revised: Complicating matters, the measured performance of a model depends not only on its architecture (and data), but it can strongly depend on hyperparameter values that affect learning, regularisation, and capacity.

Original: This hyperparameter dependence is an often inadequately controlled source of variation in experiments, which creates a risk that empirically unsound claims will be reported.
Revised: This hyperparameter dependence is an often inadequately controlled source of variation in experiments, which creates a risk that empirically unsound claims will be reported.

Original: In this paper, we use a black-box hyperparameter optimisation technique to control for hyperparameter effects while comparing the relative performance of language modelling architectures based on LSTMs, Recurrent Highway Networks (Zilly et al., 2016) and NAS (Zoph & Le, 2016).
Revised: In this paper, we use a black-box hyperparameter optimisation technique to control for hyperparameter effects while comparing the relative performance of language modelling architectures based on LSTMs, Recurrent Highway Networks (Zilly et al., 2016) and NAS (Zoph & Le, 2016).

Original: We specify ﬂexible, parameterised model families with the ability to adjust embedding and recurrent cell sizes for a given parameter budget and with ﬁne grain control over regularisation and learning hyperparameters.
Revised: We specify ﬂexible, parameterised model families with the ability to adjust embedding and recurrent cell sizes for a given parameter budget and with ﬁne grain control over regularisation and learning hyperparameters.

Original: Once hyperparameters have been properly controlled for, we ﬁnd that LSTMs outperform the more recent models, contra the published claims.
Revised: Once hyperparameters have been properly controlled for, we ﬁnd that LSTMs outperform the more recent models, contra the published claims.

Original: Our result is therefore a demonstration that replication failures can happen due to poorly controlled hyperparameter variation, and this paper joins other recent papers in warning of the under-acknowledged existence of replication failure in deep learning (Henderson et al., 2017; Reimers & Gurevych, 2017).
Revised: Our result is therefore a demonstration that replication failures can happen due to poorly controlled hyperparameter variation, and this paper joins other recent papers in warning of the under-acknowledged existence of replication failure in deep learning (Henderson et al., 2017; Reimers & Gurevych, 2017).

Original: However, we do show that careful controls are possible, albeit at considerable computational cost.
Revised: However, we do show that careful controls are possible, albeit at considerable computational cost.

Original: Several remarks can be made in light of these results.
Revised: Several remarks can be made in light of these results.

Original: First, as (conditional) language models serve as the central building block of many tasks, including machine translation, there is little reason to expect that the problem of unreliable evaluation is unique to the tasks discussed here.
Revised: First, as (conditional) language models serve as the central building block of many tasks, including machine translation, there is little reason to expect that the problem of unreliable evaluation is unique to the tasks discussed here.

Original: However, in machine translation, carefully controlling for hyperparameter effects would be substantially more expensive because standard datasets are much larger.
Revised: However, in machine translation, carefully controlling for hyperparameter effects would be substantially more expensive because standard datasets are much larger.

Original: Second, the research community should strive for more consensus about appropriate experimental methodology that balances costs of careful experimentation with the risks associated with false claims.
Revised: Second, the research community should strive for more consensus about appropriate experimental methodology that balances costs of careful experimentation with the risks associated with false claims.

Original: Finally, more attention should be paid to hyperparameter sensitivity.
Revised: Finally, more attention should be paid to hyperparameter sensitivity.

Original: Models that introduce many new hyperparameters or which perform well only in narrow ranges of hyperparameter settings should be identiﬁed as such as part of standard publication practice.
Revised: Models that introduce many new hyperparameters or which perform well only in narrow ranges of hyperparameter settings should be identiﬁed as such as part of standard publication practice.

Original: T h
Revised: T h

Original: e _ a c
Revised: e _ a c

Original: h e _ c a T t t (a) two-layer LSTM/NAS with skip connections
Revised: h e _ c a T t t (a [Figure]

Original: T h
Revised: T h

Original: e _ a c
Revised: e _ a c

Original: h e _ c a T t t (b) RHN with two processing steps per input
Revised: h e _ c a T t t (b) RHN with two processing steps per input

Original:  M ODELS
Revised:  M ODELS

Original: Our focus is on three recurrent architectures:
Revised: Our focus is on three recurrent architectures:

Original: • The Long Short-Term Memory (Hochreiter & Schmidhuber, 1997) serves as a well known and frequently used baseline.
Revised: • The Long Short-Term Memory (Hochreiter & Schmidhuber, 1997) serves as a well known and frequently used baseline.

Original: • The recently proposed Recurrent Highway Network (Zilly et al., 2016) is chosen because it has demonstrated state-of-the-art performance on a number of datasets.
Revised: • The recently proposed Recurrent Highway Network (Zilly et al., 2016) is chosen because it has demonstrated state-of-the-art performance on a number of datasets.

Original: • Finally, we also include NAS (Zoph & Le, 2016), because of its impressive performance and because its architecture was the result of an automated reinforcement learning based optimisation process.
Revised: • Finally, we also include NAS (Zoph & Le, 2016), because of its impressive performance and because its architecture was the result of an automated reinforcement learning based optimisation process.

Original: Our aim is strictly to do better model comparisons for these architectures and we thus refrain from including techniques that are known to push perplexities even lower, but which are believed to be largely orthogonal to the question of the relative merits of these recurrent cells.
Revised: Our aim is strictly to do better model comparisons for these architectures and we thus refrain from including techniques that are known to push perplexities even lower, but which are believed to be largely orthogonal to the question of the relative merits of these recurrent cells.

Original: In parallel work with a remarkable overlap with ours, Merity et al.
Revised: In parallel work with a remarkable overlap with ours, Merity et al.

Original: (2017) demonstrate the utility of adding a Neural Cache (Grave et al., 2016).
Revised: (2017) demonstrate the utility of adding a Neural Cache (Grave et al., 2016).

Original: Building on their work, Krause et al.
Revised: Building on their work, Krause et al.

Original: (2017) show that Dynamic Evaluation (Graves, 2013) contributes similarly to the ﬁnal perplexity.
Revised: (2017) show that Dynamic Evaluation (Graves, 2013) contributes similarly to the ﬁnal perplexity.

Original: As pictured in Fig.
Revised: As pictured in Fig.

Original: 1a, our models with LSTM or NAS cells have all the standard components: an input embedding lookup table, recurrent cells stacked as layers with additive skip connections combining outputs of all layers to ease optimisation.
Revised: 1a, our models with LSTM or NAS cells have all the standard components: an input embedding lookup table, recurrent cells stacked as layers with additive skip connections combining outputs of all layers to ease optimisation.

Original: There is an optional down-projection whose presence is governed by a hyperparameter from this combined output to a smaller space which reduces the number of output embedding parameters.
Revised: There is an optional down-projection whose presence is governed by a hyperparameter from this combined output to a smaller space which reduces the number of output embedding parameters.

Original: Unless otherwise noted, input and output embeddings are shared, see (Inan et al., 2016) and (Press & Wolf, 2016).
Revised: Unless otherwise noted, input and output embeddings are shared, see (Inan et al., 2016) and (Press & Wolf, 2016).

Original: Dropout is applied to feedforward connections denoted by dashed arrows in the ﬁgure.
Revised: Dropout is applied to feedforward connections denoted by dashed arrows in the ﬁgure.

Original: From the bottom up: to embedded inputs ( input dropout ), to connections between layers ( intra-layer dropout ), to the combined and the down-projected outputs ( output dropout ).
Revised: From the bottom up: to embedded inputs ( input dropout ), to connections between layers ( intra-layer dropout ), to the combined and the down-projected outputs ( output dropout ).

Original: All these dropouts have random masks drawn independently per time step, in contrast to the dropout on recurrent states where the same mask is used for all time steps in the sequence.
Revised: All these dropouts have random masks drawn independently per time step, in contrast to the dropout on recurrent states where the same mask is used for all time steps in the sequence.

Original: RHN based models are typically conceived of as a single horizontal “highway” to emphasise how the recurrent state is processed through time.
Revised: RHN based models are typically conceived of as a single horizontal “highway” to emphasise how the recurrent state is processed through time.

Original: In Fig.
Revised: In Fig.

Original: 1b, we choose to draw their schema in a way that makes the differences from LSTMs immediately apparent.
Revised: 1b, we choose to draw their schema in a way that makes the differences from LSTMs immediately apparent.

Original: In a nutshell, the RHN state is passed from the topmost layer to the lowest layer of the next time step.
Revised: In a nutshell, the RHN state is passed from the topmost layer to the lowest layer of the next time step.

Original: In contrast, each LSTM layer has its own recurrent connection and state.
Revised: In contrast, each LSTM layer has its own recurrent connection and state.

Original: The same dropout variants are applied to all three model types, with the exception of intra-layer dropout which does not apply to RHNs since only the recurrent state is passed between the layers.
Revised: The same dropout variants are applied to all three model types, with the exception of intra-layer dropout which does not apply to RHNs since only the recurrent state is passed between the layers.

Revised: unless explicitly noted otherwise.

Original:  E XPERIMENTAL S ETUP 3. D ATASETS
Revised:  E XPERIMENTAL S ETUP 3. D ATASETS

Original: We compare models on three datasets.
Revised: We compare models on three datasets.

Original: The smallest of them is the Penn Treebank corpus by Marcus et al.
Revised: The smallest of them is the Penn Treebank corpus by Marcus et al.

Original: with preprocessing from Mikolov et al.
Revised: with preprocessing from Mikolov et al.

Original: We also include another word level corpus: Wikitext-2 by Merity et al.
Revised: We also include another word level corpus: Wikitext-2 by Merity et al.

Original: It is about twice the size of Penn Treebank with a larger vocabulary and much lighter preprocessing.
Revised: It is about twice the size of Penn Treebank with a larger vocabulary and much lighter preprocessing.

Original: The third corpus is Enwik8 from the Hutter Prize dataset (Hutter, 2012).
Revised: The third corpus is Enwik8 from the Hutter Prize dataset (Hutter, 2012).

Original: Following common practice, we use the ﬁrst 90 million characters for training, and the remaining 10 million evenly split between validation and test.
Revised: Following common practice, we use the ﬁrst 90 million characters for training, and the remaining 10 million evenly split between validation and test.

Original:  T RAINING DETAILS
Revised:  T RAINING DETAILS

Original: When training word level models we follow common practice and use a batch size of 64, truncated backpropagation with 35 time steps, and we feed the ﬁnal states from the previous batch as the initial state of the subsequent one.
Revised: When training word level models we follow common practice and use a batch size of 64, truncated backpropagation with 35 time steps, and we feed the ﬁnal states from the previous batch as the initial state of the subsequent one.

Original: At the beginning of training and test time, the model starts with a zero state.
Revised: At the beginning of training and test time, the model starts with a zero state.

Original: To bias the model towards being able to easily start from such a state at test time, during training, with probability 0.01 a constant zero state is provided as the initial state.
Revised: To bias the model towards being able to easily start from such a state at test time, during training, with probability 0.01 a constant zero state is provided as the initial state.

Original: Optimisation is performed by Adam (Kingma & Ba, 2014) with β 1 = 0 but otherwise default parameters ( β 2 = 0 .
Revised: Optimisation is performed by Adam (Kingma & Ba, 2014) with β 1 = 0 but otherwise default parameters ( β 2 = 0 .

Original: 999 , (cid:15) =
Revised: 999 , (cid:15) =

Original: 10 − 9 ).
Revised: 10 − 9 ).

Original: Setting β 1 so turns off the exponential moving average for the estimates of the means of the gradients and brings Adam very close to RMSProp without momentum, but due to Adam’s bias correction, larger learning rates can be used.
Revised: Setting β 1 so turns off the exponential moving average for the estimates of the means of the gradients and brings Adam very close to RMSProp without momentum, but due to Adam’s bias correction, larger learning rates can be used.

Original: Batch size is set to 64.
Revised: Batch size is set to 64.

Original: The learning rate is multiplied by 0.1 whenever validation performance does not improve ever during 30 consecutive checkpoints.
Revised: The learning rate is multiplied by 0.1 whenever validation performance does not improve ever during 30 consecutive checkpoints.

Original: These checkpoints are performed after every 100 and 200 optimization steps for Penn Treebank and Wikitext-2, respectively.
Revised: These checkpoints are performed after every 100 and 200 optimization steps for Penn Treebank and Wikitext-2, respectively.

Original: For character level models (i.e. Enwik8), the differences 
Revised: For character level models (i.e. Enwik8), the differences are

Original: Checkpoints are only every 400 optimisation steps and embeddings are not shared.
Revised: Checkpoints are only every 400 optimisation steps and embeddings are not shared.

Original:  E VALUATION
Revised:  E VALUATION

Original: For evaluation, the checkpoint with the best validation perplexity found by the tuner is loaded and the model is applied to the test set with a batch size of 1.
Revised: For evaluation, the checkpoint with the best validation perplexity found by the tuner is loaded and the model is applied to the test set with a batch size of 1.

Original: For the word based datasets, using the training batch size makes results worse by 0.3 PPL while Enwik8 is practically unaffected due to its evaluation and training sets being much larger.
Revised: For the word based datasets, using the training batch size makes results worse by 0.3 PPL while Enwik8 is practically unaffected due to its evaluation and training sets being much larger.

Original: Preliminary experiments indicate that MC averaging would bring a small improvement of about 0.4 in perplexity and 0.005 in bits per character, similar to the results of Gal & Ghahramani (2016), while being a 1000 times more expensive which is prohibitive on larger datasets.
Revised: Preliminary experiments indicate that MC averaging would bring a small improvement of about 0.4 in perplexity and 0.005 in bits per character, similar to the results of Gal & Ghahramani (2016), while being a 1000 times more expensive which is prohibitive on larger datasets.

Original: Therefore, throughout we use the mean-ﬁeld approximation for dropout at test time.
Revised: Therefore, throughout we use the mean-ﬁeld approximation for dropout at test time.

Original: 5. H YPERPARAMETER T UNING
Revised: 5. H YPERPARAMETER T UNING

Original: Hyperparameters are optimised by Google Vizier (Golovin et al., 2017), a black-box hyperparameter tuner based on batched GP bandits using the expected improvement acquisition function (Desautels et al., 2014).
Revised: Hyperparameters are optimised by Google Vizier (Golovin et al., 2017), a black-box hyperparameter tuner based on batched GP bandits using the expected improvement acquisition function (Desautels et al., 2014).

Original: Tuners of this nature are generally more efﬁcient than grid search when the number of hyperparameters is small.
Revised: Tuners of this nature are generally more efﬁcient than grid search when the number of hyperparameters is small.

Original: To keep the problem tractable, we restrict the set of hyperparameters to learning rate , input embedding ratio , input dropout , state dropout , output dropout , weight decay .
Revised: To keep the problem tractable, we restrict the set of hyperparameters to learning rate , input embedding ratio , input dropout , state dropout , output dropout , weight decay .

Original: For deep LSTMs, there is an extra hyperparameter to tune: intra-layer dropout .
Revised: For deep LSTMs, there is an extra hyperparameter to tune: intra-layer dropout .

Original: Even with this small set, thousands of evaluations are required to reach convergence.
Revised: Even with this small set, thousands of evaluations are required to reach convergence.

Original: [Table]
Revised: [Table]

Original: Parameter budget.
Revised: Parameter budget.

Original: Motivated by recent results from Collins et al.
Revised: Motivated by recent results from Collins et al.

Original: , we compare models on the basis of the total number of trainable parameters as opposed to the number of hidden units.
Revised: , we compare models on the basis of the total number of trainable parameters as opposed to the number of hidden units.

Original: The tuner is given control over the presence and size of the down-projection, and thus over the tradeoff between the number of embedding vs. recurrent cell parameters.
Revised: The tuner is given control over the presence and size of the down-projection, and thus over the tradeoff between the number of embedding vs. recurrent cell parameters.

Original: Consequently, the cells’ hidden size and the embedding size is determined by the actual parameter budget, depth and the input embedding ratio hyperparameter.
Revised: Consequently, the cells’ hidden size and the embedding size is determined by the actual parameter budget, depth and the input embedding ratio hyperparameter.

Original: For Enwik8 there are relatively few parameters in the embeddings since the vocabulary size is only 205.
Revised: For Enwik8 there are relatively few parameters in the embeddings since the vocabulary size is only 205.

Original: Here we choose not to share embeddings and to omit the down-projection unconditionally.
Revised: Here we choose not to share embeddings and to omit the down-projection unconditionally.

Original:  R ESULTS 6. P ENN T REEBANK
Revised:  R ESULTS 6. P ENN T REEBANK

Original: We tested LSTMs of various depths and an RHN of depth 5 with parameter budgets of 10 andmillion matching the sizes of the Medium and Large LSTMs by (Zaremba et al., 2014).
Revised: We tested LSTMs of various depths and an RHN of depth 5 with parameter budgets of 10 andmillion matching the sizes of the Medium and Large LSTMs by (Zaremba et al., 2014).

Original: The results are summarised in Table 1.
Revised: The results are summarised in Table 1.

Original: Notably, in our experiments even the RHN with only 10M parameters has better perplexity than the 24M one in the original publication.
Revised: Notably, in our experiments even the RHN with only 10M parameters has better perplexity than the 24M one in the original publication.

Original: Our 24M version improves on that further.
Revised: Our 24M version improves on that further.

Original: However, a shallow
Revised: However, a shallow

Original: LSTM-based model with only 10M parameters enjoys a very comfortable margin over that, with deeper models following near the estimated noise range.
Revised: LSTM-based model with only 10M parameters enjoys a very comfortable margin over that, with deeper models following near the estimated noise range.

Original: At 24M, all depths obtain very similar results, reaching 58 .
Revised: At 24M, all depths obtain very similar results, reaching 58 .

Original: 3 at depth 4.
Revised: 3 at depth 4.

Original: Unsurprisingly, NAS whose architecture was chosen based on its performance on this dataset does almost equally well, even better than in Zoph & Le (2016).
Revised: Unsurprisingly, NAS whose architecture was chosen based on its performance on this dataset does almost equally well, even better than in Zoph & Le (2016).

Original: 6. W IKITEXT - 
Revised: 6. W IKITEXT -

Original: Wikitext-2 is not much larger than Penn Treebank, so it is not surprising that even models tuned for
Revised: Wikitext-2 is not much larger than Penn Treebank, so it is not surprising that even models tuned for

Original: Penn Treebank perform reasonably on this dataset, and this is in fact how results in previous works were produced.
Revised: Penn Treebank perform reasonably on this dataset, and this is in fact how results in previous works were produced.

Original: For a fairer comparison, we also tune hyperparameters on the same dataset.
Revised: For a fairer comparison, we also tune hyperparameters on the same dataset.

Original: In Table 2, we report numbers for both approaches.
Revised: In Table 2, we report numbers for both approaches.

Original: All our results are well below the previous state of the are for models without dynamic evaluation or caching.
Revised: All our results are well below the previous state of the are for models without dynamic evaluation or caching.

Original: That said, our best result, 65 . 9 compares
Revised: That said, our best result, 65 . 9 compares

Original: [Table]
Revised: [Table]

Original: favourably even to the Neural Cache (Grave et al., 2016) whose innovations are fairly orthogonal to the base model.
Revised: favourably even to the Neural Cache (Grave et al., 2016) whose innovations are fairly orthogonal to the base model.

Original: Shallow LSTMs do especially well here.
Revised: Shallow LSTMs do especially well here.

Original: Deeper models have gradually degrading perplexity, with
Revised: Deeper models have gradually degrading perplexity, with

Original: RHNs lagging all of them by a signiﬁcant margin.
Revised: RHNs lagging all of them by a signiﬁcant margin.

Original: NAS is not quite up there with the LSTM suggesting its architecture might have overﬁtted to Penn Treebank
Revised: NAS is not quite up there with the LSTM suggesting its architecture might have overﬁtted to Penn Treebank

Original: [Table]
Revised: [Table]

Original: but data for deeper variants would be necessary to draw this conclusion.
Revised: but data for deeper variants would be necessary to draw this conclusion.

Original: 6. E NWIK
Revised: 6. E NWIK

Original: In contrast to the previous datasets, our numbers on this task (reported in BPC, following convetion) are slightly off the state of the art.
Revised: In contrast to the previous datasets, our numbers on this task (reported in BPC, following convetion) are slightly off the state of the art.

Original: This is most likely due to optimisation being limited to 14 epochs which is about a tenth of what the model of Zilly et al.
Revised: This is most likely due to optimisation being limited to 14 epochs which is about a tenth of what the model of Zilly et al.

Original: (2016) was trained for.
Revised: (2016) was trained for.

Original: Nevertheless, we match their smaller RHN with our models which are very close to each other.
Revised: Nevertheless, we match their smaller RHN with our models which are very close to each other.

Original: NAS lags the other models by a surprising margin at this task.
Revised: NAS lags the other models by a surprising margin at this task.

Original:  A NALYSIS
Revised:  A NALYSIS

Original: On two of the three datasets, we improved previous results substantially by careful model speciﬁcation and hyperparameter optimisation, but the improvement for RHNs is much smaller compared to that for LSTMs.
Revised: On two of the three datasets, we improved previous results substantially by careful model speciﬁcation and hyperparameter optimisation, but the improvement for RHNs is much smaller compared to that for LSTMs.

Original: While it cannot be ruled out that our particular setup somehow favours LSTMs, we believe it is more likely that this effect arises due to the original RHN experimental condition having been tuned more extensively (this is nearly unavoidable during model development).
Revised: While it cannot be ruled out that our particular setup somehow favours LSTMs, we believe it is more likely that this effect arises due to the original RHN experimental condition having been tuned more extensively (this is nearly unavoidable during model development).

Original: Naturally, NAS beneﬁtted only to a limited degree from our tuning, since the numbers of Zoph & Le (2016) were already produced by employing similar regularisation methods and a grid search.
Revised: Naturally, NAS beneﬁtted only to a limited degree from our tuning, since the numbers of Zoph & Le (2016) were already produced by employing similar regularisation methods and a grid search.

Original: The small edge can be attributed to the suboptimality of grid search (see Section 7.3).
Revised: The small edge can be attributed to the suboptimality of grid search (see Section 7.3).

Original: In summary, the three recurrent cell architectures are closely matched on all three datasets, with minuscule differences on Enwik8 where regularisation matters the least.
Revised: In summary, the three recurrent cell architectures are closely matched on all three datasets, with minuscule differences on Enwik8 where regularisation matters the least.

Original: These results support the claims of Collins et al.
Revised: These results support the claims of Collins et al.

Original: that capacities of various cells are very similar and their apparent differences result from trainability and regularisation.
Revised: that capacities of various cells are very similar and their apparent differences result from trainability and regularisation.

Original: While comparing three similar architectures cannot prove this point, the inclusion of NAS certainly gives it more credence.
Revised: While comparing three similar architectures cannot prove this point, the inclusion of NAS certainly gives it more credence.

Original: This way we have two of the best human designed and one machine optimised cell that was the top performer among thousands of candidates.
Revised: This way we have two of the best human designed and one machine optimised cell that was the top performer among thousands of candidates.

Original: [Table]
Revised: [Table]

Original: 7. T HE E FFECT OF I NDIVIDUAL F EATURES
Revised: 7. T HE E FFECT OF I NDIVIDUAL F EATURES

Original: Down-projection was found to be very beneﬁcial by the tuner for some depth/budget combinations.
Revised: Down-projection was found to be very beneﬁcial by the tuner for some depth/budget combinations.

Original: On Penn Treebank, it improved results by about 2–5 perplexity points at depths 1 and 2 at 10M
Revised: On Penn Treebank, it improved results by about 2–5 perplexity points at depths 1 and 2 at 10M

Original: [Table]
Revised: [Table]

Original: and depth 1 at 24M, possibly by equipping the recurrent cells with more capacity.
Revised: and depth 1 at 24M, possibly by equipping the recurrent cells with more capacity.

Original: The very same models beneﬁted from down-projection on Wikitext-2, but even more so with gaps of about 10–18 points which is readily explained by the larger vocabulary size.
Revised: The very same models beneﬁted from down-projection on Wikitext-2, but even more so with gaps of about 10–18 points which is readily explained by the larger vocabulary size.

Original: We further measured the contribution of other features of the models in a series of experiments.
Revised: We further measured the contribution of other features of the models in a series of experiments.

Original: See
Revised: See

Original: Table 4.
Revised: Table 4.

Original: To limit the number of resource used, in these experiments only individual features were evaluated (not their combinations) on Penn Treebank at the best depth for each architecture (LSTM or RHN) and parameter budget (10M or 24M) as determined above.
Revised: To limit the number of resource used, in these experiments only individual features were evaluated (not their combinations) on Penn Treebank at the best depth for each architecture (LSTM or RHN) and parameter budget (10M or 24M) as determined above.

Original: First, we untied input and output embeddings which made perplexities worse by about 6 points across the board which is consistent with the results of Inan et al.
Revised: First, we untied input and output embeddings which made perplexities worse by about 6 points across the board which is consistent with the results of Inan et al.

Original: Second, without variational dropout the RHN models suffer quite a bit since there remains no dropout at all in between the layers.
Revised: Second, without variational dropout the RHN models suffer quite a bit since there remains no dropout at all in between the layers.

Original: The deep LSTM also sees a similar loss of perplexity as having intra-layer dropout does not in itself provide enough regularisation.
Revised: The deep LSTM also sees a similar loss of perplexity as having intra-layer dropout does not in itself provide enough regularisation.

Original: Third, we were also interested in how recurrent dropout (Semeniuta et al., 2016) would perform in lieu of variational dropout.
Revised: Third, we were also interested in how recurrent dropout (Semeniuta et al., 2016) would perform in lieu of variational dropout.

Original: Dropout masks were shared between time steps in both methods, and our results indicate no consistent advantage to either of them.
Revised: Dropout masks were shared between time steps in both methods, and our results indicate no consistent advantage to either of them.

Original: 7. M ODEL S ELECTION
Revised: 7. M ODEL S ELECTION

Original: With a large number of hyperparameter combinations evaluated, the question of how much the tuner overﬁts arises.
Revised: With a large number of hyperparameter combinations evaluated, the question of how much the tuner overﬁts arises.

Original: There are multiple sources of noise in play,
Revised: There are multiple sources of noise in play,

Original: (a) non-deterministic ordering of ﬂoating-point operations in optimised linear algebra routines, (b) different initialisation seeds, (c) the validation and test sets being ﬁnite samples from a inﬁnite population.
Revised: (a) non-deterministic ordering of ﬂoating-point operations in optimised linear algebra routines, (b) different initialisation seeds, (c) the validation and test sets being ﬁnite samples from a inﬁnite population.

Original: To assess the severity of these issues, we conducted the following experiment: models with the best hyperparameter settings for Penn Treebank and Wikitext-2 were retrained from scratch with various initialisation seeds and the validation and test scores were recorded.
Revised: To assess the severity of these issues, we conducted the following experiment: models with the best hyperparameter settings for Penn Treebank and Wikitext-2 were retrained from scratch with various initialisation seeds and the validation and test scores were recorded.

Original: If during tuning, a model just got a lucky run due to a combination of (a) and (b), then retraining with the same hyperparameters but with different seeds would fail to reproduce the same good results.
Revised: If during tuning, a model just got a lucky run due to a combination of (a) and (b), then retraining with the same hyperparameters but with different seeds would fail to reproduce the same good results.

Original: There are a few notable things about the results.
Revised: There are a few notable things about the results.

Original: First, in our environment (Tensorﬂow with a single
Revised: First, in our environment (Tensorﬂow with a single

Original: GPU) even with the same seed as the one used by the tuner, the effect of (a) is almost as large as that of (a) and (b) combined.
Revised: GPU) even with the same seed as the one used by the tuner, the effect of (a) is almost as large as that of (a) and (b) combined.

Original: Second, the variance induced by (a) and (b) together is roughly equivalent to an absolute difference of 0.4 in perplexity on Penn Treebank and 0.5 on Wikitext-2.
Revised: Second, the variance induced by (a) and (b) together is roughly equivalent to an absolute difference of 0.4 in perplexity on Penn Treebank and 0.5 on Wikitext-2.

Original: [Table]
Revised: [Table]

Original: Third, the validation perplexities of the best checkpoints are about one standard deviation lower than the sample mean of the reruns
Revised: Third, the validation perplexities of the best checkpoints are about one standard deviation lower than the sample mean of the reruns

Original: [Table]
Revised: [Table]

Original: so the tuner could ﬁt the noise only to a limited degree.
Revised: so the tuner could ﬁt the noise only to a limited degree.

Original: Because we treat our corpora as a single sequence, test set contents are not i.i
Revised: Because we treat our corpora as a single sequence, test set contents are not i [Table] i

Original: [Table]
Revised: [Table]

Original: d., and we cannot apply techniques such as the bootstrap to assess (c).
Revised: d., and we cannot apply techniques such as the bootstrap to assess (c).

Original: Instead, we looked at the gap between validation and test scores as a proxy and observed that it is very stable, contributing variance of 0.12–0.3 perplexity to the ﬁnal results on Penn Treebank and Wikitext-2, respectively.
Revised: Instead, we looked at the gap between validation and test scores as a proxy and observed that it is very stable, contributing variance of 0.12–0.3 perplexity to the ﬁnal results on Penn Treebank and Wikitext-2, respectively.

Original: We have not explicitly dealt with the unknown uncertainty remaining in the Gaussian Process that may affect model comparisons, apart from running it until apparent convergence.
Revised: We have not explicitly dealt with the unknown uncertainty remaining in the Gaussian Process that may affect model comparisons, apart from running it until apparent convergence.

Original: All in all, our ﬁndings suggest that a gap in perplexity of 1.0 is a statistically robust difference between models trained in this way on these datasets.
Revised: All in all, our ﬁndings suggest that a gap in perplexity of 1.0 is a statistically robust difference between models trained in this way on these datasets.

Original: The distribution of results was approximately normal with roughly the same variance for all models, so we still report numbers in a tabular form instead of plotting the distribution of results, for example in a violin plot (Hintze & Nelson, 1998).
Revised: The distribution of results was approximately normal with roughly the same variance for all models, so we still report numbers in a tabular form instead of plotting the distribution of results, for example in a violin plot (Hintze & Nelson, 1998).

Original: 7. S ENSITIVITY
Revised: 7. S ENSITIVITY

Original: To further verify that the best hyperparameter setting found by the tuner is not a ﬂuke, we plotted the validation loss against the hyperparameter settings.
Revised: To further verify that the best hyperparameter setting found by the tuner is not a ﬂuke, we plotted the validation loss against the hyperparameter settings.

Original: Fig.
Revised: Fig.

Original: 2 shows one such typical plot, for a 4-layer
Revised: 2 shows one such typical plot, for a 4-layer

Original: LSTM.
Revised: LSTM.

Original: We manually restricted the ranges around the best hyperparameter values to around 15–25% of the entire tuneable range, and observed that the vast majority of settings in that neighbourhood produced perplexities within 3.0 of the best value.
Revised: We manually restricted the ranges around the best hyperparameter values to around 15–25% of the entire tuneable range, and observed that the vast majority of settings in that neighbourhood produced perplexities within 3.0 of the best value.

Original: Widening the ranges further leads to quickly deteriorating results.
Revised: Widening the ranges further leads to quickly deteriorating results.

Original: Satisﬁed that the hyperparameter surface is well behaved, we considered whether the same results could have possibly been achieved with a simple grid search.
Revised: Satisﬁed that the hyperparameter surface is well behaved, we considered whether the same results could have possibly been achieved with a simple grid search.

Original: Omitting input embedding ratio because the tuner found having a down-projection suboptimal almost non-conditionally for this 4-layer
Revised: Omitting input embedding ratio because the tuner found having a down-projection suboptimal almost non-conditionally for this 4-layer

Original: LSTM, there remain six hyperparameters to tune.
Revised: LSTM, there remain six hyperparameters to tune.

Original: If there were 5 possible values on the grid for each hyperparameter (with one value in every 20% interval), then we would need 6 5 , nearly 8000 trials to get within 3.0 of the best perplexity achieved by the tuner in about 1500 trials.
Revised: If there were 5 possible values on the grid for each hyperparameter (with one value in every 20% interval), then we would need 6 5 , nearly 8000 trials to get within 3.0 of the best perplexity achieved by the tuner in about 1500 trials.

Original: 7. T YING LSTM GATES
Revised: 7. T YING LSTM GATES

Original: Normally, LSTMs have two independent gates controlling the retention of cell state and the admission of updates (Eq. 1).
Revised: Normally, LSTMs have two independent gates controlling the retention of cell state and the admission of updates (Eq. 1).

Original: A minor variant which reduces the number of parameters at the loss of some ﬂexibility is to tie the input and forget gates as in Eq.
Revised: A minor variant which reduces the number of parameters at the loss of some ﬂexibility is to tie the input and forget gates as in Eq.

Original: A possible middle ground that keeps the number of parameters the same but ensures that values of the cell state c remain in [ − 1 , 1] is to cap the input gate as in Eq.
Revised: A possible middle ground that keeps the number of parameters the same but ensures that values of the cell state c remain in [ − 1 , 1] is to cap the input gate as in Eq.

Original: [Equation]
Revised: [Equation]

Original: Where the equations are based on the formulation of Sak et al.
Revised: Where the equations are based on the formulation of Sak et al.

Original: All LSTM models in this paper use the third variant, except those titled “Untied gates” and “Tied gates” in Table 4 corresponding to Eq.
Revised: All LSTM models in this paper use the third variant, except those titled “Untied gates” and “Tied gates” in Table 4 corresponding to Eq.

Original: 1 and 2, respectively.
Revised: 1 and 2, respectively.

Original: The results show that LSTMs are insensitive to these changes and the results vary only slightly even though more hidden units are allocated to the tied version to ﬁll its parameter budget.
Revised: The results show that LSTMs are insensitive to these changes and the results vary only slightly even though more hidden units are allocated to the tied version to ﬁll its parameter budget.

Original: Finally, the numbers suggest that deep LSTMs beneﬁt from bounded cell states.
Revised: Finally, the numbers suggest that deep LSTMs beneﬁt from bounded cell states.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: During the transitional period when deep neural language models began to supplant their shallower predecessors, effect sizes tended to be large, and robust conclusions about the value of the modelling innovations could be made, even in the presence of poorly controlled “hyperparameter noise.”
Revised: During the transitional period when deep neural language models began to supplant their shallower predecessors, effect sizes tended to be large, and robust conclusions about the value of the modelling innovations could be made, even in the presence of poorly controlled “hyperparameter noise.”

Original: However, now that the neural revolution is in full swing, researchers must often compare competing deep architectures.
Revised: However, now that the neural revolution is in full swing, researchers must often compare competing deep architectures.

Original: In this regime, effect sizes tend to be much smaller, and more methodological care is required to produce reliable results.
Revised: In this regime, effect sizes tend to be much smaller, and more methodological care is required to produce reliable results.

Original: Furthermore, with so much work carried out in parallel by a growing research community, the costs of faulty conclusions are increased.
Revised: Furthermore, with so much work carried out in parallel by a growing research community, the costs of faulty conclusions are increased.

Original: Although we can draw attention to this problem, this paper does not offer a practical methodological solution beyond establishing reliable baselines that can be the benchmarks for subsequent work.
Revised: Although we can draw attention to this problem, this paper does not offer a practical methodological solution beyond establishing reliable baselines that can be the benchmarks for subsequent work.

Original: Still, we demonstrate how, with a huge amount of computation, noise levels of various origins can be carefully estimated and models meaningfully compared.
Revised: Still, we demonstrate how, with a huge amount of computation, noise levels of various origins can be carefully estimated and models meaningfully compared.

Original: This apparent tradeoff between the amount of computation and the reliability of results seems to lie at the heart of the matter.
Revised: This apparent tradeoff between the amount of computation and the reliability of results seems to lie at the heart of the matter.

Original: Solutions to the methodological challenges must therefore make model evaluation cheaper by, for instance, reducing the number of hyperparameters and the sensitivity of models to them, employing better hyperparameter optimisation strategies, or by deﬁning “leagues” with predeﬁned computational budgets for a single model representing different points on the tradeoff curve
Revised: Solutions to the methodological challenges must therefore make model evaluation cheaper by, for instance, reducing the number of hyperparameters and the sensitivity of models to them, employing better hyperparameter optimisation strategies, or by deﬁning “leagues” with predeﬁned computational budgets for a single model representing different points on the tradeoff curve