Original: R EGULARIZING AND O PTIMIZING LSTM L ANGUAGE M ODELS
Revised: R EGULARIZING AND O PTIMIZING LSTM L ANGUAGE M ODELS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we consider the speciﬁc problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models.
Revised: In this paper, we consider the speciﬁc problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models.

Original: We propose the weight-dropped LSTM, which uses DropConnect on hidden-tohidden weights, as a form of recurrent regularization.
Revised: We propose the weight-dropped LSTM, which uses DropConnect on hidden-tohidden weights, as a form of recurrent regularization.

Original: Further, we introduce NTAvSGD, a non-monotonically triggered (NT) variant of the averaged stochastic gradient method (AvSGD), wherein the averaging trigger is determined using a
Revised: Further, we introduce NTAvSGD, a non-monotonically triggered (NT) variant of the averaged stochastic gradient method (AvSGD), wherein the averaging trigger is determined using a

Original: NT condition as opposed to being tuned by the user.
Revised: NT condition as opposed to being tuned by the user.

Original: Using these and other regularization strategies, our AvSGD Weight-Dropped LSTM (AWD-LSTM) achieves state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2.
Revised: Using these and other regularization strategies, our AvSGD Weight-Dropped LSTM (AWD-LSTM) achieves state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2.

Original: In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.
Revised: In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.

Original: We also explore the viability of the proposed regularization and optimization strategies in the context of the quasi-recurrent neural network (QRNN) and demonstrate comparable performance to the AWD-LSTM counterpart.
Revised: We also explore the viability of the proposed regularization and optimization strategies in the context of the quasi-recurrent neural network (QRNN) and demonstrate comparable performance to the AWD-LSTM counterpart.

Original: The code for reproducing the results is open sourced and is available at MASKED .
Revised: The code for reproducing the results is open sourced and is available at https://github.com/salesforce/ awd-lstm-lm .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Effective regularization techniques for deep learning have been the subject of much research in recent years.
Revised: Effective regularization techniques for deep learning have been the subject of much research in recent years.

Original: Given the over-parameterization of neural networks, generalization performance crucially relies on the ability to regularize the models sufﬁciently.
Revised: Given the over-parameterization of neural networks, generalization performance crucially relies on the ability to regularize the models sufﬁciently.

Original: Strategies such as dropout (Srivastava et al., 2014) and batch normalization (Ioffe & Szegedy, 2015) have found great success and are now ubiquitous in feed-forward and convolutional neural networks.
Revised: Strategies such as dropout (Srivastava et al., 2014) and batch normalization (Ioffe & Szegedy, 2015) have found great success and are now ubiquitous in feed-forward and convolutional neural networks.

Original: Na ¨ ıvely applying these approaches to the case of recurrent neural networks (RNNs) has not been highly successful however.
Revised: Na ¨ ıvely applying these approaches to the case of recurrent neural networks (RNNs) has not been highly successful however.

Original: Many recent works have hence been focused on the extension of these regularization strategies to RNNs; we brieﬂy discuss some of them below.
Revised: Many recent works have hence been focused on the extension of these regularization strategies to RNNs; we brieﬂy discuss some of them below.

Original: A na¨ıve application of dropout (Srivastava et al., 2014) to an RNN’s hidden state is ineffective as it disrupts the RNN’s ability to retain long term dependencies (Zaremba et al., 2014).
Revised: A na¨ıve application of dropout (Srivastava et al., 2014) to an RNN’s hidden state is ineffective as it disrupts the RNN’s ability to retain long term dependencies (Zaremba et al., 2014).

Original: Gal &
Revised: Gal &

Original: Ghahramani (2016) propose overcoming this problem by retaining the same dropout mask across multiple time steps as opposed to sampling a new binary mask at each timestep.
Revised: Ghahramani (2016) propose overcoming this problem by retaining the same dropout mask across multiple time steps as opposed to sampling a new binary mask at each timestep.

Original: Another approach is to regularize the network through limiting updates to the RNN’s hidden state.
Revised: Another approach is to regularize the network through limiting updates to the RNN’s hidden state.

Original: One such approach is taken by Semeniuta et al.
Revised: One such approach is taken by Semeniuta et al.

Original: (2016) wherein the authors drop updates to network units, speciﬁcally the input gates of the LSTM, in lieu of the units themselves.
Revised: (2016) wherein the authors drop updates to network units, speciﬁcally the input gates of the LSTM, in lieu of the units themselves.

Original: This is reminiscent of zoneout (Krueger et al., 2016) where updates to the hidden state may fail to occur for randomly selected neurons.
Revised: This is reminiscent of zoneout (Krueger et al., 2016) where updates to the hidden state may fail to occur for randomly selected neurons.

Original: Instead of operating on the RNN’s hidden states, one can regularize the network through restrictions on the recurrent matrices as well.
Revised: Instead of operating on the RNN’s hidden states, one can regularize the network through restrictions on the recurrent matrices as well.

Original: This can be done either through restricting the capacity of the matrix (Arjovsky et al., 2016;
Revised: This can be done either through restricting the capacity of the matrix (Arjovsky et al., 2016;

Original: Wisdom et al., 2016; Jing et al., 2016) or through element-wise interactions (Balduzzi & Ghifary, 2016; Bradbury et al., 2016; Seo et al., 2016).
Revised: Wisdom et al., 2016; Jing et al., 2016) or through element-wise interactions (Balduzzi & Ghifary, 2016; Bradbury et al., 2016; Seo et al., 2016).

Original: Other forms of regularization explicitly act upon activations such as batch normalization (Ioffe &
Revised: Other forms of regularization explicitly act upon activations such as batch normalization (Ioffe &

Original: Szegedy, 2015), recurrent batch normalization (Cooijmans et al., 2016), and layer normalization (Ba et al., 2016).
Revised: Szegedy, 2015), recurrent batch normalization (Cooijmans et al., 2016), and layer normalization (Ba et al., 2016).

Original: These all introduce additional training parameters and can complicate the training process while increasing the sensitivity of the model.
Revised: These all introduce additional training parameters and can complicate the training process while increasing the sensitivity of the model.

Original: In this work, we investigate a set of regularization strategies that are not only highly effective but which can also be used with no modiﬁcation to existing LSTM implementations.
Revised: In this work, we investigate a set of regularization strategies that are not only highly effective but which can also be used with no modiﬁcation to existing LSTM implementations.

Original: The weightdropped LSTM applies recurrent regularization through a DropConnect mask on the hidden-tohidden recurrent weights.
Revised: The weightdropped LSTM applies recurrent regularization through a DropConnect mask on the hidden-tohidden recurrent weights.

Original: Other strategies include the use of randomized-length backpropagation through time (BPTT), embedding dropout, activation regularization (AR), and temporal activation regularization (TAR).
Revised: Other strategies include the use of randomized-length backpropagation through time (BPTT), embedding dropout, activation regularization (AR), and temporal activation regularization (TAR).

Original: As no modiﬁcations are required of the LSTM implementation these regularization strategies are compatible with black box libraries, such as NVIDIA cuDNN, which can be many times faster than na¨ıve LSTM implementations.
Revised: As no modiﬁcations are required of the LSTM implementation these regularization strategies are compatible with black box libraries, such as NVIDIA cuDNN, which can be many times faster than na¨ıve LSTM implementations.

Original: Effective methods for training deep recurrent networks have also been a topic of renewed interest.
Revised: Effective methods for training deep recurrent networks have also been a topic of renewed interest.

Original: Once a model has been deﬁned, the training algorithm used is required to not only ﬁnd a good minimizer of the loss function but also converge to such a minimizer rapidly.
Revised: Once a model has been deﬁned, the training algorithm used is required to not only ﬁnd a good minimizer of the loss function but also converge to such a minimizer rapidly.

Original: The choice of the optimizer is even more important in the context of regularized models since such strategies, especially the use of dropout, can impede the training process.
Revised: The choice of the optimizer is even more important in the context of regularized models since such strategies, especially the use of dropout, can impede the training process.

Original: Stochastic gradient descent (SGD), and its variants such as Adam (Kingma & Ba, 2014) and RMSprop (Tieleman & Hinton, 2012) are amongst the most popular training methods.
Revised: Stochastic gradient descent (SGD), and its variants such as Adam (Kingma & Ba, 2014) and RMSprop (Tieleman & Hinton, 2012) are amongst the most popular training methods.

Original: These methods iteratively reduce the training loss through scaled (stochastic) gradient steps.
Revised: These methods iteratively reduce the training loss through scaled (stochastic) gradient steps.

Original: In particular, Adam has been found to be widely applicable despite requiring less tuning of its hyperparameters.
Revised: In particular, Adam has been found to be widely applicable despite requiring less tuning of its hyperparameters.

Original: In the context of word-level language modeling, past work has empirically found that SGD outperforms other methods in not only the ﬁnal loss but also in the rate of convergence.
Revised: In the context of word-level language modeling, past work has empirically found that SGD outperforms other methods in not only the ﬁnal loss but also in the rate of convergence.

Original: This is in agreement with recent evidence pointing to the insufﬁciency of adaptive gradient methods (Wilson et al., 2017).
Revised: This is in agreement with recent evidence pointing to the insufﬁciency of adaptive gradient methods (Wilson et al., 2017).

Original: Given the success of SGD, especially within the language modeling domain, we investigate the use of averaged SGD (AvSGD) (Polyak & Juditsky, 1992) which is known to have superior theoretical guarantees.
Revised: Given the success of SGD, especially within the language modeling domain, we investigate the use of averaged SGD (AvSGD) (Polyak & Juditsky, 1992) which is known to have superior theoretical guarantees.

Original: AvSGD carries out iterations similar to SGD, but instead of returning the last iterate as the solution, returns an average of the iterates past a certain, tuned, threshold T .
Revised: AvSGD carries out iterations similar to SGD, but instead of returning the last iterate as the solution, returns an average of the iterates past a certain, tuned, threshold T .

Original: This threshold T is typically tuned and has a direct impact on the performance of the method.
Revised: This threshold T is typically tuned and has a direct impact on the performance of the method.

Original: We propose a variant of AvSGD where T is determined on the ﬂy through a non-monotonic criterion and show that it achieves better training outcomes compared to SGD.
Revised: We propose a variant of AvSGD where T is determined on the ﬂy through a non-monotonic criterion and show that it achieves better training outcomes compared to SGD.

Original:  W EIGHT - DROPPED LSTM
Revised:  W EIGHT - DROPPED LSTM

Original: We refer to the mathematical formulation of the LSTM,
Revised: We refer to the mathematical formulation of the LSTM,

Original: [Equation]
Revised: [Equation]

Original: where [ W i , W f , W o , U i , U f , U o ] are weight matrices, x t is the vector input to the timestep t , h t is the current exposed hidden state, c t is the memory cell state, and (cid:12) is element-wise multiplication.
Revised: where [ W i , W f , W o , U i , U f , U o ] are weight matrices, x t is the vector input to the timestep t , h t is the current exposed hidden state, c t is the memory cell state, and (cid:12) is element-wise multiplication.

Original: Preventing overﬁtting within the recurrent connections of an RNN has been an area of extensive research in language modeling.
Revised: Preventing overﬁtting within the recurrent connections of an RNN has been an area of extensive research in language modeling.

Original: The majority of previous recurrent regularization techniques have acted on the hidden state vector h t − 1 , most frequently introducing a dropout operation between timesteps, or performing dropout on the update to the memory state c t .
Revised: The majority of previous recurrent regularization techniques have acted on the hidden state vector h t − 1 , most frequently introducing a dropout operation between timesteps, or performing dropout on the update to the memory state c t .

Original: These modiﬁcations to a standard LSTM prevent the use of black box RNN implementations that may be many times faster due to low-level hardware-speciﬁc optimizations.
Revised: These modiﬁcations to a standard LSTM prevent the use of black box RNN implementations that may be many times faster due to low-level hardware-speciﬁc optimizations.

Original: We propose the use of DropConnect (Wan et al., 2013) on the recurrent hidden to hidden weight matrices which does not require any modiﬁcations to an RNN’s formulation.
Revised: We propose the use of DropConnect (Wan et al., 2013) on the recurrent hidden to hidden weight matrices which does not require any modiﬁcations to an RNN’s formulation.

Original: As the dropout operation is applied once to the weight matrices, before the forward and backward pass, the impact on training speed is minimal and any standard RNN implementation can be used, including inﬂexible but highly optimized black box 
Revised: As the dropout operation is applied once to the weight matrices, before the forward and backward pass, the impact on training speed is minimal and any standard RNN implementation can be used, including inﬂexible but highly optimized black box LSTM implementations such as NVIDIA’s cuDNN LSTM.

Original: By performing DropConnect on the hidden-to-hidden weight matrices [ U i , U f , U o , U c ] within the
Revised: By performing DropConnect on the hidden-to-hidden weight matrices [ U i , U f , U o , U c ] within the

Original: LSTM, we can prevent overﬁtting from occurring on the recurrent connections of the LSTM.
Revised: LSTM, we can prevent overﬁtting from occurring on the recurrent connections of the LSTM.

Original: This regularization technique would also be applicable to preventing overﬁtting on the recurrent weight matrices of other RNN cells.
Revised: This regularization technique would also be applicable to preventing overﬁtting on the recurrent weight matrices of other RNN cells.

Original: As the same weights are reused over multiple timesteps, the same individual dropped weights remain dropped for the entirety of the forward and backward pass.
Revised: As the same weights are reused over multiple timesteps, the same individual dropped weights remain dropped for the entirety of the forward and backward pass.

Original: The result is similar to variational dropout, which applies the same dropout mask to recurrent connections within the LSTM by performing dropout on h t − 1 , except that the dropout is applied to the recurrent weights.
Revised: The result is similar to variational dropout, which applies the same dropout mask to recurrent connections within the LSTM by performing dropout on h t − 1 , except that the dropout is applied to the recurrent weights.

Original: DropConnect could also be used on the non-recurrent weights of the LSTM
Revised: DropConnect could also be used on the non-recurrent weights of the LSTM

Original: [ W i , W f , W o ] though our focus was on preventing overﬁtting on the recurrent connection.
Revised: [ W i , W f , W o ] though our focus was on preventing overﬁtting on the recurrent connection.

Original:  O PTIMIZATION
Revised:  O PTIMIZATION

Original: SGD is among the most popular methods for training deep learning models across various modalities including computer vision, natural language processing, and reinforcement learning.
Revised: SGD is among the most popular methods for training deep learning models across various modalities including computer vision, natural language processing, and reinforcement learning.

Original: The training of deep networks can be posed as a non-convex empirical risk minimization problem
Revised: The training of deep networks can be posed as a non-convex empirical risk minimization problem

Original: [Equation]
Revised: [Equation]

Original: where f i is the loss function for the i th data point, w are the weights of the network, and the expectation is taken over the data.
Revised: where f i is the loss function for the i th data point, w are the weights of the network, and the expectation is taken over the data.

Original: In this context, given a sequence of learning rates, γ k , SGD iteratively takes steps of the form
Revised: In this context, given a sequence of learning rates, γ k , SGD iteratively takes steps of the form

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the subscript denotes the iteration number and the ˆ ∇ denotes a stochastic gradient that may be computed on a minibatch of data points.
Revised: where the subscript denotes the iteration number and the ˆ ∇ denotes a stochastic gradient that may be computed on a minibatch of data points.

Original: SGD demonstrably performs well in practice and also possesses several attractive theoretical properties such as linear convergence (Bottou et al., 2016), saddle point avoidance (Panageas & Piliouras, 2016) and better generalization performance (Hardt et al., 2015).
Revised: SGD demonstrably performs well in practice and also possesses several attractive theoretical properties such as linear convergence (Bottou et al., 2016), saddle point avoidance (Panageas & Piliouras, 2016) and better generalization performance (Hardt et al., 2015).

Original: For the speciﬁc task of neural language modeling, traditionally SGD without momentum has been found to outperform other algorithms such as momentum SGD (Sutskever et al., 2013), Adam (Kingma & Ba, 2014), Adagrad (Duchi et al., 2011) and RMSProp (Tieleman & Hinton, 2012) by a statistically signiﬁcant margin.
Revised: For the speciﬁc task of neural language modeling, traditionally SGD without momentum has been found to outperform other algorithms such as momentum SGD (Sutskever et al., 2013), Adam (Kingma & Ba, 2014), Adagrad (Duchi et al., 2011) and RMSProp (Tieleman & Hinton, 2012) by a statistically signiﬁcant margin.

Original: Motivated by this observation, we investigate averaged SGD (AvSGD) to further improve the training process.
Revised: Motivated by this observation, we investigate averaged SGD (AvSGD) to further improve the training process.

Original: AvSGD has been analyzed in depth theoretically and many surprising results have been shown including its asymptotic second-order convergence (Polyak & Juditsky, 1992; Mandt et al., 2017).
Revised: AvSGD has been analyzed in depth theoretically and many surprising results have been shown including its asymptotic second-order convergence (Polyak & Juditsky, 1992; Mandt et al., 2017).

Original: AvSGD takes steps identical to equation (1) but instead of returning the last iterate as the solution, returns
Revised: AvSGD takes steps identical to equation (1) but instead of returning the last iterate as the solution, returns

Original: [Equation]
Revised: [Equation]

Original: i , where K is the total number of iterations and T < K is a user-speciﬁed averaging trigger.
Revised: , where K is the total number of iterations and T < K is a user-speciﬁed averaging trigger

Original: Despite its theoretical appeal, AvSGD has found limited practical use in training of deep networks.
Revised: Despite its theoretical appeal, AvSGD has found limited practical use in training of deep networks.

Original: This may be in part due to unclear tuning guidelines for the learning-rate schedule γ k and averaging trigger T .
Revised: This may be in part due to unclear tuning guidelines for the learning-rate schedule γ k and averaging trigger T .

Original: If the averaging is triggered too soon, the efﬁcacy of the method is impacted, and if it is triggered too late, many additional iterations may be needed to converge to the solution.
Revised: If the averaging is triggered too soon, the efﬁcacy of the method is impacted, and if it is triggered too late, many additional iterations may be needed to converge to the solution.

Original: In this section, we describe a non-monotonically triggered variant of AvSGD (NT-AvSGD), which obviates the need for tuning T .
Revised: In this section, we describe a non-monotonically triggered variant of AvSGD (NT-AvSGD), which obviates the need for tuning T .

Original: Further, the algorithm uses a constant learning rate throughout the experiment and hence no further tuning is necessary for the decay scheduling.
Revised: Further, the algorithm uses a constant learning rate throughout the experiment and hence no further tuning is necessary for the decay scheduling.

Original: Ideally, averaging needs to be triggered when the SGD iterates converge to a steady-state distribution (Mandt et al., 2017).
Revised: Ideally, averaging needs to be triggered when the SGD iterates converge to a steady-state distribution (Mandt et al., 2017).

Original: This is roughly equivalent to the convergence of SGD to a neighborhood around a solution.
Revised: This is roughly equivalent to the convergence of SGD to a neighborhood around a solution.

Original: In the case of SGD, certain learning-rate reduction strategies such as the stepwise strategy analogously reduce the learning rate by a ﬁxed quantity at such a point.
Revised: In the case of SGD, certain learning-rate reduction strategies such as the stepwise strategy analogously reduce the learning rate by a ﬁxed quantity at such a point.

Original: A common strategy employed in language modeling is to reduce the learning rates by a ﬁxed proportion when the performance of the model’s primary metric (such as perplexity) worsens or stagnates.
Revised: A common strategy employed in language modeling is to reduce the learning rates by a ﬁxed proportion when the performance of the model’s primary metric (such as perplexity) worsens or stagnates.

Original: Along the same lines, one could make a triggering decision based on the performance of the model on the validation set.
Revised: Along the same lines, one could make a triggering decision based on the performance of the model on the validation set.

Original: However, instead of averaging immediately after the validation metric worsens, we propose a non-monotonic criterion that conservatively triggers the averaging when the validation metric fails to improve for multiple cycles; see Algorithm 1.
Revised: However, instead of averaging immediately after the validation metric worsens, we propose a non-monotonic criterion that conservatively triggers the averaging when the validation metric fails to improve for multiple cycles; see Algorithm 1.

Original: Given that the choice of triggering is irreversible, this conservatism ensures that the randomness of training does not play a major role in the decision.
Revised: Given that the choice of triggering is irreversible, this conservatism ensures that the randomness of training does not play a major role in the decision.

Original: Analogous strategies have also been proposed for learning-rate reduction in SGD (Keskar & Saon, 2015).
Revised: Analogous strategies have also been proposed for learning-rate reduction in SGD (Keskar & Saon, 2015).

Original: While the algorithm introduces two additional hyperparameters, the logging interval L and nonmonotone interval n , we found that setting L to be the number of iterations in an epoch and n =worked well across various models and data sets.
Revised: While the algorithm introduces two additional hyperparameters, the logging interval L and nonmonotone interval n , we found that setting L to be the number of iterations in an epoch and n =worked well across various models and data sets.

Original: As such, we use this setting in all of our NTAvSGD experiments in the following section and demonstrate that it achieves better training outcomes as compared to SGD.
Revised: As such, we use this setting in all of our NTAvSGD experiments in the following section and demonstrate that it achieves better training outcomes as compared to SGD.

Original:  E XTENDED REGULARIZATION TECHNIQUES
Revised:  E XTENDED REGULARIZATION TECHNIQUES

Original: In addition to the regularization and optimization techniques above, we explored additional regularization techniques that aimed to improve data efﬁciency during training and to prevent overﬁtting of the RNN model.
Revised: In addition to the regularization and optimization techniques above, we explored additional regularization techniques that aimed to improve data efﬁciency during training and to prevent overﬁtting of the RNN model.

Original: 4. V ARIABLE LENGTH BACKPROPAGATION SEQUENCES
Revised: 4. V ARIABLE LENGTH BACKPROPAGATION SEQUENCES

Original: Given a ﬁxed sequence length that is used to break a data set into ﬁxed length batches, the data set is not efﬁciently used.
Revised: Given a ﬁxed sequence length that is used to break a data set into ﬁxed length batches, the data set is not efﬁciently used.

Original: To illustrate this, imagine being given 100 elements to perform backpropagation through with a ﬁxed backpropagation through time (BPTT) window of 10 .
Revised: To illustrate this, imagine being given 100 elements to perform backpropagation through with a ﬁxed backpropagation through time (BPTT) window of 10 .

Original: Any element divisible by 10 will never have any elements to backprop into, no matter how many times you may traverse the data set.
Revised: Any element divisible by 10 will never have any elements to backprop into, no matter how many times you may traverse the data set.

Original: Indeed, the backpropagation window that each element receives is equal to i modwhere i is the element’s index.
Revised: Indeed, the backpropagation window that each element receives is equal to i modwhere i is the element’s index.

Original: This is data inefﬁcient, preventing 110 of the data set from ever being able to improve itself in a recurrent fashion, and resulting in 810 of the remaining elements receiving only a partial backpropagation window compared to the full possible backpropagation window of length 10 .
Revised: This is data inefﬁcient, preventing 110 of the data set from ever being able to improve itself in a recurrent fashion, and resulting in 810 of the remaining elements receiving only a partial backpropagation window compared to the full possible backpropagation window of length 10 .

Original: To prevent such inefﬁcient data usage, we randomly select the sequence length for the forward and backward pass in two steps.
Revised: To prevent such inefﬁcient data usage, we randomly select the sequence length for the forward and backward pass in two steps.

Original: First, we select the base sequence length to be seq with probability p and seq 2 with probability 1 − p , where p is a high value approaching 1 .
Revised: First, we select the base sequence length to be seq with probability p and seq 2 with probability 1 − p , where p is a high value approaching 1 .

Original: This spreads the starting point for the BPTT window beyond the base sequence length.
Revised: This spreads the starting point for the BPTT window beyond the base sequence length.

Original: We then select the sequence length according to N ( seq , s ) , where seq is the base sequence length and s is the standard deviation.
Revised: We then select the sequence length according to N ( seq , s ) , where seq is the base sequence length and s is the standard deviation.

Original: This jitters the starting point such that it doesn’t always fall on a speciﬁc word divisible by seq or seq 2 .
Revised: This jitters the starting point such that it doesn’t always fall on a speciﬁc word divisible by seq or seq 2 .

Original: From these, the sequence length more efﬁciently uses the data set, ensuring that when given enough epochs all the elements in the data set experience a full BPTT window, while ensuring the average sequence length remains around the base sequence length for computational efﬁciency.
Revised: From these, the sequence length more efﬁciently uses the data set, ensuring that when given enough epochs all the elements in the data set experience a full BPTT window, while ensuring the average sequence length remains around the base sequence length for computational efﬁciency.

Original: During training, we rescale the learning rate depending on the length of the resulting sequence compared to the original speciﬁed sequence length.
Revised: During training, we rescale the learning rate depending on the length of the resulting sequence compared to the original speciﬁed sequence length.

Original: The rescaling step is necessary as sampling arbitrary sequence lengths with a ﬁxed learning rate favors short sequences over longer ones.
Revised: The rescaling step is necessary as sampling arbitrary sequence lengths with a ﬁxed learning rate favors short sequences over longer ones.

Original: This linear scaling rule has been noted as important for training large scale minibatch SGD without loss of accuracy (Goyal et al., 2017) and is a component of unbiased truncated backpropagation through time (Tallec & Ollivier, 2017).
Revised: This linear scaling rule has been noted as important for training large scale minibatch SGD without loss of accuracy (Goyal et al., 2017) and is a component of unbiased truncated backpropagation through time (Tallec & Ollivier, 2017).

Original: 4. V ARIATIONAL DROPOUT
Revised: 4. V ARIATIONAL DROPOUT

Original: In standard dropout, a new binary dropout mask is sampled each and every time the dropout function is called.
Revised: In standard dropout, a new binary dropout mask is sampled each and every time the dropout function is called.

Original: New dropout masks are sampled even if the given connection is repeated, such as the input x 0 to an LSTM at timestep t = 0 receiving a different dropout mask than the input x 1 fed to the same LSTM at t = 1 .
Revised: New dropout masks are sampled even if the given connection is repeated, such as the input x 0 to an LSTM at timestep t = 0 receiving a different dropout mask than the input x 1 fed to the same LSTM at t = 1 .

Original: A variant of this, variational dropout (Gal & Ghahramani, 2016), samples a binary dropout mask only once upon the ﬁrst call and then to repeatedly use that locked dropout mask for all repeated connections within the forward and backward pass.
Revised: A variant of this, variational dropout (Gal & Ghahramani, 2016), samples a binary dropout mask only once upon the ﬁrst call and then to repeatedly use that locked dropout mask for all repeated connections within the forward and backward pass.

Original: While we propose using DropConnect rather than variational dropout to regularize the hidden-tohidden transition within an RNN, we use variational dropout for all other dropout operations, specifically using the same dropout mask for all inputs and outputs of the LSTM within a given forward and backward pass.
Revised: While we propose using DropConnect rather than variational dropout to regularize the hidden-tohidden transition within an RNN, we use variational dropout for all other dropout operations, specifically using the same dropout mask for all inputs and outputs of the LSTM within a given forward and backward pass.

Original: Each example within the minibatch uses a unique dropout mask, rather than a single dropout mask being used over all examples, ensuring diversity in the elements dropped out.
Revised: Each example within the minibatch uses a unique dropout mask, rather than a single dropout mask being used over all examples, ensuring diversity in the elements dropped out.

Original: 4. E MBEDDING DROPOUT
Revised: 4. E MBEDDING DROPOUT

Original: Following Gal & Ghahramani (2016), we employ embedding dropout.
Revised: Following Gal & Ghahramani (2016), we employ embedding dropout.

Original: This is equivalent to performing dropout on the embedding matrix at a word level, where the dropout is broadcast across all the word vector’s embedding.
Revised: This is equivalent to performing dropout on the embedding matrix at a word level, where the dropout is broadcast across all the word vector’s embedding.

Original: The remaining non-dropped-out word embeddings are scaled by1 − p e where p e is the probability of embedding dropout.
Revised: The remaining non-dropped-out word embeddings are scaled by1 − p e where p e is the probability of embedding dropout.

Original: As the dropout occurs on the embedding matrix that is used for a full forward and backward pass, this means that all occurrences of a speciﬁc word will disappear within that pass, equivalent to performing variational dropout on the connection between the one-hot embedding and the embedding lookup.
Revised: As the dropout occurs on the embedding matrix that is used for a full forward and backward pass, this means that all occurrences of a speciﬁc word will disappear within that pass, equivalent to performing variational dropout on the connection between the one-hot embedding and the embedding lookup.

Original: 4. W EIGHT TYING
Revised: 4. W EIGHT TYING

Original: Weight tying (Inan et al., 2016; Press & Wolf, 2016) shares the weights between the embedding and softmax layer, substantially reducing the total parameter count in the model.
Revised: Weight tying (Inan et al., 2016; Press & Wolf, 2016) shares the weights between the embedding and softmax layer, substantially reducing the total parameter count in the model.

Original: The technique has theoretical motivation (Inan et al., 2016) and prevents the model from having to learn a one-to-one correspondence between the input and output, resulting in substantial improvements to the standard
Revised: The technique has theoretical motivation (Inan et al., 2016) and prevents the model from having to learn a one-to-one correspondence between the input and output, resulting in substantial improvements to the standard

Original: LSTM language model.
Revised: LSTM language model.

Original: 4. I NDEPENDENT EMBEDDING SIZE AND HIDDEN SIZE
Revised: 4. I NDEPENDENT EMBEDDING SIZE AND HIDDEN SIZE

Original: In most natural language processing tasks, both pre-trained and trained word vectors are of relatively low dimensionality—frequently between 100 and 400 dimensions in size.
Revised: In most natural language processing tasks, both pre-trained and trained word vectors are of relatively low dimensionality—frequently between 100 and 400 dimensions in size.

Original: Most previous LSTM language models tie the dimensionality of the word vectors to the dimensionality of the LSTM’s hidden state.
Revised: Most previous LSTM language models tie the dimensionality of the word vectors to the dimensionality of the LSTM’s hidden state.

Original: Even if reducing the word embedding size was not beneﬁcial in preventing overﬁtting, the easiest reduction in total parameters for a language model is reducing the word vector size.
Revised: Even if reducing the word embedding size was not beneﬁcial in preventing overﬁtting, the easiest reduction in total parameters for a language model is reducing the word vector size.

Original: To achieve this, the ﬁrst and last LSTM layers are modiﬁed such that their input and output dimensionality respectively are equal to the reduced embedding size.
Revised: To achieve this, the ﬁrst and last LSTM layers are modiﬁed such that their input and output dimensionality respectively are equal to the reduced embedding size.

Original: 4. A CTIVATION R EGULARIZATION (AR) AND T EMPORAL A CTIVATION R EGULARIZATION (TAR)
Revised: 4. A CTIVATION R EGULARIZATION (AR) AND T EMPORAL A CTIVATION R EGULARIZATION (TAR)

Original: L 2 -regularization is often used on the weights of the network to control the norm of the resulting model and reduce overﬁtting.
Revised: L 2 -regularization is often used on the weights of the network to control the norm of the resulting model and reduce overﬁtting.

Original: In addition, L 2 decay can be used on the individual unit activations and on the difference in outputs of an RNN at different time steps; these strategies labeled as activation regularization (AR) and temporal activation regularization (TAR) respectively (Merity et al., 2017).
Revised: In addition, L 2 decay can be used on the individual unit activations and on the difference in outputs of an RNN at different time steps; these strategies labeled as activation regularization (AR) and temporal activation regularization (TAR) respectively (Merity et al., 2017).

Original: AR penalizes activations that are signiﬁcantly larger than 0 as a means of regularizing the network.
Revised: AR penalizes activations that are signiﬁcantly larger than 0 as a means of regularizing the network.

Original: Concretely, AR is deﬁned as
Revised: Concretely, AR is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where m is the dropout mask, L 2 ( · )
Revised: where m is the dropout mask, L 2 ( · )

Original: = (cid:107)·(cid:107) 2 , h t is the output of the RNN at timestep t , and α is a scaling coefﬁcient.
Revised: = (cid:107)·(cid:107) 2 , h t is the output of the RNN at timestep t , and α is a scaling coefﬁcient.

Original: TAR falls under the broad category of slowness regularizers (Hinton, 1989;
Revised: TAR falls under the broad category of slowness regularizers (Hinton, 1989;

Original: F¨oldi´ak, 1991; Luciw & Schmidhuber, 2012; Jonschkowski & Brock, 2015) which penalize the model from producing large changes in the hidden state.
Revised: F¨oldi´ak, 1991; Luciw & Schmidhuber, 2012; Jonschkowski & Brock, 2015) which penalize the model from producing large changes in the hidden state.

Original: Using the notation from AR, TAR is deﬁned as
Revised: Using the notation from AR, TAR is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where β is a scaling coefﬁcient.
Revised: where β is a scaling coefﬁcient.

Original: As in Merity et al.
Revised: As in Merity et al.

Original: (2017), the AR and TAR loss are only applied to the output of the ﬁnal RNN layer as opposed to being applied to all layers.
Revised: (2017), the AR and TAR loss are only applied to the output of the ﬁnal RNN layer as opposed to being applied to all layers.

Original:  E XPERIMENT D ETAILS
Revised:  E XPERIMENT D ETAILS

Original: For evaluating the impact of these approaches, we perform language modeling over a preprocessed version of the Penn Treebank (PTB) (Mikolov et al., 2010) and the WikiText-2 (WT2) data set (Merity et al., 2016).
Revised: For evaluating the impact of these approaches, we perform language modeling over a preprocessed version of the Penn Treebank (PTB) (Mikolov et al., 2010) and the WikiText-2 (WT2) data set (Merity et al., 2016).

Original: PTB: The Penn Treebank data set has long been a central data set for experimenting with language modeling.
Revised: PTB: The Penn Treebank data set has long been a central data set for experimenting with language modeling.

Original: The data set is heavily preprocessed and does not contain capital letters, numbers, or punctuation.
Revised: The data set is heavily preprocessed and does not contain capital letters, numbers, or punctuation.

Original: The vocabulary is also capped at 10,000 unique words, quite small in comparison to most modern datasets, which results in a large number of out of vocabulary (OoV) tokens.
Revised: The vocabulary is also capped at 10,000 unique words, quite small in comparison to most modern datasets, which results in a large number of out of vocabulary (OoV) tokens.

Original: WT2: WikiText-2 is sourced from curated Wikipedia articles and is approximately twice the size of the PTB data set.
Revised: WT2: WikiText-2 is sourced from curated Wikipedia articles and is approximately twice the size of the PTB data set.

Original: The text is tokenized and processed using the Moses tokenizer (Koehn et al., 2007), frequently used for machine translation, and features a vocabulary of over 30,000 words.
Revised: The text is tokenized and processed using the Moses tokenizer (Koehn et al., 2007), frequently used for machine translation, and features a vocabulary of over 30,000 words.

Original: Capitalization, punctuation, and numbers are retained in this data set.
Revised: Capitalization, punctuation, and numbers are retained in this data set.

Original: All experiments use a three-layer LSTM model with 1150 units in the hidden layer and an embedding of size 400 .
Revised: All experiments use a three-layer LSTM model with 1150 units in the hidden layer and an embedding of size 400 .

Original: The loss was averaged over all examples and timesteps.
Revised: The loss was averaged over all examples and timesteps.

Original: All embedding weights were uniformly initialized in the interval [ − 0 .
Revised: All embedding weights were uniformly initialized in the interval [ − 0 .

Original: 1 , 0 . 1] and all other weights were initialized between [ − 1 √ H , 1 √ H ] , where H is the hidden size.
Revised: 1 , 0 . 1] and all other weights were initialized between [ − 1 √ H , 1 √ H ] , where H is the hidden size.

Original: For training the models, we use the NT-AvSGD algorithm discussed in the previous section forepochs with L equivalent to one epoch and n = 5 .
Revised: For training the models, we use the NT-AvSGD algorithm discussed in the previous section forepochs with L equivalent to one epoch and n = 5 .

Original: We use a batch size of 80 for WT2 and 40 for
Revised: We use a batch size of 80 for WT2 and 40 for

Original: PTB.
Revised: PTB.

Original: Empirically, we found relatively large batch sizes (e.g., 40 - 80 ) performed better than smaller sizes (e.g., 10 - 20 ) for NT-AvSGD.
Revised: Empirically, we found relatively large batch sizes (e.g., 40 - 80 ) performed better than smaller sizes (e.g., 10 - 20 ) for NT-AvSGD.

Original: After completion, we run AvSGD with T = 0 and hot-started was a ﬁne-tuning step to further improve the solution.
Revised: After completion, we run AvSGD with T = 0 and hot-started was a ﬁne-tuning step to further improve the solution.

Original: For this ﬁne-tuning step, we terminate the run using the same non-monotonic criterion detailed in Algorithm 1.
Revised: For this ﬁne-tuning step, we terminate the run using the same non-monotonic criterion detailed in Algorithm 1.

Original: We carry out gradient clipping with maximum norm 0 .
Revised: We carry out gradient clipping with maximum norm 0 .

Original: and use an initial learning rate offor all experiments.
Revised: and use an initial learning rate offor all experiments.

Original: We use a random BPTT length which is N (70 , 5) with probability 0 .
Revised: We use a random BPTT length which is N (70 , 5) with probability 0 .

Original: 95 and
Revised: 95 and

Original: N (35 , 5) with probability 0 . 05 .
Revised: N (35 , 5) with probability 0 . 05 .

Original: The values used for dropout on the word vectors, the output between
Revised: The values used for dropout on the word vectors, the output between

Original: LSTM layers, the output of the ﬁnal LSTM layer, and embedding dropout where (0 . 4 , 0 . 3 , 0 . 4 , 0 . 1) respectively.
Revised: LSTM layers, the output of the ﬁnal LSTM layer, and embedding dropout where (0 . 4 , 0 . 3 , 0 . 4 , 0 . 1) respectively.

Original: For the weight-dropped LSTM, a dropout of 0 .
Revised: For the weight-dropped LSTM, a dropout of 0 .

Original: 5 was applied to the recurrent weight matrices.
Revised: 5 was applied to the recurrent weight matrices.

Original: For WT2, we increase the input dropout to 0 .
Revised: For WT2, we increase the input dropout to 0 .

Original: 65 to account for the increased vocabulary size.
Revised: 65 to account for the increased vocabulary size.

Original: For all experiments, we use AR and TAR values of 2 and 1 respectively, and tie the embedding and softmax weights.
Revised: For all experiments, we use AR and TAR values of 2 and 1 respectively, and tie the embedding and softmax weights.

Original: These hyperparameters were chosen through trial and error and we expect further improvements may be possible if a ﬁne-grained hyperparameter search were to be conducted.
Revised: These hyperparameters were chosen through trial and error and we expect further improvements may be possible if a ﬁne-grained hyperparameter search were to be conducted.

Original: In the results, we abbreviate our approach as AWD-LSTM for AvSGD Weight-Dropped LSTM.
Revised: In the results, we abbreviate [Table]

Original: The code for reproducing our results is open sourced and available at MASKED .
Revised: [Table]

Original:  E XPERIMENTAL A NALYSIS
Revised:  E XPERIMENTAL A NALYSIS

Original: We present the single-model perplexity results for both our models (AWD-LSTM) and other competitive models in Table 1 and 2 for PTB and WT2 respectively 1 .
Revised: We present the single-model perplexity results for both our models (AWD-LSTM) and other competitive models in Table 1 and 2 for PTB and WT2 respectively 1 .

Original: On both data sets we improve the[Table] [Table] state-of-the-art, with our vanilla LSTM model beating the state of the art by approximately 1 unit on
Revised: On both data sets we improve the state-of-the-art, with our vanilla LSTM model beating the state of the art by approximately 1 unit on

Original: PTB and 0.1 units on WT2.
Revised: PTB and 0.1 units on WT2.

Original: In comparison to other recent state-of-the-art models [Table] our model uses a vanilla LSTM.
Revised: In comparison to other recent state-of-the-art models, our model uses a vanilla LSTM.

Original: Zilly et al.
Revised: Zilly et al.

Original: (2016) propose the recurrent highway network, which extends the LSTM to allow multiple hidden state updates per timestep.
Revised: (2016) propose the recurrent highway network, which extends the LSTM to allow multiple hidden state updates per timestep.

Original: Zoph & Le (2016) use a reinforcement learning agent to generate
Revised: Zoph & Le (2016) use a reinforcement learning agent to generate

Original: an
Revised: an

Original: RNN cell tailored to the speciﬁc task of language modeling, with the cell far more complex than the
Revised: RNN cell tailored to the speciﬁc task of language modeling, with the cell far more complex than the

Original: LSTM.
Revised: LSTM.

Original: Independently of our work, Melis et al.
Revised: Independently of our work, Melis et al.

Original: (2017) apply extensive hyperparameter search to an LSTM based language modeling implementation, analyzing the sensitivity of RNN based language models to hyperparameters.
Revised: (2017) apply extensive hyperparameter search to an LSTM based language modeling implementation, analyzing the sensitivity of RNN based language models to hyperparameters.

Original: Unlike our work, they use a modiﬁed LSTM, which caps the input gate i t to be min(1 − f t , i t ) , use Adam with β 1 = 0 rather than SGD or AvSGD, use skip connections between
Revised: Unlike our work, they use a modiﬁed LSTM, which caps the input gate i t to be min(1 − f t , i t ) , use Adam with β 1 = 0 rather than SGD or AvSGD, use skip connections between

Original: LSTM layers, and use a black box hyperparameter tuner for exploring models and settings.
Revised: LSTM layers, and use a black box hyperparameter tuner for exploring models and settings.

Original: Of particular interest is that their hyperparameters were tuned individually for each data set compared to our work which shared almost all hyperparameters between PTB and WT2, including the embedding and hidden size for both data sets.
Revised: Of particular interest is that their hyperparameters were tuned individually for each data set compared to our work which shared almost all hyperparameters between PTB and WT2, including the embedding and hidden size for both data sets.

Original: Due to this, they used less model parameters than our model and found shallow LSTMs of one or two layers worked best for WT2.
Revised: Due to this, they used less model parameters than our model and found shallow LSTMs of one or two layers worked best for WT2.

Original: Like our work, Melis et al.
Revised: Like our work, Melis et al.

Original: ﬁnd that the underlying LSTM architecture can be highly effective compared to complex custom architectures when well tuned hyperparameters are used.
Revised: ﬁnd that the underlying LSTM architecture can be highly effective compared to complex custom architectures when well tuned hyperparameters are used.

Original: The approaches used in our work and (Melis et al., 2017) may be complementary and would be worth exploration.
Revised: The approaches used in our work and (Melis et al., 2017) may be complementary and would be worth exploration.

Original: 6. P OINTER MODELS
Revised: 6. P OINTER MODELS

Original: In past work, pointer based attention models have been shown to be highly effective in improving language modeling (Merity et al., 2016;
Revised: In past work, pointer based attention models have been shown to be highly effective in improving language modeling (Merity et al., 2016;

Original: Grave et al., 2016).
Revised: Grave et al., 2016).

Original: Given such substantial improvements to the underlying neural language model, it remained an open question as to how effective pointer augmentation may be, especially when improvements such as weight tying may act in mutually exclusive ways.
Revised: Given such substantial improvements to the underlying neural language model, it remained an open question as to how effective pointer augmentation may be, especially when improvements such as weight tying may act in mutually exclusive ways.

Original: The neural cache model (Grave et al., 2016) can be added on top of a pre-trained language model at negligible cost.
Revised: The neural cache model (Grave et al., 2016) can be added on top of a pre-trained language model at negligible cost.

Original: The neural cache stores the previous hidden states in memory cells and then uses a simple convex combination of the probability distributions suggested by the cache and the language model for prediction.
Revised: The neural cache stores the previous hidden states in memory cells and then uses a simple convex combination of the probability distributions suggested by the cache and the language model for prediction.

Original: The cache model has three hyperparameters: the memory size (window) for the cache, the coefﬁcient of the combination (which determines how the two distributions are mixed), and the ﬂatness of the cache distribution.
Revised: The cache model has three hyperparameters: the memory size (window) for the cache, the coefﬁcient of the combination (which determines how the two distributions are mixed), and the ﬂatness of the cache distribution.

Original: All of these are tuned on the validation set once a trained language model has been obtained and require no training by themselves, making it quite inexpensive to use.
Revised: All of these are tuned on the validation set once a trained language model has been obtained and require no training by themselves, making it quite inexpensive to use.

Original: The tuned values for these hyperparameters were (2000 , 0 .
Revised: The tuned values for these hyperparameters were (2000 , 0 .

Original: for PTB and (3785 , 0 . 1279 , 0 . 662) for WT2 respectively.
Revised: for PTB and (3785 , 0 . 1279 , 0 . 662) for WT2 respectively.

Original: In Tables 1 and 2, we show that the model further improves the perplexity of the language model by as much as 6 perplexity points for PTB and 11 points for WT2.
Revised: In Tables 1 and 2, we show that the model further improves the perplexity of the language model by as much as 6 perplexity points for PTB and 11 points for WT2.

Original: While this is smaller than the gains reported in Grave et al.
Revised: While this is smaller than the gains reported in Grave et al.

Original: (2016), which used an LSTM without weight tying, this is still a substantial drop.
Revised: (2016), which used an LSTM without weight tying, this is still a substantial drop.

Original: Given the simplicity of the neural cache model, and the lack of any trained components, these results suggest that existing neural language models remain fundamentally lacking, failing to capture long term dependencies or remember recently seen words effectively.
Revised: Given the simplicity of the neural cache model, and the lack of any trained components, these results suggest that existing neural language models remain fundamentally lacking, failing to capture long term dependencies or remember recently seen words effectively.

Original: To understand the impact the pointer had on the model, speciﬁcally the validation set perplexity, we detail the contribution that each word has on the cache model’s overall perplexity in Table 3.
Revised: To understand the impact the pointer had on the model, speciﬁcally the validation set perplexity, we detail the contribution that each word has on the cache model’s overall perplexity in Table 3.

Original: We compute the sum of the total difference in the loss function value (i.e., log perplexity) between the LSTM-only and LSTM-with-cache models for the target words in the validation portion of the
Revised: We compute the sum of the total difference in the loss function value (i.e., log perplexity) between the LSTM-only and LSTM-with-cache models for the target words in the validation portion of the

Original: WikiText-2 data set.
Revised: WikiText-2 data set.

Original: We present results for the sum of the difference as opposed to the mean since the latter undesirably overemphasizes infrequently occurring words for which the cache helps significantly and ignores frequently occurring words for which the cache provides modest improvements that cumulatively make a strong contribution.
Revised: We present results for the sum of the difference as opposed to the mean since the latter undesirably overemphasizes infrequently occurring words for which the cache helps significantly and ignores frequently occurring words for which the cache provides modest improvements that cumulatively make a strong contribution.

Original: The largest cumulative gain is in improving the handling of < unk > tokens, though this is overinstances.
Revised: The largest cumulative gain is in improving the handling of < unk > tokens, though this is overinstances.

Original: The second best improvement, approximately one ﬁfth the gain given by the < unk > tokens, is for Meridian, yet this word only occurs 161 times.
Revised: The second best improvement, approximately one ﬁfth the gain given by the < unk > tokens, is for Meridian, yet this word only occurs 161 times.

Original: This indicates the cache still helps signiﬁcantly even for relatively rare words, further demonstrated by Churchill, Blythe, or Sonic.
Revised: This indicates the cache still helps signiﬁcantly even for relatively rare words, further demonstrated by Churchill, Blythe, or Sonic.

Original: The cache is not beneﬁcial when handling frequent word categories, such as punctuation or stop words, for which the language model is likely well suited.
Revised: The cache is not beneﬁcial when handling frequent word categories, such as punctuation or stop words, for which the language model is likely well suited.

Original: These observations motivate the design of a cache framework that is more aware of the relative strengths of the two models.
Revised: These observations motivate the design of a cache framework that is more aware of the relative strengths of the two models.

Original: 6. AWD-QRNN
Revised: 6. AWD-QRNN

Original: Several architectures for learning sequential data based on convolutions, instead of recurrences, have been proposed recently.
Revised: Several architectures for learning sequential data based on convolutions, instead of recurrences, have been proposed recently.

Original: We brieﬂy mention experiments on the same language modeling using quasi-recurrent neural networks (QRNNs)
Revised: We brieﬂy mention experiments on the same language modeling us-

Original:  (Bradbury et al., 2016) instead of LSTMs; we label this setup the AWD-QRNN.
Revised: [Table] [Table] ing quasi-recurrent neural networks (QRNNs) (Bradbury et al., 2016) instead of LSTMs; we label this setup the AWD-QRNN.

Original: As in the case of AWD-LSTM, we regularize the network through weight, embedding and variational dropouts along with variable sequence lengths, weight tying,
Revised: As in the case of AWD-LSTM, we regularize the network through weight, embedding and variational dropouts along with variable sequence lengths, weight tying,

Original: AR and TAR.
Revised: AR and TAR.

Original: The networks were designed such that they had the same number of parameters as their LSTM counterparts and were trained using NT-AvSGD.
Revised: The networks were designed such that they had the same number of parameters as their LSTM counterparts and were trained using NT-AvSGD.

Original: Despite the same size of the network,
Revised: Despite the same size of the network,

Original: QRNNs were 2 − 4 × faster per epoch as compared to their LSTM counterparts and required fewer epochs to converge.
Revised: QRNNs were 2 − 4 × faster per epoch as compared to their LSTM counterparts and required fewer epochs to converge.

Original: We report the results in Table 4.
Revised: We report the results in Table 4.

Original: As is evident from the table, the QRNN model[Table] [Table] achieves comparable results to the LSTM suggesting the generality of the proposed regularization techniques.
Revised: As is evident from the table, the QRNN model achieves comparable results to the LSTM suggesting the generality of the proposed regularization techniques.

Original: Interestingly, the hyperparameter values for the various regularization components, including the optimization procedure, needed minimal changes from the LSTM to the QRNN models for competitive performance.
Revised: Interestingly, the hyperparameter values for the various regularization components, including the optimization procedure, needed minimal changes from the LSTM to the QRNN models for competitive performance.

Original: For full details and hyperparameters, refer to the released code.
Revised: For full details and hyperparameters, refer to the released code.

Original: 6. M ODEL A BLATION A NALYSIS
Revised: 6. M ODEL A BLATION A NALYSIS

Original: In Table 5, we present the values of validation and testing perplexity for different variants of our best-performing LSTM model.
Revised: In Table 5, we present the values of validation and testing perplexity for different variants of our best-performing LSTM model.

Original: Each variant removes a form of optimization or regularization.
Revised: Each variant removes a form of optimization or regularization.

Revised: [Table]

Original: The ﬁrst two variants deal with the optimization of the language models while the rest deal with the regularization.
Revised: The ﬁrst two variants deal with the optimization of the language models while the rest deal with the regularization.

Original: For the model using SGD with learning rate reduced by 2 using the same nonmonotonic fashion, there is a signiﬁcant degradation in performance.
Revised: For the model using SGD with learning rate reduced by 2 using the same nonmonotonic fashion, there is a signiﬁcant degradation in performance.

Original: This stands as empirical evidence regarding the beneﬁt of averaging of the iterates.
Revised: This stands as empirical evidence regarding the beneﬁt of averaging of the iterates.

Original: Using a monotonic criterion instead also hampered performance.
Revised: Using a monotonic criterion instead also hampered performance.

Original: Similarly, the removal of the ﬁne-tuning step expectedly also degrades the performance.
Revised: Similarly, the removal of the ﬁne-tuning step expectedly also degrades the performance.

Original: This step helps improve the estimate of the minimizer by resetting the memory of the previous experiment.
Revised: This step helps improve the estimate of the minimizer by resetting the memory of the previous experiment.

Original: While this process of ﬁne-tuning can be repeated multiple times, we found little beneﬁt in repeating it more than once.[Table]
Revised: While this process of ﬁne-tuning can be repeated multiple times, we found little beneﬁt in repeating it more than once.

Original: The removal of regularization strategies paints a similar picture; the inclusion of all of the proposed strategies was pivotal in ensuring state-of-the-art performance.
Revised: The removal of regularization strategies paints a similar picture; the inclusion of all of the proposed strategies was pivotal in ensuring state-of-the-art performance.

Original: The most extreme perplexity jump was in removing the hidden-to-hidden LSTM regularization provided by the weight-dropped LSTM.
Revised: The most extreme perplexity jump was in removing the hidden-to-hidden LSTM regularization provided by the weight-dropped LSTM.

Original: Without such hidden-to-hidden regularization, perplexity rises substantially, up to 11 points.
Revised: Without such hidden-to-hidden regularization, perplexity rises substantially, up to 11 points.

Original: This is in line with previous work showing the necessity of recurrent regularization in state-of-the-art models (Gal & Ghahramani, 2016; Inan et al., 2016) [Table]
Revised: This is in line with previous work showing the necessity of recurrent regularization in state-of-the-art models (Gal & Ghahramani, 2016; Inan et al., 2016).

Original: We also experiment with static sequence lengths which we had hypothesized would lead to inefﬁcient data usage.
Revised: We also experiment with static sequence lengths which we had hypothesized would lead to inefﬁcient data usage.

Original: This also worsens the performance by approximately one perplexity unit.
Revised: This also worsens the performance by approximately one perplexity unit.

Original: Next, we experiment with reverting to matching the sizes of the embedding vectors and the hidden states.
Revised: Next, we experiment with reverting to matching the sizes of the embedding vectors and the hidden states.

Original: This signiﬁcantly increases the number of parameters in the network (to 43 M in the case of PTB and 70 M for WT2) and leads to degradation by almost 8 perplexity points, which we attribute to overﬁtting in the word embeddings.
Revised: This signiﬁcantly increases the number of parameters in the network (to 43 M in the case of PTB and 70 M for WT2) and leads to degradation by almost 8 perplexity points, which we attribute to overﬁtting in the word embeddings.

Original: While this could potentially be improved with more aggressive regularization, the computational overhead involved with substantially larger embeddings likely outweighs any advantages.
Revised: While this could potentially be improved with more aggressive regularization, the computational overhead involved with substantially larger embeddings likely outweighs any advantages.

Original: Finally, we experiment with the removal of embedding dropout, AR/TAR and weight decay.
Revised: Finally, we experiment with the removal of embedding dropout, AR/TAR and weight decay.

Original: In all of the cases, the model suffers a perplexity increase of 2 – 6 points which we hypothesize is due to insufﬁcient regularization in the network.
Revised: In all of the cases, the model suffers a perplexity increase of 2 – 6 points which we hypothesize is due to insufﬁcient regularization in the network.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we discuss regularization and optimization strategies for neural language models.
Revised: In this work, we discuss regularization and optimization strategies for neural language models.

Original: We propose the weight-dropped LSTM, a strategy that uses a DropConnect mask on the hidden-tohidden weight matrices, as a means to prevent overﬁtting across the recurrent connections.
Revised: We propose the weight-dropped LSTM, a strategy that uses a DropConnect mask on the hidden-tohidden weight matrices, as a means to prevent overﬁtting across the recurrent connections.

Original: Further, we investigate the use of averaged SGD with a non-monontonic trigger for training language models and show that it outperforms SGD by a signiﬁcant margin.
Revised: Further, we investigate the use of averaged SGD with a non-monontonic trigger for training language models and show that it outperforms SGD by a signiﬁcant margin.

Original: We investigate other regularization strategies including the use of variable BPTT length and achieve a new state-of-the-art perplexity on the PTB and WikiText-2 data sets.
Revised: We investigate other regularization strategies including the use of variable BPTT length and achieve a new state-of-the-art perplexity on the PTB and WikiText-2 data sets.

Original: Our models outperform custom-built RNN cells and complex regularization strategies that preclude the possibility of using optimized libraries such as the NVIDIA cuDNN LSTM.
Revised: Our models outperform custom-built RNN cells and complex regularization strategies that preclude the possibility of using optimized libraries such as the NVIDIA cuDNN LSTM.

Original: We explore the use of a neural cache in conjunction with our proposed model and show that this further improves the performance, thus attaining an even lower state-of-the-art perplexity.
Revised: We explore the use of a neural cache in conjunction with our proposed model and show that this further improves the performance, thus attaining an even lower state-of-the-art perplexity.

Original: We also explore the viability of using the proposed regularization and optimization strategies in the context of a quasi-recurrent neural network (QRNN) and demonstrate comparable performance to the LSTM counterpart.
Revised: We also explore the viability of using the proposed regularization and optimization strategies in the context of a quasi-recurrent neural network (QRNN) and demonstrate comparable performance to the LSTM counterpart.

Original: While the regularization and optimization strategies proposed are demonstrated on the task of language modeling, we anticipate that they would be generally applicable across other sequence learning tasks.
Revised: While the regularization and optimization strategies proposed are demonstrated on the task of language modeling, we anticipate that they would be generally applicable across other sequence learning tasks