I MPROVED L ANGUAGE M ODELING BY D ECODING THE P AST
Anonymous authors
Paper under double-blind review
A BSTRACT
Highly regularized LSTMs achieve impressive results on several benchmark datasets in language modeling.
We propose a new regularization method based on decoding the last token in the context using the predicted distribution of the next token.
This biases the model towards retaining more contextual information, in turn improving its ability to predict the next token.
With negligible overhead in the number of parameters and training time, our Past Decode Regularization (PDR) method achieves a word level perplexity of 55.6 on the Penn Treebank and 63.5 on the WikiText-2 datasets using a single softmax. We also show gains by using PDR in combination with a mixture-of-softmaxes, achieving a word level perplexity of 53.8 and 60.5 on these datasets. In addition, our method achieves 1.
bits-per-character on the Penn Treebank Character dataset for character level language modeling. These results constitute a new state-of-the-art in their respective settings.
I NTRODUCTION
Language modeling is a fundamental task in natural language processing.
Given a sequence of tokens, its joint probability distribution can be modeled using the auto-regressive conditional factorization.
This leads to a convenient formulation where a language model has to predict the next token given a sequence of tokens as context.
Recurrent neural networks are an effective way to compute distributed representations of the context by sequentially operating on the embeddings of the tokens.
These representations can then be used to predict the next token as a probability distribution over a ﬁxed vocabulary using a linear decoder followed by Softmax.
Starting from the work of Mikolov et al.
(2010), there has been a long list of works that seek to improve language modeling performance using more sophisticated recurrent neural networks (RNNs) (Zaremba et al.
Zilly et al.
; Zoph & Le (2016);
Mujika et al.
However, in more recent work vanilla LSTMs (Hochreiter & Schmidhuber (1997)) with relatively large number of parameters have been shown to achieve state-of-the-art performance on several standard benchmark datasets both in word-level and character-level perplexity (Merity et al.
(2018a;b); Melis et al.
A key component in these models is the use of several forms of regularization e.g. variational dropout on the token embeddings (Gal & Ghahramani (2016)), dropout on the hidden-tohidden weights in the LSTM (Wan et al.
norm regularization on the outputs of the LSTM and classical dropout (Srivastava et al.
By carefully tuning the hyperparameters associated with these regularizers combined with optimization algorithms like NT-ASGD (a variant of the Averaged SGD), it is possible to achieve very good performance.
Each of these regularizations address different parts of the LSTM model and are general techniques that could be applied to any other sequence modeling problem.
In this paper, we propose a regularization technique that is speciﬁc to language modeling.
One unique aspect of language modeling using LSTMs (or any RNN) is that at each time step t , the model takes as input a particular token x t from a vocabulary W and using the hidden state of the
LSTM (which encodes the context till x t ) predicts a probability distribution w t +1 on the next token x t +1 over the same vocabulary as output.
Since x t can be mapped to a trivial probability distribution over W , this operation can be interpreted as transforming distributions over W (Inan et al.
Clearly, the output distribution is dependent on and is a function of x t and the context further in the past and encodes information about it.
We ask the following question – How much information is it possible to decode about the input distribution (and hence x t ) from the output distribution w t
+1 ?
In general, it is impossible to decode x t unambiguously.
Even if the language model is perfect and predicts x t +1 with probability 1, there could be many tokens preceding it.
However, in this case the number of possibilities for x t will be limited, as dictated by the bigram statistics of the corpus and the language in general.
We argue that biasing the language model such that it is possible to decode more information about the past tokens from the predicted next token distribution is beneﬁcial.
We incorporate this intuition into a regularization term in the loss function of the language model.
The symmetry in the inputs and outputs of the language model at each step lends itself to a simple decoding operation.
It can be cast as a (pseudo) language modeling problem in “reverse”, where the future prediction w t +1 acts as the input and the last token x t acts as the target of prediction.
The token embedding matrix and weights of the linear decoder of the main language model can be reused in the past decoding operation.
We only need a few extra parameters to model the nonlinear transformation performed by the LSTM, which we do by using a simple stateless layer.
We compute the cross-entropy loss between the decoded distribution for the past token and x t and add it to the main loss function after suitable weighting.
The extra parameters used in the past decoding are discarded during inference time.
We call our method Past Decode Regularization or PDR for short.
We conduct extensive experiments on four benchmark datasets for word level and character level language modeling by combining PDR with existing LSTM based language models and achieve new state-of-the-art performance on three of them.
P AST D ECODE R EGULARIZATION (PDR)
Let X =
( x 1 , x 2 , · · · , x t , · · · , x T ) be a sequence of tokens.
In this paper, we will experiment with both word level and character level language modeling.
Therefore, tokens can be either words or characters.
The joint probability P ( X ) factorizes into
[Equation]
Let c t
=
x 1 , x 2 , · · · , x t ) denote the context available to the language model for x t +1 .
Let W denote the vocabulary of tokens, each of which is embedded into a vector of dimension d .
Let E denote the token embedding matrix of dimension | W | × d and e w denote the embedding of w ∈ W .
An LSTM computes a distributed representation of c t in the form of its output hidden state h t , which we assume has dimension d as well.
The probability that the next token is w can then be calculated using a linear decoder followed by a Softmax layer as
[Equation]
where b w (cid:48) is the entry corresponding to w (cid:48) in a bias vector b of dimension | W | and | w represents projection onto w .
Here we assume that the weights of the decoder are tied with the token embedding matrix E
(Inan et al.
; Press & Wolf (2017)).
To optimize the parameters of the language model θ , the loss function to be minimized during training is set as the cross-entropy between the predicted distribution P θ ( w | c t ) and the actual token x t +1 .
[Equation]
Note that Eq.(2), when applied to all w ∈ W produces a 1 × | W | vector w t +1 , encapsulating the prediction the language model has about the next token x t +1 .
Since this is dependent on and conditioned on c t , w t +1 clearly encodes information about it; in particular about the last token x t in c t .
In turn, it should be possible to infer or decode some limited information about x t from w t +1 .
We argue that by biasing the model to be more accurate in recalling information about past tokens, we can help it in predicting the next token better.
To this end, we deﬁne the following decoding operation to compute a probability distribution over w c ∈ W as the last token in the context.
[Equation]
[Table] Here f θ r is a non-linear function that maps vectors in R d to vectors in R d and b (cid:48) θ r is a bias vector of dimension | W | , together with parameters θ r .
In effect, we are decoding the past – the last token in the context x t .
This produces a vector w rt of dimension 1 × | W | .
The cross-entropy loss with respect to the actual last token x t can then be computed as
[Equation]
Here PDR stands for Past Decode Regularization .
L PDR captures the extent to which the decoded distribution of tokens differs from the actual tokens x t in the context.
Note the symmetry between
Eqs.(2) and (5).
The “input” in the latter case is w t +1 and the “context” is provided by a nonlinear transformation of w t +1 E .
Different from the former, the context in Eq.(5) does not preserve any state information across time steps as we want to decode only using w t +1 .
The term w t E can be interpreted as a “soft” token embedding lookup, where the token vector w t is a probability distribution instead of a unit vector.
We add λ PDR L PDR to the loss function in Eq.(3) as a regularization term, where λ PDR is a positive weighting coefﬁcient, to construct the following new loss function for the language model.
[Equation]
Thus equivalently PDR can also be viewed as a method of deﬁning an augmented loss function for language modeling.
The choice of λ PDR dictates the degree to which we want the language model to incorporate our inductive bias i.e. decodability of the last token in the context.
If it is too large, the model will fail to predict the next token, which is its primary task.
If it is zero or too small, the model will retain less information about the last token which hampers its predictive performance.
In practice, we choose λ PDR by a search based on validation set performance.
Note that the trainable parameters θ r associated with PDR are used only during training to bias the language model and are not used at inference time.
This also means that it is important to control the complexity of the nonlinear function f θ r so as not to overly bias the training.
As a simple choice, we use a single fully connected layer of size d followed by a Tanh nonlinearity as f θ r .
This introduces few extra parameters and a small increase in training time as compared to a model not using PDR.
E XPERIMENTS
We present extensive experimental results to show the efﬁcacy of using PDR for language modeling on four standard benchmark datasets – two each for word level and character level language modeling.
For the former, we evaluate our method on the Penn Treebank (PTB) (Mikolov et al.
and the WikiText-2 (WT2) (Merity et al.
datasets.
For the latter, we use the Penn Treebank
Character (PTBC) (Mikolov et al. and the Hutter Prize Wikipedia Prize (Hutter (2018))
(also known as Enwik8) datasets.
Key statistics for these datasets is presented in Table 1.
As mentioned in the introduction, some of the best existing results on these datasets are obtained by using extensive regularization techniques on relatively large LSTMs (Merity et al.
(2018a;b);
Yang et al.
We apply our regularization technique to these models, the so called AWD-LSTM.
We consider two versions of the model – one with a single softmax (AWD-LSTM) and one with a mixture-of-softmaxes (AWD-LSTM-MoS).
The PDR regularization term is computed according to Eq.(4) and Eq.(5).
We call our model AWD-LSTM+PDR when using a single softmax and AWDLSTM-MoS+PDR when using a mixture-of-softmaxes. We largely follow the experimental procedure of the original models and incorporate their dropouts and regularizations in our experiments.
The relative contribution of these existing techniques and PDR will be analyzed in Section 6.
There are 7 hyperparameters associated with the regularizations used in AWD-LSTM (and one extra with MoS).
PDR also has an associated weighting coefﬁcient λ PDR .
For our experiments, we set λ PDR = 0 .
001 which was determined by a coarse search on the PTB and WT2 validation sets.
For the remaining ones, we perform light hyperparameter search in the vicinity of those reported for AWD-LSTM in Merity et al.
(2018a;b) and for AWD-LSTM-MoS in Yang et al. 3. M ODEL AND TRAINING FOR PTB AND W IKI T EXT -  For the single softmax model (AWD-LSTM+PDR), for both PTB and WT2, we use a 3-layered LSTM with 1150, 1150 and 400 hidden dimensions.
The word embedding dimension is set to d = 400 .
For the mixture-of-softmax model, we use a 3-layer LSTM with dimensions 960, 960 and 620, embedding dimension of 280 and 15 experts for PTB and a 3-layer LSTM with dimensions 1150, 1150 and 650, embedding dimension of d = 300 and 15 experts for WT2.
Weight tying is used in all the models. For training the models, we follow the same procedure as AWD-LSTM i.e. a combination of SGD and NT-ASGD, followed by ﬁnetuning. We adopt the learning rate schedules and batch sizes of Merity et al.
(2018a) and Yang et al.
(2017) in our experiments. 3. M ODEL AND TRAINING FOR PTBC AND E NWIK For PTBC, we use a 3-layer LSTM with 1000, 1000 and 200 hidden dimensions and a character embedding dimension of d = 200 .
For Enwik8, we use a LSTM with 1850, 1850 and 400 hidden dimensions and the characters are embedded in d = 400 dimensions. For training, we largely follow the procedure laid out in Merity et al.
(2018b). For each of the datasets, AWD-LSTM+PDR has less than 1% more parameters than the corresponding AWD-LSTM model (during training only).
The maximum observed time overhead due to the additional computation is less than 3%.
R ESULTS ON W ORD L EVEL L ANGUAGE M ODELING
The results for PTB are shown in Table 2.
With a single softmax, our method (AWD-LSTM+PDR) achieves a perplexity of 55.6 on the PTB test set, which improves on the current state-of-the-art with a single softmax by an absolute 1.7 points.
The advantages of better information retention due to PDR is passed on when combined with a continuous cache pointer (Grave et al.
(2016)), where our method yields an absolute improvement of 1.2 over AWD-LSTM.
Notably, when coupled with dynamic evaluation (Krause et al.
, the perplexity is decreased further to 49.3.
To the best of our knowledge, ours is the ﬁrst method to achieve a sub 50 perplexity on the PTB test set with a single softmax.
Note that, for both cache pointer and dynamic evaluation, we coarsely tune the associated hyperparameters on the validation set.
Using a mixture-of-softmaxes, our method (AWDLSTM-MoS+PDR) achieves a test perplexity of 53.8, an improvement of 0.6 points over the current state-of-the-art.
The use of dynamic evaluation pushes the perplexity further down to 47.3.
PTB is a restrictive dataset with a vocabulary of 10K words.
Achieving good perplexity requires considerable regularization.
The fact that PDR can improve upon existing heavily regularized models is empirical evidence of its distinctive nature and its effectiveness in improving language models.
Table 3 shows the perplexities achieved by our model on WT2.
This dataset is considerably more complex than PTB with a vocabulary of more than 33K words.
AWD-LSTM+PDR improves over the current state-of-the-art with a single softmax by a signiﬁcant 2.3 points, achieving a perplexity of 63.5.
The gains are maintained with the use of cache pointer (2.4) and with the use of dynamic evaluation (1.7).
Using a mixture-of-softmaxes, AWD-LSTM-MoS+PDR achieves perplexities of 60.5 and 40.3 (with dynamic evaluation) on the WT2 test set, improving upon the current state-ofthe-art by 1.0 and 0.4 points respectively.
4. P ERFORMANCE ON L ARGER D ATASETS
We consider the Gigaword dataset Chelba et al.
(2014) with a truncated vocabulary of about 100K tokens with the highest frequency and apply PDR to a baseline 2-layer LSTM language model with embedding and hidden dimensions set to 1024.
We use all the shards from the training set for training and a few shards from the heldout set for validation (heldout-0,10) and test (heldout-20,30,40).
We tuned the PDR coefﬁcient coarsely in the vicinity of 0.001. While the baseline model achieved a [Table] [Table]
continuous cache pointer
[Table]
Krause
[Table]
[Table]
Our Method (Single Softmax) AWD [Table] LSTM
[Table]
AWD-LSTM
[Table]
PDR
[Table]
continuous cache pointer [Table] [Table] PDR
[Table]
dynamic evaluation
[Table]
Our Method
[Table]
Mixture-of-Softmax)
AWD
[Table]
validation (test) perplexity of 44.3 (43.1), on applying PDR, the model achieved a perplexity of 44.0 (42.5).
Thus, PDR is relatively less effective on larger datasets, a fact also observed for other regularization techniques on such datasets (Yang et al.
R ESULTS ON C HARACTER L EVEL L ANGUAGE M ODELING
The results on PTBC are shown in Table 4.
Our method achieves a bits-per-character (BPC) performance of 1.169 on the PTBC test set, improving on the current state-of-the-art by 0.006 or 0.5%.
It is notable that even with this highly processed dataset and a small vocabulary of only 51 tokens, our method improves on already highly regularized models.
Finally, we present results on Enwik8 in Table 5.
AWD-LSTM+PDR achieves 1.245 BPC.
This is 0.012 or about 1% less than the 1.257 BPC achieved by AWD-LSTM in our experiments (with hyperparameters from Merity et al.
(2018b)).
[Table] [Table]
A NALYSIS OF PDR
In this section
[Table]
we analyze PDR by probing its performance in several ways and comparing it with current state-of-the-art models that do not use PDR
. 6. A V ALID
[Table]
To verify that indeed PDR can act as a form of regularization, we perform the following experiment.
We take the models for PTB and WT2 and turn off all dropouts and regularization and compare its performance with only PDR turned on.
The results, as shown in Table 6, validate the premise of PDR.
The model with only PDR turned on achieves 2.4 and 5.1 better validation perplexity on
PTB and WT2 as compared to the model without any regularization.
Thus, biasing the LSTM by decoding the distribution of past tokens from the predicted next-token distribution can indeed act as a regularizer leading to better generalization performance.
Next, we plot histograms of the negative log-likelihoods of the correct context tokens x t in the past decoded vector w rt computed using our best models on the PTB and WT2 validation sets in Fig.
1(a).
Indeed, for both the datasets, the NLL values are signiﬁcantly peaked near 0, which means that the past decoding operation is able to decode signiﬁcant amount of information about the last token in the context.
[Figure] [Figure]
To investigate the effect of hyperparameters on PDR
[Figure]
we pick 60 sets of random hyperparameters in the vicinity of those reported by Merity et al.
(2018a)
and compute the validation set perplexity after training (without ﬁnetuning) on PTB, for both AWD-LSTM+PDR and AWD-LSTM.
Their histograms are plotted in Fig.1(b).
The perplexities for models with PDR are distributed slightly to the left of those without PDR.
There appears to be more instances of perplexities in the higher range for models without PDR.
Note that there are certainly hyperparameter settings where adding PDR leads to lower validation complexity, as is generally the case for any regularization method.
6. C OMPARISON WITH AWD-LSTM
To show the qualitative difference between AWD-LSTM+PDR and AWD-LSTM, in Fig.2(a), we plot a histogram of the entropy of the predicted next token distribution w t
+1 for all the tokens in the validation set of PTB achieved by their respective best models.
The distributions for the two models is slightly different, with some identiﬁable patterns.
The use of PDR has the effect of reducing the entropy of the predicted distribution when it is in the higher range of 8 and above, pushing it into the range of 5-8.
This shows that one way PDR biases the language model is by reducing the entropy of the predicted next token distribution.
Indeed, one way to reduce the cross-entropy between x t and w rt is by making w t
+1 less spread out in Eq.(5).
This tends to beneﬁts the language model when the predictions are correct.
[Table] We also compare the training curves for the two models in Fig.2(b) on PTB.
Although the two models use slightly different hyperparameters, the regularization effect of PDR is apparent with a lower validation perplexity but higher training perplexity.
The corresponding trends shown in Fig.
for WT2 have similar characteristics.
6. A BLATION S TUDIES
We perform a set of ablation experiments on the best AWD-LSTM+PDR models for PTB and WT to understand the relative contribution of PDR and the other regularizations used in the model.
The results are shown in Table 7.
In both cases, PDR has a signiﬁcant effect in decreasing the validation set performance, albeit lesser than the other forms of regularization.
This is not surprising as PDR does not inﬂuence the LSTM directly.
R ELATED W ORK
Our method builds on the work of using sophisticated regularization techniques to train LSTMs for language modeling.
In particular, the AWD-LSTM model achieves state-of-the-art performance with a single softmax on the four datasets considered in this paper (Merity et al.
(2018a;b)).
Melis et al.
(2018) also achieve similar results with highly regularized LSTMs.
By addressing the so-called softmax bottleneck in single softmax models, Yang et al. (2017) use a mixture-of-softmaxes to achieve signiﬁcantly better perplexities.
PDR utilizes the symmetry between the inputs and outputs of a language model, a fact that is also exploited in weight tying (Inan et al.
; Press & Wolf (2017)).
Our method can be used with untied weights as well.
Although motivated by language modeling, PDR can also be applied to seq2seq models with shared input-output vocabularies, such as those used for text summarization and neural machine translation (with byte pair encoding of words) (Press & Wolf (2017)).
Regularizing the training of an LSTM by combining the main objective function with auxiliary tasks has been successfully applied to several tasks in NLP (Radford et al.
; Rei (2017)).
In fact, a popular choice for the auxiliary task is language modeling itself.
This in turn is related to multi-task learning (Collobert & Weston (2008)).
Specialized architectures like Recurrent Highway Networks (Zilly et al.
and NAS (Zoph &
Le (2016)) have been successfully used to achieve competitive performance in language modeling.
The former one makes the hidden-to-hidden transition function more complex allowing for more reﬁned information ﬂow.
Such architectures are especially important for character level language modeling where strong results have been shown using Fast-Slow RNNs (Mujika et al.
(2017)), a two level architecture where the slowly changing recurrent network tries to capture more long range dependencies.
The use of historical information can greatly help language models deal with long range dependencies as shown by Merity et al.
; Krause et al.
; Rae et al.
Finally, in a recent paper, Gong et al.
(2018) achieve improved performance for language modeling by using frequency agnostic word embeddings, a technique orthogonal to PDR.