C ONTEXTUAL T EMPERATURE FOR L ANGUAGE M ODELING
Anonymous authors
Paper under double-blind review
A BSTRACT
Temperature scaling is an effective technique to control the smoothness of the output distribution.
Current practices to use temperature scaling assume either a ﬁxed value or a manually-crafted, dynamically changing schedule.
Such a temperature is shared and applied to each output class.
Little has been known that an optimal trajectory of the temperature for each class could be revealed under the change of the context.
In this paper, we propose contextual temperature , a mechanism that allows the temperature to be changed over the context for each vocabulary through the co-adaption with model parameters during the training stage.
Experimental results conﬁrm that contextual temperature improves state-of-the-art language models signiﬁcantly.
The proposed model, CT-MoS, achieves a perplexity of 55 . 31 and 62 . 89 on the test set of Penn Treebank and WikiText-2, respectively.
In-depth analyses show that (a) the schedules of temperatures vary by vocabulary, and (b) optimal temperature trajectories drop as the context being longer in order to suppress the uncertainties.
I NTRODUCTION
The discrete nature of human language makes the Softmax layer a must to convert learned representations into a sequence of linguistic vocabulary.
Together with the Softmax layer, temperature scaling is often used in natural language processing. Generally, the temperature is applied as a denominator to output logits of the Softmax layer (Krizhevsky et al., 2012;
Bahdanau et al., 2014;
Hu et al., 2017; Caccia et al., 2018).
The scale of temperature controls the smoothness of the output distribution. That is, given a temperature τ , when τ → ∞ , the distribution becomes more uniform, thus increasing the uncertainty. Contrarily, when τ → 0 , the distribution collapses to a point mass.
Although temperature scaling has been justiﬁed to achieve great success, existing methods use it in extremely limited ways.
They either assume a constant temperature throughout the training stage (Norouzi et al., 2016; Ma et al., 2017; Chen et al., 2019), or use a ﬁxed schedule to control the temperature (Hu et al., 2017).
Most importantly, none of the existing works studies the effects on different word tokens when the temperature changes.
In reality, however, the temperature of each token can be dramatically different.
Figure 1(a) shows the optimized temperature for each word token during the course of training.
As shown in the ﬁgure, some certain words have a distinct heating-up temperature scaling, while the majority of words have a scaling that gradually cooling down the temperature.
We argue that existing methods limit themselves to some ﬁxed schedules, and thus have great difﬁculty to generalize.
Another example can be seen in Figure 1(b), where the average temperature drops as the length of the context increases.
This suggests that the temperature mechanism helps promote stochasticity in the beginning of a sentence, then gradually suppressing the uncertainty until the end.
All of these suggest the use of a more generalized temperature mechanism with the advantage of being able to deal with these phenomena.
We propose contextual temperature , a generalized temperature scaling method which takes both the history of a sequence and a particular vocabulary into consideration.
Through optimizing the use of temperature scaling by the change of contexts, contextual temperature has the ability to generate a unique temperature for each vocabulary.
As we parameterize contextual temperature using a deep neural network, its parameters co
-evolve with the rest of model parameters, making the temperature schedule adapt to the training procedure.
The experiment results on language modeling demonstrate signiﬁcant improvements on both Penn Treebank and WikiText-2 datasets when experiments are run with and without the ﬁne-tune setting.
Consistent improvements are shown on both validation and testing splits.
In addition, we conduct comprehensive analyses and ablation studies to conﬁrm the improvements of contextual temperature.
We observe that our method is capable of controlling the uncertainties as the patterns of contexts change, allowing language models to achieve much better performances.
To the best of our knowledge, this is the ﬁrst systematic work that studies the role of per-token temperature changing over context for training language models.
The experimental results suggest a new way of training sequential models with discrete outputs, that is, using parameterized temperatures.
We have also established the link between the control of model uncertainty and the use of temperatures, paying the way for extensions on tasks that require such long-term control, for instance summarization, translation and dialogue generation.
[Figure]
R ELATED W ORK 2. L ANGUAGE MODELING
Given a sequence of history tokens x 1 ..t − 1 , language modeling aims at predicting the next token x t in a sentence.
In other words, the goal is to model the probability of choosing the k th word,
P ( x t = k | x 1 ..t − 1 ) , by using an encoding function f and a Softmax layer σ . f is parameterized by θ to compress the history z = f ( x 1 ..t − 1 ; θ ) T · W .
[Equation]
Here W ∈ R
d × K is a matrix that converts a d dimensional output of f into a K dimensional vector.
State-of-the-art language models adopt deep neural networks as the encoding function, including the earlier feed-forward networks (Bengio et al., 2003), more recent recurrent based ones (Mikolov et al., 2010; Hochreiter & Schmidhuber, 1997) and attention based models (Devlin et al., 2018).
2. S OFTMAX L AYER
A Softmax layer σ ( z ) normalizes a K dimension, real-valued vector z to make it sum to 1 .
[Equation]
Recent progress in language modeling has suggested that Mixture of Softmaxes (MoS)
(Yang et al., 2018) would signiﬁcantly improve the performance by computing multiple Softmax distributions and use a set of weights to sum them up as the ﬁnal probability distribution.
To achieve this, a set of M matrices W m is applied to the output generated by f .
That is, z m = f
( x 1 ..t − 1 ; θ ) T · W m , where
W m ∈ R d × K has the dimension of the embedding d and the dimension of the vocabulary K .
The probability distribution of the next word x t under the MoS model is thus a mixture of M Softmaxes weighted by π .
Here Θ = ∪ Mm =1 W m ∪ θ .
[Equation]
2. T EMPERATURE S CALING
Temperature scaling is an approach used to control the smoothness of the distribution.
Instead of applying the Softmax function as suggested in Equation 1, logits here are divided by the temperature τ before passing through the Softmax layer.
[Equation]
Constant Temperature.
Early works that adopt the constant temperature can be traced back to
Model Distillation (Hinton et al., 2014).
Several works have been taking advantage of a ﬁxed temperature during training (Norouzi et al., 2016; Ma et al., 2017; Chen et al., 2019).
For instance, (Norouzi et al., 2016; Ma et al., 2017) optimize log-likelihood on augmented outputs sampled proportionally to their exponential scaled rewards, where the temperature controls the degree of augmentation.
Other works incorporate the temperature to enhance calibration during inference (Guo et al., 2017), or to achieve a great trade-off between quality and diversity (Caccia et al., 2018).
Dynamic Temperature Over Training Iterations.
A wide range of works adopt a manuallycrafted schedule to tune the temperature during training.
Notably in (Hu et al., 2017), the authors introduce a new text generation architecture that combines VAE and a discriminator.
Since text samples are discrete and non-differentiable, a continuous approximation based on the Softmax with a decreasing temperature is used to enable gradient propagation from the discriminator to the generator.
Similar techniques are adopted in the gumbel-softmax trick (Jang et al., 2017), which also allows gradients to pass through discrete sampling objectives.
Research has also shown that temperature with a heating up schedule makes the embedding vectors more compact (Zhang et al., 2018).
Dynamic Temperature Over Word Position.
Another work that is closely related to our work is the adaptive temperature over an attention model (Lin et al., 2018).
We note that contextual temperature further learns the temperature for each vocabulary in the output distribution.
M ETHODS 3. C ONTEXTUAL T EMPERATURE
Contextual temperature is a mechanism that chooses the best temperature by considering the “context” of a word x t .
A context of a word includes not only the history x 1 ..t − 1 but also the speciﬁc vocabulary k that we calculate the probability on.
Such a mechanism allows us to parameterize the temperature vector τ using a deep neural network and adapt the softness of the Softmax layer.
Our temperature vector τ ∈ R K is generated from the same mapping function f as discussed before.
Although f can be any sequential models such as RNN or LSTM, we choose to parameterize it by the AWD-LSTM architecture (Merity et al., 2018).
We omit the details of its architecture due to the limited space in this paper.
The output of AWD-LSTM is a vector with dimension D .
We multiply this vector by two matrices W τ 1 ∈ R D
× Q and W τ 2 ∈ R Q × K .
Please note that one can potentially use a single matrix to represent these two.
However, doing so can signiﬁcantly increase the number of parameters and thus in practice we factorize them into two.
And ﬁnally, we scale temperatures using a Softmax function over the dimension of the vocabulary and its range is bounded in [ αβ , 1+ αβ ].
[Equation]
3. C ONTEXTUAL T EMPERATURE M O S
We then use the Contextual Temperature Mixture of Softmaxes architecture for language modeling (CT-MoS).
The CT-MoS model extends Equation 3 by adding contextual temperature in Equation 5.
Here (cid:11) represents the element-wise division between z m and the temperature vector τ .
The results after division are then sent to the Mixture of Softmaxes model.
Since new parameters are added to the model, the CT-MoS now has a total of parameters Θ = ∪ Mm =
1 W m ∪ θ ∪ W τ 1 ∪ W τ 2 .
[Equation]
[Equation]
t
[Equation]
k
[Equation] [Equation]
t
[Equation]
One thing worth noticing is that comparing to prior works, the proposed contextual temperature model has the ability to adopt a different temperature for (a) different vocabulary in the prediction, (b) different position of the same vocabulary given the history and (c) a tunable parameter that is capable of changing as the training progresses.
The detailed architecture is illustrated in Figure 2, which highlights the difference between the prposed CT-MoS model and the MoS model.
3. T RAINING O BJECTIVES AND L OSS S CALING
We adopt the same regularization techniques in MoS (Yang et al., 2018) and AWD-LSTM (Merity et al., 2018).
Our loss function thus consists of four terms: Cross Entropy ( H ), Activation Regularization (AR), Temporal Activation Regularization (TAR), and Weight Decay (WD).
AR is used to penalize high values of outputs, and TAR is used to prevent outputs from changing too much between timesteps.
WD prevents the model from overﬁtting.
Here γ 1 , γ 2 and γ 3 are constants to scale regularization terms.
And m is the dropout mask (See in Merity et al.
[Equation]
The uniqueness in our setting is the Loss Scaling term (LS).
The use of temperature scaling makes gradients of H disproportional to that of the other three terms, so the scale of H needs to be adjusted accordingly.
A similar phenomenon is reported in (Hinton et al., 2014).
In our case, only gradients of H will be inﬂuenced by temperatures.
As for the other regularization terms, their gradients are not inﬂuenced by temperatures since they do regularization on parameters before temperature scaling.
This difference leads to the unbalance between four objectives.
Therefore, we scale H in order to redress the balance.
We empirically ﬁnd that scaling H with the average of temperatures works well in our setting.
E XPERIMENTS 4. D ATASETS
We evaluate contextual temperature on two widely-used datasets for language modeling: Penn Treebank (PTB) (Marcus et al., 1993; Mikolov et al., 2011) and WikiText-2 (WT2) (Merity et al., 2017).
The PTB dataset contains 929k training, 73k validation and 82k test tokens.
The vocabulary size is capped at 10,000 most frequent unique tokens, while the rest of tokens are replaced with the < unk > token.
We follow common practices to pre-process the dataset (Mikolov et al., 2011): (a) words are lower-cased, (b) numbers are replaced with “N”, (c) newlines are replaced with < eos > and (d) punctuation is removed.
WikiText-2 is derived from Wikipedia articles and released as a popular option to train language models.
WT2 contains 2M training tokens and a vocabulary of around 33k tokens.
Compared to PTB, WT2 is roughly two times larger in sample size and three times larger in vocabulary size.
4. E XPERIMENTAL S ETUPS
We conduct experiments on PTB and WT2 using one and four 1080 Ti GPUs, respectively.
The environment we use is PyTorch (Paszke et al., 2017).
We follow the training conﬁgurations as reported in the MoS paper and their github 1 .
For both PTB and WT2, we use the same number of parameters as the one found in the MoS paper.
We use three layers of LSTM with embedding size of 960-960-620 in PTB experiments.
That is the say, the number of embedding for functional mapping is D = 620 .
The embedding size here is Q = 280 .
For WT2, we use three layers of LSTM with embedding size of 1150-1150-650.
In both experiments, γ 1 , γ 2 , γ 3 for the regularization terms are 2.0, 1.0 and 1.2 e − 6 , respectively.
The number of Softmaxes to be mixed is M = 15 .
To ensure that temperatures are positive, we perform normalization as shown in Equation 5.
We have tried several different settings for α and β , and ﬁnd ( α, β ) =
(1 , 12 ) works best in all experiments.
4. R ESULTS
First, we show experimental results on the PTB dataset in Table 1.
The original MoS model (Yang et al., 2018) has a model size of 22M parameters.
To make a fair comparison, we augment the number of parameters of the MoS model to have 24M parameters and name it Mos + .
This is done by increasing the size of word embedding from 280 to 410 .
We show that our CT-MoS model outperforms AWD-LSTM, MoS and MoS + models on both validation and test sets consistently with lower perplexity.
The conclusion holds for the default setting with ﬁnetuning (Merity et al., 2018), the one without ﬁne-tuning, and the one with dynamic evaluation (Merity et al., 2018).
Our best model achieves 48 .
12 perplexity on the validation set and 47 .
42 on the test set, beating the best state-of-the-art model of MoS with a signiﬁcant margin.
Table 2 provides the results for WT2, a much larger language modeling dataset.
We see a similar pattern to the previous dataset that CT-MoS outperforms the state-of-the-art model on the settings with and without ﬁnetune.
When using dynamic evaluation, our model also achieves comparable results to MoS and AWD-LSTM.
4. A BLATION S TUDIES
Fixed Model Size Comparison.
As we mentioned in the previous section, we increase the number of parameters of the MoS model and name it MoS + to conduct a fair comparison with our model using the same number of parameters.
To construct MoS + with more parameters, we increase the embedding size from 280 to 410 .
In Table 1, we notice that MoS + has a higher perplexity compared to both MoS and CT-MoS, indicating that increasing model parameters cannot improve the performance in this case.
Similar observation and results are also reported by Yang et al.
This ablation study shows that CT-MoS outperforms MoS in the same number of parameters.
Temperature Normalization Methods.
Instead of using the Softmax function in Equation 5 to normalize the temperature between [ αβ , 1+ αβ ], we also consider several alternative temperature nor-
[Table]
[Table] malization methods: (a) λ Tanh ( µ ) , provided by Lin et al.
(2018), makes the range of the temperature to be ( 1 λ , λ ) .
And (b) Tanh( µ ) + λ , which generates a range that is similar to the Softmax method in
Equation 5.
To make our notations simple and clear, we deﬁne µ = f ( x 1 ..t − 1 ; θ ) T · W τ 1 · W τ 2 .
(c) ( σ ( µ ) + α ) /β , which is our method.
Results are illustrated in Table 3.
All of these three methods are evaluated on the PTB dataset [Table] and the experiments are conducted without either ﬁne-tuning or dynamic evaluation for conciseness in this ablation study.
The experimental results show that our proposed method outperforms other temperature normalization methods
[Table]
[Table]
Effects of Loss Scaling.
As discussed in Section 3. [Table] the loss values may need to be scaled since applying the temperature may lead to smaller gradients.
Here, we compare the results of whether applying loss scaling or not in Table 4.
Note that with loss scaling leads to better results (lower perplexity).
[Table]
4. A NALYSIS
Temperature Change During Training.
Going back to Figure 1(a) in the Introduction section, we have shown that contextual temperature is capable of generating a changing temperature as the training proceeds on a per vocabulary basis.
This allows each vocabulary to have a ﬂexible temperature schedule that is optimal to the model.
Tokens that have higher temperatures are common words that don’t convey much information.
This suggests that models are more conﬁdent about these words.
Our method has the advantage of determining the schedule automatically, which is not possible in the traditionally ﬁxed temperature scheduling method.
Average Temperature Over Word Positions.
We also want to dive deeper into Figure 1(b), which shows the mean and conﬁdence interval of the temperatures over different positions of the words.
We group sentences whose length ranges from 15 to 25.
For sentences whose length are greater than 15, we pick the ﬁrst 5 tokens, the middle 5 tokens, and the last 5 tokens to form a new sentence.
At the beginning of a sentence, temperatures are usually high to smooth the probabilities of the Softmax.
This is the region where the model has little conﬁdence since there is too little information in the history.
As the history builds up over time and the model becomes more conﬁdent, temperature begins to cool down, making the probability distribution spiky and trust more on the learned model.
The conﬁdence intervals of the temperature also become tighter as the length of the history goes up.
Word Frequencies And Temperature Change.
Evidence from Figure 1(a) suggests that only a small fraction of the tokens has a signiﬁcantly larger temperature.
However, even small changes in the temperature might deliver large impacts to the performance due to the facts that these tokens might be used more often than others.
To further analyze we present an analysis on the weighted temperature, which equals to the temperature multiplied by the frequencies of the tokens in Figure 3(a).
Here we see that for the majority of the words, small changes in the temperature might still be signiﬁcant as many of them occur fairly frequently.
[Figure]
Case Studies.
To visualize the effects of temperature, we analyze the performane of our model on several samples from the PTB dataset.
Table 5 shows the comparison of CT-MoS
v.s. MoS.
We highlight two differences between CT-MoS and MoS, annotated using red and blue colors.
In the ﬁrst spot, we see that both CT-MoS and MoS fail to predict the correct answer “single-a-1”, which referrs to a rating for securities.
The CT-MoS model predicts “triple-a”, which is not the same as the reference but is much closer to the answer, since “triple-a” refers to the highest rating for securities.
MoS, on the other hand, predicted < unk > , which deviates too much from the ground truth.
Taking a closer look at the temperature, the word “triple-a” has a temperature of 2 + 8 . 34 × 10 − 5 , which is a bit smaller than that of the < unk > of 2 + 8 . 6 × 10 − 5 .
This contributes to the factor that the model chooses “triple-a” over < unk > .
Another example is illustrated by the prediction of the word “standard” of the CT-MoS model.
Here the temperature of “standard” is smaller than that of “s&p”, making the model more likely to predict the prior word
[Table] [Table]
Another indicator of how contextual temperature works is to look at the change of the temperature across different positions in a sentence.
In Table
[Table]
we visualize the occurrence of the word “mortage” and its temperature.
Here we see that as the position changes, contextual temperature chooses a different value for each of the position, adjusting its conﬁdence of model’s belief.
As we analyze before, a general trend is that words appearing early in the sentence get larger temperatures while the ones approaching the end of the sentence get smaller temperatures
[Table]
[Table]
C ONCLUSION AND FUTURE WORK
In this paper
[Table]
we propose contextual temperature, which is capable of generating changing temperatures based on the history of the text.
Our temperature is parameterized using a deep neural network and generates a unique schedule for each vocabulary.
Experiments on the language modeling datasets suggest the advantages of this method on language generation and prediction problems.
Our work opens up potential new research directions along the line of fully automated temperature mechanism to replace the existing practices of manually designed ones.
A A PPENDIX - M ORE S AMPLES FROM PTB
Reference
[Table]