C ALIBRATION , E NTROPY R ATES , AND M EMORY IN
L ANGUAGE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Building accurate language models that capture meaningful long-term dependencies is a core challenge in natural language processing.
Towards this end, we present a calibration-based approach to measure long-term discrepancies between a generative sequence model and the true distribution, and use these discrepancies to improve the model.
Empirically, we show that state-of-the-art language models, including LSTMs and Transformers, are miscalibrated : the entropy rates of their generations drift dramatically upward over time.
We then provide provable methods to mitigate this phenomenon.
Furthermore, we show how this calibrationbased approach can also be used to measure the amount of memory that language models use for prediction
I NTRODUCTION
Recent advances in language modeling have resulted in signiﬁcant improvements on a wide variety of benchmarks (Dai et al., 2018;
Gong et al., 2018; Takase et al., 2018).
Capturing long-term dependencies has especially been a major focus, with approaches ranging from explicit memorybased neural networks (Grave et al., 2016; Ke et al., 2018) to optimization improvements to stabilize learning (Le et al., 2015; Trinh et al., 2018).
However, while these techniques seem to improve on standard metrics like perplexity and even produce remarkably coherent text (Radford et al., 2019), we still do not have appropriate measures to assess long-term properties in language models, making it difﬁcult to choose between different model options for downstream tasks.
Starting from Shannon’s seminal work that essentially introduced statistical language modeling (Shannon, 1951), the most classical and widely studied long-term property of a language model is its entropy rate — the average amount of information contained per word, conditioned on the preceding words.
A learned model provides an upper bound for the entropy rate of a language, via its cross-entropy loss.
The exponential of the entropy rate can be interpreted as the effective support size of the distribution of the next word (intuitively, the average number of “plausible” word choices to continue a document), and the perplexity score of a model (the exponential of the cross entropy loss) is an upper bound for this quantity.
In state-of-the-art models trained on billion-scale corpora, this number ranges between 10 and 30 (Melis et al., 2017; Radford et al., 2019).
A natural diagnostic question, with which we begin our work, is whether the long-term generations of these models exhibit the same entropy rates as the underlying languages they are modeling predictively
[Table] [Table]
Empirically, and perhaps surprisingly, it turns out that the entropy rate of generated text is substantially higher than the estimate for true text derived from the model’s one-step predictions.
As seen in
Table 1 (see also Figure 1), this is true for both state-of-the-art LSTMs and Transformers trained on a variety of datasets.
As a timely example, the GPT-2 model (Radford et al., 2019), the object of much recent attention for its seemingly coherent and on-topic generations, suffers a dramatic degradation in its entropy rate, from 23 .
7 to 61 .
This empirical ﬁnding is notable since the neural attention- and memory-based techniques have been steadily improving on standard metrics like perplexity and, in some cases, even produce remarkably coherent text (often with some heuristics to reject poor generations).
That the perplexity of generated text is so much higher than it is under the true distribution suggests that there are signiﬁcant gaps in our current methodologies in accurately learning language models, particularly if we are interested in generating text that globally resembles the modeled language itself.
Our contributions.
The focus of this work is twofold: to improve generations based on any measurement mismatch on a long-term property of the model (e.g. the entropy rate), and to quantify the way a model’s predictions depend on the distant past.
Central to both of these is a calibrationbased approach, which is utilized in statistics and other areas of machine learning (Dawid, 1982; 1985; Foster, 1991; Zadrozny and Elkan, 2002; Platt, 1999; Guo et al., 2017; Niculescu-Mizil and
Caruana, 2005).
First, we show that, from a worst-case perspective, even an extremely accurate model (with ε average
KL divergence from the true distribution) may have generated text with a substantially different entropy rate as compared to the true distribution.
Indeed, we show that this worst-case ampliﬁcation may occur for a variety of long-term properties of a probabilistic language model; this is because the one-step KL divergence does not in general provide tight control over the expectation of a bounded function.
The observed entropy rate ampliﬁcation (as seen in Table 1) demonstrates that this is not only of theoretical concern.
We then describe a calibration procedure to ﬁx this mismatch while simultaneously improving the perplexity of the language model.
From a statistical perspective, the procedure is simple, and we discuss approaches to make it computationally efﬁcient.
Second, we provide a deﬁnition for long-term memory in language models as the mutual information between the models predictions and the distant past in the input.
We then provide an upper bound on the amount of this mutual information using calibrated distributions (with a single-parameter exponent).
This allows us to estimate the amount of context used by a language model as a function of the distance of past tokens from the current prediction time step.
We perform empirical studies to accompany our theoretical results.
We ﬁrst use the entropy rate calibration algorithm to ﬁx an LSTM language model, resulting in a drop of around 20 perplexity points in the generated text (so that the entropy rate of the model more accurately matches that of the language itself).
Then, we empirically estimate and compare the long-term memory of state-ofthe-art language models.
Our insights point towards new ways of assessing (and ﬁxing) language models, especially in terms of their long-term properties, in a manner complementary to existing metrics like perplexity.
R ELATED W ORK
Improving language modeling with long-term dependencies.
Recent approaches to improving language modeling have focused on several ways to better capture long-term dependencies, from using manually-deﬁned context representations (Mikolov and Zweig, 2012; Ji et al., 2015; Wang and
Cho, 2016) or document-level topics (Wang et al., 2017) to using LSTM recurrent neural networks with careful initialization (Le et al., 2015), auxiliary loss signals (Trinh et al., 2018) or augmented memory structures (Grave et al., 2016; Ke et al., 2018).
Wiseman and Rush (2016) use scoring functions over sequences and search-based optimization to improve generation in seq2seq models.
More recent work has demonstrated the applicability of Transformer networks (Vaswani et al., 2017a) to the task, potentially side-stepping issues in training recurrent networks (e.g. vanishing/exploding gradients) and scaling to longer contexts (Dai et al., 2018; Radford et al., 2018).
All these papers propose either architectural or optimization innovations to improve language model training.
In contrast, we deﬁne and measure explicit long-term properties of language models and show that calibrating them correctly can provide improvements to any black-box language model.
Recent empirical breakthroughs have stemmed from language models which do not specify a unique autoregressive factorization (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019), and thus do not specify a unique (cid:99) Pr .
It remains an interesting problem to identify and sample from distributions induced by these models (Wang and Cho, 2019); thus, our end-to-end theoretical guarantees do not hold in this setting.
Information-theoretic approaches.
While most language models aim to predict a distribution over the next token conditioned on the context, there have been alternative approaches relying on information-theoretic measures.
Jost and Atwell (1994) propose a model which makes use of mutual information between word pairs to generate word sequences that retain longer-term dependencies.
McAllester (2018) propose a training objective based on mutual information for predictive modeling, and demonstrate its application for phoneme prediction.
Clarkson and Robinson (1999) develop a hybrid metric using both perplexity and entropy rate, and show that it correlates better with a downstream metric like word error rate.
Such works propose alternative optimization objectives; in contrast, we show how to use information-theoretic measures to improve models with respect to existing objectives like cross-entropy.
Measuring long-term statistics.
Khandelwal et al.
(2018) analyze LSTM-based language models and empirically show that such models make use of a ﬁnite context for prediction.
Lin and Tegmark (2017) measure mutual information between any two symbols in human languages, and show that it decays with distance, roughly following a power law distribution.
Takahashi and Tanaka-Ishii (2018) provide an upper bound for the entropy (character-level) of human languages by training neural language models with various context and data sizes and extrapolating to inﬁnity.
While we also make use of measures like entropy and mutual information across longer contexts, our goal is to use these to better calibrate the language model and provably improve its perplexity.
Calibration and integral probability metrics
The idea of matching properties of the models’ predictions to the empirical outcomes, in an online setting, goes back (at least) to the “prequential principle” of Dawid (1982; 1985), with subsequent work in online and game-theoretic settings (Foster, 1991; Vovk, 2001; Kalai et al., 1999).
The idea of improving probability scores is also common in machine learning (Zadrozny and Elkan, 2002; Platt, 1999; Guo et al., 2017; Niculescu-Mizil and Caruana, 2005).
Recently, Ott et al.
(2018) assessed model calibration for machine translation systems using word-level probabilities.
The notion of examining the expectation of functions as a metric for the distance between two distributions sometimes goes under the name of integral probability metrics (Mller, 1997; Sriperumbudur et al., 2009), and this notion is becoming increasingly relevant again in unsupervised learning through the connections to GANs (Mroueh and Sercu, 2017).
In this work, we directly focus on the KL divergence, where our use of calibration is largely based on basic facts about exponential families (Brown, 1986).
Relation to generation-improving heuristics.
If the sole objective is to improve the qualitative coherency of sampled generations, a wide variety of heuristics exist in the literature.
The simplest of these is a constant multiplicative adjustment to the model’s logits (known as softmax temperature Xie (2017)).
This is a speciﬁc version of our method (Algorithm 2) with a constant logistic regression feature instead of the next-token conditional entropy.
Relatedly, greedy and top- k sampling (used in state-of-the-art works such as (Radford et al., 2019)) are heuristics which make local modiﬁcations to the model’s conditional probabilities to decrease diversity and eliminate nonsensical generations.
Efforts to push the empirical state of the art in generation quality have given rise to more complex heuristics.
Bengio et al.
(2015) propose retraining the network on its own generations with a carefully scheduled probability for each token.
Some works regularize a model
’s generations with an auxiliary reverse language model Zhang et al.
; Liu et al.
Yet others promote realism using adversarial training protocols (Bahdanau et al., 2016; Lin et al., 2017; Fedus et al., 2018).
We stress that our calibration methods result in a provable improvement in the original training objective (i.e. lower perplexity).
As far as we know, none of the aforementioned heuristic methods can hope to provide such a strong guarantee, since they are fundamentally designed to bias models towards a different objective.
Our work mitigates model hallucinations for (almost) free 1 , in the sense that the global objective (entropy rate drift) is improved without worsening the local objective (perplexity).
Furthermore, calibration preserves the computational efﬁciency of density estimation: a conditional probability vector from Algorithm 2 can be computed using O (vocabulary size) inferences on the original model.
In the more advanced heuristics, the implied distribution over sequences is lost, and is only accessible by black-box sampling.
P RELIMINARIES
We ﬁrst deﬁne some useful quantities for our analyses.
Let Pr( W 1 , W 2 , . .
, W T ) represent the true underlying distribution over T length sequences of words, where the vocabulary is of size M .
Let
W 1: T denote a random sequence of length T , with distribution Pr( W 1: T ) .
For clarity of exposition, we assume that all sequences (i.e. sentences or documents or books) are of equal length T .
For any distributions D and D (cid:48) over length- T sequences, recall that the entropy H ( · ) , KLdivergence, and entropy rate are, respectively, deﬁned by: H ( D ) : = E w 1: T ∼D (cid:104) log
[Equation]
and EntRate( D ) :
= 1 T H ( D ) .
Let (cid:99)
Pr( W 1: T ) denote a learned distribution over sequences.
In the typical sequential prediction setting, the probabilistic model is implicitly deﬁned by the conditional distributions Pr( W t | W <t ) , which are typically efﬁciently computable.
It is standard for such a language model to be trained to minimize the cross entropy objective:
[Equation]
Note that for an accurate language model, we would hope that: CE(Pr (cid:107)
(cid:99) Pr)
≈ EntRate( (cid:99) Pr) , i.e. the entropy rate of the sequences generated under the learned model is nearly that of the cross entropy of the model (with respect to the true distribution Pr ).
Throughout, we assume that
[Equation]
holds for some ε .
In other words, the (unknown) ε measures the degree of sub-optimality of the learned model, this ε is often referred to as the Bayes regret.
C ALIBRATION AND E NTROPY R ATES
In this section, we assess the long-term properties of language models when generating text.
Speciﬁcally, we quantify the ampliﬁcation in the entropy rate of generations under an ε -accurate model (Eq. 1).
We then provide a procedure to ﬁx this ampliﬁcation, without increasing the perplexity of the model.
Proofs for all statements are provided in the supplementary material.
For generality, consider a function f : [ M ] T → R , deﬁned on T length sequences.
Let the mean and variance of f under distribution D be denoted by µ D ( f ) and
σ 2 D ( f )
[Equation]
4. E RROR AMPLIFICATION UNDER OUR MODEL
If our learned model (cid:99) Pr is accurate, we may hope that µ Pr ( f ) ≈ µ (cid:99)
Pr ( f ) i.e. that the expected value of f under the true distribution Pr is close to its expected value under our model.
We can quantify this gap as follows:
[Figure]
[Figure]
e H ( T r a n s f o r m e r )
Lemma 4.1.
(Pinsker’s Inequality (Csiszar and K¨orner, 2011))
Suppose that for all w 1: T , f ( w 1: T ) ≤ B .
Then:
[Equation]
Since this holds for any bounded function, we can obtain the error ampliﬁcation of the entropy rate of (cid:99) Pr simply by choosing f = − log (cid:99) Pr .
Before we proceed, in order to rule out ampliﬁcation of this entropy rate due to arbitrarily small probabilities (which can blow up − log (cid:99) Pr ), it is helpful to deﬁne the γ -mixture distribution as:
D ( γ ) :
= (1 − γ ) D + γ Uni , where the Uni is the uniform distribution over all M T sequences.
We will then consider the model (cid:99) Pr ( ε ) , which has only a minor degradation in the cross entropy compared to (cid:99) Pr , and, yet, may have a large ampliﬁcation in the entropy rate.
Corollary 4.2.
(Entropy rate ampliﬁcation under generations) Suppose the bound in equationholds.
The ε -mixture distribution has KL bounded as:
[Equation]
We have that:
[Equation]
This bound shows that, in the worst case, even a small cross entropy may provide little control over the generations under our model (in terms of entropy rate).
In fact, for ε =
O ( 1 T ) (which we may hope is an accurate model), the bound is vacuous; a later remark shows this worst case bound is unimprovable, see the supplementary material.
The above theorems suggest that entropy rate ampliﬁcation is a theoretical possibility in the worst case, which our experiments show is in fact prevalent in pratice.
These entropy rate ampliﬁcations are evident from the plots in Figure 1.
Regardless of the text corpus or the language model, we observe that the entropy rate under the model’s
generations quickly increases with time, indicating that this is a persistent problem even for state-of-the-art language models while generating text.
4. M ODEL CALIBRATION
We now describe a procedure to ﬁx this error ampliﬁcation.
First, let us deﬁne a distribution (cid:99)
Pr α such that:
[Equation]
· (cid:99) Pr( w 1: T ) .
We can then recover a calibrated model that does not suffer from error ampliﬁcation in f : Algorithm 1 (
Lemma 4.3.
(Calibration to f with model improvement)
Suppose the variance of f is uniformly bounded in that there exists σ 2+ such that the following holds for all α , σ 2Pr α ( f ) ≤ σ 2+ .
Let α ∗ = argmin α CE(Pr (cid:107) (cid:99)
Pr α ) .
We have µ Pr ( f ) − µ (cid:99)
Pr α ∗ ( f ) = 0 , and CE(Pr (cid:107) (cid:99)
Pr α ∗ ) ≤ CE(Pr (cid:107) (cid:99) Pr) − 1 T ( µ ( f ) − µ (cid:99)
Pr ( f )) 2 2 σ+ .
Entropy rate calibration.
We can now apply the previous result to ﬁx the entropy rate ampliﬁcation seen in Table 1.
Note that it is trivial to avoid the entropy rate ampliﬁcation if we were allowed to degrade the quality of our model, in terms of perplexity (e.g. a unigram model does not have this ampliﬁcation.
However, we show that it is possible to match the entropy rate without having to sacriﬁce the quality of our model.
In fact, we can both improve our model and more accurately match the entropy rate, by ﬁtting a family of one-parameter models.
Theorem 4.4.
(Entropy rate calibration)
Suppose equation 1 holds.
Algorithm 1 returns a (cid:99)
Pr α ∗ such that: the following calibration property is satisﬁed:
[Equation]
Furthermore, (cid:99)
Pr α ∗ has entropy close to the true entropy rate as speciﬁed by:
[Equation]
and (cid:99)
Pr α ∗ is an improvement over the original model as characterized by:
[Equation]
This result shows that we simply need a single parameter α to deﬁne a new model class that is a powered up version of our original model.
Then, we can ﬁt this α to minimize the cross-entropy of the new model with respect to the true distribution Pr , in order to eliminate the entropy rate ampliﬁcation.
Even though this algorithm ﬁts only a single parameter, it is not easily implementable since it requires an integration over sequences, at least in its exact form.
One future direction would be to a sample based approach.
This may be an interesting alternative to ideas like beam search (Steinbiss et al., 1994; Ortmanns and Ney, 2000; Antoniol et al., 1995), which also aims to minimize a global cost function on sequences that is inconsistent with the token-level perplexity loss used to train the underlying generative model.
Lookahead algorithms.
In order to sidestep the computational issues of Algorithm 1, we provide another simple approach based on what can be viewed as a “one-step” lookahead correction (Algorithm 2).
Let (cid:99) W t be a random variable with conditional distribution (cid:99) Pr( ·| W <t ) .
H ( (cid:99) W t +1 | w ≤ t ) denotes the entropy of this conditional distribution, i.e.
[Equation]
(cid:99) W t +1 | w ≤ t ) = E w t +1 ∼ (cid:99) Pr( ·| w ≤ t ) (cid:34) log ( cid:99) Pr( w t +1 | w ≤ t ) (cid:35) .
1: Input: Model (cid:99)
Pr ( ε ) , where (cid:99)
W t ∼ (cid:99) Pr ( ε ) ( ·| W <t ) .
2: Deﬁne a model class:
[Equation]
[Equation]
where
[Equation]
3: Fit α ∗ : α ∗ = argmin α CE(Pr (cid:107) (cid:99)
Pr α ) 4:
Return (cid:99)
Pr α ∗
Note that H ( (cid:99) W t +1 | w ≤ t ) includes the word w t , so we require computing the entropy at time t +when predicting W t using a learned model.
For a conditional distribution, D ( W 1: T ) , let us deﬁne:
[Equation]
Thus, ¯ µ D is the average of H ( (cid:99) W t +1 | w ≤ t ) with respect to a distribution which uses D for sampling the last word W t (at every timestep).
Intuitively, the resulting model (cid:99)
Pr α with a positive α would suppress sampling words leading to larger entropy but rather encourage words that stablizes the entropy 1-step ahead in the future.
Therefore, if our learned language model (cid:99) Pr was accurate, we would hope that: ¯ µ Pr ≈ ¯ µ (cid:99) Pr .
The following corollary shows that this is achievable, along with improving the model’s perplexity.
Corollary 4.5.
Suppose Equation 1 holds.
Then, Algorithm 2 returns a (cid:99)
Pr α ∗ such that: ¯ µ Pr − ¯ µ (cid:99)
Pr α ∗ = 0 , and
[Figure]
This result provides us with Algorithm 2, which is computationally quite tractable.
We ﬁrst use the learned model (cid:99) Pr to deﬁne a new model class (cid:99)
Pr α , which scales (cid:99) Pr by an exponential distribution over the weighted 1-step lookahead entropy H ( (cid:99) W t +1 | w ≤ t ) .
Then, similar to Algorithm 1, we simply ﬁt the single parameter α to minimize the cross-entropy of the new model with respect to Pr , which ﬁxes the entropy ampliﬁcation in the resulting model (cid:99)
Pr α .
We observe this empirically in Figure 2 – our calibration results in a perplexity drop of almost 20 points over long-term generations under an
LSTM model.
Model and implementation details are in the supplementary material.
Generations from a calibrated model.
Table 2 provides sample generations from a calibrated Transformer model trained on the GBW dataset, compared to its original version.
Qualitatively, the calibrated generations: (1) are shorter and more concise, and (2) display a better grasp of discourse structure across sentences.
More generations are provided in the supplementary material.
C ALIBRATION AND M EMORY
Deﬁning a notion of memory in language models is challenging, and multiple equally sensible notions may co-exist.
Here we present our choice from ﬁrst principles.
Let us say that (cid:99) W t is a sample
[Table]
Actual results could differ materially from those indicated by these forward-looking statements as a result of various important factors , including , without limitation : changes in general economic and business conditions , including more difﬁ- cult real estate environments ; [...174 tokens...] risks related to investigations by other companies ; inadequate information systems ; the impact of reduced availability of ; * assumptions upon such companies using such as ours to gauge CNET ’s ﬁnancial condition ; and other factors .
Actual results could differ materially from those in- dicated by these forward-looking statements as a result of a variety of factors , including but not lim- ited to ( i ) the risk that the tender offer could close in one or more manner or at all ; ( ii ) risks asso- ciated with conducting business in foreign jurisdictions ; ( iii ) difﬁculties in combining some or all of the businesses under one roof ; ( iv ) decreased demand for electricity , natural gas and other energy products , including adverse effects on the pricing of oil and natural gas ; and ( v ) the risks associated with doing business internationally .
Bluepoint Games , Inc. is a highly experienced and multi-faceted publisher of licensed virtual worlds for gamers , developers and technology professionals .
[...114 tokens...]
James Upon , CEO of MyNetSheltetWeb and the three previous Developers of MySQL .
Based in Redwood City , Califor- nia , BlueMountain is the leader in franchise and game development for the massively multiplayer online game .
Bluepoint Games , Inc. is a highly experienced licensing , gaming and entertainment ﬁrm focused on developing the next generation of casual games based on the PlayStation ( R ) BRAVIA family of video game machines for the North American market .
Bluepoint is a wholly owned subsidiary of Bluehill ID Holdings L.P. W t ∼ (cid:99) Pr( W t | W <t ) . Let us also assume that W <t ∼ Pr( W <t ) .
We will deﬁne the memory at gap τ as the mutual information between (cid:99) W t and the distant past (those words greater than τ steps ago) conditioned on the subsequence W t − τ : t − 1 .
Precisely,
[Equation] where we are not explicitly denoting the t dependence in this deﬁnition 2 . Intuitively, I t can be viewed as how much uncertainty (entropy) in the prediction W t the model is able to reduce by utilizing the deep past W <t − τ in addition to the recent past W t − τ : t − 1 .
The difﬁculty in estimating this mutual information is due to estimating H ( (cid:99) W t | W t − τ : t − 1 ) , which requires the marginalized model
(cid:99) Pr( W t | W t − τ : t − 1 ) .
To (even approximately) marginalize a model distribution
(cid:99) Pr( W t | W <t ) over the deep past W <t − τ is statistically difﬁcult, since it requires the access to a pool of samples of W <t that share an common recent past W t − τ : t − 1 .
Nevertheless, we now show that it is possible to obtain an upper bound (which is computationally efﬁcient to estimate).
Upper bounding mutual information using calibrated models.
In the above, we were considering the mutual information between (cid:99) W t and W <t − τ conditioned on W t − τ : t − 1 .
Let us now consider a more general setting, where we have a distribution
Pr
( Z, Y, X ) where Z , Y , and X are random variables.
We wil eventually consider Z, Y, X to be (cid:99) W t , W t − τ : t − 1 W <t − τ , respectively.
For distributions D ( ·| Y, X ) and (cid:101) D ( ·| Y, X ) and for α ∈ R , deﬁne
[Equation]
We say that D ( ·| Y, X ) is calibrated to ˜ D ( ·| Y, X ) , if D = D α =0 is unimprovable in that for all α
CE(Pr [Equation] Pr [Equation]
Note this condition is achievable due to that calibrating a model to (cid:101) D ( ·| Y, X ) involves a one dimensional (convex) estimation problem (over α ). [Figure]
Theorem 5.1.
Suppose we have a model
(cid:99) Pr( Z | X ) , and suppose (cid:101)
Z ∼ (cid:102)
Pr( ·| X ) , where (cid:101) Z is dependent only on X .
Suppose that (cid:99) Pr is calibrated to (cid:102)
Pr .
Then we have that:
[Equation]
H
[Equation]
Z | Y, X ) , where:
[Equation]
Memory estimation.
We ﬁrst learn another (cid:102) W t ∼ (cid:102)
Pr( ·| W t − τ : t − 1 ) , and then calibrate (cid:99) Pr to (cid:102) Pr .
Corollary 5.2.
Suppose (cid:99)
Pr cal ( ·| W <t ) is a model calibrated to (cid:102) Pr( ·| W t − τ : t − 1 ) .
For a random variable, (cid:99) W cal t ∼ (cid:99)
Pr cal ( ·| W <t ) , we have that:
[Equation]
cal [Equation] [Equation] <
[Equation]
t
[Equation]
, where:
[Equation]
This corollary gives us a means to efﬁciently provide upper bounds on the mutual information.
The key is that since (cid:102)
Pr is efﬁciently computable, we can directly estimate CE(Pr || (cid:102) Pr) through Monte
Carlo estimation.
We measure the upper bounds on I τ of a LSTM model with trained limitedmemory models (cid:102)
Pr (see details in the supplementary material) and report them in Figure 3.
As expected, the memory estimate gradually decays with longer τ , indicating that the models make more use of the recent past to generate text.
C ONCLUSION
We have introduced a calibration-based approach to detect and provably correct the discrepancies between the long-term generations of language models and the true distributions they estimate sequentially.
In particular, for state-of-the-art neural language models, we have observed large degradations of the entropy rate under iterative generation, and a proposed ﬁrst-order correction which is both computationally tractable and effective.
Using the same calibration approach, we have derived estimators for the amount of information extracted by these models from the deep past.
Aside from the empirical ﬁndings and improvements, we hope that this work will inspire a more principled line of discourse on the quality of long-term generations in language models.
It remains an interesting open problem to relate the plethora of “future-aware” generation-improving heuristics to our calibration framework.
A P ROOFS FOR S ECTION
Proof.
(of Lemma 4.1)
We have
[Equation]
where we have used Holder’s and Pinsker’s inequalities Cover and Thomas (2006).
Proof.
(of Corollary 4.2)
First observe:
[Equation]
and that:
[Equation]
For the ﬁrst claim
[Equation]
we have
[Equation]
using our assumption in Equation 1.
For the second claim, taking f = log
[Equation]
Pr ( ε ) ( w 1: T ) with Lemma 4.1, we have:
[Equation]
which completes the proof.
Proof.
(of Lemma 4.3)
By deﬁnition,
[Equation]
we have:
[Equation]
The ﬁrst claim now follows from optimality of α ∗ .
For the second claim,
[Equation]
By Taylor’s theorem, we have:
[Equation]
Taking the the α which minimizes the upper bound, leads to the second claim.
Remark A.1.
(Sharpness)
If ε ≥ 1 T , then there exists a problem where the bound is sharp and
EntRate( (cid:99) Pr) takes on the maximal value of O (log M ) .
As an example, consider a model (cid:99) Pr , that starts by generating words under the true distribution Pr and has a 1 T probability of transitioning into a mode in which it generates words uniformly at random thereafter.
Proof.
(of Theorem 4.4)
We can apply the previous lemma using
[Equation]
and so our calibration condition implies:
[Equation]
Now observe that:
[Equation]
and, similarly,
[Equation]
These imply:
[Equation]
which completes the proof of the ﬁrst claim.
The proof of the second claim uses
[Equation]
and, by Equation 2,
σ 2+ ≤ T log M + log
[Equation]
[Equation]
ε
[Equation]
which completes the proof.
Now we move on to the proof of Corollary 4.5.
Suppose f ( W ≤ t ) be a function of W ≤ t .
For a conditional distribution, D ( W 1: T ) , let us now deﬁne:
[Equation]
Deﬁne:
[Equation]
and
[Equation] [Equation]
Lemma A.1.
Suppose f ≤ σ 2+ .
Let
[Equation]
We have that:
[Equation]
and that
[Equation]
Proof.
(sketch)
The proof is identical to that of Lemma 4.3, with the addition of using linearity of expectation.
B P ROOFS FOR S ECTION
Proof.
(of Theorem 5.1)
It is convenient to deﬁne the distribution:
[Equation]
We then have:
[Equation]
by the deﬁntion of the mutual information
[Equation]
The proof consists of showing that:
[Equation]
Let us take (cid:99)
Pr α
( Z | X, Y ) =
(cid:99) Pr( Z | X, Y ) · (cid:16)(cid:102) Pr( Z | X ) (cid:17) α /Z
α .
The zero gradient condition for the optimality at α = 0 implies:
[Equation]
This implies:
[Equation]
where the last step uses the deﬁnition of (cid:98) Z and Jensen’s inequality.
C E XPERIMENTAL D ETAILS
In this section, we outline the experimental setups used to obtain the empirical results throughout the paper.
For the calibration and memory experiments (Table 1 row 1, Figure 1 (left), Figures 2, 3), our base model is a 3-layer LSTM with with 400 embedding dimension and 1150 hidden nodes.
We train it on the Penn Treebank (PTB) corpus Marcus et al.
(1993), following the setup of Merity et al.
(2017) and Merity et al.
(2018) for 500 epochs using SGD with batch size 20 and BPTT length 70.
The trained base model achieves 64.3 validation perplexity and 58.3 test perplexity.
The limited-memory models (cid:102)
Pr( ·| W t − τ : t − 1 ) used for the memory estimation in Section 5 share the same architecture as our base model while, during training, the hidden states is re-initialized after reading every τ tokens ( τ takes value from { 5 , 15 , . . . , 30 } ).
Finally, for the entropy rate measurements of larger-scale state-of-the-art language models (Table 1 rows 2-4, Figure 1 (right)), we used the pretrained weights published alongside Jozefowicz et al.
; Radford et al.
(2019) for rows 2 and 4, while we trained the model using the tensor2tensor framework.
The model for row 2 is an LSTM with CNN-embedded inputs, trained on the Google Billion Words (GBW) corpus.
The other two are Transformer Vaswani et al.
(2017a) models trained on GBW (row 3), and an proprietary corpus derived from a web crawl (WebText; row 4).
For GPT-2, since the authors have not published training or validation data, we used the text of several New York Times articles as a stand-in validation set; the cross entropy loss is comparable to that reported on the validation set.
The entropy rate ampliﬁcation plot in Figure(bottom) corresponds to the setup from row 4.
To measure the conditional entropy after t generations, we measured the empirical conditional entropy of the t -th word over > 500 independent generations, which were produced by the standard way of iteratively sampling from the next predicted conditional distribution, seeded with groundtruth text up to > 100 random points in the validation set.
We used the entropy rate at t = 700 as a proxy for the asymptotic limit in Table 1.
D A DDITIONAL G ENERATION S AMPLES
In this section, to provide a better sense of the qualitative effect of calibration, we provide below some additional generations, seeded by 10-token preﬁxes of the holdout (validation) sentences from the Google Billion Words dataset.
Here, we used the model we trained for row 3 of Table 1.
To identify a failure mode for the uncalibrated model, we selected the seed preﬁxes which resulted in unusually long generations by the uncalibrated model.
Original model
[Table]
Actual results could differ materially from those indicated by these forward-looking statements as a result of numerous factors including the risks associated with the timely and efﬁcient completion and integration of the Temporary Liquidity Guarantee Department ’s supervision into the commercial , open market , solar energy , energy efﬁciency , electric utility transmission , and water demands of residential and business customers , Comcast ’s ability to successfully implement its business plan , timing of completion of the acquisition and the effectiveness of the efforts and strategies involved in the integration of Rhapsody , timing of regulatory and client approvals and availability of key enhancements .
Actual results could differ materially from those indicated by these forward-looking statements as a result of a variety of factors , including but not limited to ( i ) the risk that the tender offer could close in one or more manner or at all ; ( ii ) risks associated with conducting business in foreign jurisdictions ; ( iii ) difﬁculties in combining some or all of the businesses under one roof ; ( iv ) decreased demand for electricity , natural gas and other energy products , including adverse effects on the pricing of oil and natural gas ; and ( v ) the risks associated with doing business internationally .
Actual results could differ materially from those indicated by these forward-looking statements as a result of various important factors , including , without limitation : changes in general economic and business conditions , including more difﬁcult real estate environments ; declines in information technology spending ; continued availability of capital and government regulations ; changes in general economic and business conditions ; the possibility that extended unemployment and healthcare policies may change , or may reduce access to quality care services ; failure to obtain adequate and affordable medications ; changes in certain CME / CE product mix ; disruption in
CME credit markets ; uncertainty of the outcomes of regulatory investigations of companies in which the Company has an interest ; dependence on suppliers for most of its products ; consolidation among ﬁnancial institutions ; ability to attract and retain skilled personnel ; changes in rapidly changing technology and regulatory environments ; arrogance and complacency among ﬁnancial analysts ; the impact of competition ; inability to retain and motivate senior management ; difﬁculties in the integration of acquired businesses ; the effects of redundancy and loss of key employees ; litigation , including claims and the challenge of insurance practices ; uncertainties relating to litigation ; risks related to investigations by other companies ; inadequate information systems ; the impact of reduced availability of ; * assumptions upon such companies using such as ours to gauge CNET ’s ﬁnancial condition ; and other factors .
Actual results could differ materially from those indicated by such forward-looking statements as a result of various important factors , including those discussed in the company ’s periodic reports that are ﬁled with the Securities and Exchange Commission and available on the SEC ’s website at www.sec.gov.
Actual results could differ materially from those indicated by such forward-looking statements as a result of a variety of factors , including our ability to improve our liquidity .
Among these factors are changes in the general economy , changes in political and economic conditions , changes in interest rates , changes in technology and implementation of regulatory policies and legislation , the direction of interest rates and changes in the banking industry , changes in loan prepayment activity , changes in consumer preferences and consumer and business lending markets , legislation or public compliance with applicable laws and regulations and changes in the business or regulatory environment .
We caution you that there are many uncertainties that could cause actual results to differ materially from those indicated in the forward-looking statements .
Among them are the risk factors that could cause results to differ from those expressed in the forward-looking statements .
These factors include , but are not limited to : general economic and business conditions , including the ﬁnancial markets ; ﬂuctuations in interest rates ; government regulation of the ﬁnancial services industry and possible failures ; planning assumptions and estimates ; potential funding requirements ; unexpected changes in cost increases ( including goodwill impairment ) ; competition ; the potentially lengthy , protracted U.S. recession ; and migratory consumer and business conditions .
Actual results could differ materially from those indicated by these forward-looking statements as a result of various important factors , including those discussed in the ” Risk Factors ” section of the Company ’s Annual Report on
Form 10-K for the most recently ended ﬁscal year .
Bluepoint Games , Inc. is a highly experienced and multi-faceted publisher of licensed virtual worlds for gamers , developers and technology professionals .
The company is based in Vancouver , Canada .
BlueKai ’s innovative games are distributed by Devices EA , LLC , and Club
Penguin .
BlueKai owns and is the exclusive licensor of Scrabulous .
BluetoothQ
Interactive
Inc. has acquired JoShear-Swain Media , LLC , a premier developer and publisher of community based games for the handheld game device .
For further information , please visit : www.netgear.com / ngcleveld .
Sprint ’s fantasy game publisher and Web doing business within the Entertainment Group is James Upon , CEO of MyNetSheltetWeb and the three previous Developers of MySQL .
Based in Redwood City , California , BlueMountain is the leader in franchise and game development for the massively multiplayer online game .
Bluepoint Games , Inc. is a highly experienced gaming and entertainment company with several renowned blockbuster franchises including
PC , GameHouse ( ( R ) )
GameHouse ( ( R ) ) , Heavenly Sword ( ( TM ) ) , EverQuest ( R ) , Untold Story ( TM ) and EverQuest ( R ) II .
Through its wholly-owned subsidiary , Bluehill ID ( R ) , the Bluehill ID logo and tagline are registered trademarks of Bluehill ID Corporation and its subsidiaries in the U.S. and in other countries .
Bluepoint Games , Inc. is a highly experienced gaming , entertainment and mobile games company with a vertically integrated portfolio including : games ( TM ) , social network , mobile , casual games , MMORPG , production , distribution , and licensing including its ﬂagship games , SUIT and TIMMERIX ( TM ) , as well as its award-winning gaming , basketball and entertainment network .
In order to create a highly integrated , pure and socially responsible Game ( R ) family , Bluepoint has collaborated with Amplify Systems International , Inc. on various titles for PlayStation ( R ), PLAYSTATION 3 ( R ) 5 , Wii ( TM ) 3 ,
PS3 , Wii ( TM ) ( and PS3 titles ) as well as
PC games for PC , PSP , POOL , Wii ( TM ) ( and successor title ) and IP ( R ) , in addition to its focused gaming , entertainment and communication services .
BlueBay ’s exclusive licensee worldwide licensee of the Bluepoint ( TM ) ZMFAO Gateway series , it is the world ’s leading portable gaming , PC and mobile phone company .
For more information , see UNK , Inc. and ” Oakpoint : ZWC ’s Community Health Business Development Center .
Bluepoint Games , Inc. is a highly experienced licensing , gaming and entertainment ﬁrm focused on developing the next generation of casual games based on the PlayStation ( R )
BRAVIA family of video game machines for the North American market .
Bluepoint is a wholly owned subsidiary of Bluehill ID Holdings L.P.
Bluepoint Games , Inc. is a highly experienced , innovative entertainment sports gaming company whose products and services are used by some of the most recognized and respected names in the world of gaming including : Pokemon , Macau ( Valve ) , Quattro , Super Smash Bros. , Good Neighbor
Games , IGN Games , Vail Resorts , Kania (
Ocean Spray , Pemberton and Roatenham ) ,
PURE Holdings , TeenNick , National Amuse-
ments , SEGA Games , Cirrus ( Aircraft ) and www.netapool.com.
Bluepoint Games , Inc. is a highly experienced player in the growing genre of casual games for both casual and active gaming enthusiasts .
Bluepoint is an early stage Company with a signiﬁcant following among youth and adults in Europe and the United States with an impressive track record in global on-line gaming opportunities .
Nursing Homes : Genworth ’s 2009 Cost of
Care Survey , conducted by the Robert Wood
Johnson Foundation and released today , reveals the extent to which members of the U.S. population adheres to practices recommended since 1995 , including : a rolling three-hour ”
Python for Life ” that fell asleep from 11 p.m. to 2 a.m. , sleep time from 11 p.m. to 3 a.m. , spare time from 8 a.m. to 9 p.m. , and use of state-of-the art non-invasive technologies .
A remodeling and refurbishment of hospital facilities is underway as the nation ’s economy begins to gain momentum .
Similar to the previous years , Thinking About Health - Hear how health plans are working to address various congressional proposals to advance best practices in patient care and provide greater accountability , advocacy and transparency to consumers .
Nursing Homes : Genworth ’s 2009 Cost of
Care Survey is based on interviews withfamily , friends and neighbors of insured and self-employed people conducted from Jan .
Nursing Homes : Genworth ’s 2009 Cost of
Care Survey is based on a double-blind , randomized , double-blind , placebo-controlled survey which involved an assessment of the cost-effectiveness of healthcare associated with an adequate diet and regular physical activity compared to its managed-care counterparts .
The margin of error for this survey is + / - 3.3 percentage points at the 95 percent level of conﬁdence .
Nursing Homes : Genworth ’s 2009 Cost of
Care Survey , conducted by Harris Interactive , performed signiﬁcantly worse than a control group of its peers who provided care but were not able to offer health care to their employees .
Nursing Homes : Genworth ’s 2009 Cost of
Care Survey , conducted by CareScout ( R ) and published in the April 2009 issue , evaluated ﬁndings from the 10-year , nearly 900,000-member Specialty Health Management Association ’s more than 6,000 professionals living in the United States .
Nursing Homes : Genworth ’s 2009 Cost of
Care Survey includes a series of health and medical cost reports on more than 100 home medical equipment and related products , including more than 3.9 million units of durable medical equipment .
IBC ’s cost of more than $ 100 billion is a signiﬁcant portion of Medicare spending on home health care .