A NALYZING AND I MPROVING THE O PTIMIZATION L ANDSCAPE OF N OISE -C ONTRASTIVE E STIMATION
A BSTRACT
Noise-contrastive estimation (NCE) is a statistically consistent method for learning unnormalized probabilistic models.
It has been empirically observed that the choice of the noise distribution is crucial for NCE’s performance.
However, such observations have never been made formal or quantitative.
In fact, it is not even clear whether the difﬁculties arising from a poorly chosen noise distribution are statistical or algorithmic in nature.
In this work, we formally pinpoint reasons for NCE’s poor performance when an inappropriate noise distribution is used.
Namely, we prove these challenges arise due to an ill-behaved (more precisely, ﬂat) loss landscape.
To address this we introduce a variant of NCE called eNCE which uses an exponential loss and for which normalized gradient descent addresses the landscape issues provably when the target and noise distributions are in a given exponential family.
I NTRODUCTION
Noise contrastive estimation (NCE) is a method for learning parameterized statistical models (Gutmann & Hyvärinen, 2010; 2012).
To estimate a distribution P ∗ , NCE trains a discriminant model to distinguish between samples of P ∗ and a known distribution Q of our choice, often referred to as the “noise” distribution.
If the function class for the discriminant model is representationally powerful enough, the optimal model learns the density ratio p ∗ /q , from which we can extract the density p ∗ since q is known (Menon & Ong, 2016; Sugiyama et al., 2012).
Compared to the well-studied maximum likelihood estimation (MLE), NCE avoids calculating the (often intractable) partition function,while maintaining the asymptotic consistency of MLE (Gutmann & Hyvärinen, 2012).
It is empirically well-documented that the choice of the noise distribution Q is crucial to both the statistical and algorithmic efﬁciency of NCE (Gutmann & Hyvärinen, 2010; 2012; Rhodes et al., 2020;
Goodfellow et al., 2014; Gao et al., 2020).
However, it has been observed in practice that even when following the standard guidelines for choosing Q , NCE can still yield parameter estimates far from the ground truth (Rhodes et al., 2020; Goodfellow et al., 2014; Gao et al., 2020).
Most recently,
Rhodes et al.
identiﬁed a phenomenon they call the “density chasm,” observing empirically that NCE performs poorly when the KL divergence between P ∗ and Q is large.
One example is when P ∗ , Q are both tightly concentrated unimodal distributions with faraway modes; the region between the two modes will have a small density under both distributions, thus forming a “chasm”.
While it makes intuitive sense that NCE does not perform well under such settings—since disparate
Q and P ∗ are easy to distinguish and do not require the model to learn much about P ∗ in order to do well on the classiﬁcation task—there has not been a theoretical analysis of this phenomenon.
In fact, it is not even clear whether the difﬁculty is statistical or algorithmic in nature.
In this work, we formally study the challenges for NCE with a ﬁxed Q with a focus on distributions in an exponential family.
We show that when the noise distribution Q is poorly chosen, the loss landscape can become extremely ﬂat: in particular, even when P ∗ and Q are two univariate
Gaussian with unit variance, the loss gradient and curvature can become exponentially small in the difference in their means.
We prove that this poses challenges for standard ﬁrst order and even second-order optimization methods, forcing them to take an exponential number of steps to converge to a good parameter estimate.
Thus, standard approaches to minimizing convex functions such as gradient descent—or even more advanced techniques such as momentum or Newton’s method—are not suited to the NCE objective unless Q is close to P ∗ in KL sense.
To remedy this issue, we study an alternative method for optimizing the NCE objective.
We consider instead Normalized Gradient Descent (NGD) whereby the gradient is normalized to have unit norm at each time step.
Perhaps surprisingly, we prove that this small modiﬁcation can overcome the problem of poor curvature in the Gaussian example.
In general, we show the number of steps for
NGD to converge to a good solution for the NCE loss depends on the condition number κ of the
Hessian of the loss at the optimum—the growth of this condition number is unclear for P ∗ and Q when they belong to an exponential family.
To address this, we propose the eNCE loss, a variant to NCE that replaces the log loss in NCE with an exponential loss, and we show that the resulting condition number is polynomial in the dimension and the parameter distance between P ∗ and Q when they belong to an exponential family.
Our proposed change of loss and optimization algorithm together form the ﬁrst solution that provides a provable polynomial rate for learning the parameters of the ground truth distribution.
Theoretically, both NCE and eNCE can potentially suffer from numerical issues during optimization when P ∗ and
Q are far—this is an interesting direction for future work.
Nonetheless, we ﬁnd this to be a simple and effective ﬁx to the ﬂatness of the loss landscape in many settings, as evidenced by experimental results on synthetic and MNIST dataset.
1. R ELATED W ORK
NCE and its variants have inspired a large volume of research in NLP (Mnih & Teh, 2012; Mnih &
Kavukcuoglu, 2013; Dyer, 2014; Kong et al., 2020) as well as computer vision (Oord et al., 2018;
Hjelm et al., 2018; Henaff, 2020; Tian et al., 2020).
It has been observed empirically that NCE with a ﬁxed noise Q is often insufﬁcient for learning good generative models.
The predominant class of approaches that have been proposed to overcome this issue aim to do so by not using a ﬁxed
Q but by iteratively solving multiple NCE problems with an updated Q , or equivalently updated discriminators.
This includes the famous generative adversarial network (GAN) by Goodfellow et al.
(2014), which uses a separate discriminator network updated throughout training.
In a similar vein, Gao et al.
(2020) also aimed to increase the discriminative power as the density estimator improves, and parameterize Q explicitly with a ﬂow model.
More recently, Rhodes et al.
(2020) proposed the telescoping density ratio estimation, or TRE, which sidesteps the chasm by expanding p ∗ /q into a series of intermediate density ratios, each of which is easier to estimate, leading to strong empirical performance—though their work carries no formal guarantees.
With respect to a ﬁxed Q , it remains an open question about what formally are the nature of the challenges posed by a poorly chosen Q , which could be statistical and/or algorithmic.
Various previous works have analyzed the asymptotic behavior of NCE and its variants (Gutmann & Hyvärinen, 2012; Riou-Durand et al., 2018; Uehara et al., 2020), but these do not provide guidance on the ﬁnite step convergence of NCE or its common variants.
The improvements to NCE in prior works are all borne out by the empirical observations of NCE practitioners, rather than motivated by theory, which is precisely the aim of this work.
Finally, we would like to note that prior work has proposed “generalized NCE" (Pihlaja et al., 2010;
Gutmann & Hirayama, 2011; Uehara et al., 2020), which relates the NCE objective to minimizing the Bregman divergence.
Generalized NCE says that we can design a family of training objectives by using different convex functions to deﬁne the Bregman divergence, and the proposed eNCE is an instance of the generalized NCE objective.
The difference between these prior work and ours is again the different focuses on asymptotic behavior versus ﬁnite step convergence.
P RELIMINARIES
The NCE objective Let P ∗ denote an unknown distribution in a parametric family { P θ } θ ∈ Θ , for some bounded convex set Θ , with P ∗ =
P θ ∗ .
Our goal is to estimate P ∗ via P θ for some θ ∈ Θ by solving a noise contrastive estimation task.
The noise distribution Q belongs to the same parametric family with parameters θ q ∈ Θ , so that Q = P θ q .
We use p θ , p ∗ , q to denote the probability density functions (pdfs) of P θ , P ∗ , and Q ; we may omit θ in P θ , p θ when it is clear from the context and write P, p instead.
Given P ∗ and Q , the NCE loss of P is deﬁned as follows:
Deﬁnition 2.1 (NCE Loss) .
The NCE loss of P θ w.r.t.
data distribution P ∗ and noise Q is:
[Equation]
The NCE loss can be interpreted as the binary cross-entropy loss for the classiﬁcation task of distinguishing the data samples from the noise samples.
Moreover, the NCE loss has a unique minimizer:
Lemma 2.1 (Gutmann & Hyvärinen 2012) .
The NCE objective in Deﬁnition 2.1 is uniquely minimized at P =
P ∗ , provided that the support of Q covers that of P ∗ .
Exponential family.
We focus our attention on the exponential family, where the pdf for a distribution with parameter θ is p θ ( x ) = exp (cid:16) θ (cid:62) ˜ T ( x )
− A
( θ ) (cid:17) , with ˜ T ( x ) denoting the sufﬁcient statistics and A ( θ ) the log partition function.
The partition function is treated as a parameter in
NCE, so we use τ to denote the extended parameter, i.e. τ :=
[ θ, α ] where α is the estimate for the log partition function.
We accordingly extend the sufﬁcient statistics as T ( x ) =
[ ˜ T ( x ) , − 1] to account for the log partition function.
The pdf with the extended representation is now simply p τ ( x ) =
exp( τ (cid:62) T ( x )) .
We will use the notation P θ and P τ interchangeably.
We will also use τ ( θ ) to denote the log-partition extended parameterization when the log partition function α properly normalizes the distribution speciﬁed by θ .
A compelling reason for focusing on the exponential family is the observation that the NCE loss is convex in the parameter τ :
Lemma 2.2 (NCE convexity) .
For exponential family p θ,α ( x ) = h ( x ) exp( θ (cid:62) ˜ T
( x ) − α ) , the NCE loss is convex in parameter τ :=
[ θ, α ] .
Lemma 2.2 has been stated under more general settings by Uehara et al.
; an alternative selfcontained proof is included in Appendix A for completeness.
Recall that Θ denotes the set of parameters without the extended coordinate for the log partition function.
We assume the following on distributions supported on Θ :
Assumption 2.1 (Bounded parameter norm) .
(cid:107) θ (cid:107) 2 ≤ ω , ∀ θ ∈ Θ .
Assumption 2.2 (Lipschitz log partition function) .
Assume the log partition function is β ZLipschitz, that is, ∀ θ 1 , θ 2 ∈ Θ , | log Z ( θ 1 ) −
log Z ( θ 2 ) |≤ β Z (cid:107) θ 1 − θ 2 (cid:107) .
Assumption 2.3 (Bounded singular values of the population Fisher matrix) .
There exist λ max , λ min > 0 , such that ∀ θ ∈ Θ , we have σ max ( E θ
[ T ( x ) T ( x ) (cid:62) ]) ≤ λ max , and σ min ( E θ
[ T ( x ) T ( x ) (cid:62) ]) ≥ λ min .
Assumption 2.4 (Smooth change in the Fisher matrix) .
Assume the maximum and minimum singular values of the Fisher matrix change smoothly.
Namely, there exist constants γ max , γ min > 0 s.t.
[Equation]
Assumptions 2.2-2.4 can be viewed as smoothness assumptions on the ﬁrst, second and third order derivatives of the log partition function, and can be viewed as introducing structural parameters of the distributions.
For example, distributions with ﬂatter tails will have a larger λ max , which then translates to a slower rate in the results; distributions closer to being singular will have a smaller λ min , etc.
In particular, Assumption 2.3 says the singular values of the Fisher matrix E θ [ T ( x ) T ( x ) (cid:62) ] should be bounded from above and below.
It can be shown that the Fisher matrix is proportional to the Hessian of the NCE objective when using Q =
P ∗ , which means Assumption 2.3 can be interpreted as saying the NCE task can be solved efﬁciently under the optimal choice of Q .
O VERVIEW OF RESULTS
We ﬁrst provide an informal overview of our results, focusing on learning of exponential families.
Flatness of population landscape:
Our ﬁrst contribution is a negative result identifying a key source of difﬁculty for NCE optimization to be an ill-behaved population landscape.
We show that due to an extremely ﬂat landscape, gradient descent or Newton’s method with standard choices of step sizes will need to take an exponential number of steps to ﬁnd a reasonable parameter estimate.
We emphasize that though Gaussian mean estimation is a trivial task, its simplicity strengthens the results above : we are proving a negative result so that failures with a simpler setup means a stronger result.
Moreover, the results only apply to standard choices of step sizes, such as inversely proportional to the smoothness for gradient descent, or to the ratio between the smoothness and strong convexity for Newton’s method.
This does not rule out the possibility that a cleverly designed learning rate schedule or a different algorithm would work efﬁciently; the results are however still meaningful since gradient descent with standard step sizes is the most common choice in practice.
Overcoming ﬂatness using normalized gradient descent: Our second contribution is to show that the ﬂatness problem can be solved by a simple modiﬁcation to gradient descent if the loss is well-conditioned.
Speciﬁcally, we show that the convergence rate for normalized gradient descent is polynomial in the parameter distance and κ ∗ , the condition number of the Hessian at the optimum.
One immediate consequence is that in Gaussian mean estimation, for a target error of δ ∈ (0 , 1 β Z ] in parameter distance, NCE optimized with NGD achieves a rate of O ( 1 δ 2 ) , which is the same as the optimal rate achieved by MLE.
The remaining question is then whether κ ∗ is polynomial in the parameters of interests.
We show that κ ∗ can be related to the Bhattacharyya coefﬁcient between P ∗ and Q , which indeed grows polynomially in parameter distance under certain assumptions as detailed in Section 5.2.
Polynomial condition number for the eNCE loss: Our third and ﬁnal contribution is that if we modify the NCE objective slightly—namely, use the exponential loss in place of the log loss—then the condition number at the optimum is guaranteed to be polynomial.
We call this new objective eNCE .
Combined with the NGD result, we get that running NGD on the eNCE objective achieves a polynomial convergence guarantee.
We then provide empirical evidence on synthetic and MNIST dataset that eNCE with NGD performs comparatively with NGD on the original NCE loss, and both outperform gradient descent.
F LATNESS OF THE NCE LOSS
In this section, we study the challenges posed to NCE when using a badly chosen ﬁxed Q .
The main thrust of the results is to show that both algorithmic and statistical challenges can arise because the
NCE loss is poorly behaved , particularly for ﬁrst- and second-order optimization algorithms: when
P ∗ , Q are far, the loss landscape is extremely ﬂat near the optimum.
In particular, the gradient has exponentially small norm and the strong convexity constant decreases exponentially fast, limiting the convergence rate of the excess risk.
We further show that when moving from P = Q to P = P ∗ , the loss drops from Θ(1) to a value that is exponentially small in terms of the distance between
P ∗ and Q .
Consequently, common gradient-based and second order methods will take exponential number of steps to converge.
An important note is that our analysis is at the population level, implying that the hardness comes from the landscape itself regardless of the statistical estimators used.
Setup – Gaussian mean estimation : For the negative results in this section, let’s consider an exceedingly simple scenario of 1-dimensional, ﬁxed-variance Gaussian mean estimation.
We will demonstrate the enormous difﬁculty of achieving a good parameter estimate, even for such a simple problem—this bodes ill for NCE objectives corresponding to more complex models in practice, which certainly pose a much more difﬁcult challenge.
In particular, let P ∗ , Q, P be Gaussians with identity variance.
Let θ ∗ , θ q , θ denote the respective means, with θ ∗ being the target mean that NCE aims to estimate.
When the covariance is known to be 1, we can denote h ( x ) :=
exp( − x 2 2 ) , and parametrize the pdf of a 1d Gaussian with mean θ as p ( x ) = h ( x ) exp( (cid:104) τ
( θ ) , T ( x ) (cid:105) ) , 3 where the parameter is τ ( θ ) :=
[ θ, θ 2 2 + log √ 2 π ] and the sufﬁcient statistics are T ( x ) :=
[ x, − 1] .
4 We will shorthand τ ( θ ) when it is clear from the context.
Without loss of generality, we will assume θ q = 0 , and θ ∗ > 0 , and denote R :
= θ ∗ − θ q .
We will write τ ∗ :
= τ ( θ ∗ ) =
[ R, R 2 2 + log √ 2 π ] , and τ q := τ ( θ q ) =
[0 , log √ 2 π ] .
As a clariﬁcation, the results stated in this section will be in terms of R , hence the asymptotic notations Ω , O never hide dominating dependency on R .
4. P ROPERTIES OF THE NCE LOSS
We ﬁrst describe several properties of the NCE loss that will be useful in the analysis of ﬁrst- and second-order algorithms.
To start, we show that the dynamic range of the loss is large: that is, the optimal NCE loss is exponentially small as a function of R ; on the other hand, if θ is initialized close to θ q , the initial loss would be on the order of a constant.
Precisely:
Proposition 4.1 (Range of NCE loss) .
Consider the 1d Gaussian mean estimation task with mean θ ∗ , θ q ∈ R , and a known variance of 1.
Denote R := | θ q − θ ∗ | where R (cid:29) 1 , Then, the loss at θ = θ q is log 2 , while the minimal loss L ∗ is L ∗ ( R ) =
c exp( − R 2 / 8) for some c ∈ [ 12 , 2] .
The next shows we need to decrease the loss to be on an order comparable to the optimum value.
Namely, the loss is very ﬂat close to θ ∗ , thus in order to recover a θ close to θ ∗ , we have to reach a very small value for the loss.
Precisely:
Proposition 4.2.
Under the same setup as Proposition 4.1, for a given δ ∈ (0 , 1) , if the learned parameter τ satisﬁes (cid:107) τ − τ ∗ (cid:107) 2 ≤ δ , then L ( τ ) − L ( τ ∗ ) ≤ R exp( − R 2 / 8) δ 2 .
The way we will leverage Propositions 4.1 and 4.2 to prove lower bounds is to say that if the updates of an iterative algorithm are too small, the convergence will take an exponential number of steps.
Proposition 4.2 is proven via the Taylor expansion at θ ∗ : since the gradient is 0 at θ ∗ , we just need to bound the Hessian at θ ∗ .
We show:
Lemma 4.1 (Smoothness at P =
P ∗ ) .
Under the same setup as Proposition 4.1, the smoothness at
P =
P ∗ is upper bounded as σ max ( ∇ 2 L ( τ ∗ )) ≤ R √ 2 π exp( − R 2 / 8) .
We will also need a bound on the strong convexity constant (i.e. smallest singular value) at P
= P ∗ :
Lemma 4.2 (Strong convexity at P = P ∗ ) .
Under the same setup as Proposition 4.1, the minimum singular value at P = P ∗ is σ ∗ min ( ∇
[Equation]
[Equation]
Finally, in order to estimate the choice of the step size for standard optimization methods, we will also need a bound of the smoothness at P = Q :
Lemma 4.3 (Smoothness at P = Q ) .
Under the same setup as Proposition 4.1, the smoothness at
P = Q is lower bounded as σ max ( ∇ 2 L ( τ q )) ≥ R 2 2 .
Lemma 4.1, 4.2 are proved in Appendix D.4, and Lemma 4.3 is proved in Appendix D.5.
4. L OWER BOUNDS ON FIRST - AND SECOND - ORDER METHODS
With the landscape properties at hand, we are now ready to provide lower bounds for both ﬁrst-order and second-order methods.
For ﬁrst-order methods, we show that:
Theorem 4.1 (Lower bound for gradient-based methods) .
Let P ∗ , Q, P be 1d Gaussian with variance 1.
Assume θ q = 0 , θ ∗ > 0 without loss of generality, and assume R :
= θ ∗ − θ q (cid:29) 1 .
Then, gradient descent with any step size η = o (1) from an initialization τ = τ q will need an exponential number of steps to reach some τ (cid:48) that is O (1) close to τ ∗ .
Note, the maximum step size η = o (1) the theorem applies to is actually a loose bound: the standard setting of step size for gradient descent is η ≤ 1 /λ M for λ M := max θ ∈ Θ σ max ( ∇ 2 L ( τ ( θ ))) , which is Ω( R 2 ) by Lemma 4.3.
Theorem 4.1 helps explain why NCE with a far-away Q fails in practice, if we set the budget for the number of updates to be polynomial.
A natural remedy to the drastically changing norms of the gradients is to use methods that can properly precondition the gradient.
This motivates the use of second order methods, which adapt to the geometry of the loss and hence can potentially perform more competitively.
Unfortunately, standard second-order approaches are again of no help, and the number of steps required to converge remains exponential.
Consider Newton’s method with updates of the form η ( ∇ 2 L ) − 1 ∇ L .
At ﬁrst glance, this looks like it may solve the issue of a ﬂat gradient, since the
Hessian ∇ 2 L may also be exponentially small hence canceling out with the exponentially small gradient.
However, the ﬂatness of the landscape forces us to take an exponentially small step size η , resulting in the following claim:
Theorem 4.2 (Lower bound for Newton’s method) .
Let P ∗ , Q, P satisfy the same conditions as in theorem 4.1.
Let λ ρ :=
min θ ∈ Θ σ min ( ∇ 2 L ( τ θ )) , λ M := max θ ∈ Θ σ max ( ∇ 2 L ( τ θ )) .
Then, running the Newton’s method with step size η = O
( λ ρ λ M ) from an initialization τ = τ q will need an exponential number of steps to reach some τ (cid:48) that is O (1) close to τ ∗ .
Again, the condition η
= O (cid:16) λ ρ λ M (cid:17) follows the typical step size choice for Newton’s method, i.e. the step size should be upper bounded by the ratio between the global strong convexity constant and the global smoothness of the function, which is exponentially small for this setup by Lemma 4.2, 4.3.
N ORMALIZED GRADIENT DESCENT FOR WELL - CONDITIONED LOSSES
We have seen that due to an ill-behaved landscape, NCE optimized with standard gradient descent or Newton’s method will fail to reach a good parameter estimate efﬁciently, even on a problem as simple as Gaussian mean estimation, and even with access to the population gradient.
In this section, we will show that a close relative of gradient descent, normalized gradient descent (NGD), despite its simplicity, provides a ﬁx to the ﬂatness problem to exponential family distributions when the Hessian of the loss is well-conditioned close to the optimum.
Precisely, recall that the NGD updates for a loss function L is τ t
+1 =
τ t − η ∇ L ( τ t ) (cid:107)∇ L ( τ t ) (cid:107) 2 .
We assume that in a neighborhood around τ ∗ , the change in the shape of the Hessian H is moderate:
Assumption 5.1 (Hessian in a neighborhood of τ ∗ ) .
Under assumption 2.2 with constant β Z , assume that for any τ such that (cid:107) τ − τ ∗ (cid:107) 2 ≤ 1 β Z , it holds that σ max ( H ( τ )) ≤ β u · σ max ( H ( τ ∗ )) , and σ min ( H ( τ )) ≥ β l · σ min ( H ( τ ∗ )) , for some constant β u , β l > 0 .
The main result of this section states that NGD can ﬁnd a parameter estimate efﬁciently for exponential families, where the number of steps required is polynomial in the distance between the initial estimate and the optimum:
Theorem 5.1.
Let L be any loss function that is convex in the exponential family parameter and satisﬁes Assumptions 5.1 and 2.1 - 2.4.
Furthermore, let P ∗ , Q be exponential family distributions with parameters τ ∗ , τ q and let κ ∗ be the condition number of the Hessian at P
= P ∗ .
Then, for any 0 < δ ≤ 1 β Z and parameter initialization τ 0 , with step size η ≤ (cid:113) β l β u κ ∗ δ , performing NGD on the population objective L guarantees that after T ≤ β u κ ∗ β l · (cid:107) τ 0 − τ ∗ (cid:107) 2 δsteps, there exists an iterate t ≤ T such that (cid:107) τ t − τ ∗ (cid:107) 2 ≤ δ .
The main technical ingredient for proving Theorem 5.1 is the following Lemma:
Lemma 5.1.
Suppose Assumptions 2.2 and 5.1 hold with constants β Z , β u and β l .
Let L be a convex function with minimizer τ ∗ , and let g :=
∇ L ( τ ) .
For any δ ≤ 1 β Z , let γ = (cid:113)
β
l β u κ ∗ δ .
Then for all τ s.t.
(cid:107) τ − τ ∗ (cid:107) 2 ≥
δ , we have L ( τ ∗ + γ g (cid:107) g (cid:107) )
≤ L ( τ ) .
Lemma 5.1 explains the dependency on κ ∗ in the NGD convergence rate.
The intuition of the proof is that in a small neighborhood around τ ∗ , the set of parameters that have the same loss form an “ellipsoid", and by Taylor expansion, any two points in the same set will have a distance-to- τ ∗ ratio upper bounded roughly by √ κ ∗ .
The details are in Appendix C.2.
Proof sketch for Theorem 5.1 : the proof leverages two observations: the convexity of NCE loss on exponential family parameters, and that the Hessian in a neighborhood around τ ∗ changes moderately as stated in Assumption 5.1.
One can then show there exists a global constant γ such that for any τ t satisfying (cid:107) τ t − τ ∗ (cid:107) 2 ≥ δ , one step of NGD update guarantees a decrease of γ 2 in the squared error, which means NGD must have found an δ -close estimate within (cid:107) τ 0 − τ ∗ (cid:107) 2 γsteps.
The full proof is deferred to Appendix C.1.
5. E XAMPLE : 1 D G AUSSIAN MEAN ESTIMATION
It is relatively straightforward to check that NGD addresses the ﬂatness problem faced by Gaussian
mean estimation we considered in Section 4:
Corollary 5.1.
Let P ∗ , Q be 1d Gaussian with covariance 1 and mean θ ∗ =
R where R (cid:28) 1 , and θ q = 0 .
For any given δ ≤ 1 R and initial estimate τ 0 = τ q , NGD can ﬁnd an estimate τ such that (cid:107) τ − τ ∗ (cid:107) 2 ≤
δ , with at most O ( R 6 δ 2 ) steps.
Intuitively, the effectiveness of NGD comes from the crucial observation that though the magnitude for the loss and derivatives can be exponentially small, they share the same exponential factor, making normalization effective.
Formally, it can be shown that β u β l
=
O (1) (Appendix D.6).
Corollary 5.1 then follows from Theorem 5.1 and the curvature and strong convexity from Lemma 4.1, 4.2.
B OUNDS ON THE CONDITION NUMBER OF NCE
The convergence rate in Theorem 5.1 depends on κ ∗ , the condition number of the NCE Hessian at the optimum, and Hessian-related constants β u , β l in Assumption 5.1.
We now show that under the setup of Theorem 5.1, κ ∗ and β u , β l can be related to the Bhattacharyya coefﬁcient between P ∗ and
Q , which is a similarity measure deﬁned as BC ( P ∗ , Q ) :
= (cid:82) x (cid:112) p ∗ ( x ) q ( x ) dx .
As a result, we get the following convergence guarantee:
Theorem 5.2.
Suppose Assumptions 2.1, 2.3 hold with constants ω , λ max , and λ min .
Consider a NCE task with data distribution P 1 and noise distribution P 2 , parameterized by θ 1 , θ 2 ∈ Θ respectively.
Then for any given δ ≤ 1 R and initial estimate τ 0 = τ q , NGD ﬁnds an estimate τ such that (cid:107) τ − τ ∗ (cid:107) 2 ≤ δ within T ≤ C ·
[Equation]
steps, where C := 18 exp ( 2 β Z ) · ( λ max λ min ) 3 ·
[Equation]
In particular, when P ∗ , Q are not too far, we can further show a lower bound on BC ( P ∗ , Q ) :
Lemma 5.2.
For P 1 , P 2 parameterized by θ 1 , θ 2 ∈ Θ , if (cid:107) θ 1 − θ 2 (cid:107) 22 ≤ 4 λ max , then BC ( P 1 , P 2 ) ≥ 12 .
The proofs of Theorem 5.2 and Lemma 5.2 rely on analyzing the geodesic on the manifold of square root densities √ p equipped with the Hellinger distance as a metric; the details are deferred to
Appendix C.3 and C.4.
It is also worth noting that Theorem 5.2 only requires (cid:107) θ 1 − θ 2 (cid:107) to be smaller than a constant, rather than tending to zero as usually required for analyses using Taylor expansions.
Finally, we would like to note that although our analysis can be tightened, it is unlikely to remove such dependency since NGD only uses ﬁrst-order information.
Moreover, the condition number κ ∗ also affects the practical use of Newton-like methods, since matrix inversion is widely known to be sensitive to numerical issues when the matrix is extremely ill-conditioned.
It is an interesting open question whether a non-standard preconditioning approach might be amenable to this setting.
A NALYZING E NCE : NCE WITH AN EXPONENTIAL LOSS
The previous section proved that NGD can serve as a simple ﬁx to overcome the ﬂatness problem of
NCE for well-conditioned losses.
However, though we showed κ ∗ has a polynomial growth when the distributions P , Q ∗ are sufﬁciently close —it is unclear how κ ∗ behaves beyond this threshold.
In this section, we introduce a slight modiﬁcation to the NCE objective, which we call the eNCE objective, for which κ ∗ depends polynomially on some class-related constants.
This means though eNCE may still suffer from the ﬂatness problem, eNCE and NGD together provide a solution that guarantees a polynomial convergence rate .
Towards formalizing this, the eNCE loss is deﬁned as:
Deﬁnition 6.1 (eNCE Loss) .
Let ϕ ( x ) :
= log (cid:113) p ( x ) q ( x ) , and l ( x, y ) :
= exp( − yϕ ( x )) for y ∈ {± 1 } .
The eNCE loss of P θ w.r.t.
data distribution P ∗ and noise Q is:
[Equation]
It can be checked easily that the minimizing ϕ learns ϕ ( x ) = 12 log p ∗ q .
Moreover, each ϕ is associated with an induced distribution p , deﬁned by p ( x ) = exp( ϕ ( x ))
q ( x ) .
Relation to NCE :
Same as the original NCE loss (referred to as “NCE” below), eNCE learns to solve a distinguishing task between samples from P ∗ or Q .
The difference lies only in the losses, which have analogous forms: the NCE loss described in Def. 2.1 can be rewritten in the same form with l ( x, y ) :
= log
[Equation]
yψ ( x )) and ψ ( x ) := log p ( x ) q ( x ) .
The main advantage of the exponential loss is that the Hessian at the optimum is now guaranteed to be well-conditioned:
Lemma 6.1 (Polynomial condition number for eNCE loss) .
Under Assumption 2.3 with constants λ max , λ min , the condition number of the eNCE Hessian at the optimum is bounded by κ ∗ ≤ λ max λ min .
We can also show that eNCE satisﬁes part (ii) of Assumption 5.1.
Due to space considerations, the proof is deferred to Appendix B.1.
Lemma 6.2.
Under assumption 2.2 with constant β Z , for any unit vector u and constant c ∈ [0 , 1 β Z ] , the maximum and minimum singular values of H ( τ ∗ + c u ) satisfy assumption 5.1 with constants β
[Equation]
=
[Equation]
Lemma 6.1 and Lemma 6.2 together imply the Hessian is well-conditioned around the optimum.
Combined with Theorem 5.1, we have the main result of this section:
Theorem 6.1.
Let P ∗ , Q be exponential family distributions with parameters τ ∗ , τ q under Assumption 2.1-2.4.
Let λ max , λ min be constants for Assumption 2.3.
For any given δ ≤ 1 β Z and parameter initialization τ 0 , performing NGD on the eNCE objective guarantees that when taking
[Equation]
steps, there exists an iterate t ≤ T such that (cid:107) τ t − τ ∗ (cid:107) 2 ≤ δ .
Proof.
Theorem 6.1 follows directly from Theorem 5.1, using the condition number bound from
Lemma 6.1 and constants from 6.2.
6. P ROOF OF L EMMA 6.
It can be checked that the Hessian at P
= P ∗ is H ∗ := ∇
[Equation]
p )
[Equation]
Recall that θ ∗ , θ q , ˜ T denote the parameters and sufﬁcient statistics without the partition func- tion coordinate, and τ ∗ , τ q , T denote the extended version with the partition function, e.g. τ ∗ =
[ θ ∗ , log Z ( θ ∗ )] , T ( x ) =
[ ˜ T ( x ) , − 1] .
Then, we can rewrite H ∗ as:
[Equation]
The Lemma then follows from λ min I (cid:22) E θ ∗ + θ q
[ TT (cid:62) ] (cid:22) λ max
I by Assumption 2.3.
E MPIRICAL VERIFICATION
To corroborate our theory we verify the effectiveness of NGD and eNCE on Gaussian mean estimation and the MNIST dataset.
For MNIST, we use a ResNet-18 to model the log density ratio log( p/q ) , following the setup in TRE (Rhodes et al., 2020).
Results: For Gaussian data, we run gradient descent (GD) and normalized gradient descent (NGD) on the NCE loss and eNCE loss.
Figure 1 compares the best runs under each setup given a ﬁxed computation budget (100 update steps), where “best" is deﬁned to be the run with the lowest loss on fresh samples.
The plots show the minimum parameter distance min t ∈
[ T ] (cid:107) τ ∗ − τ t (cid:107) 2 for each step T .
We ﬁnd that NGD indeed outperforms GD, and that the proposed eNCE sees a further improvement over NCE while additionally enjoying provable polynomial convergence guarantees.
For MNIST, we can no longer compare parameter distances since τ ∗ is unknown.
Instead, we compare the result of optimization directly in terms of loss achieved, again under a ﬁxed computation budget (2K steps).
The results are shown in Figure 2, with NGD converging signiﬁcantly faster for both NCE and eNCE.
We note that eNCE can be numerically unstable, especially when P ∗ , Q are well separated.
We include implementation details in Appendix E.1 and additional results in Appendix E.2.
C ONCLUSION AND D ISCUSSIONS
We provided a theoretical analysis of the algorithmic difﬁculties that arise when optimizing the NCE objective with an uninformative noise distribution, stemming from an ill-behaved loss landscape.
Our theoretical results are inspired by empirical observations in prior works (Rhodes et al., 2020;
Gao et al., 2020; Goodfellow et al., 2014) and provide the ﬁrst formal explanation on the nature of the optimization problems of NCE.
Our negative results showed that even on the simple task of
Gaussian mean estimation, and even assuming access to the population gradient, gradient descent and Newton’s method with standard step size choice still require an exponential number of steps to reach a good solution.
We then proposed modiﬁcations to the NCE loss and optimization algorithm, whose combination results in the ﬁrst provably polynomial convergence rate for NCE.
The loss we propose, eNCE, can be efﬁciently optimized using normalized gradient descent and empirically outperforms existing methods.
We hope these theoretical results will help identify promising new directions in the search for simple, effective, and practical improvements to noise-contrastive estimation.
A PPENDIX
We will ﬁst provide missing proofs for the eNCE results in section 6 in section B. Section C provides proofs for NGD convergence on the NCE loss, and section D proves the negative results of NCE in section 4).
Additional notes on the experiments are provided in section E.1.
Notation : We will use a (cid:46) b to denote a = O ( b ) with O hiding a constant less than 2.
Similarly, a (cid:38) b denotes a = Ω( b ) where Ω hides a constant greater than 12 .
A P ROOF OF CONVEXITY OF NCE (L EMMA 2.2)
As a preliminary
[Equation]
let’s ﬁrst prove that the NCE loss is convex in exponential family parameters.
Recall that the NCE loss is
[Equation]
where p ( x ) = p ( τ (cid:62) T ( x )) .
The gradient and Hessian of the NCE loss are: ∇ τ p ( x ) = p ( x ) · T ( x ) ,
[Equation]
Hence the Hessian is PSD at any τ .
B P ROOFS FOR SECTION 6 ( E NCE)
We ﬁrst write down the loss, gradient and Hessian of eNCE for the exponential family:
[Equation]
Note that the Hessian is always PSD, which means L exp is convex in the parameters of the exponential family.
B. P ROOF OF L EMMA 6.
We directly calculate the Hessian at some ˜ τ := τ ∗ + c u for some c ≤ 1 β Z and (cid:107) u
(cid:107) 2 = 1 , using the expression in equation B.1:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Note that without the term in the square brackets, the integration is exactly the same as the one for
H ∗ .
We would like to bound the ratio
[Equation]
for any unit vector v .
Denote ¯ δ :=
c u 2 , ¯ τ := τ (cid:16) θ q + θ ∗(cid:17) for notation convenience, and denote S 1 :=
{ x : ¯ δ (cid:62) T ( x ) > 0 } , S − 1 := { x : ¯ δ (cid:62) T ( x ) ≤ 0 } .
We have:
[Equation]
Recall that f (cid:39) g means functions f, g differ only by a constant factor.
This equation will be used to calculate both the upper and the lower bound
[Equation]
For the upper bound
[Equation]
let χ ∈ {± 1 }
[Equation]
we have
[Equation]
where step ( i ) uses the Lipschitz property of the log partition function in assumption 2.2.
For the lower bound, let χ ∗ :
= arg max χ ∈{± 1 } T χ .
Write ¯ δ =
[¯ θ, ¯ α ] (i.e. separating out ¯ α which is the normalizing constant), let S 12 ( v ) ⊂
S χ ∗ denote a set s.t.
[Equation]
Then T χ for χ ∈ {± 1 } can be lower bounded as:
[Equation]
where step ( i ) uses the Lipschitz property of the log partition function in assumption
[Equation]
This means for any unit vector v , we have
[Equation]
[Equation]
[Equation]
where step ( i ) , ( ii ) follow from assumption 2.3.
Hence the eNCE loss satisﬁes assumption 5.1 with constants β u = 2 exp(1) · λ max λ min , β l = 1 2exp(1) · λ min λ max .
C P ROOFS FOR S ECTION 5 (NGD AND CONDITION NUMBER AT THE OPTIMUM )
This section provides proofs for results in section 5.
Results for NGD convergence rate (Theorem 5.1 and Lemma 5.1) are proved in section C.1 and C.2.
Section C.3 proves the convergence rate stated in terms of the Bhattacharyya coefﬁcient (Theorem 5.2), and the bound on Bhattacharyya coefﬁcient (Lemma 5.2) is proved in section C.4.
C. P ROOF OF T HEOREM 5.1 (NGD CONVERGENCE RATE )
Theorem C.1 (Theorem 5.1 restated) .
Let L be any loss function that is convex in the exponential family parameter and satisﬁes Assumptions 5.1 and 2.1 - 2.4.
Furthermore, let P ∗ , Q be exponential family distributions with parameters τ ∗ , τ q and let κ ∗ be the condition number of the Hessian at P =
P ∗ .
Then, for any 0 < δ ≤ 1 β Z and parameter initialization τ 0 , let the step size be η ≤ (cid:113)
β
l β u κ ∗ δ , performing NGD on the population objective L guarantees that after T ≤ β u κ ∗ β l · (cid:107) τ 0 − τ ∗ (cid:107) 2 δsteps, there exists an iterate t ≤ T such that (cid:107) τ t − τ ∗ (cid:107) 2 ≤ δ .
Proof.
Denote g t := ∇ L ( τ t ) and R := (cid:107) τ ∗ − τ q (cid:107) 2 for notation convenience.
Recall that the NGD update with step size η is τ t
+1 =
τ t − η · g t (cid:107) g t (cid:107) 2 .
Then, (cid:107) τ t − τ ∗ (cid:107) 2 can be rewritten as:
[Equation]
If we set γ s.t.
the last term is smaller than 0 for all τ that are not within distance δ to τ ∗ , setting η = γ gives:
[Equation]
Hence the number of steps required to ﬁnd a τ s.t.
(cid:107) τ − τ ∗ (cid:107) 2 ≤ δ is at most T ≤ (cid:107) τ 0 − τ ∗ (cid:107) 2 γ.
By Lemma 5.1, setting γ = (cid:113) β l β u κ ∗ δ ensures L ( τ ∗ + γ g t (cid:107) g t (cid:107) ) ≤ L ( τ t ) for any τ t that is at least δ away from τ ∗ .
It then follows from the convexity of L that
[Equation]
Substituting this choice of γ back to the bound for T gives
[Equation]
C. P ROOF OF L EMMA 5.
Lemma C.1 (Lemma 5.1 restated) .
Suppose Assumptions 2.2 and 5.1 hold with constant β Z , β u and β l .
Let L be a convex function with minimizer τ ∗ , and let g :=
∇ L ( τ ) .
For any δ ≤ 1 β Z , let γ = (cid:113)
β
l β u κ ∗ δ , then for all τ s.t.
(cid:107) τ − τ ∗ (cid:107) 2 ≥
δ , we have L (
[Equation]
+
[Equation]
L
[Equation]
Proof.
The proof follows from the Taylor expansion around τ ∗ : for any unit vector v and any constant c ≤ γ , the Taylor remainder theorem states that there exists some constant c (cid:48) <
c and unit vector v (cid:48) such that L ( τ ∗ + c v ) − L ( τ ∗ ) =
c 2 2 v (cid:62) H ( τ ∗ + c (cid:48) v (cid:48) ) v .
For any unit vector v 1 , v 2 and constants c 1 , c 2 ≤ δ such that L ( τ ∗ + c 1 v 1 )
=
L ( τ ∗ + c 2 v 2 ) , we have
L
[Equation]
This means for any two points with the same loss, the ratio between their distances to τ ∗ will be at most (cid:113)
β
u β l κ ∗ .
Therefore setting γ = (cid:113)
β
l β u κ ∗ δ guarantees that for any τ that is at least δ away from τ ∗ , τ will have a larger loss than any point that is γ away from τ ∗ .
In other words, L ( τ 1 ) ≤ L ( τ 2 ) holds for any τ 1 ∈ B ( τ ∗ , γ ) , τ 2 (cid:54)∈ B ( τ ∗ , δ ) .
C. P ROOF OF T HEOREM 5.2 (C ONVERGENCE RATE IN TERMS OF B HATTACHARYYA COEFFICIENT )
Recall that the Bhattacharyya coefﬁcient of P ∗ , Q is deﬁned as BC ( P ∗ , Q ) :
= (cid:82) x (cid:112) p ∗ ( x ) q ( x ) dx .
Theorem C.2 (Theorem 5.2, restated) .
Suppose Assumptions 2.1, 2.3 hold with constants ω , λ max , and λ min .
Consider a NCE task with data distribution P ∗ and noise distribution Q , parameterized by θ ∗ , θ q ∈ Θ respectively.
Then for any given δ ≤ 1 R and initial estimate τ 0 = τ q , NGD ﬁnds an estimate τ such that (cid:107) τ − τ ∗ (cid:107) 2 ≤ δ within T ≤ C ·
[Equation]
steps, where C :=
[Equation]
Proof.
Proving Theorem 5.2 requires bounding the condition number κ ∗ and the Hessian-related constants β u , β l .
We ﬁrst show that κ ∗ is inversely related to BC ( P ∗ , Q ) :
Lemma C.2.
Let Θ be the set of parameters for an exponential family satisfying assumption 2.1-2.2.
Then, for any pair of P ∗ , Q parameterized by θ ∗ , θ q ∈ Θ , the NCE problem deﬁned with P ∗ , Q has κ ∗ ≤ λ max 2 λ min 1 BC ( P ∗ ,Q ) .
The next lemma provides the Hessian-related constants in Assumption 5.1:
Lemma C.3.
Let ¯ δ :=
τ − τ ∗ .
Let BC ( P ∗ , Q ) denote the Bhattacharyya coefﬁcient between P ∗ and Q , then for any τ such that (cid:107) ¯ δ (cid:107)≤ 1 β Z , we have:
[Equation]
Hence Assumption 5.1 is satisﬁed with constants β u , β l equal to the respective right hand sides.
The factor C in the theorem statement is then chosen such that
[Equation]
and the proof of
Theorem 5.2 is completed by applying Theorem 5.1 and the above lemmas.
C.3. P ROOF OF L EMMA C.
For exponential family with pdf p ( x ) = h ( x ) exp (cid:0) θ (cid:62) x − log Z ( θ ) (cid:1) , the Hessian at the optimum is:
[Equation]
We also have H ∗ (cid:23) 12 M by noting that p ∗ + q ≤ 2 max { p ∗ , q } .
Therefore in order to bound κ ∗ , it sufﬁces to analyze the condition number of M .
For any pair of distributions parameterized by θ, θ q ∈ Θ with PDFs p, q , and for any unit vector v , we have
[Equation]
where ( i ) uses Cauchy-Schwarz, and ( ii ) uses assumption 2.3.
Denote B :=
[Equation]
We have:
[Equation]
Combining equation C.6, C.7 gives a lower bound of (cid:82) x min { p, q } ( v (cid:62) T ( x )) 2 :
[Equation]
On the other hand
[Equation]
(cid:82) x min { p
[Equation]
q } ( v (cid:62) T ( x )) 2 can also be upper bounded as:
[Equation]
Hence the condition number of M is bounded as:
[Equation]
It is left to determine the value of B .
We claim that B = 1 BC ( P,Q ) , where BC ( P, Q ) is the Bhattacharyya coefﬁcient of P and Q deﬁned as BC ( P, Q ) :=
(cid:82) x (cid:112) p ( x ) q ( x ) dx .
To see this, note that it holds for any x that log Z θ
= θ (cid:62) x
+ log h ( x ) − log p θ ( x ) .
Hence for any x ,
[Equation]
Therefore
[Equation]
C.3. P ROOF FOR L EMMA C.3 (B OUND ON BC ( P ∗ , Q ) )
Proof.
For notational convenience, write ¯ δ =
[¯ θ, ¯ α ] , where ¯ α = log Z ( θ ∗ ) − log Z ( θ ) is the difference in the coordinate for the log partition function.
Upper bounding σ max ( ∇ 2 L ( τ ))
σ max ( ∇ 2 L ( τ ∗ )) :
We proceed by splitting v (cid:62) ∇ 2 L ( τ ) v into two terms:
[Equation]
C.12)
The ﬁrst term is bounded as:
[Equation]
[Table]
[Equation]
C.
[Equation]
where step ( i ) is because ¯ δ (cid:62) T ( x ) < 0 ; step ( ii ) increases the value by integrating over all x ; step ( iii ) uses Assumption 2.2 on Lipschitz log partition function; and step ( iv ) follows from the choice of ¯ δ
= [ ¯ θ, ¯ α ] that (cid:107) ¯ δ (cid:107)≤ 1 β Z .
The second term can be bounded as:
[Equation]
where step ( i ) uses Assumption 2.2 about Lipschitzness of the log partition function
[Equation]
and step ( ii ) is because we have chosen that (cid:107) ¯ δ (cid:107) 2 ≤ 1 β Z .
Substituting back to equation C.12 gives:
[Equation]
where the second inequality uses Assumption 2.3 and Assumption 2.4 for the ﬁrst and second term respectively.
Recall that v (cid:62) ∇ 2 L ( τ ∗ ) v ≥ 14 B 2 1 λ max (cid:16) E θ ∗ + θq 2 ( v (cid:62) T ( x )) 2 (cid:17) 2 .
Hence:
[Equation]
Lower bounding σ min ( ∇ 2 L ( τ ))
σ min ( ∇ 2 L ( τ ∗ )) : Let us denote S 1 :=
{ x : ¯ δ (cid:62) T ( x ) > 0 } and S − 1 :
= { x : ¯ δ (cid:62) T ( x ) ≤ 0 } .
The goal is to lower bound:
[Equation]
Let’s lower bound T 1 , T − 1 in each of the following two cases.
The ﬁrst case is when T − 1 ≥ T 1 .
Let S 12 ( v ) ⊂ S − 1 denote a set s.t.
[Equation]
Write ¯ δ =
[¯ θ, ¯ α ] as before, then
T 1 ≥
[Equation]
where step ( i ) uses p ∗ + q p + q < 1 since ¯ δ (cid:62) T ( x ) < 0 ; step ( ii ) , ( iii ) follows from the deﬁnition of
S − 1 ; step ( iv ) uses equation C.8; and step ( v ) uses Assumption 2.2 that the log partition function is
Lipschitz.
The second case is when T 1 ≥ T − 1 .
Let S 12 ( v ) ⊂
S 1 denote a set s.t.
[Equation]
Then T − 1 , T 1 can be lower bounded as: T
[Equation]
≥
[Equation]
(cid:62) T ( x )) 2 (cid:17) 2 .
(C.19)
Combining both cases and using (cid:107) ¯ δ (cid:107)≤ 1 β Z , we get:
[Equation]
≥
[Equation]
min (cid:26)(cid:16) E θ
[Equation] [Equation]
x )
[Equation]
(cid:16) E θ ∗ + θq − ¯ θ( v (cid:62) T ( x )) 2 (cid:17) 2 (cid:27) .
(C.20)
Recall that v (cid:62) ∇ 2 L ( τ ∗ ) v ≤ 2 B E θ ∗ + θq 2 [( v (cid:62) T ( x )) 2 ] .
Hence σ min ( ∇ 2 L ( τ ))
σ min ( ∇ 2 L ( τ ∗ ))
= min v v (cid:62) ∇ 2 L ( τ ) v min ˜ v (cid:48)
˜ v (cid:62) ∇ 2 L ( τ ∗ )˜ v ≥ 16 exp( − 2 − 1 β Z )
B λ min λ max · max (cid:40)
λ min λ max , min (cid:110)
σ min ( E θ ∗ + θq + ¯ θ
TT (cid:62) )
σ min ( E θ ∗ + θq 2 [ TT (cid:62) ]) , σ min ( E θ ∗ + θq − ¯ θ
TT (cid:62) )
σ min ( E θ ∗ + θq 2 [ TT (cid:62) ]) (cid:111)(cid:41) ≥ 16 exp( − 2 − 1 β Z )
B λ min λ max · max (cid:26) λ min λ max , 1 − γ min (cid:107) ¯ δ (cid:107) λ min (cid:27) .
(C.21)
C. P ROOF OF L EMMA 5.2 (B OUND ON THE B HATTACHARYYA COEFFICIENT )
Lemma C.4 (Lemma 5.2 restated) .
For P 1 , P 2 parameterized by θ 1 , θ 2 ∈ Θ , if (cid:107) θ 1 − θ 2 (cid:107) 22 ≤ 4 λ max , then BC ( P 1 , P 2 ) ≥ 12 .
Proof.
Given θ
[Equation]
θ 2 ∈ Θ
[Equation]
deﬁne a map φ from [0 , 1] to a function √ p , where p is the PDF for a distribution parameterized by some θ ∈ Θ : let Z ( θ ) denote the partition function for parameter θ ∈ Θ
[Equation]
and let δ :
= θ 2 − θ 1 , then φ ( t ) is a function of x deﬁned as:
[Equation]
[Equation]
Denote φ t ( x ) :=
φ ( t )( x ) and θ t :
= θ 1 + tδ for notation convenience.
Then [Equation] t
[Equation]
x )
[Equation]
C.23) where step ( ∗ ) used
[Equation] [Equation]
Hence
[Equation] [Equation]
Using the fundamental theorem of calculus
[Equation]
we get
[Equation]
C
[Equation]
[Equation]
Hence (cid:82) x
[Equation] p [Equation] θ 2 ≥ 1 − λ max 8 (cid:107) δ (cid:107) 2 , or 1 (cid:82) x
[Equation]
p
[Equation]
1 p
[Equation]
[Equation] 1 1 − λ max8 (cid:107) δ (cid:107) 2 for (cid:107) δ (cid:107) 2 < 8 λ max .
In particular, for any θ 1 , θ 2 satisfying (cid:107) δ (cid:107) 2 := (cid:107) θ 1 − θ 2 (cid:107) 2 ≤
[Equation] [Equation]
(cid:82)
[Equation]
[Equation]
As a side note, another bound we can get is from Lipschitzness of the log partition function:
[Equation]
which is tighter than
[Equation]
D P ROOFS FOR S ECTION 4 ( NEGATIVE RESULTS OF NCE)
This section provides proofs for the negative results in section 4, that is, the NCE landscape is ill-behaved with exponentially ﬂat loss, gradient, and curvature.
We will ﬁrst prove Theorem 4.
and properties regarding losses and gradients, then prove results related to second-order properties (Lemma 4.1, 4.3, 4.2, Theorem 4.2).
D. P ROOF OF T HEOREM 4.1 (L OWER BOUND FOR GRADIENT - BASED METHODS )
Theorem D.1 (Theorem 4.1 restated) .
Let P ∗ , Q, P be 1d Gaussian with variance 1.
Assume θ q = 0 , θ ∗ > 0 without loss of generality, and assume R :
= θ ∗ − θ q (cid:29) 1 .
Then, gradient descent with any step size η = o (1) from an initialization τ = τ q will need an exponential number of steps to reach some τ (cid:48) that is O (1) close to τ ∗ .
Proof.
The key lemma to prove Theorem 4.1 is as follows, which upper bounds the decrease in parameter distance from each gradient step:
Lemma D.1.
Consider the annulus A :
= { ( b, c ) : ( c − R 2 2 )
+ ( b − R ) 2 ∈
[(0 . 1 R ) 2 , (0 .
2 R ) 2 ] } .
Then, for any ( b, c ) ∈ A , it satisﬁes that
[Equation] [Equation]
) where κ ( b, c ) ∈ [ 34 , 54 ] is a small constant.
Lemma D.1 is proved in section D.2.
To prove Theorem 4.1, we will ﬁrst show that Lemma D.1 serves as an upper bound for the decrease in parameter distance, that is, showing η (cid:12)(cid:12)(cid:12)
[Equation] [Equation]
(cid:107)
[Equation]
(cid:107) (cid:105) (cid:12)(cid:12)(cid:12) ≥ (cid:107)
τ t − τ ∗ (cid:107)−(cid:107)
τ t +1 − τ ∗ (cid:107) .
Towards this claim, we write τ t +1 as:
[Equation]
where v := ∇ L ( τ t ) − (cid:104)∇ L ( τ t ) , τ ∗ − τ t (cid:107) τ ∗ − τ t (cid:107) (cid:105) ·
τ ∗ − τ t (cid:107) τ ∗ − τ t (cid:107) is orthogonal to τ ∗ − τ t .
Hence (cid:107)
[Equation] [Equation]
From this, we can conclude
[Equation]
The next step is to show that there is a path lying in A of length at 0 .
01 R that gradient descent has to go through.
We have the following lemma (proof in appendix D.3):
Lemma D.2.
Let η = o (1) .
For any τ s.t.
(cid:107) τ − τ ∗ (cid:107)≥ 0 .
2 R , let τ (cid:48) denote the point after one step of gradient descent from τ , then (cid:107) τ (cid:48) − τ ∗ (cid:107) > 0 .
15 R .
From any such τ (cid:48) , the shortest way to exit the annulus A is to project onto the inner circle deﬁning
A , i.e. the circle centered at τ ∗ with radius 0 .
1 R which is a convex set.
Denote this inner circle as B ( τ ∗ , 0 .
1 R ) whose projection is Π B ( τ ∗ , 0 .
1 R ) , then the shortest path is the line segment τ (cid:48) − Π B ( τ ∗ , 0 .
1 R )
( τ (cid:48) ) .
Further, this line segment is of length 0 .
05 R since (cid:107) τ (cid:48) − τ ∗ (cid:107) > 0 .
15 R by Lemma
D.2, while the decrease of the parameter distance (i.e. (cid:107) τ − τ ∗ (cid:107) ) is exponentially small at any point in A by Lemma D.1 and equation D.4.
Hence the number of steps to exit A is lower bounded by 0 .
[Equation]
D. P ROOF OF L EMMA D.
Recall that for 1d Gaussian with a known unit covariance, we can use parameter τ :=
[ b, c ] and sufﬁcient statistics T ( x ) :=
[ x, − 1] , with pdf p ( x ) = exp (cid:16) − x 2 2 (cid:17) · exp ( (cid:104) τ, T ( x ) (cid:105) ) .
For any τ such that (cid:107) τ ∗ − τ (cid:107)≥ 1 , (cid:12)(cid:12)(cid:12) (cid:104)∇ L ( τ ) , τ ∗ − τ (cid:107) τ ∗ − τ (cid:107) (cid:105) (cid:12)(cid:12)(cid:12) can be upper bounded as:
[Equation]
D.
[Equation]
Let a (cid:39) b denote a = kb for a constant k = Θ(1) .
We ﬁrst show the calculations with b > 0 for cleaner presentation; the b < 0 case is analogous and deferred to D.2.2. Bounding
[Equation]
where κ ( b, c ) ∈ [ 34 , 54 ] .
Step ( i ) uses calculations in equation D.12-D.15 (deferred to subsection
D.2.1 for cleaner presentation)
[Equation]
and assumes ( b, c ) belongs to the set V :
= { ( b, c ) : c ∈ [ b ( R − b ) , b · min { b, R } ] } .
In particular, the annulus A := { ( b, c ) : ( c − R 2 2 )
+ ( b − R ) 2 ∈
[(0 . 1 R ) 2 , (0 .
2 R ) 2 ] } is a subset of V when R (cid:29) 1 .
Step ( ii ) considers ( b, c ) ∈ A .
We can choose b, c s.t. b ≥ R 2 , c ∈
[ b ( R − b ) , b · min { b, R } ] , so that we pick up the tails in T (0) to
T (0)4 .
This means:
[Equation]
Bounding (cid:12)(cid:12)(cid:12)(cid:82)
[Equation] −
[Equation] ∗ pq
[Equation]
x
[Equation]
: Using similar calculations as before, we have that when c − log √ 2 π >(which is the case for τ =
[ b, c ] ∈ A ),
[Equation]
Below we bound the case where x > 0 ; the other case (i
[Equation]
e. x < 0 ) has an upper bound of the same order following similar calculations and is hence omitted
[Equation]
where step ( i ) uses calculations in equation D.16-D.19.
Ignoring small constants log √ 2 π in c − log √ 2 π , and denoting E 1 := exp (cid:16) − c
[Equation] b [Equation] (cid:17) , E 2 := exp (cid:16) − ( c − bR ) 2 2 b(cid:17) for notation convenience, we can substitute equation D.6 and D.8 into equation D.5 as:
[Equation]
where κ ( b, c ) ∈ [ 3 4 , 54 ] is the constant deﬁned in equation D. [Equation]
Since τ ∈ R [Equation] (cid:107) τ ∗ − τ (cid:107) =
Θ( R )
[Equation]
and the proof is completed by:
[Equation]
D.2. C ALCULATION DETAILS FOR EQUATION D.6 AND D.
We now calculate term T (0 i and T (1) i used in equation D.6 and D.8. [Equation]
  exp
[Equation]

[Equation]
[Equation]
[Equation]
≤
[Equation]
(cid:18)
[Equation]
=
[Equation]
=
[Equation]
( − c ) − 1 √
[Equation]
(cid:32) −
[Equation]
D
[Equation]
[Equation]
[Equation]
D.
[Equation]
. C ALCULATIONS FOR b <
We now calculate the gradient norm bound for the case where b < 0 .
Recall that:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
=
[Equation]
where T (0)
i
[Equation]
− terms are calculated as:
[Equation] [Equation]
Bounding (cid:12)(cid:12)(cid:12)(cid:82)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:90) x
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:90)
[Equation]
p
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:90) x p
[Equation]
≤ max (cid:40)(cid:90)
[Equation]
[Equation]
As before, we will show the bound for the case where x > 0 ; the other case (i.e. x < 0 ) follows a similar calculation and has an upper bound on the same order.
First consider b < 0 , c − log √ 2 π > 0 :
[Equation]
[Equation] where step ( i ) uses the following: [Equation] +
[Equation]
When b < 0 , c − log √ 2 π < 0 , (cid:90) x>p
[Equation]
p
[Equation]
=
[Equation]
where T (1) i, terms are calculated as:
[Equation]
where β (1) [Equation] − = O (1) is:
[Equation] [Equation]
where β (1)4 , − = O (1) is:
[Equation]
Combining equation D.21, D.
[Equation]
and D.26 we have that (cid:107)∇ L ([ b, c ]) (cid:107) 2 ≤ 32 max { R, | b |} for b < 0 .
D
P ROOF OF L EMMA D.
We ﬁrst show the following claim, and prove Lemma D.2 at the end of this subsection:
Claim D.1.
For any τ =
[ b, c ] ∈ R 2 , the gradient norm at τ is (cid:107)∇ L ( τ ) (cid:107) 2 ≤ 32 max { R, | b |} .
Proof.
For parameter τ =
[ b, c ] where b > 0 , c − log √ 2 π > 0 ,
[Equation]
[Equation]
(D.36) where step ( i ) and ( ii ) use equation D.16-D.19 and equation D.12-D.15.
Moreover, step ( ii ) increases the value
[Table]
When [Table] >
[Table]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:90) x
[Table]
− p
[Table]
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
p
[Equation] (cid:90) [Equation] (cid:90)
[Equation]
(cid:90)
[Equation]
(cid:41) .
[Table]
Let’s bound the ﬁrst term (i
[Equation]
e. x > 0 ); the bound for the second term (i
[Equation]
e. x < 0 ) follows from similar calculations and is on the same order.
[Equation]
Step ( i ) omits a factor of 1 √ 2 π and uses:
[Equation]
 
[Equation]
For b < 0 , we similarly have (cid:107)∇ L ( τ ) (cid:107)
= O (max { R, − b } ) .
The calculations are similar to the b > 0 case and hence omitted.
We are now ready to prove Lemma D.2, which we restate below.
Lemma D.3 (Lemma D.2, restated) .
Let η = o (1) .
For any τ s.t.
(cid:107) τ − τ ∗ (cid:107)≥ 0 .
2 R , let τ (cid:48) denote the point after one step of gradient descent from τ , then (cid:107) τ (cid:48) − τ ∗ (cid:107) > 0 .
15 R .
Proof of Lemma D.3.
We will prove by contradiction.
First assume that we can go from τ where (cid:107) τ − τ ∗ (cid:107) 2 ≥ 0 .
2 R to some τ (cid:48) where (cid:107) τ (cid:48) − τ ∗ (cid:107) 2 ≤ 0 .
15 R .
Then (cid:107) τ (cid:48) − τ ∗ (cid:107) is lower bounded as:
[Equation]
where step ( i ) uses (cid:107) τ − τ ∗ (cid:107)≥ | τ [1] − τ ∗ [1] |≥ ||
τ 1 |−| τ ∗ 1 || =
| b − R | , and step ( ii ) is by Claim D.1.
On the other hand, we have (cid:107) τ (cid:48) − τ ∗ (cid:107)≤ 0 .
15 R by assumption, which when combined with equation
D.40 gives b ≤ 0 .
15 R | 1 − Rb | − 32 η , or b =
O ( R ) .
This means (cid:107) τ − τ ∗ (cid:107)−(cid:107) τ (cid:48) − τ ∗ (cid:107)≤ η (cid:107)∇ L ( τ ) (cid:107) =
o (1) ·
O (max { R, | b |} )
=
o ( R ) .
However, we also have (cid:107) τ − τ ∗ (cid:107)−(cid:107) τ (cid:48) − τ ∗ (cid:107)≥ 0 .
R =
Θ( R ) by assumption.
This is a contradiction, which means the assumption must be false, i.e. τ (cid:48) cannot satisfy (cid:107) τ (cid:48) − τ ∗ (cid:107) 2 ≤ 0 .
15 R .
D. P ROOF OF L EMMA 4.1 AND L EMMA 4.
We prove Lemmas 4.1 and 4.2 in this section.
First recall the lemma statements:
Lemma D.4 (Smoothness at P =
P ∗ , Lemma 4.1 restated) .
Consider the 1d Gaussian mean estimation task with R :=
| θ ∗ − θ q |(cid:29) 1 .
Then the smoothness at P
= P ∗ is upper bounded as:
[Equation]
We will also need a bound on the strong convexity constant (i.e. smallest singular value) at P
= P ∗ :
Lemma D.5 (Strong convexity at P =
P ∗ , Lemma 4.2 restated) .
Under the same setup as lemma 4.1, the minimum singular value at P = P ∗ is σ ∗ min
[Equation]
(cid:16)
[Equation]
8 (cid:17)(cid:17) .
D.4. P ROOF OF L EMMA 4.1 ( SMOOTHNESS AT P = P ∗ )
We will show the smoothness constant (i.e. σ max ( ∇ 2 L ) ) is exponentially small at the optimum, i.e. when P = P ∗ .
The Hessian at the optimum is:
[Equation]
Recall that θ q = 0 w.l.o.g, and assume θ ∗ =
R (cid:29) 1 .
Then
[Equation]
Let S 1 ⊂ R 2 denote the circle centered as the origin with radius 1.
The maximum singular value is upper bounded by
[Equation]
where ( i ) substitutes in 1 or − 1 for a 1 , a 2 and uses the fact that the upper bounds for T 0 , T 1 , T 2 are positive.
( ii ) uses the calculations on T 0 to T 2 shown below.
We note that these calculations rely on properties of Gaussian and do not extend to general exponential families.
[Equation]
For T 2 , denote P R/ 2 :=
P ∗ (cid:16)(cid:110) x : x ≤ R 2 (cid:111)(cid:17) =
P Q (cid:16)(cid:110) x : x ≥ R 2 (cid:111)(cid:17) ; Gaussian tail bound gives
P R/ 2 ≤ 2 √ 2 π 1 R exp (cid:16) − R 2 8 (cid:17) .
Then we can calculate each term in T 2 as:
[Equation]
D.4. P ROOF OF L EMMA 4.2 ( STRONG CONVEXITY AT P = P ∗ )
Lower bounding σ ∗ min follows a similar calculation as for upper bounding σ ∗ max : σ
[Equation]
min
[Equation]
=
[Equation]
(D.48) where ( i ) uses the calculations on T 0 to T 2 stated in equation D.45 and D.46.
Step ( ii ) replaces a = 0 to remove the O ( R ) term.
D. P ROOF OF L EMMA 4.3 ( CURVATURE AT P = Q )
Lemma D.6 (Smoothness at P = Q , Lemma 4.3 restated) .
Under the same setup as Lemma 4.1, the smoothness at P = Q is lower bounded as σ max ( ∇ 2 L ( τ q )) ≥ R 2 2 .
Proof.
The result follows from direct calculation of the Hessian at P = Q :
[Equation]
(D.49)
Hence σ max ( ∇ 2 τ L ) ≥ e (cid:62) 1 ∇ 2 L ( τ ) e 1 ≥ R 2 2 .
D. P ROOF OF T HEOREM 4.2 ( LOWER BOUND FOR SECOND - ORDER METHODS )
The proof of Theorem 4.2 is similar to that of Theorem 4.1, where we show that there is a ring of width Θ( R ) in which the amount of progress at each step is exponentially small, hence the number of steps required to cross this ring is exponential.
We show that starting from τ
=
τ q , the optimization path will necessarily steps into A :
Lemma D.7.
Let η :=
O ( [Equation] M ) , where λ ρ := min θ [Equation]
Θ σ min ( ∇ 2 L ( τ θ )) , λ M := max θ ∈ Θ σ max ( ∇ 2 L ( τ θ )) as deﬁned in Section 4.
For any τ s.t.
(cid:107) τ − τ ∗ (cid:107)≥ 0 .
2 R , let τ (cid:48) denote the point after one step of gradient descent from τ , then (cid:107) τ (cid:48) − τ ∗ (cid:107) 2 > 0 .
15 R .
Proof.
First note that ∀ τ , the next point after one step of Newton update is:
[Equation]
50) where v :=
( ∇ 2 L ( τ )) [Equation] 1 ∇ L ( τ
[Equation]
2 is orthogonal to τ − τ ∗ .
Hence
[Equation]
where step ( i ) uses Claim D.1, and step ( ii ) follows from the choice of η .
Suppose (cid:107) τ (cid:48) − τ ∗ (cid:107) 2 < 0 .
15 R , then
[Equation]
∗
[Equation]
However, (cid:107) τ (cid:48)
− τ ∗ (cid:107) 2 entails b =
Θ( R ) , which is a contradiction.
Hence it must be that (cid:107) τ (cid:48) − τ ∗ (cid:107) 2 > 0 .
15 R .
Proof of Theorem 4.2.
By Lemma D.7, the optimization path will go to a point τ (cid:48) ∈
A s.t.
(cid:107) τ (cid:48)
− τ ∗ (cid:107) 2 > 0 .
15 R .
From any such τ (cid:48) , the shortest way to exit the annulus A is to project onto the inner circle deﬁning A , i.e. the circle centered at τ ∗ with radius 0 .
1 R which is a convex set.
Denote this inner circle as B ( τ ∗ , 0 .
1 R ) whose projection is Π B ( τ ∗ , 0 .
1 R ) , then the shortest path is the line segment τ (cid:48) − Π B ( τ ∗ , 0 .
1 R )
( τ (cid:48) ) .
Further, this line segment is of length 0 .
05 R since (cid:107) τ (cid:48) − τ ∗ (cid:107) > 0 .
15 R by Lemma D.7.
However, the decrease of the parameter distance (i.e. (cid:107) τ − τ ∗ (cid:107) ) is exponentially small at any point in A :
[Equation]
) where step ( i ) uses the calculations in equation D.51; step ( ii ) use the choice of η ; and step ( iii ) uses Lemma D.1.
Hence the number of steps to exit A is lower bounded by
[Equation]
[Table]
E A DDITIONAL NOTES ON EXPERIMENTS E. I MPLEMENTATION DETAILS
Parameterization :
For the 1-dimensional Gaussian, we take P ∗ , Q to have mean µ ∗ = 16 , µ q = 0 , and unit variance σ 2 ∗ =
σ 2 q = 1 ; see Figure 4a for an illustration of the ﬂat loss landscape.
We use h ( x ) :=
exp( − x 2 2 ) , T ( x ) :=
[ x, − 1] to be consistent with the notation in Section 4.
For the 16-dimensional Gaussian, P ∗ , Q share the same mean µ ∗ = µ q = 0 but have different covariance with Cov q = I d and Cov p = diag
([ s 1 , ..., s d ]) , where s i =
Uniform [8 × 0 .
75 , 8 × 1 . 5] .
For MNIST, we adapt the TRE implementation by Rhodes et al.
We model the log density ratio log( p/q ) by a quadratic of the form g ( x ) := − f ( x ) (cid:62) W f ( x ) − b (cid:62) f
( x ) − c , where f is
ResNet-18, and W , b , c are trainable parameters with W constrained to be positive deﬁnite.
Implementation notes : We include some tricks we found useful for implementation:
• Calculation in log space
: instead of dividing two pdfs, we found it more numerically stable to use subtraction between the log pdfs and then exponentiate.
• Removing common additive factors: the empirical loss is the average loss over a batch of samples where overﬂow can happen.
9 We found it more stable to calculate the mean by ﬁrst subtract the largest value of the batch, calculate the mean of the remaining values, then add back the large value—akin to the usual log-sum-exp trick.
For example, mean ([ a, b ]) =
max( a, b ) + mean ([ a − max( a, b ) , b − max( a, b )]) .
• Per-sample gradient clipping: it is sometimes helpful to limit the amount of gradient contributed by any data point in a batch.
We ensure this by limiting the norm of the gradient, that is, the gradient from a sample x is now min { 1 , K
(cid:107)∇ (cid:96) ( x ) (cid:107) }∇ (cid:96) ( x ) for some prespeciﬁed constant K (Tsai et al., 2021).
• Per-sample log ratio clipping: an alternative to per-sample gradient clipping is to upper threshold the absolute value of the log density ratio on each sample, before passing it to the loss function.
Setting a proper threshold prevents the loss from growing too large, and consequently prevents a large gradient update
E. A DDITIONAL RESULTS
Results for training with a larger computation budget : We provide additional results on Gaussian mean estimation and MNIST, both trained with a larger computation budget.
Figure 5 shows results similar to those of Figure except that we now run the optimization process for 5 times longer than in Figure and additionally show results on eNCE optimized with gradient descent (GD).
The conclusion is the same as that of Figure 1: for both NCE and eNCE , normalized
[Table]
[Table] [Table]
[Table]
gradient descent (NGD) signiﬁcantly outperforms GD.
Moreover, eNCE performs competitively compared to NCE when optimized with NGD.
Similarly, we train with a large computation budget on MNIST, whose results are shown in Figure 6.
The results are again consistent with those in Figure 2.
MNIST samples We run annealed importance sampling (AIS) Neal (2001) following Rhodes et al.
(2020) on the models trained on NCE and eNCE , optimized with GD or NGD.
Figure 7, 8 show samples generated with 4k or 10k sampling steps, from different random initialization.
We can see that eNCE gives much sharper results than NCE and eNCE with NGD results in more diverse samples.
A downside though is that eNCE samples seem to show signs of mode collapse.
However, it is unclear whether this is a problem with the model or due to the sampling procedure.
Results for training with other normalized optimization method : One interesting question to ask is, whether the results of NGD generalize to other optimizers that perform some form of normalization.
One example that is commonly used in practice is the RMSprop, which performs percoordinate normalization on the gradient.
Speciﬁcally, at the t th step with gradient g t , RMSprop ﬁrst updates a cumulative term v t :
= α v t − 1 + (1 − α ) g 2 t , where α ∈ [0 , 1] is a hyperparameter
[Figure]
[Figure]
controlling how much to “damp" the current gradient, and the square on g t is applied entrywise.
It then normalizes the gradient as ˜ g t :=
g t / v t , with division applied entry-wise.
Figure 9 compares RMSprop with GD or NGD on 1d Gaussian estimation task.
The results suggest that how well RMSprop performs may be task-speciﬁc: RMSprop performs the best when optimizing for NCE, but only slightly better than GD when optimizing for eNCE .
Moreover, NCE favors a higher value of α where the best performance is achieved by α = 0 . 99 , whereas eNCE prefers α to be small with α = 0 .
01 performing the best.
It is not yet clear what the theoretical answer should be for normalized methods in general: though methods like RMSprop perform certain form of normalization (e.g. per-coordinate normalization), so does Newton’s method, and Theorem 4.2 has shown that it still suffers from an exponentially bad convergence rate (at least with standard choices of step size).
It is unclear what quantity replaces the condition number of the Hessian for RMSProp, which governs the convergence of NGD.
Theoretical guarantees for these optimization methods is an interesting open question.