Original: P ITFALLS OF G AUSSIANS AS A NOISE DISTRIBUTION IN NCE
Revised: P ITFALLS OF G AUSSIANS AS A NOISE DISTRIBUTION IN NCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Noise Contrastive Estimation (NCE) is a popular approach for learning probability density functions parameterized up to a constant of proportionality.
Revised: Noise Contrastive Estimation (NCE) is a popular approach for learning probability density functions parameterized up to a constant of proportionality.

Original: The main idea is to design a classification problem for distinguishing training data from samples from an easy-to-sample noise distribution q , in a manner that avoids having to calculate a partition function.
Revised: The main idea is to design a classification problem for distinguishing training data from samples from an easy-to-sample noise distribution q , in a manner that avoids having to calculate a partition function.

Original: It is well-known that the choice of q can severely impact the computational and statistical efficiency of NCE.
Revised: It is well-known that the choice of q can severely impact the computational and statistical efficiency of NCE.

Original: In practice, a common choice for q is a Gaussian which matches the mean and covariance of the data.
Revised: In practice, a common choice for q is a Gaussian which matches the mean and covariance of the data.

Original: In this paper, we show that such a choice can result in an exponentially bad (in the ambient dimension) conditioning of the Hessian of the loss, even for very simple data distributions.
Revised: In this paper, we show that such a choice can result in an exponentially bad (in the ambient dimension) conditioning of the Hessian of the loss, even for very simple data distributions.

Original: As a consequence, both the statistical and algorithmic complexity for such a choice of q will be problematic in practice, suggesting that more complex and tailored noise distributions are essential to the success of NCE.
Revised: As a consequence, both the statistical and algorithmic complexity for such a choice of q will be problematic in practice, suggesting that more complex and tailored noise distributions are essential to the success of NCE.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Noise contrastive estimation (NCE), introduced in (Gutmann & Hyv ¨ arinen, 2010; 2012), is one of several popular approaches for learning probability density functions parameterized up to a constant of proportionality, i.e. p ( x ) ∝ exp( E θ ( x )) , for some parametric family { E θ } θ .
Revised: Noise contrastive estimation (NCE), introduced in (Gutmann & Hyv ¨ arinen, 2010; 2012), is one of several popular approaches for learning probability density functions parameterized up to a constant of proportionality, i.e. p ( x ) ∝ exp( E θ ( x )) , for some parametric family { E θ } θ .

Original: A recent incarnation of this paradigm is, for example, energy-based models (EBMs), which have achieved near-state-of-the-art results on many image generation tasks (Du & Mordatch, 2019; Song & Ermon, 2019).
Revised: A recent incarnation of this paradigm is, for example, energy-based models (EBMs), which have achieved near-state-ofthe-art results on many image generation tasks (Du & Mordatch, 2019; Song & Ermon, 2019).

Original: The main idea in NCE is to  train a classifier which distinguishes between samples from the data distribution P ∗ and a known, easy-to-sample distribution Q , often called the “noise”  distribution.
Revised: The main idea in NCE is to set up a self-supervised learning (SSL) task, in which we train a classifier to distinguish between samples from the data distribution P ∗ and a known, easy-to-sample distribution Q , often called the “noise” or “contrast” distribution.

Original: It can be shown that for a large choice of losses for the classification problem, the optimal classifier model is a (simple) function of the density ratio p ∗ /q , so an estimate for p ∗ can be extracted from a good classifier.
Revised: It can be shown that for a large choice of losses for the classification problem, the optimal classifier model is a (simple) function of the density ratio p ∗ /q , so an estimate for p ∗ can be extracted from a good classifier.

Original: Moreover, this strategy can be implemented while avoiding calculation of the partition function, which is necessary when using maximum likelihood to learn p ∗ .
Revised: Moreover, this strategy can be implemented while avoiding calculation of the partition function, which is necessary when using maximum likelihood to learn p ∗ .

Original: The noise distribution q is the most significant “hyperparameter” in NCE training, with both strong empirical (Rhodes et al., 2020) and theoretical (Liu et al., 2021) evidence that a poor choice of q can result in poorstatistical and algorithmic complexity.
Revised: The noise distribution q is the most significant “hyperparameter” in NCE training, with both strong empirical (Rhodes et al., 2020) and theoretical (Liu et al., 2021) evidence that a poor choice of q can result in poor algorithmic behavior.

Original: (Chehab et al., 2022) show that even the optimal q for finite number of samples can have an unexpected form (e.g., it is not equal to the true data distribution p ∗ ).
Revised: (Chehab et al., 2022) show that even the optimal q for finite number of samples can have an unexpected form (e.g., it is not equal to the true data distribution p ∗ ).

Original: Since q needs to be a distribution that one can efficiently draw samples from, as well as write an expression for the probability density function, the choices are somewhat limited.
Revised: Since q needs to be a distribution that one can efficiently draw samples from, as well as write an expression for the probability density function, the choices are somewhat limited.

Original: A particularly common way to pick q is as a Gaussian that matches the mean and covariance of the input data (Gutmann & Hyv¨arinen, 2012; Rhodes et al., 2020).
Revised: A particularly common way to pick q is as a Gaussian that matches the mean and covariance of the input data (Gutmann & Hyv¨arinen, 2012; Rhodes et al., 2020).

Original: Our main contribution in this paper is to formally show that such a choice can result in an objective that is statistically poorly behaved, even for relatively simple data distributions.
Revised: Our main contribution in this paper is to formally show that such a choice can result in an objective that is statistically poorly behaved, even for relatively simple data distributions.

Original: We show that even if p ∗ is a product distribution and a member of a very simple exponential family , the Hessian of the NCE loss, when using a Gaussian noise distribution q with matching mean and covariance has exponentially small (in the ambient dimension) spectral norm.
Revised: We show that even if p ∗ is a product distribution and a member of a very simple exponential family , the Hessian of the NCE loss, when using a Gaussian noise distribution q with matching mean and covariance has exponentially small (in the ambient dimension) spectral norm.

Original: As a consequence, the optimization landscape around the optimum will be exponentially flat, making gradient-based optimization challenging.
Revised: As a consequence, the optimization landscape around the optimum will be exponentially flat, making gradient-based optimization challenging.

Original: Even more seriously, we show the asymptotic sample efficiency of the NCE objective will be exponentially bad in the ambient dimension.
Revised: As the main result of the paper, we show the asymptotic sample efficiency of the NCE objective will be exponentially bad in the ambient dimension.

Original:  O VERVIEW OF R ESULTS
Revised:  O VERVIEW OF R ESULTS

Original: Let P ∗ be a distribution in a parametric family { P θ } θ ∈ Θ .
Revised: Let P ∗ be a distribution in a parametric family { P θ } θ ∈ Θ .

Original: We wish to estimate P ∗ via P θ for some θ ∗ ∈ Θ by solving a noise contrastive estimation task.
Revised: We wish to estimate P ∗ via P θ for some θ ∗ ∈ Θ by solving a noise contrastive estimation task.

Original: To set up the task, we also need to choose a noise distribution Q , with the constraint that we can draw samples from it efficiently, and we can evaluate the probability density function efficiently.
Revised: To set up the task, we also need to choose a noise distribution Q , with the constraint that we can draw samples from it efficiently, and we can evaluate the probability density function efficiently.

Original: We will use p θ , p ∗ , q to denote the probability density functions (pdfs) of P θ , P ∗ , and Q .
Revised: We will use p θ , p ∗ , q to denote the probability density functions (pdfs) of P θ , P ∗ , and Q .

Original: For a data distribution P ∗ and noise distribution Q , the
Revised: For a data distribution P ∗ and noise distribution Q , the

Original: NCE loss of a distribution P θ is defined as follows:
Revised: NCE loss of a distribution P θ is defined as follows:

Original: Definition 1 (NCE Loss) .
Revised: Definition 1 (NCE Loss) .

Original: The NCE loss of P θ w.r.t.
Revised: The NCE loss of P θ w.r.t.

Original: data distribution P ∗ and noise Q is
Revised: data distribution P ∗ and noise Q is

Original: [Equation]
Revised: [Equation]

Original: Moreover, the empirical version of the NCE loss when given i.i.d.
Revised: Moreover, the empirical version of the NCE loss when given i.i.d.

Original: samples ( x 1 , . . .
Revised: samples ( x 1 , . . .

Original: , x n ) ∼ P n ∗ and ( y 1 , . . .
Revised: , x n ) ∼ P n ∗ and ( y 1 , . . .

Original: , y n ) ∼ Q n is given by
Revised: , y n ) ∼ Q n is given by

Original: [Equation]
Revised: [Equation]

Revised: By a slight abuse of notation, we will use L ( θ ) , L ( p θ ) and L ( P θ ) interchangeably.

Original: The NCE loss can be interpreted as the binary cross-entropy loss for the classification task of distinguishing the data samples from the noise samples.
Revised: The NCE loss can be interpreted as the binary cross-entropy loss for the classification task of distinguishing the data samples from the noise samples.

Original: To avoid calculating the partition function, one considers it as an additional parameter, namely we consider an augmented vector of parameters ˜
Revised: To avoid calculating the partition function, one considers it as an additional parameter, namely we consider an augmented vector of parameters ˜

Original: θ =
Revised: θ =

Original: ( θ, c ) and let 
Revised: ( θ, c ) and let p ˜ θ ( x ) =

Original: [Equation]
Revised: exp( E θ ( x ) − c ) .

Original: The crucial property of the NCE loss is that it has a unique minimizer:
Revised: The crucial property of the NCE loss is that it has a unique minimizer:

Original: Lemma 2 (Gutmann & Hyv ¨ arinen 2012) .
Revised: Lemma 2 (Gutmann & Hyv ¨ arinen 2012) .

Original: The NCE objective in Definition 1 is uniquely minimized at θ = θ ∗ and c = log( (cid:82) x
Revised: The NCE objective in Definition 1 is uniquely minimized at θ = θ ∗ and c = log( (cid:82) x

Original: exp( E θ ∗ ( x ))
Revised: exp( E θ ∗ ( x ))

Original: dx ) provided that the support of Q contains that of P ∗ .
Revised: dx ) provided that the support of Q contains that of P ∗ .

Original: We will be focusing on the Hessian of the loss L , as the crucial object governing both the algorithmic and statistical difficulty of the resulting objective.
Revised: We will be focusing on the Hessian of the loss L , as the crucial object governing both the algorithmic and statistical difficulty of the resulting objective.

Original: We will show the following two main results:
Revised: We will show the following two main results:

Original: Theorem 3 (Exponentially flat Hessian) .
Revised: Theorem 3 (Exponentially flat Hessian) .

Original: For d > 0 large enough, there exists a distribution P ∗ =
Revised: For d > 0 large enough, there exists a distribution P ∗ =

Original: P θ ∗ over R d such that
Revised: P θ ∗ over R d such that

Original: • E P ∗ [ x ] = 0 and E P ∗ [ xx ⊤ ] = I d . • P ∗ is a product distribution, namely p ∗ ( x 1 , x 2 , . . .
Revised: • E P ∗ [ x ] = 0 and E P ∗ [ xx ⊤ ] = I d . • P ∗ is a product distribution, namely p ∗ ( x 1 , x 2 , . . .

Original: , x d ) = (cid:81) di =1 p ∗ ( x i ) .
Revised: , x d ) = (cid:81) di =1 p ∗ ( x i ) .

Original: • The NCE loss when using q = N (0 , I d ) as the noise distribution has the property that ∥∇ 2 L ( θ ∗ )
Revised: • The NCE loss when using q = N (0 , I d ) as the noise distribution has the property that ∥∇ 2 L ( θ ∗ )

Original: ∥ 2 ≤ exp ( − Ω( d )) .
Revised: ∥ 2 ≤ exp ( − Ω( d )) .

Original: By way of remarks, note that the above example of a problematic distribution P ∗ is extremely simple.
Revised: We remark the above example of a problematic distribution P ∗ is extremely simple.

Original: Namely, P ∗ is a product distribution, with 0 mean and identity covariance.
Revised: Namely, P ∗ is a product distribution, with 0 mean and identity covariance.

Original: It actually is also the case that P ∗ is log-concave—which is typically thought of as an “easy” class of distributions to learn due to the fact that log-concave distributions are unimodal.
Revised: It actually is also the case that P ∗ is log-concave—which is typically thought of as an “easy” class of distributions to learn due to the fact that log-concave distributions are unimodal.

Original: Also, by way of remarks, the fact that the Hessian is exponentially flat near the optimum means that gradient-descent based optimization without additional tricks (e.g., gradient normalization, second order methods like Newton’s method) will fail.
Revised: The fact that the Hessian is exponentially flat near the optimum means that gradient-descent based optimization without additional tricks (e.g., gradient normalization, second order methods like Newton’s method) will fail.

Original: (See, e.g., Theorem 4.1 and 4.2 in Liu et al.
Revised: (See, e.g., Theorem 4.1 and 4.2 in Liu et al.

Revised: For us, this will be merely an intermediate result.

Revised: We will address a more fundamental issue of the sample complexity of

Revised: NCE, which is independent of the optimization algorithm used.

Revised: Namely, we will show that without a large number of samples, the best minimizer of the empirical NCE might not be close to the target distribution.

Revised: Proving this will require the development of some technical machinery.

Original: More seriously, we use the result above to show that the asymptotic statistical complexity, using the above choice of P ∗ , Q , is exponentially bad in the dimension.
Revised: More precisely, we use the result above to show that the asymptotic statistical complexity, using the above choice of P ∗ , Q , is exponentially bad in the dimension.

Original: This substantially clarifies results in
Revised: This substantially clarifies results in

Original: Gutmann & Hyv¨arinen (2012), who provide an expression for the asymptotic statistical complexity in terms of P ∗ , Q (Theorem 3, Gutmann & Hyv¨arinen (2012)), but from which it’s very difficult to glean quantitatively how bad the dependence on dimension can be for a particular choice of P ∗ , Q .
Revised: Gutmann & Hyv¨arinen (2012), who provide an expression for the asymptotic statistical complexity in terms of P ∗ , Q (Theorem 3, Gutmann & Hyv¨arinen (2012)), but from which it’s very difficult to glean quantitatively how bad the dependence on dimension can be for a particular choice of P ∗ , Q .

Revised: Unlike the landscape issues that (Liu et al., 2021) point out, the statistical issues are impossible to fix with a better optimization algorithm: they are fundamental limitations of the NCE loss.

Original: Theorem 4 (Asymptotic Statistical Complexity) .
Revised: Theorem 4 (Asymptotic Statistical Complexity) .

Original: Let d > 0 be sufficiently large and Q = N (0 , I d ) .
Revised: Let d > 0 be sufficiently large and Q = N (0 , I d ) .

Original: Let ˆ θ n be the optimizer for the empirical NCE loss L n ( θ ) with the data distribution P ∗ given by
Revised: Let ˆ θ n be the optimizer for the empirical NCE loss L n ( θ ) with the data distribution P ∗ given by

Original: Theorem 3 above and noise distribution Q .
Revised: Theorem 3 above and noise distribution Q .

Original: Then, as n → ∞ , the mean-squared error satisfies
Revised: Then, as n → ∞ , the mean-squared error satisfies

Original: [Equation]
Revised: [Equation]

Original:  E XPONENTIALLY FLAT H ESSIAN : P ROOF OF T HEOREM
Revised:  E XPONENTIALLY FLAT H ESSIAN : P ROOF OF T HEOREM

Original: The proof of Theorem 3 consists of three ingredients.
Revised: The proof of Theorem 3 consists of three ingredients.

Original: First, in Section 3.1, we will compute an algebraically convenient upper bound for the spectral norm of the Hessian of the loss (eq.
Revised: First, in Section 3.1, we will compute an algebraically convenient upper bound for the spectral norm of the Hessian of the loss (eq.

Original: (1)) in the case when { P θ } belongs to an exponential family, in terms of the TV distance TV( P ∗ , Q )between the true data distribution P ∗ and the noise distribution Q , and the Fisher information matrix of the sufficient statistics at θ ∗ . 
Revised: We will restrict our attention to the case when { P θ } belongs to an exponential family. The upper bound will be in terms of the total variation distance TV( P ∗ , Q ) and the Fisher information matrix of the sufficient statistics at θ ∗ . Here, P ∗ denotes the true data distribution and Q denotes the noise distribution.

Original: Then, in Section 3.2, we construct a distribution P ∗ for which the TV distance between P ∗ and Q is large.
Revised: Then, in Section 3.2, we construct a distribution P ∗ for which the TV distance between P ∗ and Q is large.

Original: We do this by “tensorizing” a univariate distribution.
Revised: We do this by “tensorizing” a univariate distribution.

Original: Namely, we construct a univariate distribution with mean 0 and variance 1 that is at a constant TV distance from a standard univariate
Revised: Namely, we construct a univariate distribution with mean 0 and variance 1 that is at a constant TV distance from a standard univariate

Original: Gaussian.
Revised: Gaussian.

Original: Then, we use the fact that the Hellinger distance tensorizes, along with the relationship between TV and Hellinger distance, to show that TV ( P ∗ , Q ) ≥ 1 − δ d for some constant δ < 1 .
Revised: Then, we use the fact that the Hellinger distance tensorizes, along with the relationship between TV and Hellinger distance, to show that TV ( P ∗ , Q ) ≥ 1 − δ d for some constant δ < 1 .

Original: (We refer the reader to Wasserman (2020) for a detailed review ofthese distance measures.)
Revised: (See Wasserman (2020) for a detailed review of distance measures.)

Original: Section 3. bounds the Fisher information matrix term, completing all the components required to establish Theorem 3.
Revised: Section 3.3 bounds the Fisher information matrix term, completing all the components required to establish Theorem 3.

Original: 3. B OUNDING THE H ESSIAN IN TERMS OF TV DISTANCE
Revised: 3. B OUNDING THE H ESSIAN IN TERMS OF TV DISTANCE

Original: Suppose { P θ } is an exponential family of distributions, that is p θ ( x ) =
Revised: Suppose { P θ } is an exponential family of distributions, that is p θ ( x ) =

Original: exp( θ ⊤ T ( x )) , where T ( x ) is a known function.
Revised: exp( θ ⊤ T ( x )) , where T ( x ) is a known function.

Original: Then, a straightforward calculation (see e.g., Appendix A in Liu et al.
Revised: Then, a straightforward calculation (see e.g., Appendix A in Liu et al.

Original: (2021)) shows that the gradient and the Hessian  the NCE loss (eq.
Revised: (2021)) shows that the gradient and the Hessian of the NCE loss (eq.

Original: (1)) with respect to θ have the following form:
Revised: (1)) with respect to θ have the following forms:

Original: [Equation]
Revised: [Equation]

Original: For θ = θ ∗ and p θ =
Revised: For θ = θ ∗ and p θ =

Original: p ∗ , we have
Revised: p ∗ , we have

Original: [Equation]
Revised: [Equation]

Original: The second line holds since p ∗ q p ∗ + q = min( p ∗ , q ) · max( p ∗ ,q ) p ∗ + q ≤ min( p ∗ , q ) .
Revised: The second line holds since p ∗ q p ∗ + q = min( p ∗ , q ) · max( p ∗ ,q ) p ∗ + q ≤ min( p ∗ , q ) .

Original: Applying the vector version of the Cauchy-Schwarz inequality (Lemma 9, Appendix A) to eq.
Revised: Applying the matrix version of the Cauchy-Schwarz inequality (Lemma 9, Appendix A) to eq.

Original: (7) with two parts min( p ∗ ( x ) ,q ( x )) √ p ∗ ( x ) and T ( x ) T ( x ) ⊤ (cid:112) p ∗ ( x ) , we obtain 
Revised: (7) with two parts min( p ∗ ( x ) ,q ( x )) √ p ∗ ( x ) and T ( x ) T ( x ) ⊤ (cid:112) p ∗ ( x ) , we obtain ∥∇ [Equation] P [Equation]

Revised: (cid:18)(cid:90)

Original: [Equation]
Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: [Equation]

Original: We bound the two terms in the product above separately.
Revised: We bound the two terms in the product above separately.

Revised: The first term is small when P ∗ and Q are significantly different.

Revised: The second term is an upper bound of the Frobenius norm of the Fisher matrix at P ∗ .

Revised: We will construct P ∗ such that the first term dominates, giving us the upper bound required.

Original: 3. C ONSTRUCTING THE HARD DISTRIBUTION P ∗
Revised: 3. C ONSTRUCTING THE HARD DISTRIBUTION P ∗

Original: The hard distribution P ∗ over R d will have the property that E P ∗ [ x ] = 0 , E P ∗ [ xx ⊤ ]
Revised: The hard distribution P ∗ over R d will have the property that E P ∗ [ x ] = 0 , E P ∗ [ xx ⊤ ]

Original: = I d , but will still have large TV distance from the standard Gaussian Q = N (0 , I d ) .
Revised: = I d , but will still have large TV distance from the standard Gaussian Q = N (0 , I d ) .

Original: This distribution will simply be a product distribution—the following lemma formalizes our main trick of tensorization to construct a distribution having large TV distance with the Gaussian.
Revised: This distribution will simply be a product distribution—the following lemma formalizes our main trick of tensorization to construct a distribution having large TV distance with the Gaussian.

Original: Lemma 5.
Revised: Lemma 5.

Original: Let d > 0 be given.
Revised: Let d > 0 be given.

Original: Let Q = N (0 , I d ) be the standard Gaussian in R d .
Revised: Let Q = N (0 , I d ) be the standard Gaussian in R d .

Original: Then, for some δ < 1 , there exists a log-concave distribution P (also over R d ) with mean 0 and covariance I d satisfying TV( P, Q ) ≥ 1 − δ d .
Revised: Then, for some δ < 1 , there exists a log-concave distribution P (also over R d ) with mean 0 and covariance I d satisfying TV( P, Q ) ≥ 1 − δ d .

Original: Proof.
Revised: Proof.

Original: Let ˆ Q denote the standard normal distribution over R .
Revised: Let ˆ Q denote the standard normal distribution over R .

Original: Let ˆ P be any other distribution over
Revised: Let ˆ P be any other distribution over

Original: R with mean 0 and variance 1 that satisfies ρ ( ˆ P, ˆ Q ) =
Revised: R with mean 0 and variance 1 that satisfies ρ ( ˆ P, ˆ Q ) =

Original: δ < 1 , where ρ ( ˆ P, ˆ Q ) =
Revised: δ < 1 , where ρ ( ˆ P, ˆ Q ) =

Original: (cid:82) x √ ˆ p ˆ q dx is the
Revised: (cid:82) x √ ˆ p ˆ q dx is the

Original: Bhattacharya coefficient.
Revised: Bhattacharya coefficient.

Original: Since ρ tensorizes (Wasserman, 2020), we have that ρ ( ˆ P d , ˆ Q d ) =
Revised: Since ρ tensorizes (Wasserman, 2020), we have that ρ ( ˆ P d , ˆ Q d ) =

Original: ρ ( ˆ P, ˆ Q ) d for any d > 1 .
Revised: ρ ( ˆ P, ˆ Q ) d for any d > 1 .

Original: We can then write the Hellinger distance between P, Q as
Revised: We can then write the Hellinger distance between P, Q as

Original: [Equation]
Revised: [Equation]

Original: Further, we also know that
Revised: Further, we also know that

Original: [Equation]
Revised: [Equation]

Original: Setting P =
Revised: Setting P =

Original: ˆ P d and noting that ˆ Q d = Q = N (0 , I d ) , we have TV( P, Q ) ≥ 1 − δ d .
Revised: ˆ P d and noting that ˆ Q d = Q = N (0 , I d ) , we have TV( P, Q ) ≥ 1 − δ d .

Original: Finally, if the chosen ˆ P is a log-concave distribution, then so is ˆ P d ,  which completes the proof.
Revised: Finally, if the chosen ˆ P is a log-concave distribution, then so is ˆ P d , since the product of log-concave distributions is log-concave, which completes the proof.

Original: We will now explicitly define the distribution P ∗ that we will work with for rest of the paper.
Revised: We will now explicitly define the distribution P ∗ that we will work with for rest of the paper.

Original: Definition 6.
Revised: Definition 6.

Original: Consider the exponential family (cid:8) p θ ( x ) = exp (cid:0)
Revised: Consider the exponential family (cid:8) p θ ( x ) = exp (cid:0)

Original: θ ⊤ T ( x ) (cid:1)(cid:9) θ ∈ R d +1 given by the sufficient statistics T ( x ) =
Revised: θ ⊤ T ( x ) (cid:1)(cid:9) θ ∈ R d +1 given by the sufficient statistics T ( x ) =

Original: ( x 41 , . .
Revised: ( x 41 , . .

Original: , x 4 d , 1) .
Revised: , x 4 d , 1) .

Original: Let P ∗ =
Revised: Let P ∗ =

Original: ˆ P d where ˆ P is the distribution on R with density function ˆ p given by
Revised: ˆ P d where ˆ P is the distribution on R with density function ˆ p given by

Original: [Equation]
Revised: [Equation]

Original: We will set the constant of proportionality C and σ appropriately to ensure that ˆ P has mean 0 and variance 1.
Revised: We will set the constant of proportionality C and σ appropriately to ensure that ˆ P has mean 0 and variance 1.

Original: Note that P ∗ =
Revised: Note that P ∗ =

Original: P θ ∗ for θ ∗ = − (cid:0) 1 σ 4 , . . .
Revised: P θ ∗ for θ ∗ = − (cid:0) 1 σ 4 , . . .

Original: , 1 σ 4 , log C (cid:1) .
Revised: , 1 σ 4 , log C (cid:1) .

Original: Since d 2 log ˆ p dx= − 12 x 2 σ 4 ≤ 0 , ˆ p is log-concave.
Revised: Since d 2 log ˆ p dx= − 12 x 2 σ 4 ≤ 0 , ˆ p is log-concave.

Original: Further, symmetry of ˆ p around the origin gives
Revised: Further, symmetry of ˆ p around the origin gives

Original: E [ ˆ P ] = 0 , and the choice of σ ensures that Var [ ˆ P ] = 1 .
Revised: E [ ˆ P ] = 0 , and the choice of σ ensures that Var [ ˆ P ] = 1 .

Original: The normalizing constant C satisfies
Revised: The normalizing constant C satisfies

Original: [Equation]
Revised: [Equation]

Original: Substituting t = x 4 σ 4 , dt = 4 x 3 σ 4 dx = 4 t 3 / 4 σ dx gives
Revised: Substituting t = x 4 σ 4 , dt = 4 x 3 σ 4 dx = 4 t 3 / 4 σ dx gives

Original: [Equation]
Revised: [Equation]

Original: where Γ( z ) is the gamma function defined as Γ( z ) (cid:82) ∞ 

Revised: where Γ( z ) is the gamma function defined as Γ( z ) (cid:82) ∞

Original: x z − 1 e − x dx.
Revised: x z − 1 e − x dx.

Original: The variance is given by
Revised: The variance is given by

Original: [Equation]
Revised: [Equation]

Original: The same substitution as above gives
Revised: The same substitution as above gives

Original: [Equation]
Revised: [Equation]

Original: Thus, setting σ = (cid:113) 4Γ(5 / 4) Γ(3 / 4) results in Var [ ˆ P ] = 1 .
Revised: Thus, setting σ = (cid:113) 4Γ(5 / 4) Γ(3 / 4) results in Var [ ˆ P ] = 1 .

Original: Correspondingly, we have C = 4Γ(5 / 4) 3 / 2 √ Γ(3 / 4) .
Revised: Correspondingly, we have C = 4Γ(5 / 4) 3 / 2 √ Γ(3 / 4) .

Original: For this choice of ˆ P , the Bhattacharya coefficient ρ ( ˆ P, ˆ Q ) is given by:
Revised: For this choice of ˆ P , the Bhattacharya coefficient ρ ( ˆ P, ˆ Q ) is given by:

Original: [Equation]
Revised: [Equation]

Original: Thus, in the proof of Lemma 5, we can use this choice of ˆ P , and we have that for δ = 0 . 991 and
Revised: Thus, in the proof of Lemma 5, we can use this choice of ˆ P , and we have that for δ = 0 . 991 and

Original: P ∗ =
Revised: P ∗ =

Original: ˆ P d , TV( P ∗ , Q ) ≥ 1 − δ d , as required.
Revised: ˆ P d , TV( P ∗ , Q ) ≥ 1 − δ d , as required.

Original: 3. B OUNDING THE F ISHER INFORMATION MATRIX
Revised: 3. B OUNDING THE F ISHER INFORMATION MATRIX

Original: In this subsection, we bound the second factor in eq.
Revised: In this subsection, we bound the second factor in eq.

Original: (8), which is  the Fisher information matrix at θ ∗ .
Revised: (8), which is an upper bound on the Frobenius norm of the Fisher information matrix at θ ∗ .

Original: Lemma 7.
Revised: Lemma 7.

Original: The Fisher information matrix at θ ∗ satisfies
Revised: For some constant M > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: for some constant M > 0 .

Original: Proof.
Revised: Proof.

Original: Recall that T ( x ) =
Revised: Recall that T ( x ) =

Original: ( x 41 , . .
Revised: ( x 41 , . .

Original: , x 4 d , 1) .
Revised: , x 4 d , 1) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Therefore, by linearity of expectation, and using the fact that P ∗ is a product distribution, 
Revised: Therefore, by linearity of expectation, and using the fact that P ∗ is a product distribution, [Equation] M

Original: [Equation]
Revised: [Equation]

Original: for an appropriate choice of constant M .
Revised: for an appropriate choice of constant M .

Original: This constant exists since all the expectations above are bounded owing to the fact that the exponential density ˆ p dominates in the integrals.
Revised: This constant exists since all the expectations above are bounded owing to the fact that the exponential density ˆ p dominates in the integrals.

Original: We now have all elements ready to prove Theorem 3.

Original: 3. P UTTING THINGS TOGETHER
Revised: 3. P UTTING THINGS TOGETHER

Original: For P ∗ defined as above, and Q = N (0 , I d ) , Lemma 5 ensures that [Equation] for δ = 0 .
Revised: For P ∗ defined as above, and Q = N (0 , I d ) , Lemma 5 ensures that 1 − TV( P ∗ , Q ) ≤ δ d , for δ = 0 .

Original: From Lemma 7, we have that(cid:90)
Revised: From Lemma 7, we have that

Original: [Equation]
Revised: [Equation]

Original: Substituting these bounds in eq.
Revised: Substituting these bounds in eq.

Original: (8), we get that
Revised: (8), we get that

Original: [Equation]
Revised: [Equation]

Original: By construction, p ∗ is a product distribution with E p ∗ [ x ] = 0 and E p ∗ (cid:2) xx ⊤ (cid:3) =
Revised: By construction, p ∗ is a product distribution with E p ∗ [ x ] = 0 and E p ∗ (cid:2) xx ⊤ (cid:3) =

Original: I d , which completes the proof of the theorem.
Revised: I d , which completes the proof of the theorem.

Original:  P ROOF OF T HEOREM
Revised:  P ROOF OF T HEOREM

Original: We will bound the error of the optimizer ˆ θ n of the empirical NCE loss (eq.
Revised: We will bound the error of the optimizer ˆ θ n of the empirical NCE loss (eq.

Original: (2)) using the bias-variance decomposition of MSE.
Revised: (2)) using the bias-variance decomposition of MSE.

Original: To do this, we will reason about the random variable √ n (ˆ θ n − θ ∗ ) ; let Σ be its covariance matrix.
Revised: To do this, we will reason about the random variable √ n (ˆ θ n − θ ∗ ) ; let Σ be its covariance matrix.

Original: Since ˆ θ n is an unbiased estimate of θ ∗ , the MSE decomposes as
Revised: Since ˆ θ n is an unbiased estimate of θ ∗ , the MSE decomposes as

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 4 proceeds as follows.
Revised: The proof of Theorem 4 proceeds as follows.

Original: In Section 4.1, we show that the random variable √ n (ˆ θ n − θ ∗ ) is asymptotically normal with mean 0 and covariance matrix Σ given by
Revised: In Section 4.1, we show that the random variable √ n (ˆ θ n − θ ∗ ) is asymptotically normal with mean 0 and covariance matrix Σ given by

Original: [Equation]
Revised: [Equation]

Original: We prove that the Hessian ∇ 2 θ L ( θ ∗ ) is invertible in Appendix C, so that the above expression is well-defined.
Revised: We prove that the Hessian ∇ 2 θ L ( θ ∗ ) is invertible in Appendix C, so that the above expression is well-defined.

Original: Since Σ ⪰ 0 (it is a covariance matrix), to get a lower bound on Tr(Σ) , it suffices to get a lower bound on the largest eigenvalue of Σ .
Revised: Since Σ ⪰ 0 (it is a covariance matrix), to get a lower bound on Tr(Σ) , it suffices to get a lower bound on the largest eigenvalue of Σ .

Original: Looking at the factors on the right hand side of eq.
Revised: Looking at the factors on the right hand side of eq.

Original: (14), we note first that Theorem 3 ensures an exponential lower bound on all eigenvalues of ∇ 2 θ L ( θ ∗ ) − 1 .
Revised: (14), we note first that Theorem 3 ensures an exponential lower bound on all eigenvalues of ∇ 2 θ L ( θ ∗ ) − 1 .

Original: The bulk of the proof towards lower bounding the largest eigenvalue of Σ consists of lower bounding  the directional variance of √ n ∇ θ L n ( θ ∗ ) in a suitably chosen direction v in terms of v ⊤ ∇ 2 θ L ( θ ∗ ) v .In
Revised: The bulk of the proof towards lower bounding the largest eigenvalue of Σ consists of lower bounding Var (cid:2) v ⊤ · √ n ∇ θ L n ( θ ∗ ) (cid:3) ) , the directional variance of √ n ∇ θ L n ( θ ∗ ) along a suitably chosen direction v in terms of v ⊤ ∇ 2 θ L ( θ ∗ ) v .

Original:  Section 4.2 and Section 4.3, we use anti-concentration bounds to prove such variance lower bounds.
Revised: In Section 4.2 and Section 4.3, we use anti-concentration bounds to prove such variance lower bounds.

Original: 4. G AUSSIAN LIMIT OF √ n (ˆ θ n − θ ∗ )
Revised: 4. G AUSSIAN LIMIT OF √ n (ˆ θ n − θ ∗ )

Original: To begin, we will show that √ n (ˆ θ n − θ ∗ ) behaves as a Gaussian random variable as n → ∞ .
Revised: To begin, we will show that √ n (ˆ θ n − θ ∗ ) behaves as a Gaussian random variable as n → ∞ .

Original: Recall that the empirical NCE loss is given by eq.
Revised: Recall that the empirical NCE loss is given by eq.

Original: [Equation]
Revised: [Equation]

Original: where x i ∼ P ∗ and y i ∼ Q are i.i.d.
Revised: where x i ∼ P ∗ and y i ∼ Q are i.i.d.

Original: Let ˆ θ n be the optimizer for L n .
Revised: Let ˆ θ n be the optimizer for L n .

Original: Then, by the Taylor expansion of ∇ θ L n around θ ∗ , we have
Revised: Then, by the Taylor expansion of ∇ θ L n around θ ∗ , we have

Original: [Equation]
Revised: [Equation]

Original: by Gutmann & Hyv¨arinen (2012), who also show in their Theorem 2 that ˆ θ n is a consistent estimator of θ ∗ ; hence, as n → ∞ , (cid:13)(cid:13)(cid:13) ˆ θ n − θ ∗ (cid:13)(cid:13)(cid:13) 2 → 0 .
Revised: by Gutmann & Hyv¨arinen (2012), who also show in their Theorem 2 that ˆ θ n is a consistent estimator of θ ∗ ; hence, as n → ∞ , (cid:13)(cid:13)(cid:13) ˆ θ n − θ ∗ (cid:13)(cid:13)(cid:13) 2 → 0 .

Original: Gutmann & Hyv¨arinen (2012, Lemma 12) also assertthat the Hessian of the empirical NCE loss (eq.
Revised: Gutmann & Hyv ¨ arinen (2012, Lemma 12) also assertthat the Hessian of the empirical NCE loss (eq.

Original: at θ ∗ converges in probability to the Hessian of the true NCE loss (definition 1) at θ ∗ , i.e., ∇ 2 θ L n ( θ ∗ ) − 1 P −→ ∇ 2 θ L ( θ ∗ ) − 1 .
Revised: at θ ∗ converges in probability to the Hessian of the true NCE loss (definition 1) at θ ∗ , i.e., ∇ 2 θ L n ( θ ∗ ) − 1 P −→ ∇ 2 θ L ( θ ∗ ) − 1 .

Original: On the other hand, by the Central Limit Theorem, √ n ∇ θ L n ( θ ∗ ) converges to a Gaussian with mean E [ √ n ∇ θ L n ( θ ∗ )] = √ n ∇ θ L ( θ ∗ ) = 0 , and covariance Var [ √ n ∇ θ L n ( θ ∗ )] .
Revised: On the other hand, by the Central Limit Theorem, √ n ∇ θ L n ( θ ∗ ) converges to a Gaussian with mean E [ √ n ∇ θ L n ( θ ∗ )] = √ n ∇ θ L ( θ ∗ ) = 0 , and covariance Var [ √ n ∇ θ L n ( θ ∗ )] .

Original: With these considerations, we conclude that the random variable √ n (ˆ θ n − θ ∗ ) in eq.
Revised: With these considerations, we conclude that the random variable √ n (ˆ θ n − θ ∗ ) in eq.

Original: (15) is asymptotically a Gaussian with mean 0 and covariance Σ = ∇ 2 θ L ( θ ∗ ) − 1 Var [ √ n ∇ θ L n ( θ ∗ )] ∇ 2 θ L ( θ ∗ ) − 1 , as defined in eq.
Revised: (15) is asymptotically a Gaussian with mean 0 and covariance Σ = ∇ 2 θ L ( θ ∗ ) − 1 Var [ √ n ∇ θ L n ( θ ∗ )] ∇ 2 θ L ( θ ∗ ) − 1 , as defined in eq.

Original: Next, we introduce some quantities which will be useful in the subsequent calculations.
Revised: Next, we introduce some quantities which will be useful in the subsequent calculations.

Original: As we already have a handle on the spectrum of ∇ 2 θ L ( θ ∗ ) from Theorem 3, the main object of our focus in eq.
Revised: As we already have a handle on the spectrum of ∇ 2 θ L ( θ ∗ ) from Theorem 3, the main object of our focus in eq.

Original: (14) is the term Var [ √ n ∇ θ L n ( θ ∗ )] .
Revised: (14) is the term Var [ √ n ∇ θ L n ( θ ∗ )] .

Original: In particular, since we are concerned with the directional variance of Σ , we will reason about Var (cid:2) v ⊤ · √ n ∇ θ L n ( θ ∗ ) (cid:3) for a fixed vector of ones, i.e., v = 1 d +1 .
Revised: In particular, since we are concerned with the directional variance of Σ , we will reason about Var (cid:2) v ⊤ · √ n ∇ θ L n ( θ ∗ ) (cid:3) for a fixed vector of ones, i.e., v = 1 d +1 .

Original: This vector has the property that for all x, v ⊤ T ( x ) ≥ 1 , as all non-constant coordinates of T are non-negative, and the remaining coordinate is 1 .
Revised: This vector has the property that for all x, v ⊤ T ( x ) ≥ 1 , as all non-constant coordinates of T are non-negative, and the remaining coordinate is 1 .

Original: Writing out the variance term explicitly, we have Var (cid:2) [Equation] ⊤
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: (using linearity and independence) [Equation] (cid:123)(cid:122) (cid:125) A
Revised: where x i ∼ P ∗ and y i ∼ Q .

Original: [Equation] x ) + [Equation] 4 Var y
Revised: Writing out the variance term explicitly, we have

Original: [Equation]
Revised: [Equation]

Original: p
Revised: (using linearity and independence)

Original: [Equation]
Revised: [Equation]

Original: (cid:21) (cid:124) (cid:123)(cid:122) (cid:125)

Original: [Equation]
Revised: [Equation]

Original: ( y ) .

Original: Define A ( x ) =
Revised: Define A ( x ) =

Original: q ( x ) · v ⊤ T ( x ) p ∗ ( x )+ q ( x )
Revised: q ( x ) · v ⊤ T ( x ) p ∗ ( x )+ q ( x )

Original: =
Revised: =

Original: R 1 ( x ) 1+ R 1 ( x ) v ⊤ T ( x ) where R 1 ( x ) =
Revised: R 1 ( x ) 1+ R 1 ( x ) v ⊤ T ( x ) where R 1 ( x ) =

Original: q ( x ) p ∗ ( x ) and B (y ) = p ∗ ( y ) · v ⊤ T ( y ) p ∗ ( y )+ q ( y ) = R 2 ( y ) 1+ R 2 ( y ) v ⊤ T ( y ) where R 2 ( y ) = p ∗ ( y ) q ( y ) .
Revised: q ( x ) p ∗ ( x ) and B (

Original: To show that Var x ∼ p ∗ [ A ( x )] and Var y ∼ q [ B ( y )] are large, we will need anti-concentration bounds on R 1 ( x ) and R 2 ( y ) .

Original: 4. A NTI - CONCENTRATION OF R 1 ( x ) , R 2 ( y )
Revised: 4. A NTI - CONCENTRATION OF R 1 ( x ) , R 2 ( y )

Original: Next, we show that R 1 and R 2 satisfy (quantitative) anti-concentration.
Revised: Next, we show that R 1 and R 2 satisfy (quantitative) anti-concentration.

Original: We show this by a relatively straightforward application of the Berry-Esseen Theorem, and the proof is given in Appendix B.
Revised: We show this by a relatively straightforward application of the Berry-Esseen Theorem, and the proof is given in Appendix B.

Original: Precisely, we show:
Revised: Precisely, we show:

Original: Lemma 8.
Revised: Lemma 8.

Original: Let d > 0 be sufficiently large.
Revised: Let d > 0 be sufficiently large.

Original: Let p =
Revised: Let p =

Original: ˆ p d and q = ˆ q d be any product distributions, and define R ( x ) =
Revised: ˆ p d and q = ˆ q d be any product distributions, and define R ( x ) =

Original: q ( x ) p ( x ) .
Revised: q ( x ) p ( x ) .

Original: Suppose we have the following third moment bound: E x ∼ ˆ p [Equation]
Revised: Suppose we have the following third moment bound: E x ∼ ˆ p (cid:20)(cid:16) log ˆ q ˆ p (cid:17) 3 (cid:21) < ∞ .

Original: Then, for any ϵ , there exist constants α
Revised: Then, for any ϵ , there exist constants α

Original: = α (ˆ p, ˆ q, ϵ ) , µ = µ (ˆ p, ˆ q, ϵ ) < 0 such that
Revised: = α (ˆ p, ˆ q, ϵ ) , µ = µ (ˆ p, ˆ q, ϵ ) < 0 such that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation] [Equation] [Equation] Instantiating Lemma 8 for the pair ( p ∗ , q ) gives us the anti-concentration result for R 1 , while instantiating it for the reversed pair ( q, p ∗ ) gives us the anti-concentration result for R 2 .
Revised: Instantiating Lemma 8 for the pair ( p ∗ , q ) gives us the anti-concentration result for R 1 , while instantiating it for the reversed pair ( q, p ∗ ) gives us the anti-concentration result for R 2 .

Original: We can verify that the third moment condition holds in both instantiations, since in both the cases, log(ˆ q/ ˆ p ) is a polynomial.
Revised: We can verify that the third moment condition holds in both instantiations, since in both the cases, log(ˆ q/ ˆ p ) is a polynomial.

Original: Crucially, we will also utilize the fact that the constant µ is negative (as it equals − KL (ˆ p || ˆ q ) ).
Revised: Crucially, we will also utilize the fact that the constant µ is negative (as it equals − KL (ˆ p || ˆ q ) ).

Original: We are now ready to bound the variance of A ( x ) and B ( y ) .
Revised: We are now ready to bound the variance of A ( x ) and B ( y ) .

Original: 4. B OUNDING THE VARIANCE OF A ( x ) , B ( y )
Revised: 4. B OUNDING THE VARIANCE OF A ( x ) , B ( y )

Original: Recall that A 
Revised: Recall that A ( x ) =

Original: [Equation]
Revised: R 1 ( x ) · v ⊤ T ( x ) 1+ R 1 ( x ) and B ( y ) = R 2 ( y ) · v ⊤ T ( y ) 1+ R 2 ( y ) .

Original: Let µ, α be the constants given by
Revised: Let µ, α be the constants given by

Original: Lemma 8 for p ∗ , q, ϵ .
Revised: Lemma 8 for p ∗ , q, ϵ .

Original: Further, let L 1 = exp (cid:16) µd − α √ d (cid:17) and L 2 = exp (cid:16) µd + α √ d (cid:17) .
Revised: Further, let L 1 = exp (cid:16) µd − α √ d (cid:17) and L 2 = exp (cid:16) µd + α √ d (cid:17) .

Original: Since the mapping x (cid:55)→ x 1+ x is monotonically increasing in x ,
Revised: Since the mapping x (cid:55)→ x 1+ x is monotonically increasing in x ,

Original: [Equation]
Revised: [Equation]

Original: Let T up be such that
Revised: Let T up be such that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Revised: In Appendix D

Original: [Equation]
Revised: [Equation]

Original: In Appendix D, we show that some T up
Revised: we show that some T up

Original: = O ( σ 2 √ d ) suffices for this to hold.
Revised: = O ( σ 2 √ d ) suffices for this to hold.

Original: Then, from eq.
Revised: Then, from eq.

Original: (17), we have
Revised: (17), we have

Original: [Equation]
Revised: [Equation]

Original: [Equation] ) = ⇒ P
Revised: √ d + 1 ∥ T ( x ) ∥ 1 + Li(cid:19) ∧ (cid:18)

Original: [Equation] (cid:19)(cid:21)[Equation]
Revised: ∥ T ( x ) ∥ ≤ T up (cid:19)(cid:21)

Original: [Equation] (union bound with eq.
Revised: ≥8 − ϵ (union bound with eq.

Original: =
Revised: =

Original: ⇒
Revised: ⇒

Original: P [Equation] ∼ 
Revised: P x ∼ p ∗ (cid:20) R 1 ( x ) v ⊤ T ( x ) 1 + R 1 ( x ) ≤ √ d + 1 L 1 T up 1 + L(cid:21)

Original: [Equation] = ⇒
Revised: ≥8 − ϵ = ⇒

Original: P [Equation]
Revised: P x ∼ p ∗ (cid:20)

Original: [Equation] for ϵ ≤ 18 .
Revised: A ( x ) ≤ √ d + 1 L 1 T up 1 + L(cid:21) ≥4 , for ϵ ≤ 18 .

Original: On the other hand, recall also that v satisfies v ⊤ T ( x ) ≥ 1 for all x .
Revised: On the other hand, recall also that v satisfies v ⊤ T ( x ) ≥ 1 for all x .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: P x ∼ p ∗ (cid:20) R 1 ( x ) 1 + R 1 ( x ) ≥ L1 + L(cid:21)

Original:  = [Equation]
Revised: ≥2 − ϵ = ⇒

Original: P [Equation] 1 ( x ) ≥
Revised: P x ∼ p ∗ (cid:20) R 1 ( x ) · v ⊤ T ( x ) 1 + R 1 ( x ) ≥

Original: L1 + L(cid:21)
Revised: L1 + L(cid:21)

Original: ≥2 − ϵ = ⇒
Revised: ≥2 − ϵ = ⇒

Original: P x ∼ p ∗ (cid:20)
Revised: P x ∼ p ∗ (cid:20)

Original: A ( x ) ≥
Revised: A ( x ) ≥

Original: L1 + L(cid:21) ≥4 .
Revised: L1 + L(cid:21) ≥4 .

Original: Now, consider the event
Revised: Now, consider the event

Original: A 1 = (cid:26) A ( x ) ∈ (cid:20)2 E x ∼ p ∗
Revised: A 1 = (cid:8) A ( x ) ∈ (cid:2) 12 E x ∼ p ∗

Original: [ A ( x )] , 3 2 E x ∼ p ∗
Revised: [ A ( x )] , 32 E x ∼ p ∗

Original: [ A ( x )] (cid:21)(cid:27) .
Revised: [ A ( x )] (cid:3)(cid:9) .

Original: If this event were to intersect both the events
Revised: If this event were to intersect both the events

Original: A 2 = (cid:110)
Revised: A 2 = (cid:110)

Original: A ( x ) ≤ √ d +1 L 1 T up 1+ L(cid:111) and A 3 = (cid:110)
Revised: A ( x ) ≤ [Equation] +

Original:  A ( x ) ≥ L 2 1+ L 2 (cid:111) , then we would have2 E x ∼ p ∗ [ A ( x )] ≤ √ d + 1 L 1 T up 1 + Land 2 E x ∼ p ∗
Revised: [Equation] (cid:111) and A 3 = (cid:110) A ( x ) ≥ L 2 1+ L 2 (cid:111) , then we would have

Original: [ A ( x )] ≥
Revised: [Equation]

Original: L1 + L= ⇒ L L 1 · T up √ d + 1 ·
Revised: and

Original: L 1 + L 2 + 1 ≤ 3 .
Revised: [Equation]

Original: We will show that this cannot be the case.
Revised: We will show that this cannot be the case.

Original: Recall that µ < 0 , which means that L 2 = exp( µd + α √ d ) < 1 for sufficiently large d .
Revised: Recall that µ < 0 , which means that L 2 = exp( µd + α √ d ) < 1 for sufficiently large d .

Original: This means that for sufficiently large d we have:
Revised: This means that for sufficiently large d we have:

Original: [Equation]
Revised: [Equation]

Original: Further [Equation] since L 2 L 1 = exp(2 α √ d ) and T up
Revised: Further, since L 2 L 1 = exp(2 α √ d ) and T up

Original: = O ( σ 2 √ d ) [Equation] we get that
Revised: = O ( σ 2 √ d ) , we get that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows for large enough d since the numerator grows faster than the denominator.
Revised: where the last inequality follows for large enough d since the numerator grows faster than the denominator.

Original: Hence for large enough d [Equation] A 1 cannot intersect both A 2 and A 3 .
Revised: Hence for large enough d , A 1 cannot intersect both A 2 and A 3 .

Original: If the event A 1 is disjoint from A 2 , then
Revised: If the event A 1 is disjoint from A 2 , then

Original: [Equation]
Revised: [Equation]

Original: This finally lower-bounds the variance of A as
Revised: This finally lower-bounds the variance of A as

Original: Var p ∗ [ A ] =

Original: E

Original: [Equation]
Revised: [Equation]

Original: and thus E p ∗ ( A 2 ) − ( E p ∗ A ) 2 =
Revised: and thus E p ∗ ( A 2 ) − ( E p ∗ A ) 2 =

Original: Var p ∗ [ A ] ≥ 116 ( E p ∗ A ) 2 , so that ( E p ∗ A ) 2 ≤ 1617 E p ∗ ( A 2 ) .
Revised: Var p ∗ [ A ] ≥ 116 ( E p ∗ A ) 2 , so that ( E p ∗ A ) 2 ≤ 1617 E p ∗ ( A 2 ) .

Original: Altogether, we get Var p ∗
Revised: Altogether, we get Var p ∗

Original: [ A ] ≥ 117 E p ∗ ( A 2 ) .
Revised: [ A ] ≥ 117 E p ∗ ( A 2 ) .

Original: An analogous argument in the case when A 1 is disjoint with A 3 yields the same bound on the variance.
Revised: An analogous argument in the case when A 1 is disjoint with A 3 yields the same bound on the variance.

Original: Using an identical argument for R 2 and B , we get that for large enough d , Var q [ B ] ≥ 117 E q ( B 2 ) .
Revised: Using an identical argument for R 2 and B , we get that for large enough d , Var q [ B ] ≥ 117 E q ( B 2 ) .

Original: 4. P UTTING THINGS TOGETHER
Revised: 4. P UTTING THINGS TOGETHER

Original: Putting together the lower bounds Var p ∗ [ A ] ≥ 117 E p ∗ ( A 2 ) and Var q [ B ] ≥ 117 E q
Revised: Putting together the lower bounds Var p ∗ [ A ] ≥ 117 E p ∗ ( A 2 ) and Var q [ B ] ≥ 117 E q

Original: ( B 2 ) we showed in the previous subsection, and recalling eq.
Revised: ( B 2 ) we showed in the previous subsection, and recalling eq.

Original: (16), we get
Revised: (16), we get

Revised: Var (cid:2)

Original: [Equation]
Revised: [Table]

Original: (from eq. Finally, since ∇ 2 θ L ( θ ∗ ) is invertible as claimed earlier (Lemma 10, Appendix C), let w be such that v = ∇ 2 θ L ( θ ∗ ) − 1 w .
Revised: Finally, since ∇ 2 θ L ( θ ∗ ) is invertible as claimed earlier (Lemma 10, Appendix C), let w be such that v = ∇ 2 θ L ( θ ∗ ) − 1 w .

Original: Then, recalling the expression for Σ in eq.
Revised: Then, recalling the expression for Σ in eq.

Original: (14), we can conclude that
Revised: (14), we can conclude that

Original: [Equation]
Revised: [Equation]

Original: which gives us the desired bound on the MSE, namely
Revised: which gives us the desired bound on the MSE, namely

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from Theorem 3 and the fact that λ max ( ∇ 2 θ L ( θ ∗ )) − 1 = λ min ( ∇ 2 θ L ( θ ∗ ) − 1 ) .
Revised: where the last inequality follows from Theorem 3 and the fact that λ max ( ∇ [Equation] L [Equation]

Revised: [Equation]

Original:  This concludes the proof of Theorem 4.
Revised: − [Equation] λ min ( ∇ 2 θ L ( θ ∗ ) − 1 ) . This concludes the proof of Theorem 4.

Original:  S IMULATIONS
Revised:  S IMULATIONS

Original: [Figure]

Original: Theorem 4 suggests this plot should be linear, as is observed.

Original: We also verify our results with simulations.
Revised: We also verify our results with simulations.

Original: Precisely, we study the MSE for the empirical NCE loss as a function of the ambient dimension, and recover the dependence from Theorem 4.
Revised: Precisely, we study the MSE for the empirical NCE loss as a function of the ambient dimension, and recover the dependence from Theorem 4.

Original: For dimension d ∈ { 70 , 72 , . . .
Revised: For dimension d ∈ { 70 , 72 , . . .

Original: , 120 } , we generate n = 500 samples from the distribution P ∗(Definition 6) we construct in the theorem.
Revised: , 120 } , we generate n = 500 samples from the distribution P ∗ we construct in the theorem.

Original: We generate an equal number of samples from the noise distribution Q = N (0 , I d ) , and run gradient descent to minimize the empirical NCE loss to obtain ˆ θ n .
Revised: We generate an equal number of samples from the noise distribution Q = N (0 , I d ) , and run gradient descent to minimize the empirical NCE loss to obtain ˆ θ n .

Original: Since we explicitly know what θ ∗ is, we can compute the squared error ∥
Revised: Since we explicitly know what θ ∗ is, we can compute the squared error ∥

Original: ˆ θ n − θ ∗ ∥ 2 .
Revised: ˆ θ n − θ ∗ ∥ 2 .

Original: We run 100 trials of this, where we obtain fresh samples each time from P ∗ and Q , and average the squared errors over the trials to obtain an estimate of the MSE.
Revised: We run 100 trials of this, where we obtain fresh samples each time from P ∗ and Q , and average the squared errors over the trials to obtain an estimate of the MSE.

Original: Figure 1 shows the plot of log MSE versus dimension - we can see that the graph is  linear.
Revised: Figure 1 shows the plot of log MSE versus dimension - we can see that the graph is nearly linear.

Original: This corroborates the bound in Theorem 4, which tells us that as n → ∞ , the MSE scales exponentially with d .
Revised: This corroborates the bound in Theorem 4, which tells us that as n → ∞ , the MSE scales exponentially with d .

Original: This behavior is robust even when the proportion of noise samples to true data samples is changed to 70:30 (though our theory only addresses the 50:50 case).
Revised: This behavior is robust even when the proportion of noise samples to true data samples is changed to 70:30 (though our theory only addresses the 50: case).

Original: Finally, we note that optimizing the empirical NCE loss becomes numerically unstable with increasing d (due to very large ratios in the loss), which is why we used comparatively moderate values of d .
Revised: Finally, we note that optimizing the empirical NCE loss becomes numerically unstable with increasing d (due to very large ratios in the loss), which is why we used comparatively moderate values of d .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Despite significant interest in alternatives to maximum likelihood—for example NCE (considered in this paper), score matching, etc.—there is little understanding of what there is to “sacrifice” with these losses, either algorithmically or statistically.
Revised: Despite significant interest in alternatives to maximum likelihood—for example NCE (considered in this paper), score matching, etc.—there is little understanding of what there is to “sacrifice” with these losses, either algorithmically or statistically.

Original: In this paper, we provided formal lower bounds on the asymptotic sample complexity of NCE, when using a common choice for the noise distribution Q , a Gaussian with matching mean and covariance.
Revised: In this paper, we provided formal lower bounds on the asymptotic sample complexity of NCE, when using a common choice for the noise distribution Q , a Gaussian with matching mean and covariance.

Original: Thus, it is likely that even for moderately complex distributions in practice, more involved techniques like Gao et al.
Revised: Thus, it is likely that even for moderately complex distributions in practice, more involved techniques like Gao et al.

Original: ; Rhodes et al.
Revised: ; Rhodes et al.

Original: (2020) will have to be used, in which one learns a noise distribution Q simultaneously with the NCE minimization or “anneals” the NCE objective.
Revised: (2020) will have to be used, in which one learns a noise distribution Q simultaneously with the NCE minimization or “anneals” the NCE objective.

Original: There is very little theoretical understanding of such techniques, and this seems like a very fruitful direction for future work.
Revised: There is very little theoretical understanding of such techniques, and this seems like a very fruitful direction for future work.

Original: A B OUNDING THE MATRIX INTEGRAL IN E QUATION
Revised: A B OUNDING THE MATRIX INTEGRAL IN E QUATION

Original: We prove a variant of the Cauchy-Schwarz inequality that gives us a handle on norms of matrix integrals.
Revised: We prove a variant of the Cauchy-Schwarz inequality that gives us a handle on norms of matrix integrals.

Original: Lemma
Revised: Lemma

Original: [Equation]
Revised: [Equation]

Original: Let f : R d → R and A : R d → R n be integrable functions, with M = (cid:82) x f ( x ) A ( x ) dx .
Revised: Let f : R d → R and A : R d → R n be integrable functions, with M = (cid:82) x f ( x ) A ( x ) dx .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: if A : R d → R n × n is a matrix valued function then
Revised: if A : R d → R n × n is a matrix valued function then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof follows from the Cauchy-Schwarz inequality.
Revised: The proof follows from the Cauchy-Schwarz inequality.

Original: Since we integrate component-wise, for eq.
Revised: Since we integrate component-wise, for eq.

Original: (21) we have that
Revised: (21) we have that

Original: [Equation]
Revised: [Equation]

Original: Summing over i , we get the result.
Revised: Summing over i , we get the result.

Original: The matrix variant eq.
Revised: The matrix variant eq.

Original: (22) follows by looking at the matrix M as a vector in R n 2 .
Revised: (22) follows by looking at the matrix M as a vector in R n 2 .

Original: B P ROOF OF L EMMA
Revised: B P ROOF OF L EMMA

Original: We restate the lemma for convenience:
Revised: We restate the lemma for convenience:

Original: Lemma 8.
Revised: Lemma 8.

Original: Let d > 0 be sufficiently large.
Revised: Let d > 0 be sufficiently large.

Original: Let p =
Revised: Let p =

Original: ˆ p d and q = ˆ q d be any product distributions, and define R ( x ) =
Revised: ˆ p d and q = ˆ q d be any product distributions, and define R ( x ) =

Original: q ( x ) p ( x ) .
Revised: q ( x ) p ( x ) .

Original: Suppose we have the following third moment bound: E x ∼ ˆ p (cid:20)(cid:16) log ˆ q ˆ p (cid:17) 3 (cid:21) < ∞ .
Revised: Suppose we have the following third moment bound: E x ∼ ˆ p (cid:20)(cid:16) log ˆ q ˆ p (cid:17) 3 (cid:21) < ∞ .

Original: Then, for any ϵ , there exist constants α
Revised: Then, for any ϵ , there exist constants α

Original: = α (ˆ p, ˆ q, ϵ ) , µ = µ (ˆ p, ˆ q, ϵ ) < 0 such that
Revised: = α (ˆ p, ˆ q, ϵ ) , µ = µ (ˆ p, ˆ q, ϵ ) < 0 such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We will analyze the behaviour of R ( x ) using the Berry-Esseen theorem.
Revised: We will analyze the behaviour of R ( x ) using the Berry-Esseen theorem.

Original: Given that p ∗ =
Revised: Given that p ∗ =

Original: ˆ p d and q = ˆ q d are product distributions, let r ( x ) be the random variable defined by r ( x )
Revised: ˆ p d and q = ˆ q d are product distributions, let r ( x ) be the random variable defined by r ( x )

Original: =
Revised: =

Original: ˆ q ( x ) ˆ p ( x ) , x ∼ ˆ p .
Revised: ˆ q ( x ) ˆ p ( x ) , x ∼ ˆ p .

Original: Let y i ( x ) = log r ( x ) for 1 ≤ i ≤ d be d independent copies of the random variable r ( x ) .
Revised: Let y i ( x ) = log r ( x ) for 1 ≤ i ≤ d be d independent copies of the random variable r ( x ) .

Original: Let
Revised: Let

Original: E [ y i ] = µ r , E (cid:104) ∥
Revised: E [ y i ] = µ r , E (cid:104) ∥

Original: y i − µ r ∥ 2 (cid:105) =
Revised: y i − µ r ∥ 2 (cid:105) =

Original: σ 2 r and E (cid:104) ∥
Revised: σ 2 r and E (cid:104) ∥

Original: y i − µ r ∥ 3 (cid:105) =
Revised: y i − µ r ∥ 3 (cid:105) =

Original: γ r , all of which are well defined by the hypothesis of the lemma.
Revised: γ r , all of which are well defined by the hypothesis of the lemma.

Original: Let Y =
Revised: Let Y =

Original: (cid:80) di
Revised: (cid:80) di

Original: =1 y i , and Z be the standard Gaussian in R .
Revised: =1 y i , and Z be the standard Gaussian in R .

Original: Then, by the
Revised: Then, by the

Original: Berry-Esseen Theorem (Durrett, 2019, Theorem 3.4.17),
Revised: Berry-Esseen Theorem (Durrett, 2019, Theorem 3.4.17),

Original: [Equation]
Revised: [Equation]

Original: where C BE < 1 (van Beek, 1972) is an absolute constant.
Revised: where C BE < 1 (van Beek, 1972) is an absolute constant.

Original: We can now choose c
Revised: We can now choose c

Original: = c ( ϵ ) such that
Revised: = c ( ϵ ) such that

Original: P [ Z ≤ c ] ≥ 1 − ϵ 2 .
Revised: P [ Z ≤ c ] ≥ 1 − ϵ 2 .

Original: Further, we can choose d large enough so that C BE · γ σ 3 √ d ≤ ϵ 2 .
Revised: Further, we can choose d large enough so that C BE · γ σ 3 √ d ≤ ϵ 2 .

Original: Then for µ = µ r and α = cσ r , we have
Revised: Then for µ = µ r and α = cσ r , we have

Original: [Equation]
Revised: [Equation]

Original: Since Z is symmetric around 0, Berry-Esseen gives us the other inequality for the same choice of µ and α ,
Revised: Since Z is symmetric around 0, Berry-Esseen gives us the other inequality for the same choice of µ and α ,

Original: [Equation]
Revised: [Equation]

Original: Note that the constants µ and α are independent of d .
Revised: Note that the constants µ and α are independent of d .

Original: Further, note that µ = µ r = − KL (ˆ p || ˆ q ) < 0 .
Revised: Further, note that µ = µ r = − KL (ˆ p || ˆ q ) < 0 .

Original: C I NVERTIBILITY OF THE H ESSIAN
Revised: C I NVERTIBILITY OF THE H ESSIAN

Original: We prove that the Hessian of NCE loss for the exponential family given by T ( x ) =
Revised: We prove that the Hessian of NCE loss for the exponential family given by T ( x ) =

Original: ( x 41 , . .
Revised: ( x 41 , . .

Original: , x 4 d , 1) is invertible.
Revised: , x 4 d , 1) is invertible.

Original: In particular, we have the following lemma:
Revised: In particular, we have the following lemma:

Original: Lemma 10.
Revised: Lemma 10.

Original: Let Q = N (0 , I d ) be the standard Gaussian in R d .
Revised: Let Q = N (0 , I d ) be the standard Gaussian in R d .

Original: Let ˆ P be the log concave distribution defined in definition 6.
Revised: Let ˆ P be the log concave distribution defined in definition 6.

Original: Let P = ˆ P d .
Revised: Let P = ˆ P d .

Original: Let q and p denote the density functions of Q and P respectively.
Revised: Let q and p denote the density functions of Q and P respectively.

Original: Observe that P is in the exponential family given by T ( x ) =
Revised: Observe that P is in the exponential family given by T ( x ) =

Original: ( x 41 , . .
Revised: ( x 41 , . .

Original: , x 4 d , 1) , and equals P θ ∗ for some θ ∗ .
Revised: , x 4 d , 1) , and equals P θ ∗ for some θ ∗ .

Original: Then the hessian of the NCE loss with respect to distribution P and noise Q given by
Revised: Then the hessian of the NCE loss with respect to distribution P and noise Q given by

Original: [Equation]
Revised: [Equation]

Original: is invertible.
Revised: is invertible.

Original: Proof.
Revised: Proof.

Original: For any subset A ⊆ R d , define
Revised: For any subset A ⊆ R d , define

Original: [Equation]
Revised: [Equation]

Original: Observe that the density functions p ∗ and q of P ∗ and Q respectively are strictly positive over all of R d .
Revised: Observe that the density functions p ∗ and q of P ∗ and Q respectively are strictly positive over all of R d .

Original: Therefore, for any subset A ⊆ R d and any v ∈ R d +1 , we have
Revised: Therefore, for any subset A ⊆ R d and any v ∈ R d +1 , we have

Original: [Equation]
Revised: [Equation]

Original: Given a vector v ∈ R d +1 , we will pick A such that (cid:12)(cid:12) T ( x ) ⊤ v (cid:12)(cid:12) > 0 for all x ∈ A .
Revised: Given a vector v ∈ R d +1 , we will pick A such that (cid:12)(cid:12) T ( x ) ⊤ v (cid:12)(cid:12) > 0 for all x ∈ A .

Original: Note that the set B = { e 1 + e d +1 , . .
Revised: Note that the set B = { e 1 + e d +1 , . .

Original: , e d + e d +1 , e d +1 } is a basis.
Revised: , e d + e d +1 , e d +1 } is a basis.

Original: Therefore, if b ⊤ v = 0 for all b ∈ B , then v = 0 .
Revised: Therefore, if b ⊤ v = 0 for all b ∈ B , then v = 0 .

Original: Hence, there exists some x ∈ { e 1 , . .
Revised: Hence, there exists some x ∈ { e 1 , . .

Original: , e d } such that (cid:12)(cid:12) T
Revised: , e d } such that (cid:12)(cid:12) T

Original: [Equation]
Revised: [Equation]

Original: x )
Revised: x )

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Equation]
Revised: [Equation]

Original: > 0 .
Revised: > 0 .

Original: Since x (cid:55)→ T ( x ) ⊤ v is a continuous function, we can find an open set
Revised: Since x (cid:55)→ T ( x ) ⊤ v is a continuous function, we can find an open set

Original: A around x such that
Revised: A around x such that

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Let B = R d \ A .
Revised: Let B = R d \ A .

Original: Since v ⊤ H
Revised: Since v ⊤ H

Original: A v > 0 and v ⊤ H B v ≥ 0 , we have that v ⊤ Hv > 0 .
Revised: A v > 0 and v ⊤ H B v ≥ 0 , we have that v ⊤ Hv > 0 .

Original: Since this holds for any arbitrary non-zero vector v , the matrix H must be full rank.
Revised: Since this holds for any arbitrary non-zero vector v , the matrix H must be full rank.

Original: Since H is an integral of PSD matrices, it is a full rank PSD matrix and hence invertible.
Revised: Since H is an integral of PSD matrices, it is a full rank PSD matrix and hence invertible.

Original: D T AIL BOUNDS FOR E QUATION
Revised: D T AIL BOUNDS FOR E QUATION

Original: We prove that some T up
Revised: We prove that some T up

Original: =
Revised: =

Original: O ( σ 2 √ d ) suffices to obtain the bounds in eq.
Revised: O ( σ 2 √ d ) suffices to obtain the bounds in eq.

Original: Concretely, we prove tail bounds for ∥ T ( x ) ∥ using tail bounds for P ∗ and Q .
Revised: Concretely, we prove tail bounds for ∥ T ( x ) ∥ using tail bounds for P ∗ and Q .

Original: We will use Lemma 1 from Laurent &
Revised: We will use Lemma 1 from Laurent &

Original: Massart (2000) which proves a bound for χ 2 distributions:
Revised: Massart (2000) which proves a bound for χ 2 distributions:

Original: Lemma (Lemma 1, Laurent & Massart (2000)) .
Revised: Lemma (Lemma 1, Laurent & Massart (2000)) .

Original: If X is a χ 2 random variable with d degrees of freedom, then for any positive t ,
Revised: If X is a χ 2 random variable with d degrees of freedom, then for any positive t ,

Original: [Equation]
Revised: [Equation]

Original: Then, for x ∼ Q , ∥ x ∥ 2 is a χ 2 random variable with d degrees of freedom.
Revised: Then, for x ∼ Q , ∥ x ∥ 2 is a χ 2 random variable with d degrees of freedom.

Original: Observe that for t, d ≥ 4 , we have d + 2 t + 2 √ td ≤ 2 td .
Revised: Observe that for t, d ≥ 4 , we have d + 2 t + 2 √ td ≤ 2 td .

Original: In particular, we have the weaker bound
Revised: In particular, we have the weaker bound

Original: [Equation]
Revised: [Equation]

Original: implying that
Revised: implying that

Original: [Equation]
Revised: [Equation]

Original: Further, if ∥ x ∥ ≥ σ 2 √ d , q ( x ) ≥ p ∗ ( x ) , implying that for t ≥ σ 2 √ d
Revised: Further, if ∥ x ∥ ≥ σ 2 √ d , q ( x ) ≥ p ∗ ( x ) , implying that for t ≥ σ 2 √ d

Original: [Equation]
Revised: [Equation]

Original: In particular, for any δ such that log(1 /δ ) ≥ σ 4 , we have
Revised: In particular, for any δ such that log(1 /δ ) ≥ σ 4 , we have

Original: P x ∼ Q (cid:104)
Revised: P x ∼ Q (cid:104)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≤ δ and
Revised: ≤ δ and

Original: P x ∼ P
Revised: P x ∼ P

Original: [Equation]
Revised: [Equation]

Original: ∥ ≥ (cid:112)
Revised: ∥ ≥ (cid:112)

Original: [Equation]
Revised: [Equation]

Original: δ
Revised: δ

Original: [Equation]
Revised: [Equation]

Original: ≤ δ
Revised: ≤ δ

Original: [Equation]
Revised: [Equation]