Likelihood Score under Generalized Self-Concordance
Abstract
We show how, under a generalized self-concordance assumption and possible model misspecification, we can establish non-asymptotic bounds on the normalized likelihood score when using maximum likelihood or score matching.
The tail behavior is governed by an effective dimension corresponding to the trace of the sandwich covariance.
We also show how our non-asymptotic approach allows us to obtain confidence bounds for the estimator and analyze Rao’s score test
Introduction
The problem of statistical inference on learned parameters is regaining the importance it deserves as machine learning and data science are increasingly impacting humanity and society through a large range of successful applications from transportation to healthcare.
The classical asymptotic theory of M-estimation is well established in a general setting under the assumption that the parametric model is well-specified, i.e., the underlying data distribution belongs to the parametric family.
We mention here, among many of them, the monographs [Ibragimov and Has’minskii, 1981, van der Vaart, 2000, van de Geer, 2009].
The main tool is the local asymptotic normality (LAN) condition introduced by Le Cam [1960].
In many real problems, the parametric model is usually an approximation to the data distribution, so it is too restrictive to assume that the model is well-specified.
To relax this restriction, model misspecification has been considered in the asymptotic regime; see, e.g., [Huber, 1967, Wakefield, 2013, Dawid et al., 2016].
Another limitation of classical asymptotic theory is its asymptotic regime where n → ∞ and the parameter dimension d is fixed.
This is inapplicable in the modern context where the data are of high dimension involving a huge number of parameters.
The non-asymptotic viewpoint has been fruitful to address high dimensional problems—the results are developed for all fixed n so that it also captures the asymptotic regime where the parameter dimension can grow with n .
Early works in this line of research focus on specific models such as Gaussian models [Beran, 1996, Beran and Dumbgen, 1998, Laurent and Massart, 2000, Baraud, 2004], ridge regression [Hsu et al., 2012], logistic regression [Bach, 2010], and robust M-estimation [Zhou et al., 2018]; see [Bach, 2021] for a survey.
Spokoiny [2012] addressed the finite-sample regime in full generality in a spirit similar to the classical LAN theory.
His approach relies on heavy empirical process machinery and requires strong global assumptions on the deviation of the empirical risk process.
More recently, Ostrovskii and Bach [2021] focused on risk bounds, specializing their discussion to linear models with (pseudo) self-concordant losses and obtained a more transparent analysis under neater assumptions.
A critical characteristic shared by both works is that the neighborhood of the target parameter is defined by the so-called Dikin ellipsoid , a geometric object identified in the theory of convex optimization [Nesterov and Nemirovskii, 1994, Ben-Tal and Nemirovski, 2001, Boyd et al., 2004, Tunçel and Nemirovski, 2010, Bubeck and Eldan, 2019, Bubeck and Lee, 2016].
The Dikin ellipsoid corresponds to the distance measured by the Euclidean distance weighted by the Hessian matrix at the optimum.
This weighted Euclidean distance is adapted to the geometry near the target parameter and thus leads to sharper bounds which do not depend on the minimum eigenvalue of the Hessian.
This important property has been used fruitfully in various problems of learning theory and mathematical statistics [Zhang and Lin, 2015, Yang and Mohri, 2016, Faury et al., 2020].
Problem formulation
We briefly recall the framework of statistical inference via empirical risk minimization.
Let ( Z , Z ) be a measurable space.
Let Z ∈ Z be a random element following some unknown distribution P .
Consider a parametric family of distributions P Θ :=
{ P θ : θ ∈ Θ ⊂ R d } which may or may not contain P .
We are interested in finding the parameter θ ⋆ so that the model P θ ⋆ best approximates the underlying distribution P .
For this purpose, we choose a loss function ℓ and minimize the population risk L ( θ ) :
=
E Z ∼ P [ ℓ ( θ ; Z )] .
Through out this paper, we assume that
[Equation]
uniquely exists and satisfies θ ⋆ ∈ int (Θ) , ∇ θ L ( θ ⋆ ) = 0 , and ∇ 2 θ L ( θ ⋆ ) ≻ 0 .
Consistent loss function.
We focus on loss functions that are consistent in the following sense:
When the model is well-specified , i.e., there exists θ 0 ∈ Θ such that P
= P θ 0 , it holds that θ ⋆ = θ 0 .
We give below two popular choices of consistent loss functions.
Example 1 (Maximum likelihood estimation) .
A widely used loss function in statistical machine learning is the negative log-likelihood ℓ ( θ ; z ) :
= − log p θ ( z ) where p θ is the probability mass/density function for the discrete/continuous case.
When P = P θ 0 for some θ 0 ∈ Θ , we have L ( θ ) =
E [ − log p θ ( Z )]
= KL( p θ
∥
p θ ) − E [log p θ
( Z )] where KL is the Kullback-Leibler divergence.
As a result, θ 0 ∈ arg min θ ∈ Θ KL( p θ
∥ p θ ) =
arg min θ
∈ Θ L ( θ ) .
Moreover, if there is no θ such that p θ a.s. =
p θ 0 , then θ 0 is the unique minimizer of L .
Example 2 (Score matching estimation) .
Another important example appears in score matching [Hyvärinen and Dayan, 2005].
Assume that Z = R p and P and P θ have densities p and p θ .
Let p θ ( z ) =
q θ ( z ) / Λ( θ ) where Λ( θ ) is an unknown normalizing constant.
We can choose the loss
[Equation]
Here ∆ := (cid:80) pk =1 ∂ 2 /∂z 2 k is the Laplace operator.
Since [Hyvärinen and Dayan, 2005, Thm. 1]
[Equation]
it follows that when p = p θ 0 we have θ 0 ∈ arg min θ ∈ Θ L ( θ ) .
In fact, when q θ > 0 and there is no θ such that p θ a.s. = p θ 0 , θ 0 is the unique minimizer of L [Hyvärinen and Dayan, 2005, Thm. 2].
Empirical risk minimizer and likelihood score.
Assume that we have an i.i.d.
sample { Z i } ni
=1 from P .
To learn the parameter θ ⋆ , we minimize the empirical risk to obtain the empirical risk minimizer
[Equation]
When ℓ is chosen as the negative log-likelihood, the negative gradient −∇ ℓ n ( θ ) is known as the likelihood score .
In Sec. 3, we will establish a non-asymptotic bound for the norm of the likelihood score at θ ⋆ normalized by the Hessian ∇ 2 ℓ n ( θ ⋆ ) , i.e., (cid:13)(cid:13) ∇ 2 ℓ n ( θ ⋆ ) − 1 / 2 ∇ ℓ n ( θ ⋆ ) (cid:13)(cid:13) .
We show how this bound can be used to obtain a finite-sample confidence bound for θ ⋆ constructed from θ n .
That is, for any δ ∈ (0 , 1) , we will construct a confidence set C n ( δ ) ⊂
Θ based on θ n such that
[Equation]
In Fig.
1, we illustrate this result for a logistic regression model.
Finally, we apply our approach to analyze Rao’s score test for goodness-of-fit testing.
Main results 3.1 Preliminaries
Notation.
We denote by S ( θ ; z ) :=
∇ θ ℓ ( θ ; z ) the gradient of the loss at z and H ( θ ; z ) := ∇ 2 θ ℓ ( θ ; z ) the Hessian at z .
Their population versions are S ( θ ) :
=
E [ S ( θ ; Z )] and H ( θ ) :=
E [ H ( θ ; Z )] , θθ n δ =0 . respectively.
We assume standard regularity assumptions
so that S ( θ ) = ∇ θ L ( θ ) and H ( θ )
= ∇ 2 θ L ( θ ) .
The two optimality conditions then read S ( θ ⋆ ) = 0 and H ( θ ⋆ ) ≻ 0 .
It follows that λ
⋆ := λ min ( H ( θ ⋆ )) > 0 and λ
⋆ := λ max ( H ( θ ⋆ )) > 0 .
Furthermore, we let G ( θ ) :
= C ov( S ( θ ; Z )) be the covariance matrix of the gradient.
We define their empirical quantities as ℓ n ( θ ) :
= 1 n (cid:80) ni =1 ℓ ( θ ; Z i ) ,
S n ( θ ) :
= 1 n (cid:80) ni =1 S ( θ ; Z i ) , H n ( θ ) : = 1 n (cid:80) ni =1 H ( θ ; Z i ) , and
G n ( θ ) :
= 1 n n (cid:88) i =
[ S ( θ ; Z i ) − S ( θ )] [ S ( θ ; Z i ) − S ( θ )] ⊤ .
Recall that − S n ( θ ) is the likelihood score for maximum likelihood estimation.
Our analysis of the estimator θ n is local to a Dikin ellipsoid at θ ⋆ of radius r , i.e.,
[Equation]
where, given a positive semi-definite matrix J , we let ∥ x
∥ J
[Equation]
Effective dimension.
A quantity that plays a central role in our analysis is the effective dimension :
[Equation]
The effective dimension appears recently in non-asymptotic analyses of (penalized) M-estimation [Spokoiny, 2017, Ostrovskii and Bach, 2021].
When the model is well-specified, it can be shown that H ( θ ⋆ ) =
G ( θ ⋆ ) and thus d ⋆ = d .
When the model is misspecified, it can be much smaller than d depending on the spectrum of H ( θ ⋆ ) and G ( θ ⋆ ) ; see Appx.
B for a through discussion.
Moreover, it is closely connected to classical asymptotic theory of M-estimation under model misspecification.
Generalized self-concordance.
For a function f : R d → R , we define D x f ( x )[ u ] := dd t f ( x + tu ) | t =0 , D 2 x f ( x )[ u, v ] :
=
D x
( D x f ( x )[ u ])[ v ] for x, u, v ∈ R d , and D 3 x f ( x )[ u, v, w ] similarly.
Definition 1 (Generalized self-concordance [Sun and Tran-Dinh, 2019]) .
Let X ⊂ R d be open and f : X → R be a closed convex function.
For a constant R > 0 and an integer ν > 0 , we say f is ( R, ν ) -generalized self-concordant on X if
[Equation]
with the convention 0 / 0 = 0 for the case ν < 2 and ν > 3 .
Recall that ∥ u ∥ 2 ∇ 2 f ( x ) :=
u ⊤ ∇ 2 f ( x ) u .
Remark 2.
When ν = 3 , this definition reduces to the standard self-concordance [Nesterov and
Nemirovskii, 1994].
When ν = 2 , it recovers the pseudo self-concordance [Bach, 2010].
We give several examples in Appx.
D.
3.2 Normalized score
Our first result is a non-asymptotic bound for the normalized score at θ ⋆ , i.e., ∥ S n ( θ ⋆ )
∥ H − 1 n ( θ ⋆ ) .
Due to the space constraint, we state the precise assumptions in Appx.
A.
The problem-specific constants
K 1 , K 2 , and σ H are also defined in Appx.
A.
The proof is deferred to Appx.
C.
Proposition 3.
Under Asmps. 2 and 3 with r = 0 , it holds that, with probability at least 1 − δ , ∥ [Equation] n [Equation]
θ ⋆
[Equation]
2 H − 1 n ( θ
[Equation]
whenever n ≥ 4( K 22 + 2 σ 2 H ) log (4 d/δ ) .
3.3 Confidence set
We then construct a finite-sample confidence set for θ ⋆ using the bound for the normalized score.
We also provide a thorough discussion on this result in Appx.
B and its proof in Appx.
C.
The quantities
K ν , R ⋆ν , ω ν are defined in Cor.
20, Eq.
(6), Eq.
(12) in the appendix, respectively.
Theorem 4.
Let ν ∈
Suppose that Asmps 1,2, and 3 with r =
K ν /R
⋆ν hold true.
Let
[Equation]
where C is an absolute constant.
Then we have P ( θ ⋆ ∈ C n ( δ )) ≥ 1 − δ whenever n satisfies
[Equation]
Remark 5.
According to Huber [1967], under suitable regularity assumptions, it holds that √
[Equation]
L ∼ N (0 , H ( θ ⋆ ) − 1 / 2 G ( θ ⋆ ) H ( θ ⋆ )
[Equation]
1 / 2 ) which implies that
[Equation]
This induces an asymptotic confidence set with a similar form of (2) and
radius O
( E [ L ⊤ L ] /n ) =
O ( d ⋆ /n ) .
Our result characterizes the critical sample size enough to enter the asymptotic regime.
3.4 Rao’s score test
We illustrate how our results can be used to analyze Rao’s score test for goodness-of-fit testing.
In this subsection, we will assume that the model is well-specified.
We use θ ⋆ to denote the true parameter of P and reserve θ 0 for the parameter under the null hypothesis.
The proof is deferred to Appx.
D.
Given θ 0 ∈ Θ , a goodness-of-fit testing problem is to test the hypotheses
[Equation]
A statistical test consists of a test statistic T :=
T ( Z 1 , . . . , Z n ) and a prescribed critical value t , and we reject the null hypothesis if T > t .
Its performance is quantified by the type I error rate P ( T > t | H 0 ) and statistical power P ( T > t | H 1 ) .
Rao’s score test statistic is T Rao :=
∥ S n ( θ 0 ) ∥ 2 H − 1 n ( θ 0 ) .
Theorem 6 (Rao’s score statistic) .
Suppose that Asmps.
and 3 with r = 0 hold true.
For an arbitrary t > 0 , let Ω( θ ) :
= G ( θ ) 12 H ( θ ) − 1 G ( θ ) 12 and
[Equation]
(a) Under H 0 , we have, with probability at least 1 − δ , T Rao ≲ K 21 log ( e/δ )( d/n ) whenever n ≥ 4( K 22 + 2 σ 2 H ) log (4 d/δ ) .
(b) Under H 1 , whenever τ n ≥ 0 , we have, with c being an absolute constant,
[Equation]
Remark 7.
The bound under H 0 suggests that, for a fixed significance level α ∈ (0 , 1) , we can choose the critical value t = t n ( α )
=
O ( d ⋆ /n )
so that their type I error rates are below α .
With this choice of t n ( α ) , we can then characterize the statistical power of Rao’s score test under a fixed alternative hypothesis θ ⋆ ̸ = θ 0 —they decay to zero exponentially fast as n → ∞ .
Acknowledgments and Disclosure of Funding
L. Liu is supported by NSF CCF-2019844 and NSF DMS-2023166 and NSF DMS-2133244. Z. Harchaoui is supported by NSF CCF-2019844, NSF DMS-2134012, NSF DMS-2023166, CIFARLMB, and faculty research awards. Part of this work was done while Z. Harchaoui was visiting the
Simons Institute for the Theory of Computing.
D Examples and applications
D.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.2 Goodness-of-fit testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E Technical tools E.1 Properties of generalized self-concordant functions . . . . . . . . . . . . . . . .
E. Concentration of random vectors and matrices . . . . . . . . . . . . . . . . . .
A Assumptions
Our key assumption is the generalized self-concordance of the loss function.
Assumption 1 (Generalized self-concordance) .
For any z ∈ Z , the scoring rule ℓ ( · ; z ) is ( R, ν ) - generalized self-concordant for some R > 0 and ν ≥ 2 .
Moreover, L ( · ) is also ( R, ν ) -generalized self-concordant.
Remark 8.
When ν = 2 , it is straightforward to check that the generalized self-concordance of ℓ ( · ; z ) implies the one of L ( · ) .
In order to control the empirical gradient S n ( θ ) , we assume that the norm of the normalized gradient at θ ⋆ has a light tail.
Assumption 2 (Sub-Gaussian gradient) .
There exists a constant K 1 > 0 such that the normalized gradient at θ ∗ is sub-Gaussian with parameter K 1 , i.e., (cid:13)(cid:13) G ( θ ⋆ ) − 1 / 2 S
( θ ⋆ ; Z ) (cid:13)(cid:13)
ψ 2 ≤ K 1 .
Here ∥·∥ ψis the sub-Gaussian norm whose definition is recalled in Appx.
E.
Remark 9.
When the loss function is of the form ℓ ( θ ; z ) =
ℓ ( y, θ ⊤ x ) , we have S ( θ ; Z ) =
ℓ ′ ( Y, θ ⊤ X ) X where ℓ ′ ( y, ¯ y ) =
d ℓ ( y, ¯ y ) / d¯ y .
As a result, Asmp. 2 holds true if (i) ℓ ′ ( Y, θ ⊤ ⋆ X ) is sub-Gaussian and X is bounded or (ii) ℓ ′ ( Y, θ ⊤ ⋆ X ) is bounded and X is sub-Gaussian.
For least squares with ℓ ( y, θ ⊤ x ) = 12 ( y − θ ⊤ x ) 2 , the derivative ℓ ′ ( Y, θ ⊤ ⋆ X )
=
θ ⊤ ⋆ X − Y is the negative residual.
Asmp. 2 is guaranteed if the residual is sub-Gaussian and X is bounded.
For logistic regression with ℓ ( y, θ ⊤ x ) = − log σ ( y · θ ⊤ x ) where σ ( u ) =
(1 + e − u ) − 1 , the derivative ℓ ′ ( Y, θ ⊤ ⋆ X ) =
[ σ ( Y · θ ⊤ ⋆ X ) − 1] Y ∈
[ − 1 , 1] is bounded.
Thus, Asmp. 2 is guaranteed if X is sub-Gaussian.
In order to control the empirical Hessian, we assume that the Hessian of the loss function at a neighborhood of θ ⋆ satisfies the matrix Bernstein condition.
Assumption 3 (Matrix Bernstein of Hessian) .
There exist constants K 2 , r > 0 such that, for any θ ∈ Θ r ( θ ⋆ ) , the standardized Hessian
[Equation]
satisfies a Bernstein condition (defined in Appx.
E) with parameter K 2 .
Moreover,
[Equation]
where, for a matrix J ∈ R d × d , we define | J | 2 := max { λ max ( J ) , | λ min ( J ) |} and V ar( J ) :
=
E [ JJ ⊤ ] − E
[ J ] E
[ J ] ⊤ .
By convention, we let Θ
( θ ⋆ ) =
{ θ ⋆ } .
B Discussion
Fisher information and model misspecification.
When the model is well-specified, the covariance matrix G ( θ ) coincides with the well-known Fisher information I ( θ ) :
=
E Z ∼ P θ [ S ( θ ; Z )] at θ ⋆ .
The
Fisher information plays a central role in mathematical statistics and, in particular, M-estimation; see Pennington and Worah [2018], Kunstner et al. [2019], Ash et al. [2021], Soen and Sun [2021] for recent developments in this line of research.
It quantifies the amount of information a random variable carries about the model parameter.
Under a well-specified model, it also coincides with the Hessian matrix H ( θ ) at the optimum which captures the curvature of the population risk.
When the model is misspecified, the Fisher information deviates from the Hessian matrix.
In the asymptotic regime, this discrepancy is reflected in the limiting covariance of the weighted M-estimator which admits a sandwich form H ( θ ⋆ ) − 1 G ( θ ⋆ ) H ( θ ⋆ ) − 1 ; see, e.g., [Huber, 1967, Sec. 4].
Effective dimension.
The counterpart of the sandwitch covariance in the non-asymptotic regime is the effective dimension d ⋆ ; see, e.g., Spokoiny [2017], Ostrovskii and Bach [2021].
Our bounds also enjoy the same merit—its dimension dependency is via the effective dimension.
When the model is well-specified, the effective dimension reduces to d , recovering the same rate of convergence O ( d/n ) as in classical linear regression; see, e.g., [Bach, 2021, Prop. 3.5].
When the model is misspecified, the effective dimension provides a characterization of the problem complexity which is adapted to both the data distribution and the loss function via the matrix H ( θ ⋆ ) − 1 / 2 G ( θ ⋆ ) H ( θ ⋆ ) − 1 / 2 .
To gain a
[Table]
better understanding on the effective dimension d ⋆ , we summarize it in Tab. 1 under different regimes of eigendecay, assuming that G ( θ ⋆ ) and H ( θ ⋆ ) share the same eigenvectors.
It is clear that, when the spectrum of G ( θ ⋆ ) decays faster than the one of H ( θ ⋆ ) , the dimension dependency can be better than O ( d ) .
In fact, it can be as good as O (1) when the spectrum of G ( θ ⋆ ) and H ( θ ⋆ ) decay as e µi and i − β , respectively.
Comparison to classical asymptotic theory.
Classical asymptotic theory of M-estimation is usually based on two assumptions: (a) the model is well-specified and (b) the sample size n is much larger than the parameter dimension d .
These assumptions prevent it from being applicable to many real applications where the parametric family is only an approximation to the unknown data distribution and the data is of high dimension involving a large amount of parameters.
On the contrary, our results do not require a well-specified model and the dimension dependency is replaced by the effective dimension d ⋆ which captures the complexity of the parameter space.
Moreover, they are of non-asymptotic nature—they hold true for any n as long as it exceeds some constant factor of d ⋆ .
This allows the number of parameters to potentially grow with the same size.
Comparison to recent non-asymptotic theory.
Recently, Spokoiny [2012] achieved a breakthrough on finite-sample analysis of parametric M-estimation.
Although being fully general, their results require strong global assumptions on the deviation of empirical risk process and are built upon advanced tools from empirical process theory.
Restricting ourselves to generalized self-concordant losses, we are able to provide a more transparent analysis with neat assumptions only at the optimum parameter θ ⋆ .
Moreover, our results maintain some generality, covering several interesting examples in statistical machine learning as provided in Appx.
D.1.
Ostrovskii and Bach [2021] also considered self-concordant losses for M-estimation.
However, their results are limited to generalized linear models whose loss is (pseudo) self-concordant and admits the form ℓ ( θ ; Z ) :=
ℓ ( Y, θ ⊤ X ) .
While sharing the same rate O ( d ⋆ /n ) , our results are more general than theirs in two aspects.
First, the loss need not be of the form ℓ ( Y, θ ⊤ X ) , encompassing the score matching loss in Ex. 5 below.
Second, the notion of generalized self-concordance encompasses both the standard and pseudo self-concordance, allowing us to obtain a unified analysis rather than separate ones as in Ostrovskii and Bach [2021].
Pseudo self-concordant losses have also been considered for semi-parametric models Anonymous Author(s) [2022].
They focus on bounding the excess risk rather than providing confidence sets.
Moreover, their results require a localization assumption on θ n while this is proved in Prop. 11 in this paper.
Regularization.
Our results can also be applied to regularized empirical risk minimization by including the regularization term in the loss function.
Let θ λn and θ λ⋆ be the minimizer of the regularized empirical and population risk, respectively.
Let d λ ⋆ :=
[Equation] − [Equation] where H λ ⋆ and G λ ⋆ are the regularized Hessian and the covariance of the regularized gradient at θ λ ⋆ , respectively.
Then our results characterize the concentration of θ λn around θ λ⋆ :
[Equation]
This result coincides with Spokoiny [2017, Thm. 2.1].
If the goal is to estimate the unregularized population risk minimizer θ ⋆ , then we need to pay an additional error (cid:13)(cid:13) θ λ⋆ − θ ⋆ (cid:13)(cid:13) 2 H λ⋆ which is referred to as the modeling bias [Spokoiny, 2017, Sec. 2.5].
For instance, let Z :=
( X, Y ) where X ∈ R d with E [ XX ⊤ ] = I d and Y ∈ R .
Consider the regularized squared loss
[Equation]
where Λ = diag { λ 1 , . .
λ d } .
The regularized effective dimension is of order [Spokoiny, 2017,
Sec. 2.1] [Equation] which can be much smaller than d if { λ k } is increasing.
C Proof of main results
Our proof techniques rely on a self-concordance property to localize the estimator and control the Hessian and related quantities.
This property was, up to our knowledge, first put to use in machine learning by Abernethy et al.
[2008] in the context of sequential allocation of experiments and multi-armed bandits.
The key observation is that, within the Dikin ellipsoid, the variation of the Hessian can be easily controlled.
More recently, Ostrovskii and Bach [2021] obtained risk bounds for generalized linear models based on this observation.
Our results and proof techniques also rely on this observation.
We show how to leverage this observation to obtain confidence sets for a broad class of statistical models under a generalized self-concordance assumption owing to the use of the matrix Bernstein inequality.
For instance, we obtain confidence bounds for parameter estimation using score matching and generalized linear statistical models under possible model misspecification as provided in Appx.
D.1.
Our proofs are inspired by Ostrovskii and Bach [2021].
However, there are two key differences.
First, since they focus on loss functions of the form ℓ ( Y, θ ⊤ X ) , the Hessian is ℓ ′′ ( Y, θ ⊤ X ) XX ⊤ where ℓ ′′ ( y, ¯ y ) :
=
d 2 ℓ ( y, ¯ y ) / d¯ y 2 .
As a result, they can control the deviation of the empirical Hessian using inequalities for sample second-moment matrices of sub-Gaussian random vectors [Ostrovskii and Bach, 2021, Thm. A.2].
In contrast, we use matrix Bernstein inequality which allows us to work with a larger class of loss functions.
Second, we extend their localization result from pseudo self-concordant losses to generalized self-concordant losses (Prop. 11).
This is enabled by a new property on the existence of a unique minimizer for generalized self-concordant functions (Prop. 21).
In the remaining of this section, we first prove a localization result Prop. 11 and the score bound
Prop. 3 in Appx.
C.1.
It not only guarantees the existence and uniqueness of θ n but also localizes it.
We then, in Appx.
C.2, control the empirical Hessian at θ n using a covering number argument.
Finally, we prove Thm. 4.
C.1 Localization
We start by showing that the empirical risk ℓ n is generalized self-concordant.
Lemma 10.
Under Asmp. 1, the empirical risk ℓ n is ( n ν/ 2 − 1 R, ν ) -generalized self-concordant.
Proof.
By Asmp. 1, the loss ℓ ( · ; Z i ) is ( R, ν ) -generalized self-concordant for every i ∈ [ n ] := { 1 , . .
. , n } .
Note that ℓ n is the empirical average of { ℓ ( · ; Z i ) } ni =1 .
Hence, it follows from [Sun and
Tran-Dinh, 2019, Prop. 1] that ℓ n is ( n ν/ 2 − 1 R, ν ) -generalized self-concordant
Applying Prop. 21 to ℓ n leads to the localization result.
Let λ
n,⋆
=
λ min ( H n ( θ ⋆ )) and λ ⋆ n := λ min ( H n ( θ ⋆ )) .
Recall K ν from Cor.
Define
[Equation]
Proposition 11.
Under Asmp. 1, whenever R ⋆n,ν ∥ S n ( θ ⋆ ) ∥
H − 1 n ( θ ⋆ ) ≤ K ν , the estimator θ n uniquely exists and satisfies
[Equation]
Proof.
The claim follows directly from Lem.
10 and Prop. 21.
Prop. 11 implies that the empirical risk minimizer uniquely exists if ∥ S n ( θ ⋆ )
∥ H − 1 n ( θ ⋆ ) is small.
Hence, it remains to bound ∥
S n ( θ ⋆ ) ∥
H − 1 n ( θ ⋆ ) , which can be achieved by controlling ∥ S n ( θ ⋆ )
∥
H − 1 ( θ ⋆ ) and
H n ( θ ⋆ ) .
Recall d ⋆ from (1).
Lemma 12.
Under Asmp. 2, it holds that, with probability at least 1 − δ ,
[Equation]
Proof.
By the first order optimality condition, we have S ( θ ⋆ ) = 0 .
As a result,
[Equation]
is an isotropic random vector.
Moreover, it follows from Lem.
26 that ∥ X ∥ ψ 2 ≲ K 1 .
Define
J :=
G 1 / 2 ( θ ⋆ ) H − 1 ( θ ⋆ ) G 1 / 2 ( θ ⋆ ) /n .
Then we have
[Equation]
Invoking Thm.
27 yields the claim.
Lemma 13.
Under Asmp. 3 with r = 0 , it holds that, with probability at least 1 − δ ,
[Equation]
whenever n ≥ 4( K 22 + 2 σ 2 H ) log (2 d/δ ) .
Proof.
Due to Asmp. 3, the standardized Hessian at θ ⋆
[Equation]
satisfies a Bernstein condition with parameter K 2 .
It then follows from Thm.
that
[Equation]
As a result, whenever n ≥ 4( K 22 + 2 σ 2 H ) log (2 d/δ ) , we have
[Equation]
or equivalently
[Equation]
Proof of Prop. 3.
Define two events
[Equation]
According to Lems.
12 and 13, whenever n ≥ 4( K 22 + 2 σ 2 H ) log (4 d/δ ) , we have P ( A ) ≥ 1 − δ/and P ( B ) ≥ − δ/ 2 .
On the event AB , we have
[Equation]
log [Equation] d ⋆ . Since P ( AB ) ≥ 1 − P ( A c ) − P ( B c ) ≥ 1 − δ , we have, with probability at least 1 − δ ,
[Equation]
whenever n ≥ 4( K 22 + 2 σ 2 H ) log (4 d/δ ) .
C.2 Proof of the main theorems
Before we prove the main theorem, we control the empirical Hessian.
Recall that λ ⋆ :=
λ min ( H ( θ ⋆ )) and λ ⋆ := λ max ( H ( θ ⋆ )) .
Let
[Equation]
Proposition 14.
Fix ε ∈ (0 , K ν ] .
Under Asmps.
1 and 3 with r =
K ν /R
⋆ν , it holds that, with probability at least 1 − δ ,
[Equation]
for all θ ∈ Θ ε/R ⋆ν ( θ ⋆ ) , whenever whenever n ≥ 4( K 22 + 2 σ 2 H ) (cid:8) log (2 d/δ ) + d log (cid:2) 3(1 .
5 ω ν ( ε ) n ) ( ν/ 2 − 1) (cid:3)(cid:9) .
Proof.
We prove the result in the following steps.
Step 1.
Take a τ -covering and relate H n ( θ ) to H n ( ¯ θ ) for some ¯ θ in the covering.
Let τ :=
ε/R ⋆ν [1 . 5 ω ν ( ε ) n ] ν/ 2 − 1 .
Take an τ -covering N τ of Θ ε/R
⋆ν ( θ ⋆ ) w.r.t.
∥·∥ H ( θ ⋆ ) , and let π ( θ ) be the projection of θ onto N τ .
Let d n,
[Equation]
[Equation]
if
[Equation]
otherwise .
By Lem.
10 and Prop. 16, we have, for all θ ∈ Θ ε/R
⋆ν
( θ ⋆ ) ,
[Equation]
where it holds if d n,ν ( π ( θ ) , θ ) < 1 for the case ν > 2 .
Step 2.
Related H n ( θ ) to H ( θ ⋆ ) for all θ in the covering.
Fix an arbitrary θ ∈ N τ .
Following the same argument as Lem.
13, we have, with probability at least 1 − δ ,
[Equation]
whenever n ≥ 4( K 22 + 2 σ 2 H ) log (2 d/δ ) .
It then follows from Asmp. 1 and Lem.
18 that
[Equation]
since R ⋆ν ∥ θ − θ ⋆ ∥ H ( θ ⋆ ) ≤ ε < 1 .
By the monotonicity of ω ν , we get
[Equation]
and thus, with probability at least 1 − δ ,
[Equation]
whenever n ≥ 4( K 22 + 2 σ 2 H ) log (2 d/δ ) .
Define the event
[Equation]
for
[Equation]
Since |N τ | ≤ (3 ε/τR ⋆ν ) d Ostrovskii and Bach [2021], by a union bound, we have P ( A ) ≥ 1 − δ whenever n [Equation]
Step 3.
Combine the previous two steps.
On the event A , we have H n ( π ( θ ))
⪯ 32 ω ν ( ε ) H ( θ ⋆ ) for all θ ∈ Θ ε/R
⋆ν ( θ ⋆ ) .
A similar argument as Lem.
18 shows that
[Equation]
otherwise .
Substituting τ gives d n,ν ( π ( θ ) , θ ) ≤ R ⋆ν εR ⋆ν = ε .
Hence, by (7), we obtain
[Equation]
for all θ ∈ Θ ε/R ⋆ν ( θ ⋆ ) .
on the event A .
Theorem 15.
Let ν ∈
[2 , 3) and ε ∈ (0 , K ν ] .
Under Asmps.
1 to 3 with r = 0 , whenever
[Equation]
the empirical risk minimizer θ n uniquely exists and satisfies, with probability at least 1 − δ ,
[Equation]
[Equation]
Here C is an absolute constant.
Proof.
Similar to the proof of Prop. 3, we define two events
A :
[Equation]
e/
[Equation]
) d ⋆
[Equation]
and B :=
[Equation]
In the following, we let
[Equation]
Following the same argument as Prop. 3, we have P ( AB ) ≥ 1 − δ
and
[Equation]
Now, it suffices to prove, on the event AB ,
[Equation]
Recall R ⋆n,ν and R ⋆ν from (5) and (6).
It is straightforward to check that R ⋆n,ν ≤ √ 2 n ν/ 2 − 1 R ⋆ν for all ν ∈ [2 , 3] .
Consequently, it holds that
[Equation]
since n 3 − ν ≳
( R ⋆ν ) 2 K 21 log ( e/δ )
d ⋆ /ε 2 .
As a result, by Prop. 11, we have that θ n uniquely exists and satisfies
[Equation]
and thus, using the event B ,
[Equation]
absolute
Proof of Thm. 4.
We start by defining some events.
Let
[Equation]
In the following, we let
[Equation]
According to Lem.
12, Lem. 13, and Prop. 14, it holds that P ( A ) ≥ 1 − δ/ 3 , P ( B ) ≥ 1 − δ/ 3 , and P ( C ) ≥ 1 − δ/ 3 .
This implies that P ( ABC ) ≥ 1 − δ .
Now, it suffices to prove, on the event ABC ,
[Equation]
Following the same argument as Thm. 15, we obtain
[Equation]
Therefore, using the event C , we have
[Equation]
As a result, P ( θ ⋆ ∈ C n ( δ )) ≥ 1 − δ whenever n satisfies (3).
D Examples and applications
We give several examples from statistical machine learning and prove the results for goodness-of-fit testing in Sec. 3.4.
D.1 Examples
We begin with several standard examples in the literature of M-estimation.
Example 3 (Linear regression) .
Let Z :=
( X, Y ) be a pair of input and output, where X ∈ R d with E [ XX ⊤ ] ≻ 0 and Y ∈ R .
Consider the statistical model
[Equation]
Its log-likelihood is given by − ( Y − θ ⊤ X ) 2 / [Equation] [Equation]
+ Cσ leading to the squared loss
[Equation]
It is clear that ℓ ( · ; z ) is convex.
Moreover, since D 3 θ ℓ ( θ ; z )[ u, u, v ] = 0 for all u, v ∈ R d , the loss ℓ is generalized self-concordant for all ν ≥ 2 and R ≥ 0 .
Example 4 (Logistic regression) .
Let Z :=
( X, Y ) be a pair of input and label, where X ∈ R d with ∥ X ∥ ≤ M and Y ∈ {− 1 , 1 } .
Consider the statistical model
[Equation]
where σ ( u ) :
= (1 + e − u ) − 1 .
Its log-likelihood is given by 1 { Y = 1 } log σ ( θ ⊤ X ) + 1 { Y = − 1 } log σ ( − θ ⊤ X ) leading the loss ℓ ( θ ; z ) :
= − log (cid:0) σ
[Equation]
Y
[Equation]
⊤ X ) (cid:1) .
Take any u, v ∈ R d .
It can be shown that
[Equation]
[Table]
[Table]
Note that | 1 − 2 σ ( u ) | ≤ 1 for all u ∈ R .
It then follows from the Cauchy-Schwarz that ℓ is ( M, 2) -generalized self-concordant.
Derivation of Ex. 4.
Recall that the loss is given by
[Equation]
Take any u, v ∈ R d .
It can be computed that
[Equation]
Note that (cid:12)(cid:12) [1 − 2 σ ( y · θ ⊤ x )] y (cid:12)(cid:12) ≤ 1 .
It then follows that
[Equation]
and thus ℓ is ( M, 2) -generalized self-concordant.
We then give a popular loss function from score matching whose statistical behavior is less wellknown.
Example 5 (Score matching with exponential families) .
Assume that Z = R p .
Consider an exponential family on R d with densities
[Equation]
The non-normalized density q θ then reads log q θ ( z )
=
θ ⊤ t ( z )+ h ( z ) .
As a result, the score matching loss becomes
[Equation]
where A ( z
[Equation]
= [Equation] ⊤ is p.s.d, b ( z ) :
= (cid:80) pk =1 (cid:104) ∂
2 t ( z ) ∂z 2 k
+ ∂h ( z ) ∂z k ∂t ( z ) ∂z k (cid:105) , and c ( z ) : = (cid:80) pk =1 (cid:104) ∂
2 h ( z ) ∂z 2 k + (cid:0) ∂h
( z ) ∂z k (cid:1) 2 (cid:105) .
Therefore, the score matching loss ℓ ( θ ; z ) is convex.
Moreover, since the third derivatives of ℓ ( · ; z ) is zero, the score matching loss is generalized self-concordant for all ν ≥ 2 and R ≥ 0 .
D.2 Goodness-of-fit testing
Proof of Thm. 6.
(a) Under H 0 , we have θ ⋆ = θ 0 .
It then follows from Prop. {+
+} that, with probability at least 1 − δ ,
[Equation]
whenever n ≥ 4( K 22 + 2 σ 2 H ) log (4 d/δ ) .
(b) Define three events
[Equation]
On the event B , it holds that
[Equation]
This implies CB ⊂ AB , and thus
[Equation]
Note that C c = {∥ S n ( θ 0 ) − S ( θ 0 )
∥ 2 H − 1 ( θ 0 ) − Tr (Ω( θ 0 ))
/n ≥ τ n } where
[Equation]
It follows from Thm.
27 that, whenever τ n ≥ 0 ,
[Equation]
Moreover, due to Thm. 28, we have
[Equation]
and this proves the claim.
E Technical tools
In this section, we first recall and prove some key properties of generalized self-concordance.
We then review some key results regarding the concentration of random vectors and matrices.
E.1 Properties of generalized self-concordant functions
Throughout this section, we let f : R d → R be ( R, ν ) -generalized self-concordant as in Definition 1, where R > 0 and ν ≥ 2 .
For simplicity of the notation, we denote ∥·∥ x :=
∥·∥ ∇ 2 f ( x ) .
Let
[Equation]
and
[Equation]
with dom( ω ν ) =
R if ν = 2 and dom( ω ν ) =
( −∞ , 1) if ν > 2 .
The next proposition gives bounds for the Hessian of f .
Proposition 16 (Sun and Tran-Dinh [2019], Prop. 8) .
For any x, y ∈ dom( f ) , we have
[Equation]
where it holds if d ν ( x, y ) < 1 for the case ν > 2 .
We then give the bounds for function values.
Define two functions
[Equation]
otherwise
[Equation] 13) and [Equation] otherwise
[Equation]
Proposition 17 (Sun and Tran-Dinh [2019], Prop. 10) .
For any x, y ∈ dom( f ) , we have
[Equation]
where it holds if d ν ( x, y ) < 1 for the case ν > 2 .
In the following, we fix x ∈ dom( f ) and assume ∇ 2 f ( x ) ≻ 0 .
We denote λ min :=
λ min ( ∇ 2 f ( x )) and λ max := λ max ( ∇ 2 f ( x )) .
The next lemma bounds d ν ( x, y ) with the local norm ∥
y − x ∥ x . Let
[Equation]
Lemma 18.
For any ν ≥ 2 and y ∈ dom( f ) , we have
[Equation]
Moreover, it holds that
[Equation]
where it holds if R ν ∥
y − x ∥ x < 1 for the case ν > 2 .
Proof.
Recall the definition of d ν in (11).
If ν = 2 , then, by the Cauchy-Schwarz inequality,
[Equation]
The case ν > 2 can be proved similarly.
We then prove some useful properties for the function ¯¯
ω .
Lemma 19.
For any ν ≥ 2 , the following statements hold true:
(a) The function φ ( τ ) :
= ¯¯ ω
ν ( − τ ) is strictly decreasing on [0 , ∞ ) with φ (0) = 1 / 2 and φ ( τ ) ≥ 0 for all τ ≥ 0 .
(b) The function ψ ( τ ) :
= ¯¯ ω
ν ( − τ ) τ is strictly increasing on [0 , ∞ ) with ψ (0) = 0 .
Proof.
(a) .
By definition, ω ν is strictly increasing on ( −∞ , 1) .
As a result, for any τ ∈ ( −∞ , 1) ,
[Equation]
It then follows that, for any τ ≥ 0 ,
[Equation]
and thus φ is strictly decreasing on [0 , ∞ ) .
Note that ω ν (0)
= 1 and ω ν ( τ ) > 0 for all τ ∈ ( −∞ , 1) .
It is straightforward to check that φ (0) = 1 / 2 and φ ( τ ) > 0 for all τ ≥ 0 .
(b) Due to (13), it is clear that τ (cid:55)→ τ ¯ ω ν ( − τ ) is strictly increasing on [0 , ∞ ) and equals 0 at τ = 0 .
Note that, for any τ ≥ 0 ,
[Equation]
We get
[Equation]
By the monotonicity of τ (cid:55)→ τ ¯ ω ν ( − τ ) , it follows that ψ ′ ( τ )
Corollary 20.
Let τ ≥ 0 .
For any ν ≥ 2 , there exists K ν ∈ (0 , 1 / 2] such that
[Equation]
¯¯
[Equation]
In particular, K ν = 1 / 2 if ν = 2 and K ν = 1 / 4 if ν = 3 .
Proof.
The existence of K ν follows directly from the strict monotonicity of φ and ψ shown in
Lem.
For ν = 2 ,
[Equation]
As a result, we have ¯¯ ω
ν ( − τ ) ≥ 1 / 4 .
The case for ν = 3 can be proved similarly.
The next result shows that the local distance between the minimizer of f and x only depends on the geometry at x .
It can be used to localize the empirical risk minimizer as in Prop. 11.
Proposition 21.
Whenever R ν ∥∇ f ( x ) ∥ H − 1 ( x ) ≤ K ν , the function f has a unique minimizer ¯ x and
[Equation]
Proof.
Consider the level set
[Equation]
Take an arbitrary y ∈
L f ( f ( x )) .
According to Prop. 17, we have
[Equation]
By the Cauchy-Schwarz inequality and Lems.
18 and 19, we get
[Equation]
This implies
[Equation]
Due to Cor. 20, it holds that R ν ∥ y − x
∥ x < 1 + 1 { ν = 2 } and ¯¯ ω ν ( − R ν ∥ y − x ∥ x ) ≥ 1 / 4 .
It follows that d ν ( x, y ) < [-
-]1 + 1 { ν = 2 } and
[Equation]
Hence, the level set L f ( f ( x )) is compact so that f has a minimizer
¯ x .
Moreover, by Prop. 16 and ∇ 2 f ( x ) ≻ 0 , we obtain ∇ 2 f ( y ) ≻ 0 for all
y ∈ L f ( f ( x )) .
This yields that ¯ x is the unique minimizer of f and it satisfies
[Equation]
Remark 22.
A similar result also appears in [Ostrovskii and Bach, 2021, Prop. B.4].
We extend their result from ν ∈ { 2 , 3 } to ν ≥ 2 .
E.2 Concentration of random vectors and matrices
We start with the precise definitions of sub-Gaussian random vectors [Vershynin, 2018, Chapter 3.4] and the matrix Bernstein condition [Wainwright, 2019, Chapter 6.4].
Definition 23 (Sub-Gaussian vector) .
Let S ∈ R
d be a random vector.
We say S is sub-Gaussian if ⟨ S, s ⟩ is sub-Gaussian for every s ∈ R d .
Moreover, we define the sub-Gaussian norm of S as
[Equation]
Note that ∥·∥ ψ 2 is a norm and satisfies, e.g., the triangle inequality.
Remark 24.
When S is not mean-zero, we have
[Equation]
According to Vershynin [2018, Lemma 2.6.8], we obtain
[Equation]
where C is an absolute constant.
Definition 25 (Matrix Bernstein condition) .
Let H ∈ R
d × d be a zero-mean symmetric random matrix.
We say H satisfies a Bernstein condition with parameter b > 0 if, for all j ≥ 3 ,
[Equation]
The sum of i.i.d.
sub-Gaussian vectors is also sub-Gaussian according to the following lemma.
Lemma 26 (Vershynin [2018], Lemma 5.9) .
Let S 1 , . .
, S n be i.i.d.
random vectors, then we have
[Equation]
We call a random vector S ∈ R d isotropic if E [ S ] = 0 and E [ SS ⊤ ] = I d .
The following theorem is a tail bound for quadratic forms of isotropic sub-Gaussian random vectors.
Theorem 27 (Ostrovskii and Bach [2021], Theorem A.1) .
Let S ∈ R
d be an isotropic random vector with ∥ S ∥ ψ 2 ≤ K , and let J ∈ R d × d be positive semi-definite.
Then,
[Equation]
In other words, with probability at least 1 − δ , it holds that
[Equation]
A zero-mean symmetric random matrix H is said to be sub-Gaussian with parameter V if E [ e λH ] ⪯ e λ 2 V/ 2 for all λ ∈ R .
The next theorem is the Bernstein bound for random matrices.
Theorem 28 (Wainwright [2019], Theorem 6.17) .
Let { H i } ni
=1 be a sequence of zero-mean independent symmetric random matrices that satisfies the Bernstein condition with parameter b > 0 .
Then, for all δ > 0 , it holds that
[Equation]
where σ 2 := 1 n | (cid:80) ni
=1 V ar( H i ) | 2 .