E XPONENTIAL G ENERALIZATION B OUNDS WITH N EAR -O PTIMAL R ATES FOR L q -S TABLE A LGORITHMS
Anonymous authors
Paper under double-blind review
A BSTRACT
The stability of learning algorithms to changes in the training sample has been actively studied as a powerful proxy for reasoning about generalization.
Recently, exponential tail generalization and risk bounds with near-optimal rates have been obtained under the stringent and distribution-free notion of uniform stability (Bousquet et al., 2020; Klochkov & Zhivotovskiy, 2021).
In the meanwhile, under the notion of L q -stability, which is weaker and distribution dependent, exponential generalization bounds are also available yet so far only with sub-optimal rates.
Therefore, a fundamental question we would like to address in this paper is whether it is possible to derive near-optimal exponential generalization bounds for L q -stable learning algorithms.
As the core contribution of the present work, we give an afﬁrmative answer to this question by developing strict analogues of the near-optimal generalization and risk bounds of uniformly stable algorithms for L q -stable algorithms.
Further, we demonstrate the power of our improved L q -stability generalization theory and analysis by applying them to derive strong sparse excess risk bounds, under mild conditions, for computationally tractable sparsity estimation algorithms such as Iterative Hard Thresholding (IHT).
I NTRODUCTION
A fundamental issue in statistical learning is to bound the generalization error of a learning algorithm for understanding its prediction performance on unseen data.
It has long been recognized in literature that one of the key characteristics that permits learning algorithms to generalize is the stability of estimated model to perturbations in training data.
The idea of using algorithmic stability as a proxy for generalization performance analysis dates back to the seventies (Rogers & Wagner, 1978;
Devroye & Wagner, 1979).
Since the seminal work of Bousquet & Elisseeff (2002), the search for generalization bounds under various notions of algorithmic stability has been a ﬂourishing area of learning theory (Zhang, 2003; Mukherjee et al., 2006; Shalev-Shwartz et al., 2010; Kale et al., 2011;
Hardt et al., 2016; Celisse & Guedj, 2016; Bousquet et al., 2020).
As one may expect, the stronger an algorithmic stability criterion is, the sharper the resulting generalization bound will be.
On one end, exponential generalization bounds can be guaranteed by approaches under the most stringent notion of uniform stability (Bousquet & Elisseeff, 2002; Bousquet et al., 2020), which requires the change in the prediction loss to be uniformly small regardless data distribution.
Despite the strength of generalization, the distribution-free nature makes uniform stability too restrictive to be fulﬁlled, e.g., by learning rules with unbounded losses (Celisse &
Guedj, 2016).
On the other end, based on weaker and distribution dependent notions of stability such as hypothesis stability and mean-square stability, only polynomial generalization bounds seem possible in general cases, although the corresponding stability criteria are more amenable to veriﬁcation (Bousquet & Elisseeff, 2002).
These observations have prompted the development of L q -stability, as an in-between state, to achieve the best of two worlds (Celisse & Guedj, 2016;
Abou-Moustafa & Szepesv ´ ari, 2019): it generalizes the notion of hypothesis stability from (cid:96) 1 -norm criterion to L q -norm criterion for q ≥ 2 but remains distribution dependent and thus weaker than uniform stability; in the meanwhile it can still achieve the appealing exponential tail bounds of generalization matching those from Bousquet & Elisseeff (2002) for uniformly stable algorithms.
By far, the best known (and near-optimal) rates about exponential generalization bounds are offered by approaches based on uniform stability and certain ﬁne-grained concentration inequalities for sum of functions of independent random variables (Feldman & Vondrak, 2019; Bousquet et al., 2020).
These rates are substantially sharper than those of Bousquet & Elisseeff (2002), which are implied by a naive application of McDiarmid’s inequality, in terms of the overhead factors on stability coefﬁcients.
While for L q -stable learning algorithms, the state-of-the-art exponential generalization bounds are derived based on the moments or exponential extensions of the Efron-Stein inequality (Celisse & Guedj, 2016; Abou-Moustafa & Szepesv´ari, 2019), which yield similar rates of convergence to those of Bousquet & Elisseeff (2002) and thus are suspected to be sub-optimal .
Given the above observed gap in rates of convergence between the generalization bounds under uniform stability and L q -stability, the following question is naturally raised:
Is it possible to derive sharper exponential generalization bounds for L q -stable learning algorithms that match those recent breakthrough results for uniformly stable algorithms?
As the core contribution of the present work, we give an afﬁrmative answer to this open question by developing strict analogues of the near-optimal generalization bounds of uniformly stable algorithms for L q -stable algorithms.
To illustrate the importance of our theory, we have applied the improved L q -stability analysis techniques to derive sharper exponential risk bounds for computationally tractable sparsity recovery estimators, such as the Iterative Hard Thresholding (IHT)
algorithms widely used in high dimensional sparse learning (Blumensath & Davies, 2009; Foucart, 2011; Jain et al., 2014).
This application also serves as a key motivation of our study.
Notation.
Here we provide some notation that will be frequently used throughout the paper.
Let
S =
{ Z 1 , Z 2 , ..., Z N } be a set of independent random data samples valued in some measurable set
Z .
For any indices set I ⊆
[ N ] := { 1 , ..., N } , we denote by S
I =
{ Z i , i ∈ I } and S I =
S \ S I .
We denote by S (cid:48)
= { Z (cid:48) 1 , Z (cid:48) 2 , ..., Z (cid:48) N } another i.i.d.
sample from the same distribution as that of
S and we write S ( i ) =
{ Z 1 , ..., Z i − 1 , Z (cid:48)
i , Z i +1 , ..., Z N } .
For a real-valued random variable Y ,
its
L q -norm for q ≥ 1 is deﬁned by (cid:107) Y (cid:107) q
=
( E [ | Y | q ]) 1 /q .
By deﬁnition it can be veriﬁed that ∀ q ≥ 2 ,
[Equation]
Let g : Z N (cid:55)→ R be some measurable function and consider the random variable g ( S )
=
g ( Z 1 , ...
Z N ) .
For g ( S ) and any index set I ⊆
[ N ] , we deﬁne the following abbreviations:
[Equation]
We say a real-valued function f is G -Lipschitz continuous over the domain W if
[Equation]
For a pair of functions f, g ≥ 0 , we use f (cid:46) g to denote f ≤ cg for some universal constant c > 0 .
We denote supp ( w ) as the support of a vector w which is the index set of nonzero entries of w .
1. S ETUP AND P RIOR R ESULTS
Problem setup.
We consider a statistical learning algorithm
A : Z N (cid:55)→ W that maps a training data set S to a model
A ( S ) in a closed subset W of an Euclidean space.
The population risk and corresponding empirical risk evaluated at A ( S ) are respectively given by
[Equation]
( S [Equation]
where (cid:96) : W × Z (cid:55)→ R + is a non-negative and potentially unbounded loss function whose value (cid:96) ( w ; z ) measures the loss evaluated at z with parameter w .
As a classic fundamental issue in statistical learning, we are interested in deriving the upper bounds on the difference between population and empirical risks, i.e., | R ( A ( S )) − R S ( A ( S ))
| , which quantiﬁes the generalization error of A .
Let R ∗ :=
min w ∈W R ( w ) be the optimal value of the population risk.
We will also study how to upper bound R ( A ( S ))
− R ∗ (a.k.a.
excess risk) which is of particular interest for understanding the population risk minimization performance of A .
We ﬁrst introduce the concept of uniform stability (Bousquet & Elisseeff, 2002) which requires the change in the prediction loss to be uniformly small regardless data distribution.
Deﬁnition 1 (Uniform stability) .
A learning algorithm
A is said to have uniform stability by γ u >if it satisﬁes the following uniform bound:
[Equation]
Given that the loss function (cid:96) is almost surely bound by M , Bousquet & Elisseeff (2002) showed that a large class of regularized empirical risk minimization (ERM) algorithms has uniform stability, and using McDiarmids inequality yields the following exponential tail generalization bound that holds with probability at least 1 − δ over S for any δ ∈ (0 , 1) :
[Equation]
Recently, equipped with a strong concentration inequality for sum of random functions, Bousquet et al.
(2020) established the following moments bound of uniformly stable algorithms for all q ≥ 2 :
[Equation]
In view of the equivalence between tails and moments (see, e.g., Bousquet et al., 2020, Lemma 1), the above L q -norm bound immediately implies the following tail bound that holds with probability at least 1 − δ over S for any δ ∈ (0 , 1) :
[Equation]
This bound substantially improves the classic result in equation 2 by reducing the overhead factor on stability coefﬁcient from O (cid:0)(cid:113) N log( 1 δ ) (cid:1) to O (log( N ) log( 1 δ )) .
For example, in regimes such as regularized ERM where γ u (cid:46) 1 √ N is usually the case, the convergence rate in bound equationbecomes vacuous as it is not vanishing in sample size, while the bound in equation 4 still guarantees
O (cid:0) log( N )log( 1 δ ) √ N (cid:1) rate of convergence.
Indeed, up to logarithmic factors on sample size and tail bounds, the bound in equation 4 is nearly optimal in the sense of a lower bound on sum of random functions by Bousquet et al.
Under the generalized Bernstein condition (Koltchinskii, 2006) and using the sharp concentration inequality for sum of random functions by Bousquet et al.
Klochkov & Zhivotovskiy (2021) alternatively established the following deviation optimal excess risk bound that holds with probability at least 1 − δ over S :
[Equation]
where ∆ opt := R S ( A ( S )) − min w ∈W R S ( w ) represents the empirical risk sub-optimality of the algorithm on training data, and B is the Bernstein condition constant as deﬁned in Assumption 1.
While implying strong generalization guarantees, the uniform stability is also most stringent in the sense that it is distribution independent and hard to be fulﬁlled, e.g., by learning rules with unbounded losses.
To address such an unpleasant restrictiveness, the notion of L q -stability was alternatively introduced by Celisse & Guedj (2016) as a relaxation of uniform stability.
Deﬁnition 2 ( L q -Stability) .
For q ≥ 1 , a learning algorithm
A is said to have L q -stability by coefﬁcient γ q >
if it satisﬁes the following moment bound:
[Equation]
In the above deﬁnition, the expectation associated with L q -norm is taken over S, S ( i ) , Z , and the internal random bits of A , if any (such as in the case of stochastic optimization algorithms).
Note that slightly different from that of Celisse & Guedj (2016), the random variable Z in the above deﬁnition is not necessarily required to be independent of S and S ( i ) .
By deﬁnition, L q -stability is distribution dependent and thus is weaker than uniform stability which can be regarded as a special case of L qstability with γ q ≡ γ u for all q ≥ 1 .
Particularly for q = 1 and q = 2 , the L q -stability reduces to the notions of hypothesis stability (Bousquet & Elisseeff, 2002) and mean-square stability (Kale et al., 2011), respectively.
For an instance, it has been shown that the classical ridge regression model with unbounded responses has L q -stability for all q ≥ 1 rather than uniform stability (Celisse & Guedj, 2016).
As a novel and concrete example, we will see shortly in Section 3 that L q -stability plays a crucial role for deriving strong sparse excess risk bounds for sparsity estimation algorithms such as
IHT (Jain et al., 2014; Yuan et al., 2018).
Alternatively, the deﬁnition of L q -stability can be extended to the L q -argument-stability as (cid:13)(cid:13)
A ( S ) − A
( S ( i ) )
(cid:13)(cid:13)
q ≤ γ q , which generalizes the concept of uniform argument stability (Bassily et al., 2020) to the L q -norm criterion.
Obviously L q -argument-stability is not at all relying on the random argument Z
and it implies L q -stability for Lipschitz losses.
The following is by far the best known moments generalization bound under L q -stability that holds for all q ≥ 2 and potentially unbounded losses (Celisse & Guedj, 2016; Abou-Moustafa & Szepesv´ari, 2019):
[Equation]
As one can see that the L q -stability generalization bound equation 6 is signiﬁcantly inferior to the near-optimal uniform stability generalization bound equation 3 in terms of the overhead on stability coefﬁcient.
Such a gap in rate of convergence is indeed unsurprising: the bound equation 6 was derived via more or less directly applying moments or exponential extensions of Efron-Stein inequality to generalization error (Celisse & Guedj, 2016; Abou-Moustafa & Szepesv ´ ari, 2019), and thus yields about the same overhead factor on stability coefﬁcient as that of the sub-optimal exponential bound in equation 2 for uniform stable algorithms.
In light of these observations, we are naturally motivated to derive sharper exponential generalization bounds for L q -stable algorithms hopefully to match the near-optimal bound equation 3 achievable by uniformly stable algorithms.
1. O UR C ONTRIBUTION
The core contribution of the present work is a set of substantially improved exponential generalization bounds for L q -stable algorithms.
The key ingredient of our analysis is a sharper concentration bound on sum of functions of independent random variables under the L q -norm bounded difference conditions, which generalizes a previous counterpart under the uniform bounded difference conditions (Bousquet et al., 2020).
With this generic concentration bound in hand, we are able to derive sharper generalization and excess risk bounds for L q -stable learning algorithms that match those best known for uniform stable algorithms.
The power of our results is demonstrated through deriving stronger exponential sparse excess risk bounds for computationally tractable sparsity estimation algorithms (such as IHT).
Below we sketch the main results obtained in this work:
• In Section 2, we ﬁrst establish in Theorem 1 an L q -norm inequality for sum of functions of random variables with L q -norm bounded difference.
Then equipped with such a general-purpose concentration inequality, we prove in Theorem 2 the following L q -norm generalization bound for L q -stable learning algorithms for all q ≥ 2 :
[Equation]
where M q is an upper bound of moments (cid:107) (cid:96) ( A ( S ); Z ) (cid:107) q .
Compared to equation 6, the preceding bound improves the overhead factor on γ q from √ N to log( N ) .
As another consequence of our L q -norm concentration inequality, we further derive in Theorem 3 the following excess risk bound for L q -stable algorithms under B -Bernstein-condition with M -bounded losses (where C = M + B ), or µ -quadratic-growth condition with G -Lipschitz losses (where C = G 2 µ ):
[Equation]
Based on the equivalence between moments and tails, this above result implies an identical deviation optimal risk bound in equation 5 for uniformly stable algorithms.
• In Section 3, based on our L q -stability generalization theory, we show in Theorem 4 a novel exponential sparse excess risk bound for inexact L 0 -estimators.
A key insight here is that L 0estimators are in many cases “almost always” stable over any ﬁxed supporting set, and thus can be shown to have L q -stability over the same supporting set, which consequently makes our analysis techniques developed for L q -stable algorithms applicable there.
S HARPER E XPONENTIAL B OUNDS FOR L q -S TABLE A LGORITHMS 2. A M OMENT I NEQUALITY FOR S UM OF R ANDOM F UNCTIONS
We start by presenting in the following theorem a moment inequality for sum of random functions of N independent random variables that satisfy the L q -norm bounded difference condition.
See
Appendix B.1 for its proof which follows the techniques by Bousquet et al.
(2020, Theorem 4).
Theorem 1.
Let S
=
{ Z 1 , Z 2 , ..., Z N } be a set of independent random variables valued in Z .
Let g 1 , ..., g N be a set of measurable functions
g i : Z N (cid:55)→ R that satisfy the following conditions for any i ∈ [ N ] :
• E [ g i ( S ) | S \ Z i ] = 0 almost surely; • g i ( S ) has the following L q -norm bounded difference property with respect to all variables in S except Z i , i.e., ∀ j (cid:54)
=
i , for all q ≥ 1 :
[Equation]
Then there exists a universal constant κ < 1 . 271 such that for all q ≥ 2 ,
[Equation]
Additionally, if (cid:107) E [ g i ( S ) | Z i ] (cid:107) q ≤ M q , then for all q ≥(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Remark 1.
Theorem 1 extends the moment inequality of Bousquet et al.
(2020, Theorem 4) from under the distribution-free uniform bounded difference property to under the L q -norm bounded difference property which is distribution dependent.
Specially if g i ( S ) have uniformly bounded difference property, then Theorem 1 reduces to the result of Bousquet et al.
(2020, Theorem 4).
Remark 2.
The L q -norm boundedness condition (cid:107) E [ g i ( S )
| Z i ] (cid:107) q ≤ M q in our theorem allows g i to be potentially unbounded over domain of interest, which is weaker than the corresponding almost sure boundedness condition on | E
[ g i ( S ) |
Z i ] | as imposed by Bousquet et al.
(2020, Theorem 4).
2. G ENERALIZATION B OUNDS FOR L q -S
As a consequence of Theorem 1, we can show the following main result on the generalization bound of L q -stable learning algorithms.
See Appendix B.2 for a proof of this result.
Theorem 2.
Let A : Z N (cid:55)→ W be a learning algorithm that has L q -stability by γ q > 0 for q ≥ 1 .
Suppose that (cid:107) (cid:96) ( A ( S ); Z ) (cid:107) q ≤ M q for any Z ∈ Z .
Then for all q ≥ 2 , (cid:107) [Equation] S ) [Equation]
qγ
[Equation]
N + M q (cid:114) q
N .
Remark 3.
The L q -norm boundedness condition (cid:107) (cid:96) ( A ( S ); Z ) (cid:107) q ≤ M q allows for learning with unbounded losses over, e.g., data distribution with sub-Gaussian or sub-exponential tail bounds.
To compare with the best known moments generalization bound equation 6 under the notion of L q -stability, our bound in Theorem 2 substantially improves the overhead factor on γ q from √ N to log( N ) .
Specially when reduced to regime of uniform stability where γ q ≡ γ u for all q ≥ 1 , our result revisits the moments generalization bound in equation 3 which is nearly tight, up to logarithmic factors on sample size, in the sense of a lower bound on sum of random functions from Bousquet et al.
More broadly, suppose that the following exponential stability bound holds with probability at least 1 − δ for any δ ∈ (0 , 1) with a mixture of sub-Gaussian and subexponential tails 1 over S, S ( i ) , Z :
[Equation]
Then according to the equivalence of tails and moments, as summarized in Lemma 4 (see Appendix A), we must have that A is L q -stable by γ q = aq + b √ q .
Assume that the loss is bounded almost surely over data.
Then the L q -norm generalization bound in Theorem 2 combined with
Lemma 4 immediately implies the following generalization bound:
[Equation]
Compared with the uniform stability implied tail bound equation 4, the preceding L q -stability bound is nearly identical up to slightly worse conﬁdence tail terms which are caused by the uncertainty of L q -stability with respect to data distribution.
We conjecture that such a slight deterioration in tail bounds might possibly be remedied by using the exponential versions of Efron-Stein inequality (Boucheron et al., 2003; Abou-Moustafa & Szepesv ´ ari, 2019) instead of its currently used moments variant.
We leave the improvement over poly-logarithmic terms for future investigation.
2. E XCESS R ISK B OUNDS FOR L q -S
In addition to the generalization bounds, we further apply Theorem 1 to study the excess risk bounds of an L q -stable learning algorithm which are of particular interest for understanding its population risk minimization performance.
Let us denote W ∗ :=
Argmin w ∈W R ( w ) as the optimal solution set of the population risk.
In order to get sharper risk bounds, we need to impose some structural conditions on risk functions.
Particularly, the following deﬁned generalized Bernstein condition (Koltchinskii, 2006) is conventionally used with multiple global risk minimizers allowed.
Assumption 1 (Generalized Bernstein condition) .
For some B > 0 and for any w ∈ W , there exists w ∗ ∈ W ∗ such that the following holds:
[Equation]
We will also consider the quadratic growth condition which is widely used as an alternative condition for establishing fast rates of convergence in learning theory.
Assumption 2 (Quadratic growth condition) .
For some µ > 0 and for any w ∈ W , there exists w ∗ ∈ W ∗ such that the following holds:
[Equation]
Remark 4.
Clearly, when the loss is G -Lipschitz, the quadratic growth condition with parameter µ implies the Bernstein condition with parameter B = 2 G 2 µ .
The following is our main result on the excess risk bound of L q -stable algorithms, which extends the near-optimal exponential risk bounds of Klochkov & Zhivotovskiy (2021) from uniform stable algorithms to L q -stable algorithms.
A proof of this theorem can be found in Appendix B.3.
Theorem 3.
Let A : Z N (cid:55)→ W be a learning algorithm that has L q -stability by γ q for q ≥ 1 .
(a) If Assumption 1 holds and (cid:96) ( · ; · ) ≤ M , then ∀ q ≥ 2 ,
[Equation]
Assumption and · ) is G Lipschitz with respect to its ﬁrst argument,
[Equation]
Compute an inexact k -sparse L 0 -ERM estimation ˜ w S,k such that
• ˜ w S,k is optimal over its support ˜ J = supp ( ˜ w S,k ) , i.e., ˜ w S,k = arg min w ∈W , supp ( w ) ⊆ ˜ J R S ( w ) ; • ˜ w S,k attains certain ¯ k -sparse sub-optimality level ∆ ¯ k, opt ≥ 0 for some ¯ k ≤ k such that [Equation]
˜
[Equation] S [Equation] [Equation]
[Equation] ∆ ¯ k, opt . Remark 5. Suppose that A satisﬁes the exponential stability bound in equation 7, and thus A has L q -stability by γ q = aq + b √ q .
Then combined with Lemma 4, the L q -norm risk bounds in Theorem 3 suggest that the following exponential tail bound holds:
[Equation]
Remark 6.
In part (a), the M -bounded-loss condition is not essential and it can be relaxed to a subexponential (or sub-Gaussian) variant by alternatively using the general Bernstein-type inequalities for sums of independent sub-exponential random variables (Vershynin, 2018) . Concerning part (b), under the quadratic growth condition, the loss is allowed to be unbounded but Lipschitz continuous.
A PPLICATION TO I NEXACT L 0 -ERM
In this section, we demonstrate an application of our L q -stability and generalization theory to the following problem of high-dimensional stochastic risk minimization under hard sparsity constraint: [Equation] ] subject to (cid:107)
[Equation]
(cid:107) [Equation] , where W ⊆ R d the cardinality constraint (cid:107) w (cid:107) 0 ≤ k for k (cid:28) d is imposed for enhancing the interpretability and learnability of model in cases where there are no clear favourite explanatory variables, or the model is over-parameterized.
We consider the following L 0 -ERM problem over training set S
=
{ Z i } i ∈ [ N ] :
[Equation]
Since the problem is known to be NP-hard (Natarajan, 1995), it is computationally intractable to solve it exactly in general cases.
Alternatively, we consider the inexact L 0 -ERM oracle as a metaalgorithm outlined in Algorithm 1.
In order to avoid assuming unrealistic conditions like restricted isometry property (RIP), it is typically needed to allow sparsity level relaxation for approximate algorithms like IHT to achieve favorable converge behavior (Jain et al., 2014; Shen & Li, 2016; Yuan et al., 2018;
Zhou et al., 2018).
Therefore, we are particularly interested in the inexact L 0ERM oracle with ¯ k -sparse sub-optimality ∆ ¯ k, opt ≥ 0 for some ¯ k ≤ k such that the output ˜ w S,k of
Algorithm 1 satisﬁes
[Equation]
It is typical that ∆ ¯ k, opt is a random value over the training set S .
For example, the sub-optimality guarantees of IHT for empirical risk usually hold with high probability over training data (Jain et al., 2014).
Let w ∗ ¯ k := arg min (cid:107) w (cid:107)
≤ ¯ k R ( w ) be the ¯ k -sparse minimizer of population risk for some ¯ k ≤ k .
We are interested in deriving exponential upper bounds for the ¯ k -sparse excess risk given by R ( ˜ w S,k ) − R ( w ∗ ¯ k ) .
Our analysis also relies on the conditions of Restricted Strong Convexity (RSC) which extends the concept of strong convexity to the analysis of sparsity recovery methods (Bahmani et al., 2013;
Blumensath & Davies, 2009; Jain et al., 2014; Yuan et al., 2020).
Deﬁnition 3 (Restricted Strong Convexity) .
For any sparsity level 1 ≤ s ≤ d , we say a function f is restricted µ s -strongly convex if there exists some µ s > 0 such that f ( w ) − f ( w (cid:48) )
{+−(cid:104)∇ f ( w (cid:48) ) , w − w (cid:48) (cid:105) ≥ µ s 2 (cid:107) w − w (cid:48) (cid:107) 2 , ∀(cid:107) w − w (cid:48) (cid:107) 
 ≤ s .+}
Specially when s = d , we say f is µ -strongly convex if it is µ d -strongly convex.
The following basic assumptions will be made in our theoretical analysis.
Assumption 3.
The loss function (cid:96) ( · ; · ) is convex and G -Lipschitz with respect to its ﬁrst argument.
Assumption 4.
The population risk R is µ -strongly convex and the empirical risk R S is µ k -strongly convex with probability at least 1 − δ N over sample S for some δ N ∈ (0 , 1) .
Assumption 5.
The domain of interest is uniformly bounded such that (cid:107) w (cid:107) ≤ D, ∀ w ∈ W .
Let w ∗ :=
arg min w ∈W R ( w ) be the global minimizer which is unique due to the strong convexity of R .
For a given index set J ⊆
[ d ] , let us consider the following restrictive estimator over J :
[Equation]
We ﬁrst present the following lemma that guarantees the L q -stability of w ∗ S | J for any ﬁxed J with | J | = k .
See Appendix C.1 for its proof.
Lemma 1.
Asssume that Assumptions 3, 4 and 5 hold and log(1 /δ
N ) log( N ) ≥ 2 .
Let J ⊆
[ d ] , | J |
= k be a set of indices of cardinality k .
Then for any q ≥ 2 , the oracle estimator w ∗ S | J has L q -stability by
[Equation]
Remark 7.
For sparse linear regression models, it can be veriﬁed based on the result by Agarwal et al.
(2012, Lemma 6) that Assumptions 4 holds with δ N = e − c 0 N for some universal positive constant c 0 .
Then we have log(1 /δ
N ) log( N ) =
c 0 N log( N ) ≥ 2 for sufﬁciently large N , and Lemma 1 implies that γ q (cid:46) 1 N (cid:0) G 2 µ
[Equation]
The following theorem is our main result on the sparse excess risk of the inexact L 0 -ERM oracle as deﬁned in Algorithm 1.
See Appendix C.2 for its proof which is stimulated by that of Theorem 3.
Theorem 4.
Suppose that Assumptions 3, 4, 5 hold.
Assume that log(1 /δ N ) log( N ) ≥ 2 .
Then for any δ ∈ (0 , e − 1 ) , the following ¯ k -sparse excess risk bound holds with probability at least 1 − δ over S :
[Equation]
A few remarks are in order.
Remark 8.
For sparse linear regression models in which Assumptions 4 holds with δ N = e − c 0 N (Agarwal et al., 2012), we have that log(1 /δ
N ) log( N ) =
c 0 N log( N ) ≥ 2 can always be fulﬁlled for sufﬁciently large sample size N , and the sparse excess risk bound in Theorem 4 roughly scales as
[Equation]
In the well-speciﬁed ¯ k -sparse models such that R ( w ∗ ¯ k ) =
R ( w ∗ ) , the above rate improves to R
[Equation]
S,
[Equation]
Remark 9.
For the IHT-style algorithms, under proper sparsity level relaxation, the sparse optimization sub-optimality ∆ ¯ k, opt in the above sparse excess risk bounds can be arbitrarily small (with high probability) after sufﬁcient iteration (Jain et al., 2014).
Remark 10.
For misspeciﬁed sparsity models, the dominant rate in Theorem 4 matches the O (cid:0) 1 √ N (cid:1) sparse excess risk bound (Yuan & Li, 2022, Theorem 1) for IHT under similar conditions. As discussed in Remark 8, our bound is more appealing in the sense that it naturally adapts to wellspeciﬁed models to attain an improved O ( 1 N ) rate. Compared to the O (cid:0) 1 N (cid:1) bound also available in that paper (Yuan & Li, 2022, Theorem 3), our bound is generally slower in rate but more broadly applicable without imposing any strong-signal or bounded-loss conditions as required therein.
O THER R ELATED W ORK
Uniform stability and exponential generalization.
Stimulated by a recent landmark work of Hardt et al.
, there is a renewed interest in the use of uniform stability for deriving generalization bounds for various learning algorithms and paradigms including stochastic gradient descent (SGD) (Kuzborskij & Lampert, 2018; Charles & Papailiopoulos, 2018; Bassily et al., 2020; Lei &
Ying, 2020), stochastic model based optimization (Wang et al., 2017; Deng & Gao, 2021), optimization based meta learning (Zhou et al., 2019), and differential privacy stochastic optimization (Bassily et al., 2019; Feldman et al., 2020).
Compared to other stability arguments, uniform stability is notorious for implying high-probability generalization bounds in addition to the traditional in-expectation bounds.
Until very recently, the basic result of Bousquet & Elisseeff (2002), as expressed in equation 2, remains the best known exponential generalization bound for uniformly stable algorithms.
Using some elegant proof techniques from adaptive data analysis, Feldman & Vondrak (2018) managed to replace the ﬁrst term in equation 2 by (cid:113) γ u log( 1 δ ) , which leads to an improvement whenever γ u (cid:38) 1 N .
Soon after, a series of breakthrough results were obtained (Feldman & Vondrak, 2019; Bousquet et al., 2020) using tighter concentration bounds for sum of random functions, which eventually improve the stability dependent rate to a near-optimal one γ u log( N ) log (cid:0) 1 δ (cid:1) as shown in equation 4.
Additionally under the generalized Bernstein condition, these state-of-the-art results lead to O ( 1 N ) excess risk bounds for uniformly stable algorithms (Klochkov & Zhivotovskiy, 2021).
Non-uniform stability and exponential generalization.
More broadly for non-uniformly stable algorithms, exponential generalization bounds have also been shown to be possible under various weaker and distribution-dependent notions of stability.
As an early work in this line, Kutin & Niyogi (2002) showed that under the so called “almost-everywhere” stability, which is a high-probability counterpart of uniform stability, generalization bounds that hold with overwhelming probability are still possible in view of certain modiﬁed McDiarmid’s inequality (Kutin, 2002).
Later, Rakhlin et al.
(2005) revisited the bounded-difference results of Kutin & Niyogi (2002) in a more straightforward manner by using a powerful moment extension of Efron-Stein inequality (Boucheron et al., 2005).
Recently for general L q -stable algorithms, the exponential leave-one-out generalization bounds were derived using moment or exponential extensions of Efron-Stein inequality, with applications found in ridge regression, k -nearest neighbor classiﬁcation and k -folds cross-validation (Celisse & Guedj, 2016; Celisse & Mary-Huard, 2018; Abou-Moustafa & Szepesv ´ ari, 2019).
We refer the interested readers to Boucheron et al.
; Kontorovich (2014); Combes (2015); Warnke (2016); Maurer & Pontil (2021) for more results on concentration inequalities beyond bounded-difference conditions, which are fundamental for deriving exponential bounds for non-uniformly stable algorithms.
Generalization analysis of (cid:96) 0 -estimators.
We further brieﬂy review some prior results on the generalization guarantees for statistical learning under cardinality constraints, which is the theme of the application part of our work.
For the (cid:96) 0 -ERM estimator as expressed in equation 8, provided that the solution is exactly known, a series of uniform excess risk bounds were derived over binary prediction classes (Chen & Lee, 2018; 2020) and bounded liner prediction classes (Foster & Syrgkanis, 2019).
The exact solutions of (cid:96) 0 -ERM, however, is computationally intractable in general high-dimensional cases due to the NP-hardness of problem.
Therefore, it is more realistic and desirable to establish generalization bounds for approximate (cid:96)
-estimators such as the IHT-style algorithms (Yuan et al., 2018; Garg & Khandekar, 2009; Li et al., 2016).
Particularly for misspeciﬁed sparsity models, a set of sparse excess risk bounds with slow and fast rates were established through the lens of uniform stability theory under proper regularity conditions (Yuan & Li, 2022).
C ONCLUSIONS
There exits a clear discrepancy between the recently developed near-optimal generalization bounds for uniformly stable algorithms and the best known yet sub-optimal bounds for L q -stable algorithms.
Aiming at closing such a theoretical gap, we for the ﬁrst time derive in this paper a set of near-optimal exponential generalization bounds for L q -stable algorithms that match those of the uniformly stable algorithms.
As a concrete application of our L q -stability theory, we have applied our improved analysis tools to derive strong exponential risk bounds for inexact sparsity-constrained ERM estimators under mild conditions.
To conclude, L q -stable algorithms generalize almost as sharply as uniformly stable algorithms, though the notion of L q -stability is weaker and distribution-dependent.
A P RELIMINARIES
In this section, we collect some preliminary results that will be used in our analysis.
We start by introducing the following L q -norm generalization of the celebrated Efron-Stein inequality, which is a corollary of Boucheron et al. (2005, Theorem 2).
Proposition 1 (Generalized Efron-Stein inequality (Celisse & Guedj, 2016)) .
Let S
=
{ Z 1 , ..., Z N } be a set of independent random variables valued in Z and g : Z N
(cid:55)→ R be some measurable function.
Then there exists a universal constant κ < 1 . 271 such that for all q ≥ 2 ,
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N (cid:88) i = (cid:0)
[Equation]
( S ) − g ( S ( i ) ) (cid:1) 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) q/.
The following result is an immediate consequence of Proposition 1 when applied to sum of independent random variables, which revisits a version of Marcinkiewicz-Zygmund inequality (Chow &
Teicher, 2003).
Proposition 2.
Let Z 1 , ..., Z N be a set of independent centered random variables.
Then for all q ≥ 2 , (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) N (cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) q
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) q
[Equation]
The following lemma is simple yet useful in our analysis.
Lemma 2.
Let S
=
{ Z 1 , ..., Z N } be a set of independent random variables valued in some measure space Z and g : Z N
(cid:55)→ R be some measurable function.
Then for all I ⊆
[ N ] and q ≥ 1 , we have
[Equation]
Proof.
Recall g ( S I ) =
E [ g ( S ) | S I ] .
Then using Jensen’s inequality we can show that
[Equation]
By deﬁnition we can also express (cid:107) g ( S ) (cid:107) q
= ( E
[Equation]
g
[Equation]
(cid:107) q
( S I ) (cid:107) q .
As a direct consequence of Lemma 2, the following result indicates that conditional expectation does not expand the differences in L q -norm.
Lemma 3.
Let S
=
{ Z 1 , ..., Z N } be a set of independent random variables valued in some measure space Z and g : Z N
(cid:55)→ R be some measurable function.
Let I ⊆
[ N ] be an index set.
Then for all i ∈ I and q ≥ 1 ,
[Equation]
(cid:13)(cid:13) g ( S I ) − g S ( i ) I ) (cid:13) (cid:13)(cid:13) q ≤ (cid:13) (cid:13)(cid:13) g ( S ) − g ( S ( i ) ) (cid:13) (cid:13)(cid:13) q .
Proof.
For each i ∈ I , by applying Lemma 2 to g ( S ) − g ( S ( i ) )
we can show that (cid:107) g
( S I ) − g ( S ( i ) I ) (cid:107) q ≤ (cid:107) g ( S )
− g ( S ( i ) )
(cid:107) q , which gives the desired result.
We also need the following lemma about the equivalence between tails and moments (see, e.g.,
Bousquet et al., 2020).
Lemma 4.
Let Y be a real-valued random variable.
• Suppose that Y satisﬁes the following inequality for some a, b ≥ 0 with probability at least 1 − δ for any δ ∈ (0 , 1) ,
[Equation]
Then, for any q ≥ 1 it holds that
[Equation]
• Suppose that Y satisﬁes (cid:107) Y (cid:107) q ≤ f ( q ) for any 1 ≤ q l ≤ q < q u and some non-negative real function f .
Then the following holds with probability at least 1 − δ for any δ ∈ ( e 1 − q u , e
[Equation]
l ] :
[Equation]
Proof.
We only prove the second part which slightly generalizes the corresponding result of (Bousquet et al., 2020, Lemma 1).
For any δ ∈ ( e 1 − q u , e 1 − q l ] , we choose q = log( e/δ ) ∈
[ q l , q u ) .
Using the condiiton (cid:107) Y (cid:107) q ≤ f ( q ) and Markov inequality yields
[Equation]
This proves the desired bound in the second part.
Remark 11.
Suppose that it holds with probability at least 1 − δ over S, S ( i ) ,
Z such that
[Equation]
then according to the ﬁrst part of Lemma 4 we have that A is L q -stable by γ q = 3 aq + 9 b √ q .
Remark 12. Specially if q l = 1 and q u = ∞ is allowed, then the second bound in Lemma 4 holds with an arbitrary tail bound δ ∈ (0 , 1) Finally, we present the following technical lemma about self-bounding inequalities to be used for showing fast rates of excess risk bounds under Bernstein or quadratic growth conditions.
Lemma 5.
Let x, a, b, c be a set of non-negative quantities satisfying x ≤ a + (cid:112) b ( x + c ) . Then it must hold that x ≤ 3 a +2 b + c. Proof.
If x ≤ a , then the claim holds trivially.
In the complementary case of x > a , by condition we must have ( x − a ) 2 ≤ b ( x + c ) , which then implies
[Equation]
where we have used the basic fact 2 (cid:112) b ( a + c ) ≤ b + a + c .
B P ROOFS FOR S ECTIONB. P ROOF OF T HEOREM
The proof is a generalization of that of Bousquet et al.
(2020, Theorem 4) under the considered property of L q -norm bounded difference, by using the generalized Efron-Stein inequality in places of McDiarmid’s inequality.
We reproduce the proof here for the sake of completeness.
Proof of Theorem 1.
Consider k such that 2 k − 1 < N ≤ 2 k .
If N < 2 k , we pad the training set S with extra zero-functions so that N = 2 k .
Consider the partition I 0 ,..., I k of [ N ] given by
{+I 
 = {{ 1 } , ..., { 2 k }} , I 1 = {{ 1 , 2 } , { 3 , 4 } ..., { 2 k − 1 , 2 k }} , I k = {{ 1 , ..., 2 k }} .+}
For any i ∈ [ N ] and l = 0 , ..., k , we denote by I l ( i
) ∈
I l the only set from I l that contains i and consider the following random variables
[Equation]
In particular, g 0 i =
g i and g ki
= E [ g i |
Z i ] .
Clearly we have the following telescope sum:
[Equation]
It follows that
[Equation]
[Equation]
We need to upper bound the right hand side of the above inequality.
To this end, it can be veriﬁed that
[Equation]
Since g i has a bounded L q -difference by β q with respect to all variables except the i -th variable, it is known from Lemma 3 that so is g li for each l = 0 , ..., k .
Conditioned on Z i , S I l +1 ( i ) , invoking
Proposition 1 to g li yields
[Equation]
as there are 2 l indices in I l +1 ( i ) \
I l
( i ) .
It follows from Lemma 2 that
[Equation]
l β q .
Now consider any I l ∈ I l .
Since for each i ∈ I l , g li − g l +1 i depends only on Z i , Z I l , these terms are independent and centered conditioned on S I l .
Therefore, applying Proposition 2 yields (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i ∈ I
l g li − g {+l +
+}
[-[Equation] +
-] i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) q (cid:0) S I l
[Equation]
≤
[Equation]
[-(cid:13)(cid:13)(cid:13) q ≤ (cid:88) I l ∈I l (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) (cid:88) i ∈ I l g li − g l +
 i (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) q-](cid:112)
[Equation]
l ×
[Equation]
β q , which according to Lemma 2 implies that (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i ∈ I
l g li − g l {++
+} i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) q ≤ 4 κq 2 l β q .
Then based on the triangle inequality we get (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:88)
i ∈
[ N ] g li − g l +1 i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) q ≤ (cid:88) I l ∈I l (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i ∈ I
l g {+li − g l +
+}
i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) q ≤ 2 k − l × [Equation]
Finally, the right hand side of equation 10 can be bounded as
[Equation]
which gives the ﬁrst desired bound
In view of equation 11 and the triangle inequality we have
[Equation]
Since (cid:107) E
[ g i ( S )
|
Z i ] (cid:107) q ≤ M q and E [ g i ( S ) | S \ Z i ] = [Equation] it follows from Proposition 2 that the ﬁrst term at the right hand side of equation 12 can be bounded as
[Equation]
The second desired bound is obtained by plugging equation 13 into equation 12.
B. P ROOF OF T HEOREM
The proof technique follows that of Bousquet et al.
(2020, Lemma 7) developed for uniformly stable algorithms, with natural adaptation to the distribution-dependent notion of L q -stability.
Proof.
Let us consider
[Equation]
Then the L q -norm of the generalization gap can be bounded as
[Equation]
We next respectively upper bound the two terms A and B in equation 14.
To bound the term A , by deﬁnition it holds that E [ g i ( S ) | S \ Z i ] = 0 .
Based on the triangle inequality we can show that
[Equation]
where in the ﬁrst and second inequalities we have twice used Lemma 2.
Next we further show that g i has a bounded L q -norm difference by 2 γ q with respect to all variables in S except Z i .
Indeed, for each j (cid:54) =
i it can be veriﬁed that
[Equation]
q
[Equation]
where in the last but one inequality we have used Lemma
[Equation]
while in the last equality we have used the L q -stability assumption on the algorithm A .
Therefore [Equation] { g i } satisfy the conditions of Theoremand thus [Equation]
[Equation]
Now we proceed to bound the term B .
It can be veriﬁed that
[Equation]
[Table]
[Equation]
where in the last but one inequality we have used Lemma 2, and in the last equality we have used the L q -stability assumption.
Plugging bounds equation 15 and equation 16 into equation 14 and preserving leading terms yields the desired result.
B. P ROOF OF T HEOREM
We need the following lemma which plays a fundamental role in proving the main result.
Lemma 6.
Let A : Z N (cid:55)→ W be a learning algorithm that has L q -stability by γ q for q ≥ 1 .
Suppose that (cid:107) (cid:96) ( A ( S ); Z ) (cid:107) q ≤ M q for any Z ∈ Z .
Let S (cid:48) be an independent copy of S .
Then the following bound holds for all q ≥ 2 :
[Equation]
Proof.
Let us again consider g i ( S ) =
E Z (cid:48)
i (cid:2) R ( A ( S ( [Equation] )) − (cid:96) ( A [Equation]
S (
[Equation] ) ); Z i ) (cid:3) . Then using similar proof arguments to those of Theorem 2 we can show that [Equation]
[Table]
[Equation]
which implies
[Equation]
Also [Equation] g i ( S ) satisﬁes the conditions of Theorem 1 with β q = 2 γ q
and it follows from the second bound of Theorem 1 that for all q ≥ 2 ,
[Equation]
Combining the above two yields
[Equation]
The desired result follows by noting that
[Equation]
This completes the proof.
With Lemma 6 in place, we are ready to prove the main result of Theorem [Equation]
Proof of Theorem 3.
Consider any w ∗ ∈ W ∗ .
It is standard to decompose and bound the excess risk as
[Equation]
where
[Equation]
Since we have the freedom to choose w ∗
[Equation]
let us specify it in the above as w ∗ ( S (cid:48) )
∈ W ∗ which is the minimizer that satisﬁes the Bernstein condition in Assumption 1 associated with A ( S (cid:48) ) .
Then, it follows from equation 17 that
[Equation]
Consequently,
[Equation]
where in “ ζ 1 ” we have applied Lemma 6 to obtain (cid:107) Γ( S ) (cid:107) q ≤ qγ q log( N )
[Equation]
and the fact E [ R ( A ( S ))]
=
E [ R ( A ( S (cid:48) ))] .
Part (a) :
To bound the term T , using Bernstein’s inequality for sum of independent bounded variables 2 together with the generalized Bernstein condition we can show (see the proof arguments of Klochkov & Zhivotovskiy (2021, Theorem 1.1) for the details) that
[Equation]
where the last inequality is due to Jensen’s inequality.
Therefore, combining the above and equation 18 yields that for some universal constant C :
[Equation]
By invoking Lemma 5 to the above self-bounding inequality with x = (cid:107) R ( A ( S )) − R ∗ − ∆ opt (cid:107) q , a = C ( qγ q log( N ) + qMN ) , b = qBN , and c = E [∆ opt ] we immediately obtain that
[Equation]
This gives the desired bound in part (a).
Part (b) : Under the given conditions in part (b), we can bound the term T in equation 18 as follows for q ≥ 2 :
[Table]
where in “ ζ 1 ” we have used Proposition 2, in “ ζ 2 ” we have used the Lipschitz-loss condition, in “ ζ 3 ” we have used (cid:13)(cid:13) E 2 [ G (cid:107) w ∗ ( S (cid:48) )
− A ( S (cid:48) )
(cid:107) | Z i ] (cid:13)(cid:13) [Equation] =
[Equation] [ G (cid:107) w
[Equation]
S [Table]
) − A ( S (cid:48) ) (cid:107) ] ≤
G 2 E (cid:2) (cid:107) w ∗ ( S (cid:48) ) − A ( S (cid:48) )
(cid:107) 2 (cid:3) , in “ ζ 4 ” we have used Assumption 2, and the last inequality is due to
Jensen’s inequality.
Then, plugging the above bound of term T into equation 18 yields that for some universal constant C :
[Equation]
Invoking Lemma 5 to the above inequality with x = (cid:107) R ( A ( S )) − R ∗ − ∆ opt (cid:107) q , a = Cqγ q log( N ) , b = qG 2 µN , and c = E [∆ opt ] yields
[Equation]
This gives the desired bound in part (b).
The proof is completed.
C P ROOFS FOR S ECTIONC. P ROOF OF L EMMA
Proof.
Let us consider the following event about the restricted strong convexity of R S :
E : R S is µ k -strongly convex .
Let Y =
1 E be the indication random variable associated with E .
Then by Assumption 4 we have P ( Y = 1) ≥ 1 − δ N .
Suppose that E occurs such that Y = 1 .
Then,
[Equation]
where we have used the optimality of w ∗ S ( i ) | J with respect to R S ( i ) ( w ) and the Lipschitz continuity of loss.
Since E occurs by assumption, R S is µ k -strongly convex.
Since w ∗ S | J is optimal for R S ( w ) over the supporting set J , we have
[Equation]
Combing the preceding two inequalities yields (cid:13)(cid:13)(cid:13) w ∗ S ( i ) | J − w ∗ S | J (cid:13)(cid:13)(cid:13) ≤ 4 G µ k N .
Consequently from the
Lipschitz continuity of (cid:96) we have that for any Z ∈ Z , the following holds conditioned on Y = 1 :
[Equation]
In the complementary case of Y = 0 , in view of Assumption 5, it always holds that
[Equation]
Let us consider q u := log
[Equation]
(cid:17) log( N ) .
By assumption q u ≥ 2 .
Then for 2 ≤ q ≤ q u , it can be veriﬁed that E
[Equation]
(cid:96) ( w
[Equation]
S ( i )
[Equation]
; Z )
[Equation]
w
[Equation]
S | J
[Equation]
Z )
[Equation]
q
[Equation]
= P ( Y = 1) E (cid:104)(cid:12)(cid:12) (cid:12) (cid:96) ( w
[Equation]
S ( i ) |
[Equation]
≤ [Equation] It follows that for all 2 ≤ q ≤ q u [Equation]
≤
[Equation]
µ
[Equation]
(cid:19)
[Equation]
N
[Equation]
(cid:19)
[Equation]
q ≤ N [Equation] where we have used a q + b q ≤ ( a + b ) q for a, b > 0 and q ≥ 2 .
For the complementary case q > q u , it is trivial to show that
[Equation]
Assembling the preceding two bounds yields the desired L q -stability bound.
C. P ROOF OF T HEOREM
Let us denote a + = max { a, 0 } .
We need the following key result in our analysis, whose proof idea draws large inspiration from that of Theorem 3 with proper modiﬁcations for handing the challenges imposed by the combinatorial optimization nature of L 0 -ERM.
Lemma 7.
Suppose that Assumptions 3, 4, 5 hold.
Assume that log(1 /δ N ) log( N ) ≥ 2 .
Then for any δ ∈ (0 , e − 1 ) [Equation] it holds with probability at least 1 − δ that
[Equation]
Proof.
Given a ﬁxed index set J ⊆
[ d ] with | J | = k
[Equation]
we can show that the following holds for any q ≥ 1 : [Equation] N
[Equation]
i
[Equation]
[Equation]
[Equation]
q (cid:124)
[Equation]
T , (21) where Γ S |
[Equation]
= R
[Equation]
N
N
[Equation]
i =
E [ (cid:96) ( w ∗ S (cid:48) |
[Equation] i ) | Z i ] . In view of Lemma 1 we have that w ∗ S | J has L q -stability by
[Equation]
Then invoking Lemma 6 over the supporting set J yields
[Equation]
We now bound the term T in equation 21 as follows:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where in “ ζ 1 ” we have used Proposition 2, in “ ζ 2 ” we have used the Lipschitz-loss assumption, in “ ζ 3 ” we have used (cid:13)(cid:13)(cid:13)
[Equation]
≤
G 2 E (cid:104) (cid:107) w ∗ − w ∗ S (cid:48) | J (cid:107) 2 (cid:105) , in “ ζ 4 ” we have used the strong convexity condition in Assumption 4.
Plugging equation 22 and equation 23 into equation 21 yields that for any q ≥ 2 ,
[Equation]
Next we need to upper bound the factor E (cid:104) R ( w ∗ S | J ) − R ( w ∗ ) (cid:105) in the second term of the above bound.
To do so, let us consider q = 2 in equation 24.
It follows from the optimality of w ∗ and w ∗ S | J that
[Equation]
equation 24 ≤
[Equation]
where in the ﬁrst inequality we have used Cauchy-Schwarz inequality, and in the last inequality we have used the fact √ ab ≤ a 2 t + bt 2 for any a, b, t > 0 .
Rearranging both sides of the above inequality to
[Equation]
+ log
[Equation]
N )
N
[Equation]
Nµ
[Equation]
Plugging equation 25 into equation 24 yields that for any q ≥(cid:13)(cid:13)(cid:13)
R ( w ∗ S | J ) − R ( w ∗ ) − R S ( w ∗ S | J ) + R S ( w ∗ ) (cid:13)(cid:13)(cid:13) q ≤ q log( N )
N (cid:18) 4 Gµ k + 2 GD (cid:19) + 2 GDq 2 log 2 ( N ) log(1 /δ N ) + 8 G (cid:115) κq ( T 1 + T 2 + T 3 )
Nµ ≤ q log( N )
N (cid:18) 4 Gµ k + 2 GD (cid:19) + 2 GDq 2 log 2 ( N ) log(1 /δ N ) + 8 G (cid:115) κq ( T 1 + T 3 )
Nµ + 8 G (cid:115) κqT
Nµ ζ 1 ≤ q log( N )
N (cid:18) 4 Gµ k + 2 GD (cid:19) + 2 GDq 2 log 2 ( N ) log(1 /δ N ) + 8 G (cid:115) κqT
Nµ + T 1 + T 3 + 16 κqG 2 Nµ (cid:46) q log( N )
N (cid:18) Gµ k + GD (cid:19) +
[Equation]
where in “ ζ 1 ” we have again used the fact √ ab ≤ a 2 t + bt 2 for a, b, t > 0 .
Now we are in the position to ﬁnally upper bound the desired sparse excess risk with respect to w ∗ ¯ k , which can be decomposed in the following way:
[Equation]
where in the ﬁrst inequality we have used the fact R S ( w ∗ S
[Equation]
¯ k ) ≤ R S ( w ∗ ¯ k ) .
We are going to bound the above two terms T (cid:48) 1 and T (cid:48) 2 respectively with high probability.
Concerning T (cid:48) 1 , since equationholds for all q ≥ 2 , in view of the second part of Lemma 4 (with q l = 2 and q u = ∞ ) we can show that the following holds with probability at least 1 − δ 2 for any δ ∈ (0 , e − 1 ) :
[Equation]
Regarding the term T (cid:48)
[Equation]
similar to the argument of equation
[Equation]
we can bound its L q -norm for any q ≥ 2 as follows:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
By invoking the second part of Lemma 4 with q l = 2 and q u = ∞ we can translate the above moment bound into the following exponential tail bound that holds with probability at least 1 − δfor any δ ∈ (0 , e − 1 ) :
[Equation]
By plugging equation 28 and equation 29 into equation 27 and applying union probability argument we obtain that the following holds with probability at least 1 − δ for any δ ∈ (0 , e − 1 ) :
[Equation]
This proves the desired sparse excess risk bound over the supporting set J .
As the ﬁnal step, since there are at most (cid:0) dk (cid:1) ≤ (cid:0) edk (cid:1) k different J with | J | = k , by union probability we can show that the following holds with probability 1 − δ for any δ ∈ (0 , e − 1 ) :
[Equation]
This completes the proof.
Now we are in the position to prove the main result in Theorem 4.
Proof of Theorem 4.
Let us consider
˜ J
= supp ( ˜ w S,k ) .
By deﬁnition we have ˜ w S,k
= w ∗ S | ˜ J .
Then invoking Lemma 7 yields that for any δ ∈ (0 , e − 1 )
[Equation]
the following holds with probability at least 1 − δ :
[Equation]
N
[Equation]
This proves the desired sparse excess risk bound
[Equation]