B OOSTING THE C ONFIDENCE OF N EAR -T IGHT G ENERALIZATION B
OUNDS
FOR U NIFORMLY
S
TABLE
R ANDOMIZED A LGORITHMS
Anonymous authors
Paper under double-blind review
A BSTRACT
High probability generalization bounds of uniformly stable learning algorithms have recently been actively studied with a series of near-tight results established by Feldman & Vondrak (2019); Bousquet et al.
However, for randomized algorithms with on-average uniform stability, such as stochastic gradient descent (SGD) with time decaying learning rates, it still remains less well understood if these deviation bounds still hold with high conﬁdence over the internal randomness of algorithm.
This paper addresses this open question and makes progress towards answering it inside a classic framework of conﬁdence-boosting.
To this end, we ﬁrst establish an in-expectation ﬁrst moment generalization error bound for randomized learning algorithm with on-average uniform stability, based on which we then show that a properly designed subbagging process leads to near-tight high probability generalization bounds over the randomness of data and algorithm.
We further substantialize these generic results to SGD to derive improved high probability generalization bounds for convex or non-convex optimization with natural time decaying learning rates, which have not been possible to prove with the existing uniform stability results.
Specially for deterministic uniformly stable algorithms, our conﬁdence-boosting results improve upon the best known generalization bounds in terms of a logarithmic factor on sample size, which moves a step forward towards resolving an open question raised by Bousquet et al.
I NTRODUCTION
In many statistical machine learning problems, the ultimate goal is to design a suitable algorithm
A : Z N (cid:55)→ W that maps a training data set S =
{ z i } i ∈ [ N ] ∈ Z N to a model
A ( S ) in a closed subset
W of an Euclidean space such that the following population risk function evaluated at the model is as small as possible:
[Equation]
, where (cid:96) : W × Z (cid:55)→ R
+ is a non-negative bounded loss function whose value (cid:96) ( w ; z ) measures the loss evaluated at z with parameter w , and D represents a distribution over Z .
It is generally the case that the underlying data distribution is unknown, and in this case the training data is usually assumed to be an i.i.d.
set, i.e., S i.i.d.
∼ D N .
Then, a natural alternative measurement that mimics the computationally intractable population risk is the empirical risk deﬁned by
[Equation]
The bound on the difference between population and empirical risks is of central interest in understanding the generalization performance of learning algorithm A .
Particularly, we hope to derive a suitable law of large numbers, i.e., a sample size vanishing rate b N such that the generalization bound | R S ( A ( S ))
− R ( A ( S ))
| (cid:46) b N holds with high probability over the randomness of S and potentially the randomness of A as well.
Provided that A ( S ) is an almost minimizer of the empirical risk function R S , say R S
( A ( S ))
(cid:46) min w ∈W R S
w ) + b N , the generalization bound immediately implies an excess risk bound R ( A ( S ))
− min w ∈W R ( w ) (cid:46) b N + 1 √ N due to the standard risk decomposition and Hoeffding’s inequality.
Therefore, generalization bounds also play a crucial role in understanding the stochastic optimization performance of a learning algorithm.
The generalization bounds can be naturally implied by uniform bounds on sup w ∈W | R ( w ) −
R S ( w ) | (Bartlett et al., 2006; Bottou & Bousquet, 2008).
While broadly applicable (e.g., to nonconvex problems) and leading to tight generalization in some speciﬁc regimes (e.g., margin-based learning (Kakade et al., 2009)), uniform convergence bounds in general cases might suffer from the polynomial dependence on dimensionality and thus are not suitable for high-dimensional models which are ubiquitous in modern machine learning.
Alternatively, a powerful proxy for analyzing the generalization bounds is the stability of learning algorithms to changes in the training dataset.
Since the seminal work of Bousquet & Elisseeff (2002), stability has been extensively demonstrated to beget dimension-independent generalization bounds for deterministic learning algorithms (Mukherjee et al., 2006; Shalev-Shwartz et al., 2010), as well as for randomized learning algorithms (such as bagging and SGD) (Elisseeff et al., 2005; Hardt et al., 2016).
So far, the best known results about generalization bounds are offered by approaches based on the notion of uniform stability (Feldman & Vondrak, 2018; 2019; Bousquet et al., 2020).
Inspired by these recent breakthrough results, we seek to derive sharper high-probability generalization bounds for randomized learning algorithms with on-average uniform stability.
A concrete working example of our study is the widely used stochastic gradient descent (SGD) algorithm that carries out the following recursion for all t ≥with learning rate η t > 0 :
[Equation]
where ξ t ∈ [ N ] is a uniform random index of data under with or without replacement sampling, and Π W is the Euclidean projection operator associated with W .
Despite its popularity in the study of stability theory (Hardt et al., 2016; Zhou et al., 2019; Lei & Ying, 2020), the high probability generalization bounds of SGD are still relatively under explored through the lens of uniform stability.
1. P RIOR R ESULTS
Let us now brieﬂy review some state-of-the-art generalization bounds for uniformly stable algorithms and their randomized variants.
We denote by S .
= S (cid:48)
if a pair of data sets S and S (cid:48) differ in a single data point.
A randomized learning algorithm
A is said to have on-average γ N -uniform stability if it satisﬁes the following uniform bound
[Equation]
This deﬁnition is equivalent to the concept of uniform stability deﬁned over the on-average loss E
A [ (cid:96) ( A ( S ); z )] .
Suppose that the loss function is Lipschitz and bounded in the interval (0 , 1] .
Then essentially it has been shown in Feldman & Vondrak (2019) that for any δ ∈ (0 , 1) , with probability at least 1 − δ over S , the on-average generalization error is upper bounded by
[Equation]
Recently, Bousquet et al.
(2020) derived a slightly improved uniform stability bound that implies
[Equation]
These generalization bounds are near-tight (up to a logarithmic factor log( N ) ) in the sense of a lower bound on sum of random functions provided in that paper.
While sharp in the dependence on sample size, one common limitation of the above uniform stability implied generalization bounds lies in that these high-probability results only hold in expectation with respect to the internal randomness of algorithm.
Further suppose that A has γ N -uniform stability with probability at least 1 − δ (cid:48) for some δ (cid:48) ∈ (
[Equation]
, 1) over the randomness of A , i.e.,
[Equation]
Suppose that the randomness of A is independent of the training set S .
Then with probability at least 1 − δ − δ (cid:48) over S and A , the bound of Bousquet et al.
(2020) naturally implies
[Equation]
Input : A learning and a training data set d
D
Output : A k ∗ ( S k ∗ ) .
Uniformly divide S into K disjoint subsets such that S
=
(cid:83)
k ∈
[ K ] S k and | S k | =
NK , ∀ k ∈
[ K ] .
for k = 1 , 2 , ...
, K do
Estimate A k
( S k ) as an output of the randomized algorithm A over subset S k .
end
Compute k ∗ = arg min k ∈
[ K ] (cid:12)(cid:12) R S \ S k
( A k ( S k ))
− R S k
( A k ( S k )) (cid:12)(cid:12) .
This is by far the best known generalization bound of randomized stable algorithms that hold with high probability jointly over data and algorithm.
The result, however, relies heavily on the highprobability uniform stability condition expressed in equation 4.
For the SGD (see equation 1) with ﬁxed learning rate η t ≡ η , it is possible to show that γ N (cid:46) η √ T + ηTN and δ (cid:48)
=
N exp( − N 2 ) in equation 4 (Bassily et al., 2020).
For SGD with time decaying learning rate that has been widely studied in theory Harvey et al. ; Rakhlin et al. and applied in practice for training popular deep nets such as ResNet and DenseNet Bengio et al. (2017), it is not clear if the condition in equation 4 is still valid for γ N and δ (cid:48) of interest.
On the other hand, it is possible to show (see the proofs of Corollary 1 and 2) that SGD with time decaying learning rate has desirable on-average uniform stability.
More specially for randomized learning methods such as bagging (Breiman, 1996) and SGD equation 1, the randomness of algorithm can be precisely characterized by a vector of i.i.d.
parameters ξ = { ξ 1 , ..., ξ T } which are independent on data S .
In such cases, suppose that A ( S ; ξ ) has uniform stability with respect to ξ at any given S , i.e., sup ξ .
= ξ (cid:48) | (cid:96) ( A ( S ; ξ )) − (cid:96) ( A ( S ; ξ (cid:48) ))
| ≤ ρ T .
Then the high probability bound established in Elisseeff et al.
(2005) shows that with probability at least 1 − δ ,
[Equation]
Provided that γ N (cid:46) 1 N and ρ T (cid:46) 1 T , the above bound shows that the generalization bound scales as O ( 1 √ N + 1 √ T ) with high probability.
However, the above bound will show no guarantee on convergence if γ N (cid:38) 1 √ N and/or ρ T (cid:38) 1 √ T .
For example, this is actually the case for SGD with time decaying learning rate η t = O ( 1 t ) on non-convex loss functions in which γ N (cid:46) √ TN and ρ T can scale as large as O (1) .
Open problem and motication. Keeping the merits and deﬁciencies of above recalled prior results in mind, it still remains an open issue if the existing deviation bounds can hold with high conﬁdence for randomized algorithms with on-average uniform stability (such as SGD with decaying learning rates). The goal of the present study is to derive sharper high-probability generalization bounds for randomized algorithms that hold jointly over the randomness of data and algorithm, based on the relatively weaker notion of on-average uniform stability rather than its high probability counterpart.
1. O VERVIEW OF O UR R ESULTS
The conﬁdence-boosting technique of Schapire (1990) is a classical meta approach that allows us to boost the dependence of a learning algorithm on the failure probability δ from 1 /δ to log(1 /δ ) , at a certain cost of computational complexity.
The fundamental contribution of our work is to reveal that the conﬁdence-boosting trick yields near-tight high probability generalization bounds for uniformly stable randomized learning algorithms.
The novelty lies in a reﬁned analysis of the in-expectation ﬁrst moment generalization error bound for a randomized learning algorithm with onaverage uniform stability, which leads to improved high-probability generalization bounds over the randomness of data and algorithm via conﬁdence-boosting.
More speciﬁcally, given a randomized learning algorithm A , we consider the following subbagging process over training set:
Boosting the Conﬁdence via Subbagging .
We independently run A over K disjoint and uniformly divided training subsets { S k } k ∈ [ K ] to obtain solutions { A k ( S k ) } k ∈ [ K ] .
Then we e- valuate the validation error of each candidate solution over its complementary training subset, and output A k ∗ ( S k ∗ ) that has the smallest gap between training error and validation error, i.e., k ∗ =
arg min k ∈ [ K ] (cid:12)(cid:12) R S \ S k ( A k ( S k )) − R S k ( A k ( S k )) (cid:12)(cid:12) .
Specially when A is deterministic, this reduces to a standard subbagging process, namely a variation of bagging using without-replacement sampling for subsets generation (see, e.g., Andonova et al., 2002).
In general, this is essentially a subbagging procedure with greedy model selection for randomized algorithms over multiple disjoint subsets.
Here we have assumed without loss of generality that N is a multiplier of K .
The considered procedure of conﬁdence-boosting for randomized learning is outlined in Algorithm 1.
Main Results.
In what follows, we highlight our main results on the generalization bounds of the output of Algorithm 1 along with the implications for SGD and deterministic algorithms:
• General results.
Suppose that the loss is Lipschitz and bounded in the range of (0 , 1] .
Our main result
in
Theorem
show
that for any δ ∈ setting K (cid:16) log( 1 δ ) yields the following generalization bound of the output of Algorithm 1 that holds with probability at least 1 − δ over S and { A k } k ∈
[ K ] :
[Equation]
where γ m ,N and γ m 2 ,N are respectively mean-uniform stability and mean-square-uniform stability bounds introduced in Deﬁnition 1.
In contrast to the bound in equation 6, our bound is not relying on the uniform stability with respect to the random bits of algorithm.
• Stronger generalization bounds for SGD via conﬁdence-boosting. We
then use our general result- s to study the beneﬁt of conﬁdence-boosting on the generalization bounds of SGD-w (SGD via with-replacement sampling as outlined in Algorithm 2).
The main results are a series of corollaries of Theorem 1 when substantialized to SGD with smooth (Corollary 1) or non-smooth (Corollary 2) convex loss, and smooth non-convex loss functions (Corollary 3).
For an instance, our result in Corollary 1 showcases that when invoked to SGD-w on smooth convex loss with learning rates η t =
O ( 1 √ t ) , the generalization bound of the output of Algorithm 1 with K (cid:16) log( 1 δ ) is upper bounded by
[Equation]
Compared with the O ( √ TN ) in-expectation bound of smooth convex SGD (Hardt et al., 2016), our above bound is competitive in order while it holds with high conﬁdence.
• Sharper bounds for uniformly stable algorithms via conﬁdence-boosting.
Specially for a deterministic learning algorithm
A that has γ N -
uniform
stability for each data set size N
≥ 1 , we further show through Corollary 4 that the following bound holds for the output of Algorithmwith K = log 1 δ ) :
[Equation]
In the case of γ N (cid:46) 1 √ N which holds in some popular learning paradigms such as regularized
ERM, the above bound implies (recall K (cid:16) log( 1 δ ) )
[Equation]
which is sharper than the best known result in equation 5 (under δ (cid:48)
= 0 ) in the sense of the removal of a log( N ) factor.
This is a side contribution of our work that might be of independent interest towards answering an open question raised by Bousquet et al.
In addition to the generalization bounds, we have also derived a high probability excess risk bound for uniformly stable randomized learning with conﬁdence-boosting.
More speciﬁcally, with a proper modiﬁcation of the output of Algorithm 1, we can show that with probability at least 1 − δ over S and { A k } k ∈ [ K ] :
[Equation]
where ∆ opt is the in-expectation empirical risk optimization error as given by equation 7.
G ENERALIZATION A NALYSIS WITH C ONFIDENCE -B OOSTING
In this section, we present a set of generic results on the generalization bounds of randomized learning algorithms with conﬁdence-boosting as described in Algorithm 1.
2. P RELIMINARIES AND A K EY L EMMA
We ﬁrst introduce the following concept of mean-uniform stability that serves as a powerful tool for analyzing the generalization bounds of randomized learning algorithms (Hardt et al., 2016; Elisseeff et al., 2005; Bassily et al., 2020).
Deﬁnition 1 (Mean and Mean-Square Uniform Stability of Randomized Algorithms) .
Let A :
Z N (cid:55)→ W be a randomized learning algorithm that maps a data set S ∈ Z N to a model
A ( S ) ∈ W .
Then A is said to have γ m ,N -mean-uniform stability if for every N ≥ 1 ,
[Equation]
Moreover, A is said to have γ m 2 ,N -mean-square-uniform stability if for every N ≥ 1 ,
[Equation]
Here the algorithm outputs A ( S ) and A ( S (cid:48) ) share the same random bits associated with the algorithm.
The above deﬁned notion of mean-uniform stability is also known as Uniform Argument
Stability (UAS) (Bassily et al., 2020), which was originally introduced by Liu et al.
(2017) for nonparametric hypotheses.
The notion of mean-square uniform stability is stronger than mean-uniform stability in the sense that the former naturally implies the latter such that γ m ,N ≤ √ γ m 2 ,N .
Moreover, we say a function f is G -Lipschitz continuous over W
if | f ( w ) − f ( w (cid:48) ) | ≤ G (cid:107) w − w (cid:48) (cid:107) for all w, w (cid:48) ∈ W , and it is L -smooth if (cid:107)∇ f ( w ) − ∇ f ( w (cid:48) )
(cid:107) ≤ L (cid:107) w − w (cid:48)
(cid:107) for all w, w (cid:48) ∈ W
Inspired by a second moment bound for generalization error of uniformly stable algorithms from Bousquet et al.
(2020, Section 5), we ﬁrst establish the following lemma which states that if a randomized learning algorithm has γ m ,N -mean-uniform stability, then its on-average ﬁrst moment generalization error bound will be as small as O ( γ m ,N + √ γ m 2 ,N + 1 √ N ) when the loss function is Lipschitz continuous.
This result is an adaptation of the second moment bound derived by Bousquet et al.
(2020) to on-average uniform stable randomized algorithms, and that bound is also the source of γ m 2 ,N entering into play.
For completeness, we provide a proof for this result in Appendix B.1.
Lemma 1.
Suppose that a randomized learning algorithm A : Z N (cid:55)→ W has γ m ,N -mean-uniform stability and γ m 2 ,N -mean-square-uniform stability.
Assume that the loss function (cid:96) is G -Lipschitz with respect to its ﬁrst argument and is bounded in the range of [0 , M ] .
Then we have
[Equation]
Remark 1.
In comparison to the on-average bound | E A,S [ R ( A ( S )) − R S ( A ( S ))] | ≤ Gγ m ,N (see, e.g., Hardt et al., 2016, Theorem 2.2), our on-average ﬁrst moment bound in Lemma 1 is substantially stronger and it turns out to play a crucial role in deriving high probability bounds for randomized algorithms.
When A is deterministic, the above bound reduces to the explicitly or implicitly known ﬁrst moment bound for uniformly stable algorithms (see, e.g., Feldman & Vondrak, 2018; Bousquet et al., 2020) which is tighter in logarithmic factors than the one implied by the p -th moment inequality of (Bousquet et al., 2020, Theorem 4).
In this case, our bound is stronger than the bound from Bousquet & Elisseeff (2002, Lemma 9) which essentially scales as 1 √ N + √ γ m ,N in our notation.
2. M AIN R ESULTS ON G ENERALIZATION B OUND
Let us recall the subbagging process as described in Algorithm 1: we independently run A over
K even and disjoint training subsets { S k } k ∈ [ K ] to obtain solutions { A k ( S k ) } k ∈ [ K ] , and then pick
A k ∗ ( S k ∗ ) that has the smallest difference between training error and validation error (over the complementary training subset S \ S k ∗ ).
The following theorem is our main result about the high probability generalization bound of the output A k ∗ ( S k ∗ ) evaluated over the entire training set S .
See
Appendix B.2 for its proof which builds largely on the ﬁrst moment bound in Lemma 1 and the fact that at least one of the solutions generated by subbagging generalizes well with high probability.
Theorem 1.
Suppose that a randomized learning algorithm A : Z N (cid:55)→ W has γ m ,N -mean-uniform stability and γ m 2 ,N -mean-square-uniform stability as well.
Assume that the loss function (cid:96) is GLipschitz with respect to its ﬁrst argument and is bounded in the range of [0 , M ] .
Then for any α, δ ∈ (0 , 1) and K ≥ 1 1 − α log( 4 δ ) , with probability at least 1 − δ over the randomness of S and { A k } k ∈ [ K ] , the output of Algorithm 1 satisﬁes
[Equation]
Remark 2.
To gain some intuition on the superiority of our bound in Theorem 1, let us consider
K (cid:16) log (cid:0) 1 δ (cid:1) as allowed in the conditions.
If γ m ,N (cid:46) 1 √ N and γ m 2 ,N (cid:46) 1 N , then our high-probability bound in Theorem 1 roughly scales as (cid:113) log(1 /δ )
N which is sharper than the on-average bounds in equation 2 and equation 3 with γ N (cid:46) 1 √ N .
More precise consequences of these general results on SGD and deterministic uniformly stable estimators such as ERMs and full gradient descent method will be discussed shortly in the sections to follow.
Remark 3.
In sharp contrast to the bound in equation 5 that requires high probability uniform stability and the bound in equation 6 that assumes uniform stability over the random bits of algorithm, our bound in Theorem 1 holds under a substantially milder notion of mean(-square)-uniform stability over data.
In terms of the tightness of bound, note that the conﬁdence term (cid:113)
[Equation]
is necessary even for an algorithm with ﬁxed output. The uniform stability terms γ m , NK and γ m 2 , NK are also near-tight as the the algorithm output can change arbitrarily with respect to these quantities.
2. O N E XCESS R ISK B OUNDS
To understand the optimization performance of a randomized learning algorithm A with conﬁdenceboosting, we further study here the excess risk bounds of Algorithm 1 which are of special interest for stochastic convex optimization problems.
In the following analysis, the global minimizer of the population risk and in-expectation empirical risk sub-optimality of the randomized algorithm are respectively denoted by
[Equation]
In order to derive the excess risk guarantees, we ﬁrst need to slightly modify the output of Algorithm 1 as A k ∗ ( S k ∗ ) where k ∗ = arg min k ∈
[ K ] R S \ S k
( A k ( S k )) .
The following theorem is our main result about the high probability excess risk bounds of such a modiﬁed output of conﬁdenceboosting.
See Appendix B.3 for its proof.
Theorem 2.
Suppose that a randomized learning algorithm A : Z N (cid:55)→ W has γ m ,N -mean-uniform stability.
Assume that the loss function (cid:96) is G -Lipschitz with respect to its ﬁrst argument and is bounded in the range of [0 , M ] .
Then for any α, δ ∈ (0 , 1) and K ≥ 1 1 − α log( 4 δ ) , with probability at least 1 − δ over the randomness of S and { A k } k ∈ [ K ] , the modiﬁed output of Algorithm 1 satisﬁes
[Equation]
N .
Remark 4.
Unlike the generalization error bounds, the excess risk bounds established in Theoremare not relying on the mean-square uniform stability of the algorithm, but with an additional term of in-expectation optimization error for minimizing the empirical risk.
For deterministic optimization algorithms such as ERMs with ∆ opt = 0 , similar excess risk bounds can be implied by the generic results of Shalev-Shwartz et al.
(2010, Theorem 26) developed for the conﬁdence-boosting approach.
Remark 5.
Consider K (cid:16) log (cid:0) 1 δ (cid:1) and γ m ,N (cid:46) 1 √ N .
Then the bound in Theorem 2 roughly scales as O ( (cid:113) log(1 /δ )
N + ∆ opt ) .
Finally, we comment on the difference between the generalization error and excess risk analysis inside the considered conﬁdence-boosting framework.
Since the excess risk is non-negative and
Input : Data set z
} i N ] i.i.d.
∼ D N , step-sizes { η t } t ≥ #iterations T initialization w 0 .
Output : ¯ w T = 1 T (cid:80) t ∈
[ T ] w t .
for t = 1 , 2 , ..., T do
Uniformly randomly sample an index ξ t ∈ [ N ] with replacement;
Compute w t
=
Π W ( w t − 1 − η t ∇ w (cid:96) ( w t − 1 ; z ξ t )) .
end its in-expectation bound is standardly known for uniformly stable learning algorithms, the highconﬁdence bound in Theorem 2 can be easily derived via invoking Markov inequality to the independent runs of algorithm over K disjoint subsets.
The generalization error analysis, however, is way more challenging in the sense that establishing tight in-expectation ﬁrst moment generalization bound (see Lemma 1) for uniformly stable randomized algorithms is by itself highly non-trivial.
I MPLICATIONS FOR S TOCHASTIC G RADIENT D ESCENT
In this section we demonstrate the applications of the generic bound in Theorem 1 to the widely used SGD algorithm.
We focus on a variant of SGD under with-replacement sampling as outlined in Algorithm 2, which we call A SGD-w .
In what follows, we denote by { A SGD-w ,k } k ∈ K the outputs of A SGD-w over subsets { S k } k ∈ K when applied with Algorithm 1.
Our results readily extend to the without-replacement variant of SGD and the corresponding results can be found in Appendix D.
C ONVEX O PTIMIZATION WITH S MOOTH L OSS
For smooth and convex losses such as logistic loss, we can derive the following result as a direct consequence of Theorem 1 with α = 1 / 2 to A SGD-w .
See Appendix C.1 for its proof.
Corollary 1.
Suppose that the loss function is (cid:96) ( · ; · ) is convex, G -Lipschitz and L -smooth with respect to its ﬁrst argument, and is bounded in the range of [0 , M ] .
Consider Algorithm 1 speciﬁed to A SGD-w with learning rate η t ≤ 2 /L for all t ≥ 1 .
Then for any δ ∈ (0 , 1) and K (cid:16) log( 4 δ ) , with probability at least 1 − δ over the randomness of S and { A SGD-w ,k } k ∈ [ K ] , the generalization error is upper bounded as | R ( A SGD-w ,k ∗ ( S k ∗ ))
− R S ( A SGD-w ,k ∗ ( S k ∗ ))
| (cid:46)
[Equation]
Remark 6.
For the conventional step-size choice of η t = 2 L √ t , the high probability generalization bound in Corollary 1 is dominated by O ( (cid:113) log( T ) N + √ T + √ N log(1 /δ )
N ) , which matches the corresponding O ( √ TN ) in-expectation bound for SGD with smooth convex losses (Hardt et al., 2016).
3. C ONVEX O PTIMIZATION WITH N ON - SMOOTH L OSS
Now we turn to study the case where the loss is convex but not necessarily smooth, such as the hinge loss and absolute loss.
The following result as a direct consequence of Theorem 1 for the speciﬁcation of Algorithm 1 to A SGD-w with non-smooth convex loss and time varying learning rate { η t } t ≥ 1 .
Its proof is provided in Appendix C.2.
Corollary 2.
Suppose that the loss function is (cid:96) ( · ; · ) is convex and G -Lipschitz with respect to its ﬁrst argument, and is bounded in the range of [0 , M ] .
Consider Algorithm 1 speciﬁed to A SGD-w .
Then for any δ ∈ (0 , 1) and K (cid:16) log( 4 δ ) , with probability at least 1 − δ over the randomness of S and { A SGD-w ,k } k ∈ [ K ] , the generalization error satisﬁes | R ( A SGD-w ,k ∗ ( S k ∗ )) −
[Equation]
( A SGD
[Equation]
k ∗ ( S k ∗ ))
| (cid:46)
[Equation]
Remark 7.
For constant rates η t ≡ η , Corollary 2 admits a high probability generalization bound of scale O ( η √ T + η TN + (cid:113) log(1 /δ )
N ) which matches the near-optimal rate by Bassily et al.
Theorem 3.3).
More importantly, our deviation bound in Corollary 2 still holds for time varying learning rates.
3. N ON - CONVEX O PTIMIZATION WITH S MOOTH L OSS
We further study the performance of Algorithm 1 for SGD on smooth but not necessarily convex loss functions, such as normalized sigmoid loss (Mason et al., 1999).
The following result is a direct application of Theorem 1 to A SGD-w with smooth non-convex loss.
See Appendix C.3 for its proof.
Corollary 3.
Suppose that the loss function is (cid:96) ( · ; · ) is G -Lipschitz and L -smooth with respect to its ﬁrst argument, and is bounded in the range of [0 , M ] .
Consider Algorithm 1 speciﬁed to A SGD-w with η t ≤ 1 L .
Let u t :=
η 2 t + 2 η t (cid:80) t − 1 τ =1 exp( L (cid:80) t − 1 i =
τ +
η i )
η τ for all t ≥ 1 .
Then for any δ ∈ (0 , 1) and K (cid:16) log( 4 δ ) , with probability at least 1 − δ over the randomness of S and { A SGD-w ,k } k ∈ [ K ] , the generalization error is upper bounded as | R ( A SGD-w ,k ∗ ( S k ∗ ))
− R S ( A SGD-w ,k ∗ ( S k ∗ ))
| (cid:46)
G(cid:118)(cid:117) (cid:117)(cid:116)
N
[Equation]
(cid:88) t = exp (cid:32)
L
[Equation]
(cid:88)
[Equation]
N
[Equation]
(cid:88) t = exp (cid:32)
L
T (cid:88) τ
= t + η τ (cid:33) η t + M (cid:114) log(1 /δ )
N .
Remark 8.
For the constant learning rates η t ≡ 1 LT , Corollary 3 admits high probability generalization bound of scale O ( (cid:113) log(1 /δ ) N ) .
For time decaying learning rates η t = 1 Lνt with arbitrary ν ≥ 1 , it can be veriﬁed that the corresponding bound is of scale O ( (cid:113) T 1 /ν log( T ) νN + (cid:113) log(1 /δ )
N ) .
I MPLICATIONS FOR D ETERMINISTIC U NIFORMLY S
This section is devoted to showing that conﬁdence-boosting is also beneﬁcial for deriving stronger generalization bounds for uniformly stable deterministic learning algorithms.
First we note that when there is no internal randomness in A , the deﬁnition of mean(-square)-uniform stability reduces to the conventional concept of γ N -uniform stability for deterministic algorithms given by sup
[Equation]
Let us now consider a speciﬁcation of Algorithm 1 to a uniformly stable deterministic algorithm A .
Since there is no randomness contained in A , we have that A k ( S k )
=
A ( S k ) for all k ∈
[ K ] in such a deterministic case.
Then the following result is a direct consequence of Theorem 1 when applied to the considered deterministic learning regime.
Corollary 4.
Suppose that a deterministic learning algorithm A : Z N (cid:55)→ W has γ N -uniform stability.
Assume that the loss function (cid:96) is G -Lipschitz with respect to its ﬁrst argument and is bounded in [0 , M ] .
Then for any α, δ ∈ (0 , 1) and K ≥ log(4 /δ )
1 − α , with probability at least 1 − δ over the randomness of S , the output of Algorithm 1 satisﬁes
[Equation]
To demonstrate the superiority of our bound over prior ones, let us consider α = 0 .
5 and K (cid:16) log( 1 δ ) in the above corollary.
In the regime γ N (cid:46) 1 √ N which is of interest in many popular deterministic learning paradigms/algorithms such as regularized ERM (Shalev-Shwartz et al., 2009) and full gradient descent (Feldman & Vondrak, 2019), Corollary 4 implies a generalization bound for A ( S k ∗ ) over the data set S that scales as | R ( A ( S k ∗ ))
[Equation]
A
[Equation]
|
[Equation]
log
[Equation] [Equation] N . In comparison, the best known bound in equation 5 essentially from Bousquet et al.
(2020) gives (keep in mind that δ (cid:48)
=in the deterministic case)
| R ( A ( S ))
− R S ( A ( S ))
| (cid:46) log( N ) √ N log( 1 δ ) .
As we can see that inside the carefully designed framework of conﬁdence-bossting via subbagging, our generalization bound gets rid of the logarithmic factor log( N ) from the above best known result, though the generalization is with respect to the estimation over a speciﬁc part of the sample.
We expect this result will fuel future research towards fully resolving the corresponding open question raised in Bousquet et al.
O THER R ELATED W ORK
The idea of using stability of a learning algorithm, namely the sensitivity of estimated model to the changes in training data, for generalization performance analysis dates back to the seventies (Vapnik & Chervonenkis, 1974; Rogers & Wagner, 1978; Devroye & Wagner, 1979).
For deterministic learning algorithms, algorithmic stability has been extensively studied with a bunch of applications to establishing strong generalization and excess risk bounds for stable learning models like k -NN and regularized ERMs (Bousquet & Elisseeff, 2002; Zhang, 2003; Klochkov & Zhivotovskiy, 2021).
The stability theory for randomized learning algorithms was formally introduced and investigated by Elisseeff et al.
In a recent breakthrough work (Hardt et al., 2016), it was shown in that the solution obtained via stochastic gradient descent is expected to be stable and generalize well for smooth convex and non-convex loss functions.
For non-smooth convex losses, the stability induced generalization bounds of SGD have been established in expectation (Lei & Ying, 2020) or deviation (Bassily et al., 2020).
In Kuzborskij & Lampert (2018), a set of data-dependent generalization bounds for SGD were derived based on the stability of algorithm.
More broadly, generalization bounds for stable learning algorithms that converge to global minima were established in Charles & Papailiopoulos (2018);
Lei & Ying (2021).
For non-convex sparse learning, algorithmic stability theory has been applied to derive the generalization bounds of the popularly used iterative hard thresholding (IHT) algorithm
(Yuan & Li, 2021).
The uniform stability bounds on SGD have also been extensively used for designing differential privacy stochastic optimization algorithms (Bassily et al., 2019; Feldman et al., 2020).
Bagging (or bootstrap aggregating) is one of the earliest and most popular ensemble methods that has been widely applied to reduce the variance for unstable learning algorithms such as decision tree and neural networks (Breiman, 1996; Opitz & Maclin, 1999), and sometimes stable algorithms such as SVMs (Valentini & Dietterich, 2003).
As an important variant of bagging, subbagging has been proposed to reduce the computational cost of bagging via training base models under withoutreplacement sampling (B¨uhlmann, 2012).
The stability and generalization bounds of bagging have been analyzed for both uniform (Elisseeff et al., 2005) and non-uniform (Foster et al., 2019) averaging schemes.
Unlike these prior results for bagging with averaging aggregation, our bounds are obtained based on a conﬁdence-boosting greedy aggregation scheme which turns out to yield sharper dependence on the uniform stability parameter.
The conﬁdence-boosting technique has long been applied for obtaining sharp high-probability excess risk bounds from the corresponding strong in-expectation bounds (Shalev-Shwartz et al., 2010;
Mehta, 2017).
For generic statistical learning problems, conﬁdence-boosting has been used to convert any low-conﬁdence learning algorithm with linear dependence on 1 /δ to a high-conﬁdence algorithm with logarithmic factor log(1 /δ ) .
For learning with exp-concave losses, a relevant ERM estimator with in-expectation fast rate of convergence was converted to a high-conﬁdence learning algorithm with an almost identical fast rate of convergence up to a logarithmic factor on 1 /δ (Mehta, 2017).
While sharing a similar spirit of boosting the conﬁdence, our generalization analysis is substantially more challenging than those prior excess risk analysis in terms of tightly deriving inexpectation ﬁrst moment generalization bound for uniformly stable randomized algorithms.
C ONCLUSIONS
In this paper we presented a generic conﬁdence-boosting method for deriving near-optimal high probability generalization bounds for uniformly stable randomized learning algorithms.
At a nutshell, our main results in Theorem 1 and Theorem 2 reveal that a carefully designed subbagging process in Algorithm 1 can yield high-conﬁdence generalization and risk bounds under the notion of mean(-square)-uniform stability.
Our theory has been substantialized to SGD on both convex and non-convex losses to obtain stronger generalization bounds especially in the case of time decaying learning rates.
When reduced to deterministic algorithms, the proposed method removes a logarithmic factor on sample size from the best known bounds.
While sharper in the dependence on sample size, our conﬁdence-boosting results are only applicable to one of the independent runs of algorithm
A over K disjoint training subsets of equal size with K (cid:16) log (cid:0) 1 δ (cid:1) .
It is so far not clear if these near-optimal bounds can be further extended to the full-batch setting where the generalization is with respect to the evaluation of algorithm over the entire sample.
We leave the full understanding of such an open issue raised by Bousquet et al.
(2020) for future investigation.
A A UXILIARY L EMMAS
We need the following lemma from Hardt et al.
(2016) which shows that SGD iteration is nonexpansive on convex and smooth loss.
Lemma 2 (Hardt et al.
Assume that f is convex and L -smooth.
Then for any w, w (cid:48) ∈ W and α ≤ 2 /L , we have the following bound holds
[Equation]
The following lemma, which can be proved by induction (see, e.g., Schmidt et al., 2011), will be used to prove the main results in Section 3.
Lemma 3.
Assume that the nonnegative sequence { u τ } τ ≥ 1 satisﬁes the following recursion for all t ≥ 1 :
[Equation]
with { S τ } τ ≥ 1 an increasing sequence, S 0 ≥ u 20 and α τ ≥ 0 for all τ .
Then, the following inequality holds for all t ≥ 1 :
[Equation]
B P ROOFS FOR THE R ESULTS IN S ECTION
In this section, we present the technical proofs for the main results stated in Section 2.
B. P ROOF OF L EMMA
We need the following lemma essentially from Bousquet et al.
(2020) that provides a ﬁrst moment bound for the sum of random functions.
Lemma 4.
Let S
=
{ Z 1 , Z 2 , ..., Z N } be a set of i.i.d.
random variables valued in Z .
Let g 1 , ..., g N be a set of measurable functions
g i : Z N (cid:55)→ R that satisfy E S [ g 2 i ( S )]
≤ M 2 and E Z i [ g i ( S )]
=for all i ∈ [ N ] .
Then we have
[Equation]
where S ( j ) =
{ Z 1 , ..., Z j − 1 , Z (cid:48)
j , Z j +1 , ..., Z N } and S (cid:48)
=
{ Z (cid:48) 1 , Z (cid:48) 2 , ..., Z (cid:48) N } is another i.i.d.
sample from the same distribution as that of S .
Proof.
We reproduce the proof in view of the argument in Bousquet et al.
(2020, Section 5) showing that { g i } are weakly correlated.
For any i (cid:54) =
j , since E Z i [ g i ( S )]
= 0 and E Z j
[ g j ( S )]
= 0 , we can verify that
[Equation]
where we have used the independence of the elements in S ∪ { Z (cid:48) j } .
Similarly, we can show that
[Equation]
Then it follows that for any i (cid:54) =
j ,
[Equation]
[Table]
[Equation]
This proves the desired bound.
Now we are ready to prove the result in Lemma 1.
Proof of Lemma 1.
Let us consider h i ( S ) :
= R ( A ( S )) − (cid:96) ( A ( S ); Z i ) and g i ( S )
=
h i ( S ) − E
Z i [ h i ( S )] for i ∈ [ N ] .
Then by assumption we have
[Equation]
For each i ∈ [ N ] , let S ( i ) denote a random data set that is identical to S except that one of the
Z i is replaced by another random sample Z (cid:48)
i .
For any i (cid:54) =
j , since the loss is non-negative and
G -Lipschitz, it can be veriﬁed that
[Equation]
which readily implies
[Equation]
Then invoking Lemma 4 to { g i } yields
[Equation]
Further, it can be veriﬁed that
[Equation]
i
[Equation] [Equation]
)] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:35) =
E
[Equation]
(cid:34)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12)
[Equation]
i =
E Z i [ E
[Equation]
(cid:48)
i [ (cid:96) ( A ( S ); Z (cid:48) i )] − (cid:96) (
[Equation]
A.
[Equation]
By we obtain
[Equation]
where in the last inequality we have used the stability conditions on A .
The proof is completed
[Equation]
B. P ROOF OF T HEOREM
We ﬁrst establish the following intermediate result that captures the effects of subbagging on randomized algorithms: it basically tells that with K (cid:16) log( 1 δ ) , at least one of the solutions generated by subbagging generalizes well with high probability.
Lemma 5.
Suppose that a randomized learning algorithm A : Z N (cid:55)→ W has γ m ,N -mean-uniform stability and γ m 2 ,N -mean-square-uniform stability as well.
Assume that the loss function (cid:96) is GLipschitz with respect to its ﬁrst argument and is bounded in the range of [0 , M ] .
Then for any α, δ ∈ (0 , 1) and K ≥ log(2 /δ )
1 − α , with probability at least 1 − δ over the randomness of { ( A k , S k ) } k ∈
[ K ] , the sequence { A k ( S k ) } k ∈ [ K ] generated by Algorithm 1 satisﬁes
[Equation]
Proof.
From Lemma 1 we have that over randomized algorithm A and data ˜ S with | ˜ S
| =
NK ,
[Equation]
Since { A k , S k } k ∈ [ K ] are independent to each other, by Markov inequality we know that
P { A k ,S k }
[Equation]
which implies the desired bound.
Next we proceed to prove the main result in Theorem 1.
Proof of Theorem 1.
Let us consider the following three events:
[Equation]
max
[Equation]
We can show that E ⊇ E 1 ∩ E 2 .
Indeed, suppose that E 1 and E 2 simultaneously occur.
Consequently the following inequality is valid:
[Equation]
where in “ ζ 1 ” we have used the deﬁnition of k ∗
[Equation]
and “ ζ 2 ” follows from E 1 , E 2 .
With leading terms preserved in the above we can see that E occurs.
Next we can show that P S, { A k } ( E 1 ) ≤ δ 2 .
Toward this end, let us consider the following events for all k ∈
[ K ] :
[Equation]
Clearly, it is true that E 1 = (cid:84)
Kk =1 E k 1 .
It is sufﬁcient to prove that P S,A k (cid:16) E k 1 (cid:17) ≤
δ 2 K holds for each k ∈
[ K ] .
Indeed, consider the random indication function β ( S, A k ) :
= 1 E k 1 associated with the event E k 1 .
Then we have the following holds for each k ∈
[ K ] :
[Equation]
where in “ ζ 1 ” we have used the independence between { A k , S k } and S \ S k , and “ ζ 2 ” is due to
Hoeffding’s inequality conditioned on { A k , S k } , keeping in mind that A k ( S k ) is independent on the data set S \ S k of size (1 − 1 /K ) N .
It follows by union probability that
[Equation]
Further, from the part(b) of Lemma 5 we have P S, { A k } ( ¯ E 2 ) ≤ δ 2 .
Combining this and the preceding bound yields
[Equation]
This implies the desired result in part(b) as K/ ( K − 1) ≤ 2 for K ≥ 2 .
B. P ROOF OF T HEOREM
We ﬁrst present the following simple lemma about the in-expectation risk bounds of a randomized algorithm which will be used in our analysis.
Lemma 6.
Suppose that a randomized learning algorithm A : Z N (cid:55)→ W has γ m ,N -mean-uniform stability.
Assume that the loss function (cid:96) is G -Lipschitz with respect to its ﬁrst argument.
Then we have
[Equation]
Proof.
By risk decomposition we can show that
[Equation]
where in the last inequality we have used the on-average generalization bound by Hardt et al.
Theorem 2.2).
Lemma 7.
Suppose that a randomized learning algorithm A : Z N (cid:55)→ W has γ m ,N -mean-uniform stability.
Assume that the loss function (cid:96) is G -Lipschitz with respect to its ﬁrst argument.
Then for any α, δ ∈ (0 , 1) and K ≥ log(2 /δ )
1 − α , with probability at least 1 − δ over the randomness of { ( A k , S k ) } k ∈
[ K ] , the sequence { A k ( S k ) } k ∈ [ K ] generated by Algorithm 1 with the modiﬁed output satisﬁes
[Equation]
Proof.
Recall the modiﬁed output A k ∗ ( S k ∗ ) where k ∗ =
arg min k ∈ [ K ] R S \ S k
( A k ( S k )) .
From
Lemma 6 we have that over randomized algorithm A and data ˜ S with | ˜ S
| =
NK ,
[Equation]
Since { A k , S k } k ∈ [ K ] are independent to each other, by Markov inequality we know that
[Equation]
which implies the desired bound in part(b).
Next we proceed to prove the main result in Theorem 2.
Proof of Theorem 2.
Let us consider the following three events:
[Equation]
Similarly, we show that E ⊇ E 1 ∩ E 2 .
Indeed, suppose that E 1 and E 2 simultaneously occur.
Consequently the following inequality is valid:
[Equation]
where in “ ζ 1 ” we have used the deﬁnition of k ∗
[Equation]
and “ ζ 2 ” follows from E 1 , E 2 .
With leading terms preserved in the above we can see that E occurs.
Based on the same proof argument as that of the part(b) of Theorem 1 we can show that P
S, { A k } ( E 1 ) ≤ δ 2 .
Further, from the part(b) of Lemma 7 we have P S, { A k } ( ¯ E 2 ) ≤ δ 2 .
Combining this and the preceding bound yields
[Equation]
This implies the desired result in part(b) as K/ ( K − 1) ≤ 2 for K ≥ 2 .
C P ROOFS FOR THE R ESULTS IN S ECTION
In this section, we present the technical proofs for the main results stated in Section 3.
C. P ROOF OF C OROLLARY
We begin with presenting and proving the following lemma that gives the mean(-square)-uniform stability bounds for A SGD-w on convex and smooth loss functions such as logistic loss.
Lemma 8.
Suppose that the loss function is (cid:96) ( · ; · ) is convex, G -Lipschitz and L -smooth with respect to its ﬁrst argument.
Assume that η t ≤ 2 /L for all t ≥ 1 .
Then A SGD-w has mean-uniform stability such that sup
[Equation]
and has mean-square-uniform stability such that sup
[Equation]
Proof.
The ﬁrst mean-uniform stability bound can be straightforwardly derived based on the argument of Hardt et al.
(2016, Theorem 3.7).
We focus on proving the second mean-square-uniform stability bound.
For any pair of S, S (cid:48) , let us deﬁne the sequences { w t } t ∈ [ T ] and { w (cid:48) t } t ∈
[ T ] that are respectively generated over S and S (cid:48) via A SGD-w via sample path ξ = { ξ t } t ∈ [ T ] .
Note by assumption that w 0 = w (cid:48) 0 .
We distinguish the following two complementary cases.
Case I: z ξ t = z (cid:48) ξ t .
In this case, by invoking Lemma 2 we immediately get
[Equation]
A.
[Equation]
Case II: z ξ t (cid:54) =
z (cid:48) ξ t .
In this case, we have
[Equation]
where in the last but inequality we have used (cid:96) ( · ; · ) is G -Lipschitz with respect to its ﬁrst argument.
Let β t = β t ( S, S (cid:48) , ξ ) :
= 1 { z ξt (cid:54) =
z (cid:48) ξt } be the random indication function associated with event z ξ t (cid:54) =
z (cid:48) ξ t .
Based on the recursion forms equation A.3 and equation A.4 and the condition w 0 = w (cid:48) 0 we can show that for all t ≥ 1 ,
[Equation]
Then applying Lemma 3 with simple algebraic manipulation yields
[Equation]
Since by assumption S and S (cid:48) differ only in a single element, under the scheme of uniform sampling without replacement, we can see that β t ( S, S (cid:48) , ξ ) ∼ Bernoulli (1 /N ) and { β t ( S, S (cid:48) , ξ ) } is an i.i.d.
sequence of Bernoulli random variables.
It follows that
[Equation]
where we have used E ξ t [ β t ]
=
E ξ t [ β 2 t ] = 1 N .
The convexity of squared Euclidean norm leads to
[Equation]
Note that the above holds for any S .
= S (cid:48) , i.e.,
[Equation]
The proof is concluded.
With Lemma 8 in place
[Equation]
we are ready to prove Corollary 1.
Proof of Corollary 1.
From Lemma 8 we know that A SGD-w has mean-uniform stability with parameter
[Equation]
] η t ,
[Equation]
The desired results then follow immediately via invoking Theorem 1 with α = 1 / 2 .
C. P ROOF OF C OROLLARY
We ﬁrst establish the following lemma on the mean(-square)-uniform stability of A SGD-w in the case of non-smooth convex loss.
Lemma 9.
Suppose that the loss function is (cid:96) ( · ; · ) is convex and G -Lipschitz with respect to its ﬁrst argument.
Then A SGD-w has mean-uniform stability such that sup
[Equation]
and has mean-square-uniform stability such that sup
[Equation]
Proof.
Let us deﬁne the sequences { w t } t ∈ [ T ] and { w (cid:48) t } t ∈
[ T ] that are respectively generated over S and S (cid:48) via A SGD-w via sample path ξ = { ξ t } t ∈ [ T ] .
Suppose that S .
= S (cid:48) and consider a hitting time variable t 0 = inf { t : z ξ t (cid:54) =
z (cid:48) ξ t } .
Let β t =
β t ( S, S (cid:48) , ξ ) :
= 1 { z ξt (cid:54) =
z (cid:48) ξt } be the random indication function associated with event z ξ t (cid:54) =
z (cid:48) ξ t .
Conditioned on t 0 , it has been shown by Bassily et al.
(2020, Lemma 3.1) that
[Equation]
Then we can show the following for all t ≤ T
[Equation]
where we have used the fact that { β t } is an i.i.d.
sequence of Bernoulli (1 /N ) random variables.
The convexity of Euclidean norm leads to
[Equation]
which is the ﬁrst desired bound.
Similarly, based on the square of the bound equation A.5 we can show that
[Equation]
where we have used E ξ t [ β t ]
=
E ξ t [ β 2 t ] = 1 N .
It follows directly from the convexity of loss that
[Equation]
The proof is concluded.
Equipped with Lemma 9, we are now in the position to prove Corollary 2.
Proof of Corollary 2.
From Lemma 9 we know that A SGD-w with non-smooth convex loss has mean(-square)-uniform stability with parameters
[Equation]
The desired results then follow immediately via invoking Theorem 1 with α = 1 / 2 .
C. P ROOF OF C OROLLARY
We ﬁrst establish the following lemma on the mean(-square)-uniform stability of A SGD-w in the considered non-convex regime.
Lemma 10.
Suppose that the loss function is (cid:96) ( · ; · ) is G -Lipschitz and L -smooth with respect to its ﬁrst argument.
Consider η t ≤ 1 /L .
Let
[Equation]
for all t ≥ 1 .
Then A SGD-w has mean-uniform stability such that sup
[Equation]
and has mean-square-uniform stability such that sup
[Equation]
Proof.
Let us deﬁne the sequences { w t } t ∈ [ T ] and { w (cid:48) t } t ∈
[ T ] that are respectively generated over
S and S (cid:48) via A SGD-w via sample path ξ = { ξ t } t ∈ [ T ] .
Suppose that S .
= S (cid:48) .
Let us consider ∆ t :=
E ξ [ t ] [ (cid:107) w t − w (cid:48) t
(cid:107) ] .
Then based on the arguments of Hardt et al.
(2016, Theorem 3.8)
we know that with probability 1 − 1 N over ξ t , (cid:107) w t − w (cid:48) t (cid:107) ≤ (1+ η t L ) (cid:107) w
[Equation]
−
[Equation]
− w
[Equation]
− 1 (cid:107)
[Equation]
and (cid:107) w t − w (cid:48) t (cid:107) ≤
[Equation]
w t − 1 − w (cid:48)
t − 1 (cid:107) + 2 Gη t with probability 1 N .
Therefore we have ∆
[Equation]
where we have used 1 + x ≤ exp( x ) .
Then we can unwind the above recursion form to obtain that for all t ≥ 1 ,
[Equation]
where we have used ∆ 0 = 0 .
The convexity of Euclidean norm leads to
[Equation]
which immediately implies the ﬁrst desired bound as it holds for all S .
= S (cid:48) .
To show the mean-square-uniform stability bound, let us consider ˜∆ t :=
E ξ [ t ] (cid:2) (cid:107) w t − w (cid:48) t
(cid:107) 2 (cid:3) .
Then we can verify that with probability 1 − 1 N over ξ t , (cid:107) w
[Equation]
− w
[Equation]
t (cid:107)
[Equation]
− w
[Equation]
1 (cid:107) 2 , and with probability 1 N ,
[Equation]
Therefore we have
[Equation]
where in the second inequality we have used the bound equation A.6 on ∆ t .
Recall that ˜∆ 0 = 0 .
Then we can unwind the above recursion form to obtain ˜∆
[Equation]
N
[Equation]
where we have used η t ≤ 1 /L .
It follows immediately from the convexity that
[Equation]
which is the second desired bound.
The proof is completed.
With Lemma 10 in place
[Equation]
we proceed to prove the main result in Corollary 3.
Proof of Corollary 3.
From Lemma 10 we know that A SGD-w with smooth non-convex loss has mean(-square)-uniform stability with parameters
[Equation]
The desired results then follow immediately via invoking Theorem 1 with α = 1 / 2 .
Input : Data set S =
{ z
} i ∈ [ N ] i.i.d.
∼ D N , step-sizes { η t } t ≥ 1 , #iterations T , initialization w 0 .
Output : ¯ w T = 1 T (cid:80) t ∈
[ T ] w t .
for t = 1 , 2 , ..., T do
Uniformly randomly sample an index ξ t ∈ [ N ] with or without replacement;
Compute w t
=
Π W ( w t − 1 − η t ∇ w (cid:96) ( w t − 1 ; z ξ t )) .
end
D A UGMENTED R ESULTS FOR SGD UNDER W ITHOUT -R EPLACEMENT S AMPLING
In this section, we further consider applying our main results in Theorem 1 to the variant of SGD under without-replacement sampling ( A SGD-wo ), as is outlined in Algorithm 3.
For the sake of simplicity and readability, we only consider single-epoch processing with T ≤ N , and we focus on the case where the loss is convex but non-smooth.
The extensions of our analysis to multi-epoch processing, i.e., T ≤ rN for some integer r ≥ 1 , and to convex or non-convex smooth loss functions are more or less straightforward and thus are omit.
We start by establishing the following lemma on the mean(-square)-uniform stability of A SGD-wo which can be easily proved based on the result from Bassily et al.
(2020, Lemma 3.1).
Lemma 11.
Suppose that the loss function is (cid:96) ( · ; · ) is convex and G -Lipschitz with respect to its ﬁrst argument.
Consider T ≤ N .
Then A SGD-wo has mean-uniform stability such that sup
[Equation]
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
and has mean-square-uniform stability such that sup
[Equation]
Proof.
Let ¯ w T ( S, ξ ) and ¯ w T ( S (cid:48) , ξ ) respectively be the output generated over S
= { z i } i ∈ [ N ] and
S (cid:48)
= { z (cid:48) i } i ∈
[ N ] by A SGD-wo via sample path ξ
= { ξ t } t ∈ [ T ] .
Recall that T ≤ N .
Let us deﬁne a hitting time variable t 0 = inf { t : z ξ t (cid:54) =
z (cid:48) ξ t } .
Since S .
= S (cid:48) , the uniform randomness of ξ t implies that
[Equation]
Given t ∈ [ T ] , it follows from Bassily et al.
(2020, Lemma 3.1) that
[Equation]
Then we have
[Equation]
The convexity of squared Euclidean norm leads to
[Equation]
(cid:107) ¯ w
[Equation]
≤ (cid:80) Tt =1 E ξ [ t ] (cid:2) (cid:107)
[Equation]
t
[Equation]
(cid:48) t (cid:107)
[Equation]
(cid:3)
T ≤ 4 G
N
T (cid:88) t 0 =
T (cid:88) t = tη 2 t .
Similarly we can show
E
[Equation]
G N
T (cid:88) t
= (cid:118)(cid:117)(cid:117)(cid:116) T (cid:88) t = tη 2 t .
The proof is completed.
The following result is a direct consequence of Theorem 1 when invoking Algorithm 1 to A SGD-wo with non-smooth convex loss.
Corollary 5.
Suppose that the loss function is (cid:96) ( · ; · ) is convex and G -Lipschitz with respect to its ﬁrst argument, and is bounded in the range of [0 , M ] .
Consider Algorithm 1 speciﬁed to A SGD-wo with T ≤ N .
Then for any δ ∈ (0 , 1) and K ≥ 2 log( 4 δ ) , with probability at least 1 − δ over the randomness of S and { A SGD-wo ,k } k ∈ [ K ] , the output of A SGD-wo satisﬁes
[Equation]
Proof.
From Lemma 11 we know that A SGD-wo has mean(-square)-uniform stability with parameters
[Equation]
The results then follow immediately via invoking Theorem 1 with α = 1 / 2 .
Remark 9.
Specially for constant learning rates η t ≡ η , Corollary 5 admits a high probability generalization bound of scale O
[Equation]
(cid:19) .
For general time varying learning rates, our bound in Corollary 5 still holds with high probability.
For example, when η t ∝ 1 t , the generalization bound scales as O (cid:18)(cid:113) log
[Equation]
T
[Equation]
N
[Equation]