Stability and Generalization for Markov Chain Stochastic Gradient Methods
Anonymous Author(s)
AffiliationAddress email
Abstract
Recently there is a large amount of work devoted to the study of Markov chainstochastic gradient methods (MC-SGMs) which mainly focus on their convergenceanalysis for solving minimization problems.
In this paper, we provide a comprehensive generalization analysis of MC-SGMs for both minimization and minimaxproblems through the lens of algorithmic stability in the framework of statisticallearning theory.
For empirical risk minimization (ERM) problems, we establish theoptimal excess population risk bounds for both smooth and non-smooth cases byintroducing on-average argument stability.
For minimax problems, we develop aquantitative connection between on-average argument stability and generalizationerror which extends the existing results for uniform stability [38].
We further develop the first nearly optimal convergence rates for convex-concave problems bothin expectation and with high probability, which, combined with our stability results,show that the optimal generalization bounds can be attained for both smooth andnon-smooth cases.
To the best of our knowledge, this is the first generalizationanalysis of SGMs when the gradients are sampled from a Markov process.
Introduction
Stochastic gradient methods (SGMs) have been the workhorse behind the success of many machinelearning (ML) algorithms due to their simplicity and high efficiency.
As opposed to the deterministic(full) gradient methods, SGMs only require a small batch of random example(s) to update the modelparameters at each iteration, making them amenable for solving large-scale problems.
There are mainly two notable types of SGMs which are inherent for different learning problems.
Inparticular, stochastic gradient descent (SGD) is widely used for solving the empirical risk minimization (ERM) problem and the theoretical convergence has been extensively studied [e g.,5, 17, 21, 37, 42, 43, 53, 57, 73, 80, 82, 86].
Concomitantly, the minimax problems instantiatemany ML problems such as Generative Adversarial Networks (GANs) [2, 30], AUC maximization[29, 45, 81], and algorithmic fairness [16, 41, 52, 51].
Stochastic gradient descent ascent (SGDA) isan off-the-shelf algorithm for solving minimax problems.
The convergence of SGDA and its variantsis also widely studied in the literature [e.g., 44, 49, 54, 56].
On the other important front, the ultimate goal of learning is to achieve good generalization from thetraining data to the unknown test data.
Along this line, generalization analysis of SGMs has attractedconsiderable attention using the algorithmic stability approach [11, 24].
In particular, stability andgeneralization of SGD have been studied using the uniform argument stability [6, 7, 13, 32, 36]and on-average stability [39, 40].
In [25, 38, 85], different stability and generalization measures areinvestigated for SGDA under both convex-concave and non-convex-non-concave settings.
A criticalassumption in most of the above studies about SGD and SGDA is the i.i.d.
sampling scheme wherethe randomly sampled mini-batch or datum at each iteration is i.i.d.
drawn from the given trainingdata, guaranteeing that the stochastic gradient is an unbiased estimator of the true gradient.
Markov chain naturally appears in many important problems, such as decentralized consensusoptimization, which finds applications in various areas including wireless sensor networks, smartgrid implementations and distributed statistical learning [4, 14, 23, 48, 50, 58, 61, 63] as well aspairwise learning [78] which instantiates AUC maximization [1, 29, 46, 81, 87] and metric learning[35, 75, 76, 79].
A common example is a distributed system in which each node stores a subset ofthe whole data, and one aims to train a global model based on these data. We let a central nodethat stores all model parameters walk randomly over the system, in which case the samples areaccessed according to a Markov chain. Several works studied this kind of model [33, 34, 48, 50, 58]. Markov chains also arise extensively in thermodynamics, statistical mechanics dynamic systemsand so on [59, 67].
In addition, it was observed in [69, 78] that SGD with Markov chain sampling(MC-SGD) performs more efficiently than SGD with the common i.i.d.
sampling scheme in variouscases.
Hence, studying the performance of MC-SGMs has certain theoretical and application values.
The key difference from the i.i.d.
sampling scheme is that the stochastic gradient at each iteration issampled on the trajectory of a Markov chain, in which the stochastic gradient estimators are neitherunbiased nor independent.
Recent studies [4, 18, 22, 33, 34, 58, 68] overcame this technical hurdleand provided the convergence rates of MC-SGD.
However, to the best of our knowledge, there is nowork on the generalization performance of SGMs with Markov sampling.
Main contribution: In this paper, we provide a comprehensive study of the stability and generalization for both SGD and SGDA with Markov sampling in the framework of statistical learning theory[72, 10].
Our main contribution can be summarized as follows.
• We develop stability and generalization results of MC-SGD for solving ERM problems in bothsmooth and non-smooth cases.
In particular, we show that MC-SGD can achieve competitive stabilityresults as SGD with i.i.d.
sampling scheme.
By trading off the generalization and optimizationerrors appropriately, we establish the first-ever-known excess generalization bound O (1 / √ n ) forMC-SGD where n is the size of training data.
The key idea for handling Markov sampling structureof MC-SGD is to use the concept of on-average argument stability.• We first establish the connection between on-average argument stability and generalization forminimax optimization algorithms, which extends the existing work on uniform argument stability[38].
We further develop stability bounds of SGDA with Markov sampling (MC-SGDA) for bothsmooth and non-smooth cases and obtain the nearly optimal convergence rates ˜
O (1 / √ T ) for convexconcave problems in the form of both expectation and high probability, where T is the number ofiterations, from which its optimal population risk bound is established.
Specifically, we considerseveral measures of generalization performance and show that the optimal population risk boundsO (1 / √ n ) can be derived even in the non-smooth case.• To the best of our knowledge, this is the first-ever-known work on stability and generalization ofSGD and SGDA under the Markov chain setting.
Our results show that, despite the stochastic gradientestimator is biased and dependent across iterations due to the Markov sampling scheme, the generalization performance of MC-SGD and MC-SGDA enjoys the same optimal excess generalization ratesas the i.i.d.
sampling setting.
Organization of the paper:
We discuss the related work in Subsection 1.1 and formulate theproblem in Section 2.
Section 3 presents the stability and generalization results of MC-SGD for bothsmooth and non-smooth losses.
Section 4 develops the first nearly optimal convergence rates forconvex-concave problems of MC-SGDA, and show that the optimal risk bounds can be derived inboth smooth and non-smooth cases.
Section 5 concludes the paper.
1.1 Related Work
In this subsection, we review some further works which are closely related to our paper.
Algorithmic Stability .
Algorithmic stability characterizes the sensitivity of a learning algorithmwhen the inputs to the algorithm are slightly perturbed.
The framework of algorithmic stabilitywas established in a seminal paper [11] for the exact minimizer of the ERM problem, where theuniform stability was established for strongly convex objective functions.
Recent work [12, 27, 28]derived sharper generalization bounds for uniformly stable algorithms with high probability.
Severalother stability measures were later developed for studying the generalization of different learningalgorithms including the hypothesis stability [11], on-average stability [64], argument stability [47]and total variation stability [8, 71].
Stability and Generalization Analysis of SGMs.
[32] established generalization error bounds oforder O (1 / √ n ) in expectation for SGD for convex and smooth problems using uniform stability.[-
-]
The on-average variance of stochastic gradients was used to refine the generalization analysis of SGDfor non-convex problems [88].
The results were improved and refined by [36] using a data-dependentnotion of algorithmic stability for SGD.
[39] introduced on-average argument stability and studied thestability and generalization of SGD for a general class of non-smooth convex losses, i.e., the gradientof the loss function is α -Hölder continuous.
They also established fast generalization bounds O (1 /n )for smooth convex losses in a low-noise setting.
The same authors also extended the analysis to thenon-convex loss functions in [40].
Meanwhile, [6] addressed uniform argument stability of SGD with
Lipschitz-continuous convex losses.
Optimal generalization bounds were also developed for SGDin different settings [6, 7, 26, 74, 77].
Stability and generalization for SGMs have been studied forpairwise learning [65, 78] where the loss involves a pair of examples.
In particular, [78] introduced asimple MC-SGD algorithm for pairwise learning where pairs of examples form a special Markovchain { ξ t =
( z i t , z i t − 1 ) : t ∈ N } .
Here, z i t and z i t − 1 are i.i.d. sampled from the training data ofsize n at time t and t − 1 , respectively. The uniform argument stability and generalization have beenestablished (see more discussion on the difference between our work and [78] in Remark 8 below). For minimax problems, [85] studied the weak generalization and strong generalization bounds in thestrongly-convex-concave setting.
[25] established the optimal generalization bounds for proximalpoint method, while gradient descent ascent (GDA) is not guaranteed to have a vanishing excessrisk in convex-concave case.
[38] proved that SGDA can achieve the optimal excess risk boundsof order O (1 / √ n ) for both smooth and non-smooth problems in the convex-concave setting.
Theyalso extended their work to the nonconvex-nonconcave problems.
However, all the above studies thestability and generalization of SGD and SGDA under the assumption of the i.i.d.
sampling scheme.
Convergence Analysis of MC-SGMs.
The convergence analysis of SGD and its variants when thegradients are sampled from a Markov chain have been studied in different settings [3, 18, 19, 22,34, 58, 66, 69, 70].
Specifically, [34, 58] studied the Markov subgradient incremental methods in adistributed system under time homogeneous and time non-homogeneous settings, respectively.
[22]studied the convergence of stochastic mirror descent under the ergodic assumption.
[69] establishedthe convergence rate O (1 /T 1 − q ) with some q ∈ (1 / 2 , 1) for convex problems.
They also developedthe convergence result for non-convex problems.
In addition, decentralized SGD methods with thegradients sampled from a non-reversible Markov chain have been studied in [68].
[18] consideredan accelerated ergodic Markov chain SGD for both convex and non-convex problems.
[19] furtherstudied the convergence rates without the bounded gradient assumption.
All these studies focused onthe convergence analysis of MC-SGD for solving the ERM problems.
Problem Setting and Target of Analysis
In this section, we introduce the SGD for ERM and SGDA for solving minimax problems with
Markov Chain, and describe the target of generalization analysis for both optimization algorithms.
Target of Generalization Analysis.
Let W be a parameter space in R d and D be a populationdistribution defined on a sample space Z .
Let f : W × Z → [0 , ∞ ) be a loss function.
In thestandard framework of Statistical Learning Theory (SLT) [10, 72], one aims to minimize the expectedpopulation risk, i.e., F ( w ) :
=
E z [ f ( w ; z )] , where the model parameter w belongs to W , and theexpectation is taken with respect to (w.r.t.)
z according to D .
However, the population distribution isoften unknown.
Instead, we have access to a training dataset S =
{ z i ∈ Z} ni =1 with size n , where z iis independently drawn from D .
Then consider the following ERM problem
[Equation]
136 For a randomized algorithm A to solve the above problem, let A ( S ) be the output of algorithm Abased on the dataset S .
Then its statistical generalization performance (prediction ability) is measuredby its excess population risk F ( A ( S ))
− F ( w ∗ ) , i.e., the discrepancy between the expected risks ofthe model
A ( S ) and the best model w ∗ ∈ W .
We are interested in studying the excess population risk.
Let E S, A [ · ] denote the expectation w.r.t.
both the randomness of data S and the internal randomnessof A .
To analyze the excess population error, we use the following error decomposition
E S, A [
[Equation]
A ( [Equation] [Equation] The first term is called the generalization error of the algorithm A measuring the difference betweenthe expected risk and empirical one, for which we will handle using stability analysis as shown soon.
The second term is the optimization error, which is induced by running the randomized algorithm
Ato minimize the empirical objective.
It can be estimated by tools from optimization theory.
As discussed in the introduction, many machine learning problems can be formulated as minimax problems including adversarial learning [30], reinforcement learning [15, 20] and AUCmaximization [29, 46, 81, 87].
We are also interested in solving this type of problem.
Let W and V be parameter spaces in R d .
Let D be a population distribution defined on a samplespace Z , and f : W × V × Z → [0 , ∞ ) .
We consider the minimax optimization problems:
[Equation]
(cid:8) F ( w , v ) :
= E z
∼D [ f ( w , v ; z )] } .
In practice, we only have a training dataset
S = { z 1 , . . .
, z n } independently drawn from D and hence the minimax problem is reduced to thefollowing empirical version:
[Equation]
Since minimax problems involve the primal variable and dual variable, we have different measuresof generalization [38, 85].
For a randomized algorithm A ( S ) solving the problem (3), we denotethe output of A as A ( S ) =
( A w ( S ) , A v ( S )) for notation simplicity.
Let E [ · ] denote the expectationw.r.t.
the randomness of both A and S .
We are particularly interested in the following two metrics.
Definition 1 (Weak Primal-Dual (PD) Risk) .
The weak Primal-Dual population risk of A ( S ) ,denoted by △ w ( A w , A v ) , is defined as max v ∈V E (cid:2) F ( A w ( S ) , v ) (cid:3) − min w ∈W E (cid:2) F ( w , A v ( S ))
(cid:3) .
The corresponding (expected) weak PD empirical risk, denoted by △ w emp ( A w , A v ) , is defined by max v ∈V E (cid:2)
F S
A w ( S )
, v ) (cid:3) − min w ∈W E (cid:2) F S ( w , A v ( S ))
(cid:3) .
We refer to △ w ( A w , A v ) −△ w emp ( A w , A v ) as the weak PD generalization error of the model ( A w ( S ) , A v ( S )) .
Definition 2 (Primal Risk) .
The primal population and empirical risks of A ( S ) are respectivelydefined by R ( A w ( S ))
=
max v ∈V F
( A w ( S ) , v ) , and R S ( A w ( S ))
=
max v ∈V F S
A w ( S ) , v ) .
Werefer to R ( A w ( S ))
− R S ( A w ( S ))
as the primal generalization error of the model A w ( S ) , and
R ( A w ( S ))
− min w ∈W R ( w ) as the excess primal population risk .
SGD and SGDA with Markov Sampling.
One often considers SGD to solve the ERM problem (1).
Specifically, let W ⊆ R d be convex, Proj W ( · ) denote the projection to W , and ∂f ( w ; z ) denote asubgradient of f ( w ; z ) at w .
Let w 0 ∈ W be an initial point, and { η t } is a stepsize sequence.
Forany t ∈ N , the update rule of SGD is given by
[Equation]
(4) where { i t } is generated from [ n ] = { 1 , 2 , . . . , n } with some sampling scheme.
A typically samplingscheme is the uniform i.i.d.
sampling, i.e., i t is drawn randomly from [ n ] according to a uniformdistribution with/without replacement.
In this paper, we are particularly interested in the case when i t ∈ [ n ] is drawn from a Markov Chainwhich is widely used in practice [3, 4, 18, 22, 34, 66, 69, 70].
Let P be an n × n -matrix withreal-valued entries.
We say a Markov chain { X k } with finite state [ n ] and transition matrix P istime-homogeneous if, for k ∈ N , i, j ∈ [ n ] , and i 1 , . . .
, i k − 1 ∈ [ n ] , there holds Pr ( X k {++
+} =
j | X
=i 1 , . . .
, X k = i )
=
Pr ( X k
+
=
j | X k = i )
=
[ P ] i,j .
Likewise, the SGDA algorithm with Markovsampling scheme is defined as follows.
Specifically, let ∂ w f and ∂ v f denote the subgradients of fw.r.t.
the arguments w and v , respectively.
We initialize ( w 0 , v 0 ) ∈ W × V , for any t ∈ N , let { i t }is drawn from [ n ] according to a Markov Chain. The update rule of SGDA is given by
[Equation]
[Equation]
For brevity, we refer to the above algorithms as Markov chain-SGD (MC-SGD) and Markov chainSGDA (SGDA), respectively.
There are two types of randomness in MC-SGD/MC-SGDA.
The firstrandomness is due to training dataset S which is i.i.d.
from the population distribution D .
The otherrandomness arises from the internal randomness of the MC-SGD/MC-SGDA algorithm, i.e., therandomness of the indices { i t } , which is a Markov chain.
Remark 1.
Convergence analysis mainly considers the empirical optimization gap, i.e., the discrepancy between F S ( A ( S )) and F S ( w ∗ ) .
Here, we are mainly interested in the generalizationerror which measures the prediction ability of the trained model on the test (future) data.
As such,the purpose of this paper is to provide a comprehensive generalization analysis of MC-SGD and
MC-SGDA in the framework of statistical learning theory.
Specifically, given a finite training data S ,let A ( S ) be the output of the MC-SGD for solving the ERM problem (1).
Our target is to analyzethe excess population risk E S, A [ F ( A ( S ))]
− F ( w ∗ ) .
Let A ( S ) =
( A w ( S ) , A v ( S )) be the outputof MC-SGDA for solving the empirical minimax problem (3), our aim is to analyze the weak PDpopulation risk △ w ( A w , A v ) and the excess primal population risk R ( A w ( S )) − min w ∈W R ( w ) .
In both cases, the generalization analysis will be conducted using the algorithmic stability [11, 32].
As we show soon below, the final rates are obtained through trade-offing the optimization error(convergence rate) and the generalization error (stability results).
Properties of Markov Chain.
Denote the probability distribution of X k as the non-negative rowvector π k = ( π k (1) , π k (2) , . .
, π k ( n )) , i.e., Pr ( X k = j ) =
π k ( j ) .
Further, we have (cid:80) ni
{+=
 π+} k ( i ) =1 .
For the time-homogeneous Markov chain, it holds π k = π k − 1 P = · · · = π 1 P k − 1 for all k ∈ N .[-
-]
Here, π 1 is an initial distribution and P k denotes the k -th power of P .
A Markov chain is irreducibleif, for any i, j ∈ [ n ] , there exists k such that [ P k ] i,j > 0 .
That is, the Markov process can go fromany state to any other state.
State i ∈ [ n ] is said to have a period τ if [ P k ] i,i = 0 whenever k is not amultiple of τ and τ is the greatest integer with this property.
If τ = 1 for every state i ∈ [ n ] , thenwe say the Markov chain is aperiodic.
We say a Markov chain with stationary distribution Π ∗ isreversible if Π ∗ ( i )
[ P ] i,j = Π ∗ ( j )
[ P ] j,i for all i, j ∈ [ n ] .
We need the following assumption for studying optimization error of MC-SGMs.
Assumption 1.
Assume the Markov chain { i t } with finite state [ n ] is time-homogeneous, irreducibleand aperiodic.
It starts from an initial distribution π 1 , and has transition matrix P and stationarydistribution Π ∗ with Π ∗ ( i ) =
1 n for any i ∈ [ n ] , i.e., lim k →∞ P k = 1 n 1 n 1 ⊤ n , where 1 n ∈ R n is thevector with each entry being 1 and 1 ⊤ n denotes its transpose. Remark 2.
Our assumptions on Markov chains listed above are standard in the literature [18, 34, 50,69 , 68, 78]. For instance, Markov chain-type SGD was proposed for pairwise learning which canapply to various learning task such as AUC maximization and bipartite ranking [1, 83, 87, 29, 46] andmetric learning [35, 75, 76, 79]. This pairwise learning algorithm forms pairs of examples followinga special Markov chain { ξ t = ( z i t , z i t − 1 ) : t ∈ N } where z i t and z i t − 1 are i.i.d.
sampled from thetraining data of size n at time t and t − 1 , respectively and, at time t , the model parameter is updatedusing gradient descent based on ξ t .
As mentioned in Remark 3 of [78], { ξ t : t ∈ N } is a Markov
Chain satisfying all of our assumptions.
Another notable example is the decentralized consensusoptimization in a multi-agent network, where the samples are accessed according to a Markov chainand the number of states of the Markov chain equals the number of nodes in the network, which isfinite.
One always considers the same transition matrix P for each node and assumes the Markovchain is irreducible and aperiodic [50, 84].
Results for Markov Chain SGD
In this section, we present the stability and generalization results of MC-SGD.
Our analysis requiresthe following definition and assumptions.
Let G, L > 0 and ∥ · ∥ 2 denote the Euclidean norm.
Definition 3.
We say f is convex w.r.t.
the first argument if, for any z ∈ Z and w , w ′ ∈ W , thereholds f ( w ; z ) ≥ f ( w ′ ; z ) + ⟨ ∂f ( w ′ ; z ) , w − w ′ ⟩ .
Assumption 2.
Assume f is G -Lipschitz continuous, i.e., for any z ∈ Z and w , w ′ ∈ W , thereholds | f ( w ; z ) − f ( w ′ ; z ) | ≤ G ∥ w − w ′ ∥ 2 .
Assumption 3.
Assume f is L -smooth, i.e., for any z ∈ Z and w , w ′ ∈ W , there holds f ( w ; z ) −f ( w ′ ; z ) ≤ ⟨ ∂f ( w ′ ; z ) , w − w ′ ⟩ + L 2 ∥ w − w ′ ∥ 22 .
3.1 Stability and Generalization of MC-SGD
Let w ∗ =
arg min w ∈W F ( w ) be the best model in W and ¯ w T
= (cid:80)
Tj =1 η j w j / (cid:80) Tj =1 η j be theoutput of MC-SGD with T iterations.
We will use algorithmic stability to study the generalizationerrors, which measures the sensitivity of the output model of an algorithm.
Below we give thedefinition of on-average argument stability [39].
Definition 4.
(On-average argument stability) Let S
=
{ z 1 , . . .
, z n } and (cid:101) S =
{ ˜ z 1 , . . .
, ˜ z n } be drawnindependently from D .
For any i ∈ [ n ] , define S ( i ) = { z 1 , . . .
, z i − 1 , ˜ z
i , z i +1 , . . . , z n } as the setformed from S by replacing the i -th element with ˜ z i .
We say a randomized algorithm A is on-averageϵ -argument-stable if E S, (cid:101)
[Table]
To obtain on-average argument stability bounds of MC-SGD, our idea is to first write the stability asa deterministic function according to whether the different data point is selected, and then take theexpectation w.r.t.
the randomness of the algorithm.
The detailed proofs are given in Appendix B.1.
Theorem 1 (Stability bounds) .
Suppose f is convex and Assumption 2 holds.
Let W = R d and let
A be MC-SGD with T iterations.
(a) (Smooth case)
Suppose Assumption 3 holds and η j ≤ 2 /L .
Then A is on-average ϵ -argumentstable with ϵ ≤ 2 Gn (cid:80)
Tj =1 η j .(b)
(Non-smooth case)
A is on-average ϵ -argument-stable with ϵ ≤ 2 G (cid:113)(cid:80) Tj =1 η 2 j [-⌘ 1 / 2 +
-]+ 4 Gn (cid:80)
Tj =1 η j .
Remark 3.
Without any assumption on Markov chain, Theorem 1 shows that argument stabilitybounds of MC-SGD are in the order of O ( Tη/n ) and O ( √ Tη + Tη/n ) with a constant stepsize η forsmooth and non-smooth losses, respectively.
Both of them match the corresponding bounds for SGDwith i.i.d.
sampling [6, 32, 39, 74], which imply that stability of MC-SGD is at least not worse thanthat of the i.i.d.
sampling case.
The technical novelty here is to observe that, in the sense of on-averageargument stability, we can use the calculation of E A
[ (cid:80) ni
=
I [ i t = i ] ] to replace that of E A [ I [ i t
= i ] ] , where 257 I [ · ] is the indicator function.
This key step avoids the complicated calculations about E A
[ I [ i t = i ] ] .
Taking the uniform stability as example, we need to consider neighboring datasets differing by the i -thdata, and can get E A [ ∥ w t − w ′ t ∥ 2 ]
= O ( η (cid:80) t j =1 E A
[ I i j = i ]) =
{+O ( η (cid:80) t j =1 (cid:80) n k =
 [ P j − 1 ] k,i π 1 ( k ))+} ,which depends on the transition matrix P and is not easy to control.
In contrast, with the on-averagestability we get stability bounds depending on (cid:80) ni
[-P ni =
-]=1 I [ i t
= i ] , which is always 1 , i.e., the on-averagestability allows us to ignore the effect of sampling process.
The following theorem presents generalization bounds for MC-SGD in both smooth and non-smoothcases, which directly follows from Lemma A.4 and Theorem 1.
Theorem 2 (Generalization error bounds) .
Suppose f is convex and Assumption 2 holds.
Let
W = R d and let A be MC-SGD with T iterations.
(a) (Smooth case)
Suppose Assumption 3 holds and let η j ≡ η ≤ 2 /L .
Then there holdsE S, A [ F ( ¯ w T ) − F S ( ¯ w T )] ≤ 2 G 2 Tηn .(b)
(Non-smooth case)
If η j ≡ η , then there holds E S, A [ F ( ¯ w T ) − F S ( ¯ w T )]
= O (cid:0) √ Tη + Tηn (cid:1) .
3.2 Excess Population Risk of MC-SGD
In this subsection, we present excess population risk bounds for MC-SGD in both smooth andnon-smooth cases.
The proofs are given in Appendix B.3.
We use the notation B ≍ ˜ B if there existuniversal constants c 1 , c 2 >
such that c 1 ˜ B ≤ B ≤ c 2 ˜ B .
Let λ i ( P ) be the i -th largest eigenvalueof the transition matrix P and λ ( P ) =
(max {| λ 2 ( P ) | , | λ n ( P ) |} + 1) / 2 ∈ [1 / 2 , 1) .
Let K P bethe mixing time and C P be a constant depending on P and its Jordan canonical form (detailedexpressions are given in Lemma A.1).
We assume sup z ∈Z f (0; z ) and ∥ w ∗ ∥ 2 are bounded.
Assumption 4.
Assume the Markov chain { i t } is reversible with P = P ⊤ .
Theorem 3 (Excess population risk for smooth losses) .
Suppose f is convex and Assumptions 1,2, 3 and 4 hold.
Let W ∈ R d .
Let A be MC-SGD with T iterations, and { w j } Tj =1 be producedby A with w 0 = 0 and η j ≡ η ≤ 2 /L .
If we select T ≍ n and η = ( T log( T )) − 1 / 2 , then
E S, A [ F ( ¯ w T )] − F ( w ∗ ) =
O (cid:0)(cid:112) log( n ) / ( √ n log(1 /λ ( P ))) (cid:1) .
Remark 4. A term K P / (cid:0) M 0 n 34 log 14 ( n ) (cid:1) with M 0 = min { (cid:112) n log( n ) C P nλ ( P ) K P , 1 } appearsin the excess risk bound of MC-SGD (see the proof of Theorem 3), which will be worse than 284 (cid:112) log( n ) / √ n when K P is large. Note Lemma A.1 implies that this term will disappear if P issymmetric. Hence, we introduce Assumption 4 to get the nearly optimal rate.
Theorem 4 (Excess population risk for non-smooth losses) .
Suppose f is convex and Assumptions 1, 2,4 hold.
Let W ∈ R d .
Let A be MC-SGD with T iterations, and { w j } T j =1 be produced by A with η j ≡η .
If we select T ≍ n 2 and η = T − 3 / 4 , then E S, A [ F ( ¯ w T )] − F ( w ∗ ) =
O (cid:0) 1 / (cid:0) √ n log(1 /λ ( P ))
(cid:1)(cid:1) .
Remark 5.
To estimate the excess population risk, we need the convergence rates of MC-SGDwhich can be found in Appendix B.2.
[69] established a convergence rate of O (1 /T 1 − q ) with someq ∈ (1 / 2 , 1) under the bounded parameter domain assumption.
We remove this assumption by showing ∥ w t ∥ 22 = O ( (cid:80) Tk =1 η k ) and obtain the nearly optimal convergence rate ˜ O (1 / √ T ) with acareful choice of η = 1 / (cid:112) T log( T ) .
To understand the variation of the algorithm, we present aconfidence-based bound for optimization error, which matches the bound in expectation up to aconstant factor.
We also provide the convergence analysis for non-convex problems in Appendix B.2.
Remark 6.
Theorems 3 and 4 show, after carefully selecting the iteration number T and stepsizeη , that the excess population risk rate O (1 / √ n ) is achieved in both smooth and non-smooth cases.
Note [6, 32, 39] show that the excess population risk rate O (1 / √ n ) is optimal for the i.i.d.
samplingcase.
Therefore, our results for MC-SGD are also optimal since the i.i.d.
sampling is a special caseof Markov sampling.
Our results imply that despite the gradients are biased and dependent acrossiterations in Markov sampling, the generalization performance of SGD is competitive with the i.i.d.sampling case.
Theorems 3 and 4 also show the impact of the smoothness in achieving the optimalrate.
The rate for the non-smooth case in Theorem 4 looks slightly better than the smooth case(Theorem 3) with a logarithmic term.
However, the optimal rate can be achieved with a linear gradientcomplexity (i.e., the total number of computing the gradient) for smooth losses, while Theorem 4implies that gradient complexity O ( n 2 ) is required for non-smooth losses.
Remark 7.
According to Theorems 3 and 4, we can further observe how the transition matrix P affects the excess population risks.
Indeed, the excess population risk rates are monotonicallyincreasing w.r.t λ ( P ) .
Particularly, the closer λ ( P ) is to 1 / 2 , the better the rate is.
Let us consider twoextreme examples.
Suppose the Markov chain starts from the uniform distribution and has transitionmatrix P = 1 n 1 n 1 Tn .
MC-SGD degenerates to SGD with i.i.d.
sampling in this case.
The excesspopulation risk rate O (1 / √ n ) is obtained from Theorem 4 with λ ( P ) = 1 / 2 .
For a Markov chainmoving on a circle (i.e., if the chain is currently at state i , then it goes to states i + 1 , i and i − 1 withequal probability), we can verify that λ ( P ) =
O (1 − 1 /n 2 ) , which implies a bad rate in this case. Remark 8.
[78] proposed a simple MC-SGD algorithm for pairwise learning associated with apairwise loss f ( w , z, z ′ ) .
Specifically, at iteration t , the algorithm update the model parameter asfollows: w t = w t − 1 − η t ∇ w f ( w t − 1 , z i t , z i t − 1 ) where z i t and z i t − 1 are i.i.d.
sampled from the 318 training data of size n at time t and t − 1 , respectively.
In Remark 3 of [78], it was shown that{ ξ t = ( i t , i t − 1 ) ∈
[ n ] × [ n ] } does form a time-homogeneous, irreducible and aperiodic Markovchain. There are two key differences between our work and [78]. Firstly, the work [78] used uniformstability directly due to E A [ I i t
= i ]
= 1 /n , while this term is not easy to control in our general setting(see Remark 3 for details). To overcome this hurdle, we resort to the on-average stability and showthat MC-SGD achieves the optimal excess risk rate. Secondly, the proofs there critically rely onthe fact f ( w t − 1 ; z i t , z i t − 1 ) = f ( w t − 2 ; z i t , z i t − 1 )
+ O ( η t − 1 ) and the independence of w t − 2 w.r.t.
325 i t
and i t − 1 .
However, these specially tailored techniques for pairwise learning do not apply to thegeneral Markov setting as we considered here.
Results for Markov Chain SGDA
In this section, we study the generalization analysis of MC-SGDA for minimax optimization problems.
Let ( ¯ w T , ¯ v T ) be the output of MC-SGDA with T iterations, where
[Equation]
We first introduce some necessary definitions and assumptions.
Definition 5.
Let ρ ≥ 0 and g : W × V
(cid:55)→ R .
We say g is ρ -strongly-convex-strongly-concave( ρ -SC-SC) if, for any v ∈ V , the function w (cid:55)→ g ( w , v ) is ρ -strongly-convex and, for any w ∈ W ,the function v (cid:55)→ g ( w , v ) is ρ -strongly-concave.
We say g is convex-concave if g is 0 -SC-SC.
The following two assumptions are standard [25, 85].
Assumption 5 amounts to saying f is Lipschitzcontinuous w.r.t.
both w and v , while Assumption 6 considers smoothness conditions.
Assumption 5.
Assume for all w ∈ W , v ∈ V and z ∈ Z , (cid:13)(cid:13) ∂ w f ( w , v ; z ) (cid:13)(cid:13) 2 ≤
G and (cid:13) (cid:13) ∂ v f ( w , v ; z ) (cid:13) (cid:13) 2 ≤ G.
Assumption 6.
For any z , assume the function ( w , v ) (cid:55)→ f ( w , v ; z ) is L -smooth, i.e., the followinginequality holds for all w ∈ W , v ∈ V and z ∈ Z
[Equation]
[Table]
4.1 Stability and Generalization Measures
We use algorithmic stability to study the generalization of minimax learners.
To this end, we firstintroduce the stability for minimax optimization problems.
Definition 6 (Argument stability for minimax problems) .
Let S, (cid:101) S and S ( i ) be defined as Definition 4.
Let A be a randomized algorithm and ϵ > 0 .
We say A is on-average ϵ -argument-stable for minimaxproblems if 1 n (cid:80) ni =1 E (cid:2) ∥A w ( S ( i ) )
−A w ( S ) ∥
2 + ∥A v ( S ( i ) ) −A v ( S ) ∥ 2 (cid:3) ≤ ϵ.
The following theorem establishes a connection between stability and generalization.
Part (a) showsthat on-average argument stability implies generalization measured by the weak PD risk, while Part(b) shows that on-average argument stability guarantees a strong notion of generalization in terms ofthe primal risk under a strong concavity assumption.
Theorem 5 will be proved in Appendix C.1.
Theorem 5 (Generalization via argument stability) .
Let A be a randomized algorithm and ϵ > 0 .
(a) If A is on-average ϵ -argument-stable and Assumption 5 holds, then there holds △ w ( A w , A v ) −△ w emp ( A w , A v ) ≤ Gϵ.
(b) If A is on-average ϵ -argument-stable, the function v (cid:55)→ F ( w , v ) is ρ -strongly-concave and Assumptions 5, 6 hold, then we have E S, A (cid:2) R ( A w ( S ))
{+− R S ( A w ( S )) (cid:3) ≤ (cid:0) 
 + L/ρ (cid:1) Gϵ.+}
In the following theorem we develop stability bounds for MC-SGDA applied to convex-concaveproblems.
The proof is given in Section C.1 of the Appendix.
Theorem 6 (Stability bounds) .
Assume for all z , the function ( w , v ) (cid:55)→ f ( w , v ; z ) is convex-concave.
Let W = R d
and Assumption 5 hold, and let A be MC-SGDA with T iterations.
(a) (Smooth case)
If
Assumption
holds
and
(cid:80)
Tj =1 η 2 j ≤ 1 / (2 L 2 ) , then A is on-average ϵ -argument stable with ϵ ≤ 4 G (cid:0) 1 n (cid:80) Tj =1 η 2 j (cid:1) 1 / 2 + 8 √ 2 G n (cid:80) Tj =1 η j .
(b) (Non-smooth case)
A is on-average ϵ -argument stable with ϵ ≤ 2 G (cid:113) 2 (cid:80) Tj =1 η 2 j + 4 √ 2[-+
 p-] G n (cid:80) Tj =1 η j .
Remark 9.
For convex-concave and Lipschitz problems, the stability bound of the order O ( η ( √ T + T/n )) was established for SGDA with a constant stepsize under the uniformly i.i.d.
samplingsetting.
Under a further smoothness assumption, the stability bound was improved to the order of O ( ηT/n ) [38].
Our stability bounds in Theorem 6 match these results up to a constant factor andextend them to the Markov sampling case.
Remark 10.
Let { ( w ( i ) t , v ( i ) t ) } be the SGDA sequence based on S ( i ) .
The existing stability analysis [38] builds a recursive relationship for E A (cid:2)
∥ w t − w ( i ) t ∥ 22 + ∥ v t − v ( i ) t ∥ 22 (cid:3) , which cruciallydepends on the i.i.d.
sampling property of i t ∈ [ n ] .
This strategy does not apply to MC-SGDA sincethe conditional expectation over i t is in a much complex manner due to the Markov Chain sampling.
We bypass this difficulty by building a recursive relationship for ∥ w t − w ( i ) t ∥ 22 + ∥ v t − v ( i ) t ∥ 22 interms of a sequence of random variables
I [ i t = i ] .
A key observation is that the effect of randomnesswould disappear if we consider on-average argument stability since (cid:80) ni
=1 I [ i t = i ]
= 1 for any t ∈ N .
We can combine the stability bounds in Theorem 6 and Theorem 5 to develop generalization bounds.
We first establish weak PD risk bounds in Theorem 7, and then move on to primal population riskbounds in Theorem 8.
The proofs are given in Section C.1 of the Appendix.
Theorem 7 (Weak PD risk bounds) .
Suppose Assumption 5 holds.
Assume for all z , the function( w , v ) (cid:55)→ f ( w , v ; z ) is convex-concave.
Let W = R d and { w j , v j } Tj =1 be produced by MC-SGDAwith η j ≡ η .
Let A be defined by A w ( S )
=
¯ w T and A v ( S ) =
¯ v T for ( ¯ w T , ¯ v T ) in (6) .
Denoteϵ wgen := △ w ( ¯ w T , ¯ v T ) − △ w emp ( ¯ w T , ¯ v T ) .
(a) (Smooth case)
If Assumption 6 holds and (cid:80) Tj =1 {+η 
+} j [Equation] (2 L 2 ) , then ϵ wgen
[Equation]
[Equation]
(b) (Non-smooth case)
The weak PD risk [-satisﬁes ✏ wgen  2 p 2 G P Tj =1 ⌘ 2 j  1 / 
-]satisfies
[Equation]
Theorem 8 (Primal risk bounds) .
Suppose Assumption 5 holds.
Assume for all z , the function( w , v ) (cid:55)→ f ( w , v ; z ) is convex-concave, and the function v (cid:55)→ F ( w , v ) is ρ -strongly-concave.
Let W = R d and let { w j , v j } Tj =1 be produced by MC-SGDA with η j ≡ η .
Let A be defined by
A w ( S ) =
¯ w T and A v ( S ) =
¯ v T for ( ¯ w T , ¯ v T ) in (6) .
Denote ϵ pgen := E S, A (cid:2) R ( ¯ w T ) − R S ( ¯ w T ) (cid:3) .
(a) (Smooth case)
If Assumption 6 holds and (cid:80) Tj =1 η 2 j ≤ 1 / (2 L 2 ) , then ϵ pgen ≤ 4 G 2 (1 +L/ρ ) (cid:0)(cid:0) 1 n (cid:80)
Tj =1 η 2 j (cid:1) 12 + 2 √ 2 n (cid:80) Tj =1 η j (cid:1) .(b)
(Non-smooth case)
The primal population risk satisfies ϵ pgen
[Equation]
+L/ρ ) (cid:0)(cid:0) (cid:80)
[Equation]
4.2 Population Risks of MC-SGDA
Now we establish the population risk bounds for MC-SGDA.
The following theorem establishes theweak PD population risk of MC-SGDA for both smooth and non-smooth problems.
Let D w and D vbe the diameters of W and V .
The proof for Theorem 9 is provided in Appendix C.3.
Theorem 9 (Weak PD population risk) .
Suppose Assumptions 1, 4 and 5 hold.
Assume for all z , thefunction ( w , v ) (cid:55)→ f ( w , v ; z ) is convex-concave.
Let { w j , v j } Tj =1 be produced by MC-SGDA withη j ≡ η .
Let A be defined by A w ( S )
=
¯ w T and A v ( S ) =
¯ v T for ( ¯ w T , ¯ v T ) in (6) .
(a) (Smooth case) Let Assumption 6 hold.
If T ≍ n and η ≍ ( T log( T )) − 12 , then △ w ( ¯ w T , ¯ v T ) =
O (cid:0) log( n ) / (cid:0) √ n log(1 /λ ( P )) (cid:1)(cid:1) .(b)
(Non-smooth case)
If we select T ≍ n 2 and η ≍ T − 34 , then we have △ w ( ¯ w T , ¯ v T ) =
O (cid:0) 1 / (cid:0) √ n log(1 /λ ( P )) (cid:1)(cid:1) .
Remark 11.
The above excess population risk bounds are obtained through the trade-off betweenthe optimization errors (convergence analysis) and stability results of MC-SGDA.
The convergencerates ˜ O
(1 / √ T ) of MC-SGDA for minimax problems in both expectation and high probability areprovided in Theorem C.3 and C.4 in Appendix C.2.
With gradient complexity O ( n ) , the minimaxoptimal excess risk bound O (1 / √ n ) for SGDA with uniform sampling for smooth problems wasestablished in [38].
We show SGDA with Markov sampling can achieve the nearly optimal boundwith the same gradient complexity.
For non-smooth problems, part (b) shows that the optimal excessrisk bound can be exactly achieved with the gradient complexity O ( n 2 ) .
Finally, we establish the following bounds for excess primal population risk under a strong concavitycondition on v (cid:55)→ F ( w , v ) , which measures the performance of the primal variable.
The proof for Theorem 10 is provided in Appendix C.3.
Theorem 10 (Excess primal population risk) .
Suppose Assumptions 1, 4, 5 and 6 hold.
Assumefor all z , the function ( w , v ) (cid:55)→ f ( w , v ; z ) is convex-concave.
Assume v (cid:55)→ F ( w , v ) is ρ -stronglyconcave.
Let { w j , v j } Tj =1 be produced by MC-SGDA with η j ≡ η .
Let A be defined by A w ( S )
=
¯ w T and A v ( S ) =
¯ v T for ( ¯ w T , ¯ v T ) in (6) .
If we choose T ≍ n, η ≍ ( T log( T )) − 1 / 2 , then
E S, A
[ R ( ¯ w T )] − min w ∈W R ( w ) = O (cid:0) ( L/ρ ) (cid:112) log( n ) / ( √ n log(1 /λ ( P ))) (cid:1) .
Remark 12.
We show MC-SGDA attains population risk bounds of the order ˜ O
(1 / √ n ) with a lineargradient complexity O ( n ) , which are minimax optimal up to a logarithmic factor.
This implies thatconsidering sampling with a Markov chain does not weaken the learnability.
Theorems 9 and 10 alsoshow the effect of P on the population risk rates, i.e., the rates get better as λ ( P ) decreases.
Conclusion
We develop the first-ever-known stability and generalization analysis of Markov chain stochasticgradient methods for both minimization and minimax objectives.
In particular, we establish theoptimal excess population bounds O (1 / √ n ) for MC-SGD for both smooth and non-smooth cases.
We also develop the first nearly optimal convergence rates ˜
O (1 / √ T ) for convex-concave problemsof MC-SGDA, and show that the optimal risk bounds O (1 / √ n ) can be derived even in the nonsmooth case.
Although the gradients from Markov sampling are biased and not independent acrossthe iterations, we show the performance of MC-SGMs is competitive compared to SGMs with theclassical i.i.d.
sampling scheme.
An interesting direction is to consider other variants of SGMs withvariance reduction techniques and differentially private SGMs under the Markov sampling scheme.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? N/ [Table]
[Table]
) Did you specify all the training details (e g., data splits, hyperparameters, how they [Table]
were chosen)?
[N/A [Table]
) Did you report error bars (e g., with respect to the random seed after running experi- [Table] ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[Yes]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable [Table] information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount [Table] spent on participant compensation?
[N/A]
[Table]