Stability and Generalization for Markov Chain Stochastic Gradient Methods
Anonymous Author(s) Afﬁliation
Address email
Abstract
Recently there are a large amount of work devoted to the study of Markov chainstochastic gradient methods (MC-SGMs) which mainly focus on the convergenceanalysis of MC-SGMs for solving minimization problems.
In this paper, we providea comprehensive generalization analysis of MC-SGMs for both minimization andminimax problems through the lens of algorithmic stability in the framework ofstatistical learning theory.
For empirical risk minimization (ERM) problems, weestablish the optimal excess population risk bounds for both smooth and nonsmooth cases by introducing on-average argument stability.
For minimax problems,we develop a quantitative connection between on-average argument stability andgeneralization error which extends the existing results for uniform stability [36].
We further develop the ﬁrst nearly optimal convergence rates for convex-concaveproblems both in expectation and with high probability, which, combined with ourstability results, show that the optimal generalization bounds can be attained forboth smooth and non-smooth cases.
To the best of our knowledge, this is the ﬁrstgeneralization analysis of SGMs when the gradients are sampled from a Markovprocess.
Introduction
Stochastic gradient methods (SGMs) have been the workhorse behind the success of many machinelearning (ML) algorithms due to their simplicity and high efﬁciency.
As opposed to the deterministic(full) gradient methods, SGMs only require a small batch of random example(s) to update the modelparameters at each iteration, making them amenable for solving large-scale problems.
There are mainly two notable types of SGMs which are inherent for different learning problems.
In particular, stochastic gradient descent (SGD) is widely used for solving the empirical riskminimization (ERM) problem and the theoretical convergence has been extensively studied [e g.,4, 16, 20, 34, 40, 41, 51, 55, 67, 70, 72, 74]).
Concomitantly, the minimax problems instantiatemany ML problems such as Generative Adversarial Networks (GANs) [1, 28], AUC maximization[27, 43, 71], and algorithmic fairness [15, 39, 49, 48].
Stochastic gradient descent ascent (SGDA) isan off-the-shelf algorithm for solving minimax problems.
The convergence of SGDA and its variantsis also widely studied in the literature [e.g., 42, 46, 52, 54].
On the other important front, the ultimate goal of learning is to achieve good generalization from thetraining data to the unknown test data.
Along this line, generalization analysis of SGMs has attractedconsiderable attention using the algorithmic stability approach [10, 22].
In particular, stability andgeneralization of SGD have been studied using the uniform argument stability [5, 6, 12, 30, 33]and on-average stability [37, 38].
In [23, 36, 73], different stability and generalization measures areinvestigated for SGDA under both convex-concave and non-convex-non-concave settings.
A criticalassumption in most of the above studies about SGD and SGDA is the i.i.d.
sampling scheme where the randomly sampled mini-batch or datum at each iteration is i.i.d.
drawn from the given trainingdata, guaranteeing that the stochastic gradient is an unbiased estimator of the true gradient.
At the same time, Markov chain offers a natural alternative of the common i.i.d. sampling scheme,which commonly appears in many application domains [3, 32, 47, 56, 59, 62].
For instance, [32]presented a variant of SGD algorithms using the Markov chain sampling approach for decentralizedimplementation in distributed systems which can be naturally applied to solve the parameter estimationproblems in sensor networks.
In [63], it was observed that SGD with Markov chain sampling (MCSGD) performs more efﬁciently than SGD with the common i.i.d.
sampling scheme in various cases.
The key difference from the i.i.d.
sampling scheme is that the stochastic gradient at each iteration issampled on the trajectory of a Markov chain, in which the stochastic gradient estimators are neitherunbiased nor independent.
Recent studies [3, 17, 18, 21, 31, 32, 56, 62] overcame this technicalhurdle and provided the convergence rates of MC-SGD.
However, to the best of our knowledge, thereis no work on the generalization performance of SGMs with Markov sampling.
In this paper, we provide a comprehensive study of the stability and generalization for both SGDand SGDA with Markov sampling in the framework of statistical learning theory [66, 9].
Our maincontribution can be summarized as follows
We develop stability and generalization results of MC-SGD for solving ERM problems in bothsmooth and non-smooth cases.
In particular, we show that MC-SGD can achieve competitive stabilityresults as SGD with i.i.d.
sampling scheme.
By trading off the generalization and optimizationerrors appropriately, we establish the ﬁrst-ever-known excess generalization bound O (1 / p n ) for 56 MC-SGD where n is the size of training data.
The key idea for handling Markov sampling structureof MC-SGD is to use the concept of on-average argument stability. We establish the connection between on-average argument stability and generalization for minimaxoptimization algorithms, which extends the existing work on uniform argument stability [36].
Wefurther develop stability bounds of SGDA with Markov sampling (MC-SGDA) for both smooth andnon-smooth cases and obtain the nearly optimal convergence rates ˜
O (1 / p T ) for convex-concaveproblems in the form of both expectation and high probability, where T is the number of iterations,from which its optimal population risk bound is established.
Speciﬁcally, we consider severalmeasures of generalization performance and show that the optimal population risk bounds O (1 / p n ) 65 can be derived even in the non-smooth case.3. To the best of our knowledge, this is the ﬁrst-ever-known work on stability and generalization of SGD and SGDA under the Markov chain setting.
Our results show that, despite the stochastic gradientestimator is biased and dependent across iterations due to the Markov sampling scheme, the generalization performance of MC-SGD and MC-SGDA enjoys the same optimal excess generalization ratesas the i.i.d.
sampling setting.
Organization of the paper:
The paper is organized as follows.
We discuss the related work in 72 Subsection 1.1 and formulate the problem in Section 2.
The main theoretical results of MC-SGD and MC-SGDA are presented in Sections 3 and 4, respectively.
We conclude the paper in Section 5.
1.1 Related Work
In this subsection, we review some further works which are closely related to our paper.
Algorithmic Stability .
Algorithmic stability characterizes the sensitivity of a learning algorithmwhen the inputs to the algorithm are slightly perturbed.
The framework of algorithmic stabilitywas established in a seminal paper [10] for the exact minimizer of the ERM problem, where theuniform stability was established for strongly convex objective functions.
Recent work [11, 25, 26]derived sharper generalization bounds for uniformly stable algorithms with high probability.
Severalother stability measures were later developed for studying the generalization of different learningalgorithms including the hypothesis stability [10], on-average stability [60], argument stability [45]and total variation stability [7, 65].
Stability and Generalization Analysis of SGMs.
[30] established generalization error bounds of 85 order O (1 / p n ) in expectation for SGD for convex and smooth problems using uniform stability. {+
+}
The on-average variance of stochastic gradients was used to reﬁne the generalization analysis of SGDfor non-convex problems [76].
The results were improved and reﬁned by [33] using a data-dependentnotion of algorithmic stability for SGD.
[37] introduced on-average argument stability and studied the stability and generalization of SGD for a general class of non-smooth convex losses, i.e., thegradient of the loss function is ↵ -Hölder continuous.
They also established fast generalization bounds O (1 /n ) for smooth convex losses in a low-noise setting.
The same authors also extended the analysis 92 to the non-convex loss functions in [38].
Meanwhile, [5] addressed uniform argument stability of
SGD with Lipschitz-continuous convex losses.
Optimal generalization bounds were also developedfor SGD in different settings [5, 6, 24, 68, 69].
For minimax problems, [73] studied the weak generalization and strong generalization bounds in thestrongly-convex-concave setting.
[23] established the optimal generalization bounds for proximalpoint method (PPM), while gradient descent ascent (GDA) is not guaranteed to have a vanishingexcess risk in convex-concave case.
[36] proved that SGDA can achieve the optimal excess riskbounds of order O (1 / p n ) for both smooth and non-smooth minimax problems in the convex-concave 100 setting.
They also extended their work to the nonconvex-nonconcave problems.
However, all theabove studies the stability and generalization of SGD and SGDA under the assumption of the i.i.d.sampling scheme. Under this i.i.d.
sampling scheme
, the stochastic gradient is an unbiased estimatorof the true gradient which is critical for the involved analysis.
Convergence Analysis of MC-SGMs.
The convergence analysis of SGD and its variants when the 105 gradients are sampled from a Markov chain have been studied in different settings [2, 3, 17, 18, 21,32, 56, 61, 63, 64].
Speciﬁcally, [32, 56] studied the Markov subgradient incremental methods in adistributed system under time homogeneous and time non-homogeneous settings, respectively.
[21]studied the convergence of stochastic mirror descent under the ergodic assumption.
[63] establishedthe convergence rate O (1 /T 1  q ) with some 1 / 2 < q < 1 when the Markov chain is non-reversiblefor convex problems.
They also developed the convergence result for non-convex problems.
Inaddition, decentralized SGD methods with the gradients sampled from a non-reversible Markovchain have been studied in [62].
[17] considered an accelerated ergodic Markov chain SGD for bothconvex and non-convex problems.
[18] further studied the convergence rates when the boundedgradient assumption is removed.
All these studies focused on the convergence analysis of MC-SGDfor solving the ERM problems.
Problem Setting and Target of Analysis
In this section, we introduce the SGD for ERM and SGDA for solving minimax problems with
Markov Chain, and describe the target of generalization analysis for both optimization algorithms.
Target of Generalization Analysis.
Let W be a parameter space in R d and D be a population 120 distribution deﬁned on a sample space Z .
Let f : W ⇥ Z ! [0 , 1 ) be a loss function.
In the 121 standard framework of Statistical Learning Theory (SLT) [9, 66], one aims to minimize the expectedpopulation risk, i.e., F ( w ) :
=
E z [ f ( w ; z )] , where the model parameter w belongs to W , and theexpectation is taken with respect to (w.r.t.)
z according to D .
However, the population distribution is 124 often unknown.
Instead, we have access to a training dataset S =
{ z i 2 Z} ni =1 with size n , where z i 125 is independently drawn from D .
Then consider the following ERM problem
[Equation]
For a randomized algorithm A to solve the above problem, let A ( S ) be the output of algorithm A 127 based on the dataset S .
Then its statistical generalization performance (prediction ability) is measuredby its excess population risk F ( A ( S ))
F ( w ⇤ ) , i.e., the discrepancy between the expected risks of 129 the model
A ( S ) and the best model w ⇤ 2 W .
We are interested in studying the excess population risk.
130 Let
E S, A [ · ] denote the expectation w.r.t.
both the randomness of data S and the internal randomness 131 of A .
To analyze the excess population error, we use the following error decomposition
[Equation]
The ﬁrst term is called the generalization error of the algorithm A measuring the difference between 133 the expected risk and empirical one, for which we will handle using stability analysis as shown soon.
The second term is the optimization error, which is induced by running the randomized algorithm
A 135 to minimize the empirical objective.
It can be estimated by tools from optimization theory.
As discussed in the introduction, many machine learning problems can be formulated as minimaxproblems including adversarial learning [28], robust optimization [13, 50], reinforcement learning [14, 19] and AUC maximization [27, 44, 71, 75].
We are also interested in solving this type of problem.
Let W and V be parameter spaces in R d .
Let D be a population distribution deﬁned on a sample 140 space Z , and f : W ⇥ V ⇥ Z !
We consider the following minimax optimization problems:
[Equation]
In practice, we only have a training dataset S
= { z 1 , . . .
, z n } independently drawn from D and hence 142 the minimax problem (3) is reduced to the following empirical version:
[Equation]
Since minimax problems involve the primal variable and dual variable, we have different measuresof generalization [36, 73].
For a randomized algorithm A ( S ) solving the problem (4), we denote 145 the output of A as A ( S ) =
( A w ( S ) , A v ( S )) for notation simplicity.
Let E [ · ] denote the expectation 146 w.r.t.
the randomness of both A and S .
We are particularly interested in the following two metrics.
Deﬁnition 1 (Weak Primal-Dual (PD) Risk) .
The weak Primal-Dual population risk of A ( S ) ,denoted by 4 w ( A w , A v ) , is deﬁned as max v 2 V E
[Equation]
⇤ .
The corresponding (expected) weak PD empirical risk, denoted by 4 w emp ( A w , A v ) , is deﬁned by max v 2 V E [Equation]
S
[Equation] A
[Equation] S
[Equation]
⇤ .
We refer to 4 w ( A w , A v ) 4 w emp ( A w , A v ) as the weak PD generalization error of the model ( A w ( S ) , A v ( S )) .
Deﬁnition 2 (Primal Risk) .
The primal population and empirical risks of A ( S ) are respectively 153 deﬁned by R ( A w ( S ))
=
max v 2 V F
( A w ( S ) , v ) , and R S ( A w ( S ))
=
max v 2 V F S
A w ( S ) , v ) .
We 154 refer to R ( A w ( S ))
R S ( A w ( S ))
as the primal generalization error of the model A w ( S ) , and 155 R (
A w ( S ))
min w 2 W R ( w ) as the excess primal population risk .
SGD and SGDA with Markov Sampling.
One often considers SGD to solve the ERM problem (1).
157 Speciﬁcally, let W ✓ R d be convex, Proj W ( · ) denote the projection to W , and @ f ( w ; z ) denote a 158 subgradient of f ( w ; z ) at w .
Let w2 W be an initial point, and { ⌘ t } is a stepsize sequence.
For 159 any t 2 N , the update rule of SGD is given by
[Equation]
where { i t } is generated from [ n ] = { 1 , 2 , . . . , n } with some sampling scheme.
A typically sampling 161 scheme is the uniform i.i.d.
sampling, i.e., i t is drawn randomly from [ n ] according to a uniformdistribution with/without replacement.
In this paper, we are particularly interested in the case when i t 2 [ n ] is drawn from a Markov Chain 164 which is widely used in practice [2, 3, 17, 21, 32, 61, 63, 64].
Let P be an n ⇥ n -matrix with 165 real-valued entries.
We say a Markov chain { X k } with ﬁnite state [ n ] and transition matrix P istime-homogeneous if, for k 2 N , i, j 2 [ n ] , and i 1 , . . .
, i k  1 2 [ n ] , there holds Pr ( X k [-+
-]+1 =
j | X
= i 1 , . . .
, X k = i )
=
Pr ( X k
+
=
j | X k = i )
=
[ P ] i,j .
Likewise, the SGDA algorithm with Markov sampling scheme is deﬁned as follows.
Speciﬁcally, let 168 @ w f and @ v f denote the subgradients of f w.r.t.
the arguments w and v , respectively.
We initialize 169 ( w 0 , v 0 ) 2 W ⇥ V , for any t 2 N , the update rule of SGDA is given by
[Equation]
where { i t } is drawn from [ n ] according to a Markov Chain.
For brevity, we refer to the above algorithms as Markov chain-SGD (MC-SGD) and Markov chainSGDA (SGDA), respectively.
There are two types of randomness in MC-SGD/MC-SGDA.
The ﬁrstrandomness is due to training dataset S which is i.i.d.
from the population distribution D .
The other 174 randomness arises from the internal randomness of the MC-SGD/MC-SGDA algorithm, i.e., therandomness of the indices { i t } , which is a Markov chain.
Remark 1.
Convergence analysis mainly considers the empirical optimization gap, i.e., the discrepancy between F S ( A ( S )) and F S ( w ⇤ ) .
Here, we are mainly interested in the generalization 178 error which measures the prediction ability of the trained model on the test (future) data.
As such, the purpose of this paper is to provide a comprehensive generalization analysis of MC-SGD and
MC-SGDA in the framework of statistical learning theory.
Speciﬁcally, given a ﬁnite training data S ,let A ( S ) be the output of the MC-SGD for solving the ERM problem (1).
Our target is to analyze 182 the excess population risk E S, A [ F ( A ( S ))]
F ( w ⇤ ) .
Let A ( S ) =
( A w ( S ) , A v ( S )) be the output 183 of MC-SGDA for solving the empirical minimax problem (4), our aim is to analyze the weak PDpopulation risk 4 w ( A w , A v ) and the excess primal population risk R ( A w ( S ))  min w 2 W R ( w ) .
In 185 both cases, the generalization analysis will be conducted using the algorithmic stability [10, 30, 35].
As we show soon below, the ﬁnal rates are obtained through trade-ofﬁng the optimization error(convergence rate) and the generalization error (stability results).
Properties of Markov Chain.
Denote the probability distribution of X k as the non-negative row 189 vector ⇡ k = ⇡ k (1) , ⇡ k (2) , . . . ,
⇡ k ( n )) , i.e., Pr ( X k = j ) =
⇡ k ( j ) .
Further, we have
[-=
 π-]P ni =1 ⇡ k ( i ) =1 .
For the time-homogeneous Markov chain, it holds ⇡ k = ⇡ k  1 P = · · · = ⇡ 1 P k  1 for all k 2 N . {+
+}
Here, ⇡ 1 is an initial distribution and P k denotes the k -th power of P .
A Markov chain is irreducibleif, for any i, j 2 [ n ] , there exists k such that [ P k ] i,j > 0 .
That is, the Markov process can go fromany state to any other state.
State i 2 [ n ] is said to have a period ⌧ if [ P k ] i,i = 0 whenever k is not amultiple of ⌧ and ⌧ is the greatest integer with this property.
If ⌧ = 1 for every state i 2 [ n ] , thenwe say the Markov chain is aperiodic.
We say a Markov chain with stationary distribution ⇧ ⇤ isreversible if ⇧ ⇤ ( i )
[ P ] i,j = ⇧ ⇤ ( j )
[ P ] j,i for all i, j 2 [ n ] .
We need the following assumption [17, 63] for studying optimization error of MC-SGMs.
Assumption 1.
Assume the Markov chain { i t } with ﬁnite state [ n ] is time-homogeneous, irreducible 199 and aperiodic.
It starts from an initial distribution ⇡ 1 , and has transition matrix P and stationarydistribution ⇧ ⇤ with ⇧ ⇤ ( i ) =
1 n for any i 2 [ n ] , i.e., lim k !
1 P k = 1 n 1 n 1 > n , where 1 n 2 R n is thevector with each entry being 1 and 1 > n denotes its transpose.
Results for Markov Chain SGD
In this section, we present the stability and generalization results of MC-SGD.
Our analysis requiresthe following deﬁnition and assumptions.
Let G, L > 0 and k · k 2 denote the Euclidean norm.
Deﬁnition 3.
We say f is convex w.r.t.
the ﬁrst argument if, for any z 2 Z and w , w 0 2 W , there 206 holds f ( w ; z ) [Equation]
Assumption 2.
Assume f is G -Lipschitz continuous, i.e., for any z 2 Z and w , w 0 2 W , there 208 holds | f ( w ; z )  f ( w 0 ; z ) |  G k w  w 0 k 2 .
Assumption 3.
Assume f is L -smooth, i.e., for any z 2 Z and w , w 0 2 W , there holds f ( w ; z ) f ( w 0 ; z )  h @ f ( w 0 ; z ) ,
w  w 0 i + L 2 k w  w 0 k 22 .
3.1 Stability and Generalization of MC-SGD
Let ¯ w T =
P Tj
[Equation]
j / P Tj =1 ⌘ j be the output of MC-SGD with T iterations, and w ⇤ =
arg min w 2 W F ( w ) be the best model in W .
We will use algorithmic stability to study the gen- 214 eralization errors, which measures the sensitivity of the output model of an algorithm.
Below we givethe deﬁnition of on-average argument stability [37].
Deﬁnition 4.
(On-average argument stability) Let S
=
{ z 1 , . . .
, z n } and e S =
{ ˜ z 1 , . . .
, ˜ z n } be drawnindependently from D .
For any i 2 [ n ] , deﬁne S ( i ) = { z 1 , . . .
, z i  1 , ˜ z
i , z i +1 , . . . , z n } as the set 218 formed from S by replacing the i -th element with ˜ z i .
We say a randomized algorithm A is on-average 219 ✏ -argument-stable if E S,
[Equation]
The connection between on-average argument stability for Lipschitz continuous losses and itsgeneralization error has been studied in [37].
Lemma 1 (Generalization via argument stability) .
Let S, e
S and S ( i ) be deﬁned as Deﬁnition 4.
If
A is on-average ✏ -argument-stable and Assumption 2 holds, then there holds
E S, A ⇥
F ( A ( S ))  F S ( A ( S )) ⇤  G ✏ . To obtain on-average argument stability bounds of MC-SGD, our idea is to ﬁrst write the stability asa deterministic function according to whether the different data point is selected, and then take theexpectation w.r.t.
the randomness of the algorithm.
The detailed proofs are given in Appendix B.1.
Theorem 2 (Stability bounds) .
Suppose f is convex and Assumption 2 holds.
Let W = R d and let
A be MC-SGD with T iterations.
(a) (Smooth case)
Suppose Assumption 3 holds and ⌘ j  2 /L .
Then A is on-average ✏ -argumentstable with ✏  2 Gn P
Tj =1 ⌘ j .(b)
(Non-smooth case)
A is on-average ✏ -argument-stable with ✏  2 G ⇣ P Tj =1 ⌘ 2 j {+⌘ 1 / 2 +
+} Gn P
Tj =1 ⌘ j .
Remark 2.
Without any assumption on Markov chain, Theorem 2 shows that argument stabilitybounds of MC-SGD are in the order of O ( T ⌘ /n ) and O ( p T ⌘ + T ⌘ /n ) with a constant stepsize ⌘ for 236 smooth and non-smooth losses, respectively.
Both of them match the corresponding bounds for SGDwith i.i.d.
sampling [5, 30, 37, 68], which imply that stability of MC-SGD is at least not worse thanthat of the i.i.d.
sampling case.
The technical novelty here is to observe that, in the sense of on-averageargument stability, we can use the calculation of E A [
P ni
=
I [ i t = i ] ] to replace that of E A [ I [ i t
= i ] ] , where I [ · ] is the indicator function.
This key step avoids the complicated calculations about E A
[ I [ i t = i ] ] .
241 Taking the uniform stability as example, we need to consider neighboring datasets differing by the i -thdata, and can get E A [ k w t  w 0 t k 2 ]
= O ( ⌘
[Equation]
[-O ( η (cid:80) t j =1 (cid:80) n k =
 [ P j − 1 ] k,i π 1 ( k ))-]) ,which depends on the transition matrix P and is not easy to control.
In contrast, with the on-averagestability we get stability bounds depending on
{+P ni =
+}
I [ i t = i ] , which is always 1 , i.e., the on-average 245 stability allows us to ignore the effect of sampling process.
The following theorem presents generalization bounds for MC-SGD in both smooth and non-smoothcases, which directly follows from Lemma 1 and Theorem 2.
Theorem 3 (Generalization error bounds) .
Suppose f is convex and Assumption 2 holds.
Let
W = R d and let A be MC-SGD with T iterations.(a)
[Equation] case)
Suppose Assumption 3 holds and let ⌘ j ⌘ ⌘  2 /L .
Then there holds [Equation]
(b) (Non-smooth case)
If ⌘ j ⌘ ⌘ , then there holds E S, A [ F [Equation] ¯ w T
[Equation]
3.2 Excess Population Risk of MC-SGD
In this subsection, we present excess population risk bounds for MC-SGD in both smooth andnon-smooth cases.
The proofs are given in Appendix B.3.
We use the notation B ⇣ ˜ B if there existuniversal constants c 1 , c 2 >
such that c 1 ˜ B  B  c 2 ˜ B .
Let  i ( P ) be the i -th largest eigenvalue 257 of the transition matrix P and  ( P ) = (max {|  [Equation] P
[Equation] | , |  n ( P [Equation] |
[Equation] + [Equation] Let K P be 258 the mixing time and C P be a constant depending on P and its Jordan canonical form (detailedexpressions are given in Lemma A.1 in the Appendix).
Let D be the diameter of W , and [Equation]
We introduce the following assumption to handle the extreme setting (i.e., the mixing time of Markovchain is too large) when estimating optimization errors (see a detailed explanation given in Remark B.1in Appendix B.2).
Assumption 4.
Assume the Markov chain { i t } is reversible with P = P > .
Theorem 4 (Excess population risk bounds for smooth losses) .
Suppose f is convex and Assumptions 265 1, 2, 3 hold.
Let
A be MC-SGD with T iterations, and { w j } Tj =1 be produced by A with ⌘ j ⌘ ⌘  2 /L .
Let k j be deﬁned in (7) . Then
[Equation] Further, suppose Assumption 4 holds.
If we select T ⇣ n and ⌘ = ( T log( T ))  1 / 2 , then
[Equation]
Theorem 5 (Excess population risk bounds for non-smooth losses) .
Suppose f is convex and 269 Assumptions 1, 2 hold.
Let
A be MC-SGD with T iterations, and { w j } Tj =1 be produced by A with ⌘ j ⌘ ⌘ . Let k j be deﬁned in (7) . Then
[Equation]
Further, suppose Assumption 4 holds.
If we select T ⇣ n 2 and ⌘ = T  3 / 4 , then E S, A [ F ( ¯ w T )]  F [Equation]
⇤
[Equation]
Remark 3.
To estimate the excess population risk, we need the convergence rates of MC-SGDfor convex problems which can be found in Appendix B.2.
The convergence proof follows from[63], who established a convergence rate of O (1 /T 1  q ) with some 1 / 2 < q < 1 .
Indeed, witha careful choice of ⌘ = 1 / p T log( T ) , a faster rate ˜ O (1 / p T ) can be obtained under the sameassumption. Here the notation ˜ O ( · ) means O ( · ) up to some logarithmic terms.
To understand 277 the variation of the algorithm, we present a conﬁdence-based bound for optimization error, whichmatches the bound in expectation up to a constant factor.
We also provide the convergence analysisfor non-convex problems in Appendix B.2 .
With a suitable choice of stepsize ⌘ j , we can show that
[Equation]
Remark 4.
Theorems 4 and 5 show, after carefully selecting the iteration number T and stepsize 282 ⌘ , that the excess population risk rate O (1 / p n ) is achieved in both smooth and non-smooth cases.
283 Note [5, 30, 37] show that the excess population risk rate O (1 / p n ) is optimal for the i.i.d.
sampling 284 case.
Therefore, our results for MC-SGD are also optimal since the i.i.d.
sampling is a special caseof Markov sampling.
Our results imply that despite the gradients are biased and dependent acrossiterations in Markov sampling, the generalization performance of SGD is competitive with the i.i.d.sampling case.
Theorems 4 and 5 also show the impact of the smoothness in achieving the optimalrate.
The rate for the non-smooth case in Theorem 5 looks slightly better than the smooth case(Theorem 4) with a logarithmic term.
However, the optimal rate can be achieved with a linear gradientcomplexity (i.e., the total number of computing the gradient) for smooth losses, while Theorem 5implies that gradient complexity O ( n 2 ) is required for non-smooth losses.
Remark 5.
According to Theorems 4 and 5, we can further observe how the transition matrix 293 P affects the excess population risks.
Indeed, the excess population risk rates are monotonicallyincreasing w.r.t  ( P ) .
Particularly, the closer  ( P ) is to 1 / 2 , the better the rate is.
Let us consider twoextreme examples.
Suppose the Markov chain starts from the uniform distribution and has transitionmatrix P = 1 n 1 n 1 Tn .
MC-SGD degenerates to SGD with i.i.d.
sampling in this case.
The excesspopulation risk rate O (1 / p n ) is obtained from Theorem 5 with  ( P ) = 1 / 2 .
For a Markov chainmoving on a circle (i.e., if the chain is currently at state i , then it goes to states i + 1 , i and i  1 withequal probability), we can verify that  ( P ) =
O (1  1 /n 2 ) , which implies a bad rate in this case.
Results for Markov Chain SGDA
In this section, we study the generalization analysis of MC-SGDA for minimax optimization problems.
Let ( ¯ w T , ¯ v T ) be the output of MC-SGDA with T iterations, where¯ w T =
P
Tj =1 ⌘ j w j
P
Tj =1 ⌘ j and ¯ v T = P
Tj =1 ⌘ j v j P Tj = [Equation]
We ﬁrst introduce some necessary deﬁnitions and assumptions.
Deﬁnition 5.
Let ⇢  0 and g : W ⇥ V 7!
R .
We say g is ⇢ -strongly-convex-strongly-concave 305 ( ⇢ -SC-SC) if, for any v 2 V , the function w 7! g ( w , v ) is ⇢ -strongly-convex and, for any w 2 W , 306 the function v 7! g ( w , v ) is ⇢ -strongly-concave.
We say g is convex-concave if g is 0 -SC-SC.
The following two assumptions are standard [23, 73].
Assumption 5 amounts to saying f is Lipschitzcontinuous w.r.t.
both w and v , while Assumption 6 considers smoothness conditions.
Assumption 5.
Assume for all w 2 W , v 2 V and z 2 Z , [Equation]
[Equation]
Assumption 6.
For any z , assume the function ( w , v ) 7! f ( w , v ; z ) is L -smooth, i.e., the following 312 inequality holds for all w [Equation] , v
[Equation]
and z
[Equation] [Equation]
4.1 Stability and Generalization Measures
We use algorithmic stability to study the generalization of minimax learners.
To this end, we ﬁrstintroduce the stability for minimax optimization problems.
Deﬁnition 6 (Argument stability for minimax problems) .
Let S, e S and S ( i ) be deﬁned as Deﬁnition 4.
Let A be a randomized algorithm and ✏ > 0 .
We say A is on-average ✏ -argument-stable for minimaxproblems if 1 n
[Equation]
The following theorem establishes a connection between stability and generalization.
Part (a) showsthat on-average argument stability implies generalization measured by the weak PD risk, while Part(b) shows that on-average argument stability guarantees a strong notion of generalization in terms ofthe primal risk under a strong concavity assumption.
Theorem 6 will be proved in Appendix C.1.
Theorem 6 (Generalization via argument stability) .
Let A be a randomized algorithm and ✏ > 0 .
(a) If A is on-average ✏ -argument-stable and Assumption 5 holds, then there holds 4 w ( A w , A v )  [Equation] ✏ .
[Equation] (b) If A is on-average ✏ -argument-stable, the function v 7! F ( w , v ) is ⇢ -strongly-concave andAssumptions 5, 6 hold, then we have E S, A
[-− R S ( A w ( S )) (cid:3) ≤ (cid:0) 
 + L/ρ (cid:1) Gϵ.-][Equation]
In the following theorem we develop stability bounds for MC-SGDA applied to convex-concaveproblems.
The proof is given in Section C.1 of the Appendix.
Theorem 7 (Stability bounds) .
Assume for all z , the function ( w , v ) 7! f ( w , v ; z ) is convex-concave.
Let W = R d
and Assumption 5 hold, and let A be MC -SGDA with T iterations.(a)
(Smooth case)
If Assumption 6 holds and P Tj =1 ⌘ 2 j  1 / (2 L 2 ) , then A is on-average ✏ -argumentstable with ✏  4 G  1 n P Tj =1 ⌘ 2 j  1 / 2 + 8 p 2 G n P Tj =1 ⌘ j .(b)
(Non-smooth case)
A is on-average ✏ -argument stable with ✏  2 p 2 G  P Tj =1 ⌘ 2 j  1 / 2 {++
 p+} G n P Tj =1 ⌘ j .
Remark 6.
For convex-concave and Lipschitz problems, the stability bound of the order O ( ⌘ ( p T + 337 T/n )) was established for SGDA with a constant stepsize under the uniformly i.i.d.
samplingsetting.
Under a further smoothness assumption, the stability bound was improved to the order of 339 O ( ⌘ T/n ) [36].
Our stability bounds in Theorem 7 match these results up to a constant factor andextend them to the Markov sampling case.
Remark 7.
Let { ( w ( i ) t , v ( i ) t ) } be the SGDA sequence based on S ( i ) .
The existing stability analysis [36] builds a recursive relationship for E A
[Equation] , which cruciallydepends on the i.i.d.
sampling property of i t 2 [ n ] .
This strategy does not apply to MC-SGDA since 344 the conditional expectation over i t is in a much complex manner due to the Markov Chain sampling.
We bypass this difﬁculty by building a recursive relationship for k w t  w ( i ) t k 22 + k v t  v ( i ) t k 22 in 346 terms of a sequence of random variables
I [ i t = i ] .
A key observation is that the effect of randomnesswould disappear if we consider on-average argument stability since
[Equation]
1 for any t 2 N .
We can combine the stability bounds in Theorem 7 and Theorem 6 to develop generalization bounds.
We ﬁrst establish weak PD risk bounds in Theorem 8, and then move on to primal population riskbounds in Theorem 9.
The proofs are given in Section C.1 of the Appendix.
Theorem 8 (Weak PD risk bounds) .
Suppose Assumption 5 holds.
Assume for all z , the function 352 ( w , v ) 7! f ( w , v ; z ) is convex-concave.
Let W = R d and { w j , v j } Tj =1 be produced by MC-SGDAwith ⌘ j ⌘ ⌘ .
Let A be deﬁned by A w ( S )
=
¯ w T and A v ( S ) =
¯ v T for ( ¯ w T , ¯ v T ) in (8) .
Denote 354 ✏ wgen := 4 w ( ¯ w T , ¯ v T )  4 w emp ( ¯ w T , ¯ v T ) .
(a) (Smooth case)
If Assumption 6 holds and P Tj =1 [-η 
-]⌘ 2 j  1 / (2 L 2 ) , then ✏ wgen 4 G 1 n P Tj =1 ⌘ 2 j  1 / 2 + 8 p 2 G 2 n P Tj =1 ⌘ j .
(b) (Non-smooth case)
The weak PD risk {+satisﬁes ✏ wgen  2 p 2 G P Tj =1 ⌘ 2 j  1 / 
+}
+ 2 p 2 G 2 n P Tj =1 ⌘ j .
Theorem 9 (Primal risk bounds) .
Suppose Assumption 5 holds.
Assume for all z , the function 359 ( w , v ) 7! f ( w , v ; z ) is convex-concave, and the function v 7! F ( w , v ) is ⇢ -strongly-concave.
Let W = R d and let { w j , v j } Tj =1 be produced by MC-SGDA with ⌘ j ⌘ ⌘ .
Let A be deﬁned by
A w ( S ) =
¯ w T and A v ( S ) =
¯ v T for ( ¯ w T , ¯ v T ) in (8) .
Denote ✏ pgen := E S, A ⇥ R ( ¯ w T )  R S ( ¯ w T ) ⇤ .(a)
(Smooth case)
If Assumption 6 holds and [Equation]  [Equation]
/ (2 L 2 ) , then ✏ pgen  [Equation] G 2 (1 + [Equation]
(b) (Non-smooth case)
The primal population risk satisﬁes ✏
[Equation]
4.2 Population Risks of MC-SGDA
Now we establish the population risk bounds for MC-SGDA.
The following theorem establishes theweak PD population risk of MC-SGDA for both smooth and non-smooth problems.
The proof for Theorem 10 is provided in Appendix C.3.
Theorem 10 (Weak PD population risk) .
Suppose Assumptions 1, 4 and 5 hold.
Assume for all z , thefunction ( w , v ) 7! f ( w , v ; z ) is convex-concave.
Let { w j , v j } Tj =1 be produced by MC-SGDA with⌘ j ⌘ ⌘ .
Let A be deﬁned by A w ( S )
=
¯ w T and A v ( S ) =
¯ v T for ( ¯ w T , ¯ v T ) in (8) .(a)
(Smooth case) Let Assumption 6 hold.
If T ⇣ n and ⌘ ⇣ ( T log( T ))  12 , then 4 w ( ¯ w T , ¯ v T ) =
O  log( n ) /  p n log(1 /  ( P ))  .(b)
(Non-smooth case)
If we select T ⇣ n 2 and ⌘ ⇣ T  34 , then we have 4 w ( ¯ w T , ¯ v T ) =
O  1 /  p n log(1 /  ( P ))  .
Remark 8.
The above excess population risk bounds are obtained through the trade-off betweenthe optimization errors (convergence analysis) and stability results of MC-SGDA.
The convergencerates ˜ O
(1 / p T ) of MC-SGDA for minimax problems in both expectation and high probability areprovided in Theorem C.3 and C.4 in Appendix C.2.
With gradient complexity O ( n ) , the minimax 381 optimal excess risk bound O (1 / p n ) for SGDA with uniform sampling for smooth problems was 382 established in [36].
We show SGDA with Markov sampling can achieve the nearly optimal boundwith the same gradient complexity.
For non-smooth problems, part (b) shows that the optimal excessrisk bound can be exactly achieved with the gradient complexity O ( n 2 ) .
Finally, we establish the following bounds for excess primal population risk under a strong concavitycondition on v 7! F ( w , v ) , which measures the performance of the primal variable.
The proof for 387 Theorem 11 is provided in Appendix C.3.
Theorem 11 (Excess primal population risk) .
Suppose Assumptions 1, 4, 5 and 6 hold.
Assume for all 389 z , the function ( w , v ) 7! f ( w , v ; z ) is convex-concave.
Assume v 7! F ( w , v ) is ⇢ -strongly-concave.
390 Let { w j , v j } Tj =1 be produced by MC-SGDA with ⌘ j ⌘ ⌘ .
Let A be deﬁned by A w ( S )
=
¯ w T and A v ( S ) =
¯ v T for ( ¯ w T , ¯ v T ) in (8) .
If we choose T ⇣ n, ⌘ ⇣ ( T log( T ))  1 / 2 , then
E
[Equation]
Remark 9.
We show MC-SGDA attains population risk bounds of the order ˜ O
(1 / p n ) with a lineargradient complexity O ( n ) , which are minimax optimal up to a logarithmic factor.
This implies that 394 considering sampling with a Markov chain does not weaken the learnability.
Theorems 10 and 11also show the effect of P on the population risk rates, i.e., the rates get better as  ( P ) decreases.
Conclusion
We develop the ﬁrst-ever-known stability and generalization analysis of Markov chain stochasticgradient methods for both minimization and minimax objectives.
In particular, we establish theoptimal excess population bounds O (1 / p n ) for MC-SGD for both smooth and non-smooth cases.
We also develop the ﬁrst nearly optimal convergence rates ˜
O (1 / p T ) for convex-concave problemsof MC-SGDA, and show that the optimal risk bounds O (1 / p n ) can be derived even in the non- 402 smooth case.
Although the gradients from Markov sampling are biased and not independent acrossthe iterations, we show the performance of MC-SGMs is competitive compared to SGMs with theclassical i.i.d.
sampling scheme.
An interesting direction is to consider other variants of SGMs withvariance reduction techniques and differentially private SGMs under the Markov sampling scheme.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[N/A](c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Figure]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Figure]
(b) Did you mention the license of the assets?
[Yes]
[Figure]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]
[Figure]
If you used crowdsourcing or conducted research with human subjects...
[Figure]
(a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
[N/A]
[Figure]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Figure]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]