Stability and Generalization for Markov Chain Stochastic Gradient Methods
Abstract
Recently there is a large amount of work devoted to the study of Markov chain stochastic gradient methods (MC-SGMs) which mainly focus on their convergence analysis for solving minimization problems.
In this paper, we provide a comprehensive generalization analysis of MC-SGMs for both minimization and minimax problems through the lens of algorithmic stability in the framework of statistical learning theory.
For empirical risk minimization (ERM) problems, we establish the optimal excess population risk bounds for both smooth and non-smooth cases by introducing on-average argument stability.
For minimax problems, we develop a quantitative connection between on-average argument stability and generalization error which extends the existing results for uniform stability [38].
We further develop the ﬁrst nearly optimal convergence rates for convex-concave problems both in expectation and with high probability, which, combined with our stability results, show that the optimal generalization bounds can be attained for both smooth and non-smooth cases.
To the best of our knowledge, this is the ﬁrst generalization analysis of SGMs when the gradients are sampled from a Markov process.
Introduction
Stochastic gradient methods (SGMs) have been the workhorse behind the success of many machine learning (ML) algorithms due to their simplicity and high efﬁciency.
As opposed to the deterministic (full) gradient methods, SGMs only require a small batch of random example(s) to update the model parameters at each iteration, making them amenable for solving large-scale problems.
There are mainly two notable types of SGMs which are inherent for different learning problems.
In particular, stochastic gradient descent (SGD) is widely used for solving the empirical risk minimization (ERM) problem and the theoretical convergence has been extensively studied [e.g., 5, 17, 21, 37, 42, 43, 53, 57, 73, 80, 82, 86].
Concomitantly, the minimax problems instantiate many ML problems such as Generative Adversarial Networks (GANs) [2, 30], AUC maximization [29, 45, 81], and algorithmic fairness [16, 41, 52, 51].
Stochastic gradient descent ascent (SGDA) is an off-the-shelf algorithm for solving minimax problems.
The convergence of SGDA and its variants is also widely studied in the literature [e.g., 44, 49, 54, 56].
On the other important front, the ultimate goal of learning is to achieve good generalization from the training data to the unknown test data.
Along this line, generalization analysis of SGMs has attracted considerable attention using the algorithmic stability approach [11, 24].
In particular, stability and generalization of SGD have been studied using the uniform argument stability [6, 7, 13, 32, 36]
The corresponding author is Yiming Ying.
and on-average stability [39, 40]. In [25, 38, 85], different stability and generalization measures are investigated for SGDA under both convex-concave and non-convex-non-concave settings.
A critical assumption in most of the above studies about SGD and SGDA is the i.i.d.
sampling scheme where the randomly sampled mini-batch or datum at each iteration is i.i.d.
drawn from the given training data, guaranteeing that the stochastic gradient is an unbiased estimator of the true gradient.
Markov chain naturally appears in many important problems, such as decentralized consensus optimization, which ﬁnds applications in various areas including wireless sensor networks, smart grid implementations and distributed statistical learning [4, 14, 23, 48, 50, 58, 61, 63] as well as pairwise learning [78] which instantiates AUC maximization [1, 29, 46, 81, 87] and metric learning [35, 75, 76, 79].
A common example is a distributed system in which each node stores a subset of the whole data, and one aims to train a global model based on these data.
We let a central node that stores all model parameters walk randomly over the system, in which case the samples are accessed according to a Markov chain.
Several works studied this kind of model [33, 34, 48, 50, 58].
Markov chains also arise extensively in thermodynamics, statistical mechanics dynamic systems and so on [59, 67].
In addition, it was observed in [69, 78] that SGD with Markov chain sampling (MC-SGD) performs more efﬁciently than SGD with the common i.i.d.
sampling scheme in various cases.
Hence, studying the performance of MC-SGMs has certain theoretical and application values.
The key difference from the i.i.d.
sampling scheme is that the stochastic gradient at each iteration is sampled on the trajectory of a Markov chain, in which the stochastic gradient estimators are neither unbiased nor independent.
Recent studies [4, 18, 22, 33, 34, 58, 68] overcame this technical hurdle and provided the convergence rates of MC-SGD.
However, to the best of our knowledge, there is no work on the generalization performance of SGMs with Markov sampling.
Main contribution: In this paper, we provide a comprehensive study of the stability and generaliza- tion for both SGD and SGDA with Markov sampling in the framework of statistical learning theory [72, 10].
Our main contribution can be summarized as follows.
• We develop stability and generalization results of MC-SGD for solving ERM problems in both smooth and non-smooth cases.
In particular, we show that MC-SGD can achieve competitive stability results as SGD with i.i
d. sampling scheme.
By trading off the generalization and optimization errors appropriately, we establish the ﬁrst-ever-known excess generalization bound O (1 / p n ) for MC-SGD where n is the size of training data.
The key idea for handling Markov sampling structure of MC-SGD is to use the concept of on-average argument stability. We ﬁrst establish the connection between on-average argument stability and generalization for minimax optimization algorithms, which extends the existing work on uniform argument stability [38].
We further develop stability bounds of SGDA with Markov sampling (MC-SGDA) for both smooth and non-smooth cases and obtain the nearly optimal convergence rates ˜
O (1 / p T ) for convex- concave problems in the form of both expectation and high probability, where T is the number of iterations, from which its optimal population risk bound is established.
Speciﬁcally, we consider several measures of generalization performance and show that the optimal population risk bounds O (1 / p n ) can be derived even in the non-smooth case. • To the best of our knowledge, this is the ﬁrst-ever-known work on stability and generalization of SGD and SGDA under the Markov chain setting.
Our results show that, despite the stochastic gradient estimator is biased and dependent across iterations due to the Markov sampling scheme, the generalization performance of MC-SGD and MC-SGDA enjoys the same optimal excess generalization rates as the i.i.d.
sampling setting.
Organization of the paper:
We discuss the related work in Subsection 1.1 and formulate the problem in Section 2.
Section 3 presents the stability and generalization results of MC-SGD for both smooth and non-smooth losses.
Section 4 develops the ﬁrst nearly optimal convergence rates for convex-concave problems of MC-SGDA, and show that the optimal risk bounds can be derived in both smooth and non-smooth cases.
Section 5 concludes the paper.
1.1 Related Work
In this subsection, we review some further works which are closely related to our paper.
Algorithmic Stability .
Algorithmic stability characterizes the sensitivity of a learning algorithm when the inputs to the algorithm are slightly perturbed.
The framework of algorithmic stability was established in a seminal paper [11] for the exact minimizer of the ERM problem, where the uniform stability was established for strongly convex objective functions.
Recent work [12, 27, 28] derived sharper generalization bounds for uniformly stable algorithms with high probability.
Several other stability measures were later developed for studying the generalization of different learning algorithms including the hypothesis stability [11], on-average stability [64], argument stability [47] and total variation stability [8, 71].
Stability and Generalization Analysis of SGMs.
[32] established generalization error bounds of order O (1 / p n ) in expectation for SGD for convex and smooth problems using uniform stability.
The on-average variance of stochastic gradients was used to reﬁne the generalization analysis of SGD for non-convex problems [88].
The results were improved and reﬁned by [36] using a data-dependent notion of algorithmic stability for SGD.
[39] introduced on-average argument stability and studied the stability and generalization of SGD for a general class of non-smooth convex losses, i.e., the gradient of the loss function is ↵ -Hölder continuous.
They also established fast generalization bounds O (1 /n ) for smooth convex losses in a low-noise setting.
The same authors also extended the analysis to the non-convex loss functions in [40].
Meanwhile, [6] addressed uniform argument stability of SGD with
Lipschitz-continuous convex losses.
Optimal generalization bounds were also developed for SGD in different settings [6, 7, 26, 74, 77].
Stability and generalization for SGMs have been studied for pairwise learning [65, 78] where the loss involves a pair of examples.
In particular, [78] introduced a simple MC-SGD algorithm for pairwise learning where pairs of examples form a special Markov chain { ⇠ t =
( z i t , z i t  1 ) : t 2 N } .
Here, z i t and z i t  1 are i.i.d.
sampled from the training data of size n at time t and t  1 , respectively.
The uniform argument stability and generalization have been established (see more discussion on the difference between our work and [78] in Remark 8 below).
For minimax problems, [85] studied the weak generalization and strong generalization bounds in the strongly-convex-concave setting.
[25] established the optimal generalization bounds for proximal point method, while gradient descent ascent (GDA) is not guaranteed to have a vanishing excess risk in convex-concave case.
[38] proved that SGDA can achieve the optimal excess risk bounds of order O (1 / p n ) for both smooth and non-smooth problems in the convex-concave setting.
They also extended their work to the nonconvex-nonconcave problems.
However, all the above studies the stability and generalization of SGD and SGDA under the assumption of the i.i.d.
sampling scheme.
Convergence Analysis of MC-SGMs.
The convergence analysis of SGD and its variants when the gradients are sampled from a Markov chain have been studied in different settings [3, 18, 19, 22, 34, 58, 66, 69, 70].
Speciﬁcally, [34, 58] studied the Markov subgradient incremental methods in a distributed system under time homogeneous and time non-homogeneous settings, respectively.
[22] studied the convergence of stochastic mirror descent under the ergodic assumption.
[69] established the convergence rate O (1 /T 1  q ) with some q 2 (1 / 2 , 1) for convex problems.
They also developed the convergence result for non-convex problems.
In addition, decentralized SGD methods with the gradients sampled from a non-reversible Markov chain have been studied in [68].
[18] considered an accelerated ergodic Markov chain SGD for both convex and non-convex problems.
[19] further studied the convergence rates without the bounded gradient assumption.
All these studies focused on the convergence analysis of MC-SGD for solving the ERM problems.
Problem Setting and Target of Analysis
In this section, we introduce the SGD for ERM and SGDA for solving minimax problems with
Markov Chain, and describe the target of generalization analysis for both optimization algorithms.
Target of Generalization Analysis.
Let W be a parameter space in R d and D be a population distribution deﬁned on a sample space Z .
Let f : W ⇥ Z ! [0 , 1 ) be a loss function.
In the standard framework of Statistical Learning Theory (SLT) [10, 72], one aims to minimize the expected population risk, i.e., F ( w ) :=
E z [ f ( w ; z )] , where the model parameter w belongs to W , and the expectation is taken with respect to (w.r.t.)
z according to D .
However, the population distribution is often unknown.
Instead, we have access to a training dataset S =
{ z i 2 Z} ni =1 with size n , where z i is independently drawn from D .
Then consider the following ERM problem
[Equation]
For a randomized algorithm A to solve the above problem, let A ( S ) be the output of algorithm A based on the dataset S .
Then its statistical generalization performance (prediction ability) is measured by its excess population risk F ( A ( S ))
F ( w ⇤ ) , i.e., the discrepancy between the expected risks of the model
A ( S ) and the best model w ⇤ 2 W .
We are interested in studying the excess population risk.
Let E S, A [ · ] denote the expectation w.r.t.
both the randomness of data S and the internal randomness of A .
To analyze the excess population error, we use the following error decomposition
E
[Equation]
The ﬁrst term is called the generalization error of the algorithm A measuring the difference between the expected risk and empirical one, for which we will handle using stability analysis as shown soon.
The second term is the optimization error, which is induced by running the randomized algorithm A to minimize the empirical objective.
It can be estimated by tools from optimization theory.
As discussed in the introduction, many machine learning problems can be formulated as minimax problems including adversarial learning [30], reinforcement learning [15, 20] and AUC maximization [29, 46, 81, 87].
We are also interested in solving this type of problem.
Let
W and V be parameter spaces in R d .
Let D be a population distribution deﬁned on a sample space Z , and f : W ⇥ V ⇥ Z !
We consider the minimax optimization problems:
[Equation]
= E z
[Equation]
[Equation] , v ; z )] } .
In practice, we only have a training dataset
S = { z 1 , . . .
, z n } independently drawn from D and hence the minimax problem is reduced to the following empirical version:
[Equation]
Since minimax problems involve the primal variable and dual variable, we have different measures of generalization [38, 85].
For a randomized algorithm A ( S ) solving the problem (3), we denote the output of A as A ( S ) =
( A w ( S ) , A v ( S )) for notation simplicity.
Let E [ · ] denote the expectation w.r.t.
the randomness of both A and S .
We are particularly interested in the following two metrics.
Deﬁnition 1 (Weak Primal-Dual (PD) Risk) .
The weak Primal-Dual population risk of A ( S ) , denoted by 4 w ( A w , A v ) , is deﬁned as max v 2 V E ⇥
F ( A w ( S ) , v ) ⇤  min w 2 W E ⇥ F ( w , A v ( S ))
⇤ .
The corresponding (expected) weak PD empirical risk, denoted by 4 w emp ( A w , A v ) , is deﬁned by max v 2 V E ⇥
F S ( A w ( S ) , v ) ⇤  min w 2 W E ⇥ F S ( w , A v ( S ))
⇤ .
We refer to 4 w ( A w , A v )  4 w emp ( A w , A v ) as the weak PD generalization error of the model ( A w ( S ) , A v ( S )) .
Deﬁnition 2 (Primal Risk) .
The primal population and empirical risks of A ( S ) are respectively deﬁned by R ( A w ( S ))
=
max v 2 V F
( A w ( S ) , v ) , and R S ( A w ( S ))
=
max v 2 V F S
A w ( S ) , v ) .
We refer to R ( A w ( S ))
R S ( A w ( S ))
as the primal generalization error of the model A w ( S ) , and R (
A w ( S ))
min w 2 W R ( w ) as the excess primal population risk .
SGD and SGDA with Markov Sampling.
One often considers SGD to solve the ERM problem (1).
Speciﬁcally, let W ✓ R d be convex, Proj W ( · ) denote the projection to W , and @ f ( w ; z ) denote a subgradient of f ( w ; z ) at w .
Let w2 W be an initial point, and { ⌘ t } is a stepsize sequence.
For any t 2 N , the update rule of SGD is given by
[Equation]
where { i t } is generated from [ n ] = { 1 , 2 , . . . , n } with some sampling scheme.
A typically sampling scheme is the uniform i.i.d.
sampling, i.e., i t is drawn randomly from [ n ] according to a uniform distribution with/without replacement.
In this paper, we are particularly interested in the case when i t 2 [ n ] is drawn from a Markov Chain which is widely used in practice [3, 4, 18, 22, 34, 66, 69, 70].
Let P be an n ⇥ n -matrix with real-valued entries.
We say a Markov chain { X k } with ﬁnite state [ n ] and transition matrix P is time-homogeneous if, for k 2 N , i, j 2 [ n ] , and i 1 , . . .
, i k  1 2 [ n ] , there holds Pr ( X k +
=
j | X
= i 1 , . . .
, X k = i )
=
Pr ( X k
+
=
j | X k = i )
=
[ P ] i,j .
Likewise, the SGDA algorithm with Markov sampling scheme is deﬁned as follows.
Speciﬁcally, let @ w f and @ v f denote the subgradients of f w.r.t.
the arguments w and v , respectively.
We initialize ( w 0 , v 0 ) 2 W ⇥ V , for any t 2 N , let { i t } is drawn from [ n ] according to a Markov Chain.
The update rule of SGDA is given by
[Equation]
For brevity, we refer to the above algorithms as Markov chain-SGD (MC-SGD) and Markov chainSGDA (SGDA), respectively.
There are two types of randomness in MC-SGD/MC-SGDA.
The ﬁrst randomness is due to training dataset S which is i.i.d.
from the population distribution D .
The other randomness arises from the internal randomness of the MC-SGD/MC-SGDA algorithm, i.e., the randomness of the indices { i t } , which is a Markov chain.
Remark 1.
Convergence analysis mainly considers the empirical optimization gap, i.e., the dis- crepancy between F S ( A ( S )) and F S ( w ⇤ ) .
Here, we are mainly interested in the generalization error which measures the prediction ability of the trained model on the test (future) data.
As such, the purpose of this paper is to provide a comprehensive generalization analysis of MC-SGD and
MC-SGDA in the framework of statistical learning theory.
Speciﬁcally, given a ﬁnite training data S , let A ( S ) be the output of the MC-SGD for solving the ERM problem (1).
Our target is to analyze the excess population risk E S, A [ F ( A ( S ))]
F ( w ⇤ ) .
Let A ( S ) =
( A w ( S ) , A v ( S )) be the output of MC-SGDA for solving the empirical minimax problem (3), our aim is to analyze the weak PD population risk 4 w ( A w , A v ) and the excess primal population risk R ( A w ( S ))
min w 2 W R ( w ) .
In both cases, the generalization analysis will be conducted using the algorithmic stability [11, 32].
As we show soon below, the ﬁnal rates are obtained through trade-ofﬁng the optimization error (convergence rate) and the generalization error (stability results).
Properties of Markov Chain.
Denote the probability distribution of X k as the non-negative row vector ⇡ k = ( ⇡ k (1) , ⇡ k (2) , . .
⇡ k ( n )) , i.e., Pr ( X k = j ) =
⇡ k ( j ) .
Further, we have
[-=
 π-]P ni =1 ⇡ k ( i ) = 1 .
For the time-homogeneous Markov chain, it holds ⇡ k = ⇡ k  1 P = · · · = ⇡ 1 P k  1 for all k 2 N .
Here, ⇡ 1 is an initial distribution and P k denotes the k -th power of P .
A Markov chain is irreducible if, for any i, j 2 [ n ] , there exists k such that [ P k ] i,j > 0 .
That is, the Markov process can go from any state to any other state.
State i 2 [ n ] is said to have a period ⌧ if [ P k ] i,i = 0 whenever k is not a multiple of ⌧ and ⌧ is the greatest integer with this property.
If ⌧ = 1 for every state i 2 [ n ] , then we say the Markov chain is aperiodic.
We say a Markov chain with stationary distribution ⇧ ⇤ is reversible if ⇧ ⇤ ( i )[ P ] i,j = ⇧ ⇤ ( j )[ P ] j,i for all i, j 2 [ n ] .
We need the following assumption for studying optimization error of MC-SGMs.
Assumption 1.
Assume the Markov chain { i t } with ﬁnite state [ n ] is time-homogeneous, irreducible and aperiodic.
It starts from an initial distribution ⇡ 1 , and has transition matrix P and stationary distribution ⇧ ⇤ with ⇧ ⇤ ( i ) =
1 n for any i 2 [ n ] , i.e., lim k ! 1 P k = 1 n 1 n 1 > n , where 1 n 2 R n is the vector with each entry being 1 and 1 > n denotes its transpose.
Remark 2.
Our assumptions on Markov chains listed above are standard in the literature [18, 34, 50, 69, 68, 78].
For instance, Markov chain-type SGD was proposed for pairwise learning which can apply to various learning task such as AUC maximization and bipartite ranking [1, 83, 87, 29, 46] and metric learning [35, 75, 76, 79].
This pairwise learning algorithm forms pairs of examples following a special Markov chain { ⇠ t =
( z i t , z i t  1 ) : t 2 N } where z i t and z i t  1 are i.i.d.
sampled from the training data of size n at time t and t  1 , respectively and, at time t , the model parameter is updated using gradient descent based on ⇠ t .
As mentioned in Remark 3 of [78], { ⇠ t : t 2 N } is a Markov Chain satisfying all of our assumptions.
Another notable example is the decentralized consensus optimization in a multi-agent network, where the samples are accessed according to a Markov chain and the number of states of the Markov chain equals the number of nodes in the network, which is ﬁnite.
One always considers the same transition matrix P for each node and assumes the Markov chain is irreducible and aperiodic [50, 84].
Results for Markov Chain SGD
In this section, we present the stability and generalization results of MC-SGD.
Our analysis requires the following deﬁnition and assumptions.
Let G, L > 0 and k · k 2 denote the Euclidean norm.
Deﬁnition 3.
We say f is convex w.r.t.
the ﬁrst argument if, for any z 2 Z and w , w 0 2 W , there holds f ( w ; z ) [Equation]
Assumption 2.
Assume f is G -Lipschitz continuous, i.e., for any z 2 Z and w , w 0 2 W , there holds | f ( w ; z )  f ( w 0 ; z ) |  G k w  w 0 k 2 .
Assumption 3.
Assume f is L -smooth, i.e., for any z 2 Z and w , w 0 2 W , there holds f ( w ; z )  f ( w 0 ; z )  h @ f ( w 0 ; z ) , w  w 0 i + L 2 k w  w 0 k 22 .
3.1 Stability and Generalization of MC-SGD
Let w ⇤ =
arg min w 2 W F ( w ) be the best model in W and ¯ w T
= P
Tj =1 ⌘ j w j / P
Tj =1 ⌘ j be the output of MC-SGD with T iterations.
We will use algorithmic stability to study the generalization errors, which measures the sensitivity of the output model of an algorithm.
Below we give the deﬁnition of on-average argument stability [39].
Deﬁnition 4.
(On-average argument stability) Let S
=
{ z 1 , . . .
, z n } and e S =
{ ˜ z 1 , . . .
, ˜ z n } be drawn independently from D .
For any i 2 [ n ] , deﬁne S ( i ) = { z 1 , . . .
, z i  1 , ˜ z
i , z i +1 , . . . , z n } as the set formed from S by replacing the i -th element with ˜ z i .
We say a randomized algorithm A is on-average ✏ -argument-stable if E S, e
[Equation]
To obtain on-average argument stability bounds of MC-SGD, our idea is to ﬁrst write the stability as a deterministic function according to whether the different data point is selected, and then take the expectation w.r.t.
the randomness of the algorithm.
The detailed proofs are given in Appendix B.1.
Theorem 1 (Stability bounds) .
Suppose f is convex and Assumption 2 holds.
Let W = R d and let
A be MC-SGD with T iterations.
(a) (Smooth case)
Suppose Assumption 3 holds and ⌘ j  2 /L .
Then A is on-average ✏ -argumentstable with ✏  2 Gn [Equation]
1 ⌘ j . (b) (Non-smooth case)
A is on-average ✏ -argument-stable with ✏  2 G
[Equation]
Remark 3.
Without any assumption on Markov chain, Theorem 1 shows that argument stability bounds of MC-SGD are in the order of O ( T ⌘ /n ) and O ( p T ⌘ + T ⌘ /n ) with a constant stepsize ⌘ for smooth and non-smooth losses, respectively.
Both of them match the corresponding bounds for SGD with i.i.d.
sampling [6, 32, 39, 74], which imply that stability of MC-SGD is at least not worse than that of the i.i.d.
sampling case.
The technical novelty here is to observe that, in the sense of on-average argument stability, we can use the calculation of E A [
P ni
=
I [ i t = i ] ] to replace that of E A [ I [ i t
= i ] ] , where I [ · ] is the indicator function.
This key step avoids the complicated calculations about E A
[ I [ i t = i ] ] .
Taking the uniform stability as example, we need to consider neighboring datasets differing by the i -th data, and can get E A [ k w t  w 0 t k 2 ]
= O ( ⌘ P tj =1 E A
[ I i j
= i ])
=
O ( ⌘ P tj =
P nk =
[ P j  1 ] k,i ⇡ 1 ( k )) , which depends on the transition matrix P and is not easy to control.
In contrast, with the on-average stability we get stability bounds depending on
{+P ni =
+}
I [ i t = i ] , which is always 1 , i.e., the on-average stability allows us to ignore the effect of sampling process.
The following theorem presents generalization bounds for MC-SGD in both smooth and non-smooth cases, which directly follows from Lemma A.4 and Theorem 1.
Theorem 2 (Generalization error bounds) .
Suppose f is convex and Assumption 2 holds.
Let
W = R d and let A be MC-SGD with T iterations.
(a) (Smooth case)
Suppose Assumption 3 holds and let ⌘ j ⌘ ⌘  L
Then there holds E [Equation]
(b) (Non-smooth case)
If ⌘ j ⌘ ⌘ , then there holds E S, A [ F [Equation] ¯ w T
[Equation]
F S [Equation]
3.2 Excess Population Risk of MC-SGD
In this subsection, we present excess population risk bounds for MC-SGD in both smooth and non-smooth cases.
The proofs are given in Appendix B.3.
We use the notation B ⇣ ˜ B if there exist universal constants c 1 , c 2 > [-
-]0 such that c 1 ˜ B  B  c 2 ˜ B .
Let  i ( P ) be the i -th largest eigenvalue of the transition matrix P and  ( P ) =
(max {|  2 ( P ) | , |  n ( P ) |} + 1) / 2 2 [1 / 2 , 1) .
Let K P be the mixing time and C P be a constant depending on P and its Jordan canonical form (detailed expressions are given in Lemma A.1).
We assume sup z 2 Z f (0; z ) and k w ⇤ k 2 are bounded.
Assumption 4.
Assume the Markov chain { i t } is reversible with P = P > .
Theorem 3 (Excess population risk for smooth losses) .
Suppose f is convex and Assumptions 1, 2, 3 and 4 hold.
Let W 2 R d .
Let A be MC-SGD with T iterations, and { w j } Tj =1 be produced by A with w 0 = 0 and ⌘ j ⌘ ⌘  2 /L .
If we select T ⇣ n and ⌘ = ( T log( T ))  1 / 2 , then
E S, A [ F ( ¯ w T )]  F ( w ⇤ ) =
O p log( n ) / ( p n log(1 /  ( P )))  .
Remark 4.
A term K P /  M 0 n 34 log 14 ( n )  with M 0 = min { p n log( n ) C P n  ( P ) K P , 1 } appears in the excess risk bound of MC-SGD (see the proof of Theorem 3), which will be worse than p log( n ) / p n when K P is large.
Note Lemma A.1 implies that this term will disappear if P is symmetric.
Hence, we introduce Assumption 4 to get the nearly optimal rate.
Theorem 4 (Excess population risk for non-smooth losses) .
Suppose f is convex and Assumptions 1, 2, 4 hold.
Let W 2 R d .
Let A be MC-SGD with T iterations, and { w j } Tj =1 be produced by A with ⌘ j ⌘ ⌘ .
If we select T ⇣ n 2 and ⌘ = T  3 / 4 , then E S, A [ F ( ¯ w T )]  F ( w ⇤ )
= O  1 /  p n log(1 /  ( P ))  .
Remark 5.
To estimate the excess population risk, we need the convergence rates of MC-SGD which can be found in Appendix B.2.
[69] established a convergence rate of O (1 /T 1  q ) with some q 2 (1 / 2 , 1) under the bounded parameter domain assumption. {+We remove this assumption by showing k w t k 
 =+}
O (
P Tk =1 ⌘ k ) and obtain the nearly optimal convergence rate ˜ O (1 / p T ) with a careful choice of ⌘ = 1 / p T log( T ) .
To understand the variation of the algorithm, we present a conﬁdence-based bound for optimization error, which matches the bound in expectation up to a constant factor.
We also provide the convergence analysis for non-convex problems in Appendix B.2.
Remark 6.
Theorems 3 and 4 show, after carefully selecting the iteration number T and stepsize ⌘ , that the excess population risk rate O (1 / p n ) is achieved in both smooth and non-smooth cases.
Note [6, 32, 39] show that the excess population risk rate O (1 / p n ) is optimal for the i.i.d.
sampling case.
Therefore, our results for MC-SGD are also optimal since the i.i.d.
sampling is a special case of Markov sampling.
Our results imply that despite the gradients are biased and dependent across iterations in Markov sampling, the generalization performance of SGD is competitive with the i.i.d. sampling case.
Theorems 3 and 4 also show the impact of the smoothness in achieving the optimal rate.
The rate for the non-smooth case in Theorem 4 looks slightly better than the smooth case (Theorem 3) with a logarithmic term.
However, the optimal rate can be achieved with a linear gradient complexity (i.e., the total number of computing the gradient) for smooth losses, while Theoremimplies that gradient complexity O ( n 2 ) is required for non-smooth losses.
Remark 7.
According to Theorems 3 and 4, we can further observe how the transition matrix P affects the excess population risks.
Indeed, the excess population risk rates are monotonically increasing w.r.t  ( P ) .
Particularly, the closer  ( P ) is to 1 / 2 , the better the rate is.
Let us consider two extreme examples.
Suppose the Markov chain starts from the uniform distribution and has transition matrix P = 1 n 1 n 1 Tn .
MC-SGD degenerates to SGD with i.i.d.
sampling in this case.
The excess population risk rate O (1 / p n ) is obtained from Theorem 4 with  ( P ) = 1 / 2 .
For a Markov chain moving on a circle (i.e., if the chain is currently at state i , then it goes to states i + 1 , i and i  1 with equal probability), we can verify that  ( P ) =
O (1  1 /n 2 ) , which implies a bad rate in this case.
Remark 8.
[78] proposed a simple MC-SGD algorithm for pairwise learning associated with a pairwise loss f ( w , z, z 0 ) .
Speciﬁcally, at iteration t , the algorithm update the model parameter as follows: w t = w t  1  ⌘ t r w f ( w t  1 , z i t , z i t  1 ) where z i t and z i t  1 are i.i.d.
sampled from the training data of size n at time t and t  1 , respectively.
In Remark 3 of [78], it was shown that { ⇠ t = ( i t , i t  1 ) 2 [ n ] ⇥ [ n ] } does form a time-homogeneous, irreducible and aperiodic Markov chain.
There are two key differences between our work and [78].
Firstly, the work [78] used uniform stability directly due to E A
[ I i t
= i ]
= 1 /n , while this term is not easy to control in our general setting (see Remark 3 for details).
To overcome this hurdle, we resort to the on-average stability and show that MC-SGD achieves the optimal excess risk rate.
Secondly, the proofs there critically rely on the fact f ( w t  1 ; z i t , z i t  1 ) =
f ( w t  2 ; z i t , z i t  1 ) + O ( ⌘ t  1 ) and the independence of w t  2 w.r.t.
i t and i t  1 .
However, these specially tailored techniques for pairwise learning do not apply to the general Markov setting as we considered here.
Results for Markov Chain SGDA
In this section, we study the generalization analysis of MC-SGDA for minimax optimization problems.
Let ( ¯ w T , ¯ v T ) be the output of MC-SGDA with T iterations, where
[Equation]
We ﬁrst introduce some necessary deﬁnitions and assumptions.
Deﬁnition 5.
Let ⇢  0 and g : W ⇥ V 7!
R .
We say g is ⇢ -strongly-convex-strongly-concave ( ⇢ -SC-SC) if, for any v 2 V , the function w 7! g ( w , v ) is ⇢ -strongly-convex and, for any w 2 W , the function v 7! g ( w , v ) is ⇢ -strongly-concave.
We say g is convex-concave if g is 0 -SC-SC.
The following two assumptions are standard [25, 85].
Assumption 5 amounts to saying f is Lipschitz continuous w.r.t.
both w and v , while Assumption 6 considers smoothness conditions.
Assumption 5.
Assume for all w 2 W , v 2 V and z 2 Z ,  @ w f ( w , v ; z )  2  G and  [Equation] f ( w [Equation]
[Equation] G [Equation]
Assumption 6.
For any z , assume the function ( w , v ) 7! f ( w , v ; z ) is L -smooth, i.e., the following inequality holds for all w [Equation] V and z
[Equation]
Z
[Equation]
4.1 Stability and Generalization Measures
We use algorithmic stability to study the generalization of minimax learners.
To this end, we ﬁrst introduce the stability for minimax optimization problems.
Deﬁnition 6 (Argument stability for minimax problems) .
Let S, e S and S ( i ) be deﬁned as Deﬁnition 4.
Let A be a randomized algorithm and ✏ > 0 .
We say A is on-average ✏ -argument-stable for minimax problems if 1 n
[Equation]
The following theorem establishes a connection between stability and generalization.
Part (a) shows that on-average argument stability implies generalization measured by the weak PD risk, while Part (b) shows that on-average argument stability guarantees a strong notion of generalization in terms of the primal risk under a strong concavity assumption.
Theorem 5 will be proved in Appendix C.1.
Theorem 5 (Generalization via argument stability) .
Let A be a randomized algorithm and ✏ > 0 .
(a) If A is on-average ✏ -argument-stable and Assumption 5 holds, then there holds 4 w ( A w , A v )  4 w emp ( A w , A v )  G ✏ .
(b) If A is on-average ✏ -argument-stable, the function v 7! F ( w , v ) is ⇢ -strongly-concave and
Assumptions 5, 6 hold, then we have E S, A ⇥
R ( A w ( S ))  R S ( A w ( S ))[-(cid:3) ≤ (cid:0) 
-]
⇤   1 + L/ ⇢  G ✏ .
In the following theorem we develop stability bounds for MC-SGDA applied to convex-concave problems.
The proof is given in Section C.1 of the Appendix.
Theorem 6 (Stability bounds) .
Assume for all z , the function ( w , v ) 7! f ( w , v ; z ) is convex-concave.
Let W = R d
and Assumption 5 hold, and let A be MC-SGDA with T iterations.
(a) (Smooth case)
If Assumption 6 holds and
[Equation]
2 L [Equation] , then A is on-average ✏ -argument stable with ✏  4 G
[Equation]
(b) (Non-smooth case)
A is on-average ✏ -argument stable with ✏  2 G
[Equation]
Remark 9.
For convex-concave and Lipschitz problems, the stability bound of the order O ( ⌘ ( p T + T/n )) was established for SGDA with a constant stepsize under the uniformly i.i.d.
sampling setting.
Under a further smoothness assumption, the stability bound was improved to the order of O ( ⌘ T/n ) [38].
Our stability bounds in Theorem 6 match these results up to a constant factor and extend them to the Markov sampling case.
Remark 10.
Let { ( w ( i ) t , v ( i ) t ) } be the SGDA sequence based on S ( i ) .
The existing stability analysis [38] builds a recursive relationship for E A
[Equation] , which crucially depends on the i.i.d.
sampling property of i t 2 [ n ] .
This strategy does not apply to MC-SGDA since the conditional expectation over i t is in a much complex manner due to the Markov Chain sampling.
We bypass this difﬁculty by building a recursive relationship for k w t  w ( i ) t k 22 + k v t  v ( i ) t k 22 in terms of a sequence of random variables
I [ i t = i ] .
A key observation is that the effect of randomness would disappear if we consider on-average argument stability since
P [Equation]
for any [Equation]
We can combine the stability bounds in Theorem 6 and Theorem 5 to develop generalization bounds.
We ﬁrst establish weak PD risk bounds in Theorem 7, and then move on to primal population risk bounds in Theorem 8.
The proofs are given in Section C.1 of the Appendix.
Theorem 7 (Weak PD risk bounds) .
Suppose Assumption 5 holds.
Assume for all z , the function ( w , v ) 7! f ( w , v ; z ) is convex-concave.
Let W = R d and { w j , v j } Tj =1 be produced by MC-SGDA with ⌘ j ⌘ ⌘ .
Let A be deﬁned by A w ( S )
=
¯ w T and A v ( S ) =
¯ v T for ( ¯ w T , ¯ v T ) in (6) .
Denote ✏ w gen := 4 w ( ¯ w T , ¯ v T )  4 w emp ( ¯ w T , ¯ v T ) .
(a) (Smooth case)
If Assumption 6 holds and[-(cid:80) Tj =1 η 
-]
P
Tj =1 ⌘ 2 j  1 / ( L , then ✏ wgen
 4 G  1 n
P Tj =1 ⌘ 2 j  1 / 2 + 8 p 2 G 2 n P Tj =1 ⌘ j .
(b) (Non-smooth case)
The weak PD risk satisﬁes
wgen  2 p 2 G P
Tj =1 ⌘ 2 j  1 / 2 +
[Equation]
p
[Equation]
G 2 n
P
Tj =1 ⌘ j .
Theorem 8 (Primal risk bounds) .
Suppose Assumption 5 holds.
Assume for all z , the function ( w , v ) 7! f ( w , v ; z ) is convex-concave, and the function v 7! F ( w , v ) is ⇢ -strongly-concave.
Let W = R d and let { w j , v j } Tj =1 be produced by MC-SGDA with ⌘ j ⌘ ⌘ .
Let A be deﬁned by
A w ( S ) =
¯ w T and A v ( S ) =
¯ v T for ( ¯ w T , ¯ v T ) in (6) .
Denote ✏ pgen := E S, A ⇥ R ( ¯ w T )  R S ( ¯ w T ) ⇤ .
(a) (Smooth case)
If Assumption 6 holds and P Tj =1 ⌘ 2 j  1 / (2 L 2 ) , then ✏ pgen  4 G 2 (1 + L/ ⇢ )
1 n
P
Tj =1 ⌘ 2 j  12 + 2 p 2 n P Tj =1 ⌘ j  . (b
) (Non-smooth case)
The primal population risk satisﬁes ✏
pgen  2 p 2 G 2 (1 + L/ ⇢ )
P Tj =1 ⌘ 2 j  1 / 2 + 2 n P Tj =1 ⌘ j  .
4.2 Population Risks of MC-SGDA
Now we establish the population risk bounds for MC-SGDA.
The following theorem establishes the weak PD population risk of MC-SGDA for both smooth and non-smooth problems.
Let D w and D v be the diameters of W and V .
The proof for Theorem 9 is provided in Appendix C.3.
Theorem 9 (Weak PD population risk) .
Suppose Assumptions 1, 4 and 5 hold.
Assume for all z , the function ( w , v ) 7! f ( w , v ; z ) is convex-concave.
Let { w j , v j } Tj =1 be produced by MC-SGDA with ⌘ j ⌘ ⌘ .
Let A be deﬁned by A w ( S )
=
¯ w T and A v ( S ) =
¯ v T for ( ¯ w T , ¯ v T ) in (6) .
(a) (Smooth case) Let Assumption 6 hold.
If T ⇣ n and ⌘ ⇣ ( T log( T ))  1 2 , then 4 w ( ¯ w T , ¯ v T ) =
O  log( n ) /  p n log(1 /  ( P ))  .
(b) (Non-smooth case)
If we select T ⇣ n 2 and ⌘ ⇣ T  3 4 , then we have 4 w ( ¯ w T , ¯ v T ) =
O  1 /  p n log(1 /  ( P ))  .
Remark 11.
The above excess population risk bounds are obtained through the trade-off between the optimization errors (convergence analysis) and stability results of MC-SGDA.
The convergence rates ˜ O
(1 / p T ) of MC-SGDA for minimax problems in both expectation and high probability are provided in Theorem C.3 and C.4 in Appendix C.2.
With gradient complexity O ( n ) , the minimax optimal excess risk bound O (1 / p n ) for SGDA with uniform sampling for smooth problems was established in [38].
We show SGDA with Markov sampling can achieve the nearly optimal bound with the same gradient complexity.
For non-smooth problems, part (b) shows that the optimal excess risk bound can be exactly achieved with the gradient complexity O ( n 2 ) .
Finally, we establish the following bounds for excess primal population risk under a strong concavity condition on v 7! F ( w , v ) , which measures the performance of the primal variable.
The proof for Theorem 10 is provided in Appendix C.3.
Theorem 10 (Excess primal population risk) .
Suppose Assumptions 1, 4, 5 and 6 hold.
Assume for all z , the function ( w , v ) 7! f ( w , v ; z ) is convex-concave.
Assume v 7! F ( w , v ) is ⇢ -strongly- concave.
Let { w j , v j } Tj =1 be produced by MC-SGDA with ⌘ j ⌘ ⌘ .
Let A be deﬁned by A w ( S )
=
¯ w T and A v ( S ) =
¯ v T for ( ¯ w T , ¯ v T ) in (6) .
If we choose T ⇣ n, ⌘ ⇣ ( T log( T ))  1 / 2 , then
E S, A
[ R ( ¯ w T )]  min w 2 W R ( w ) =
O  ( L/ ⇢ ) p log( n ) / ( p n log(1 /  ( P )))  .
Remark 12.
We show MC-SGDA attains population risk bounds of the order ˜ O
(1 / p n ) with a linear gradient complexity O ( n ) , which are minimax optimal up to a logarithmic factor.
This implies that considering sampling with a Markov chain does not weaken the learnability.
Theorems 9 and 10 also show the effect of P on the population risk rates, i.e., the rates get better as  ( P ) decreases.
Conclusion
We develop the ﬁrst-ever-known stability and generalization analysis of Markov chain stochastic gradient methods for both minimization and minimax objectives.
In particular, we establish the optimal excess population bounds O (1 / p n ) for MC-SGD for both smooth and non-smooth cases.
We also develop the ﬁrst nearly optimal convergence rates ˜
O (1 / p T ) for convex-concave problems of MC-SGDA, and show that the optimal risk bounds O (1 / p n ) can be derived even in the non- smooth case.
Although the gradients from Markov sampling are biased and not independent across the iterations, we show the performance of MC-SGMs is competitive compared to SGMs with the classical i.i.d.
sampling scheme.
An interesting direction is to consider other variants of SGMs with variance reduction techniques and differentially private SGMs under the Markov sampling scheme.
Acknowledgement.
The work described in this paper is partially done when the last author, Ding- Xuan Zhou, worked at City University of Hong Kong, supported by the Laboratory for AI-Powered
Financial Technologies under the InnoHK scheme, the Research Grants Council of Hong Kong [Projects No.
CityU 11308121, No. N_CityU102/20, and No. C1013-21GF], the National Science
Foundation of China [Project No. 12061160462], and the Hong Kong Institute for Data Science.
The corresponding author is Yiming Ying whose work is supported by SUNY-IBM AI Alliance Research and NSF grants (IIS-2103450, IIS-2110546 and DMS-2110836).