H IGH P ROBABILITY G ENERALIZATION B OUNDS WITH F AST R ATES FOR M INIMAX P ROBLEMS
A BSTRACT
Minimax problems are receiving an increasing amount of attention in a wide range of applications in machine learning (ML), for instance, reinforcement learning, robust optimization, adversarial learning, and distributed computing, to mention but a few.
Current studies focus on the fundamental understanding of general minimax problems with an emphasis on convergence behavior.
As a comparison, there is far less work to study the generalization performance.
Additionally, existing generalization bounds are almost all derived in expectation, and the high probability bounds are all presented in the slow order O (1 / √ n ) , where n is the sample size.
In this paper, we provide improved generalization analyses and obtain sharper high probability generalization bounds for most existing generalization measures of minimax problems.
We then use the improved learning bounds to establish high probability generalization bounds with fast rates for classical empirical saddle point (ESP) solution and several popular gradient-based optimization algorithms, including gradient descent ascent (GDA), stochastic gradient descent ascent (SGDA), proximal point method (PPM), extra-gradient (EG), and optimistic gradient descent ascent (OGDA).
In summary, we provide a systematical analysis of sharper generalization bounds of minimax problems.
I NTRODUCTION
Minimax learning problems have achieved great success over a broad range of learning tasks in machine learning, with examples including reinforcement learning (Du et al., 2017; Dai et al., 2018), robust optimization (Chen et al., 2017; Namkoong & Duchi, 2017), adversarial learning (Goodfellow et al., 2014), distributed computing (Razaviyayn et al., 2020; Shamma, 2008; Mateos et al., 2010), and AUC maximization (Lei & Ying, 2021b), to just name a few.
This framework is formulated as a zero-sum game characterized as two groups of decision variables, one for minimization and one for maximization.
The coupling of the two groups of variables makes analysis of minimax problems more complex than the standard statistical learning theory setting, with only one minimization operator (Liu et al., 2021b;
Yin et al., 2020; Li & Liu, 2021;
Li et al., 2018; Liu et al., 2020; Li & Liu, 2021).
Researchers have designed various optimization algorithms, for instance, gradient descent ascent (GDA), stochastic gradient descent ascent (SGDA), proximal point method (PPM), extra-gradient (EG), and optimistic gradient descent ascent (OGDA), to solve the minimax optimization problem (Farnia & Ozdaglar, 2021).
Current theoretical research in ML literature is mainly devoted to the convergence rate and optimality of these minimax optimization algorithms in different setting, such as convex-concave settings (Nemirovski et al., 2008), nonconvex-concave setting (Raﬁque et al., 2018), strongly convex-strongly-concave setting (Balamurugan & Bach, 2016), and nonconvex-nonconcave setting (Liu et al., 2021a;
Yang et al., 2020).
In contrast, there is far less work on the generalization performance analysis, which is an important measure to indicate the performance of the learned model based on training samples when generalized to the test data.
To the best of our knowledge, there is only three work on the generalization bounds of minimax optimization algorithms (Zhang et al., 2021a; Farnia & Ozdaglar, 2021;
Lei et al., 2021).
Among them,
(Zhang et al., 2021a) studies the generalization bounds for ESP solution to minimax problems, (Farnia & Ozdaglar, 2021) analyzes the generalization properties of several gradient-based optimization algorithms: GDA, SGDA, GDmax and PPM, and (Lei et al., 2021) provides a systematical generalization analysis of SGDA.
However, in the above-mentioned papers, almost all generalization bounds are derived in expectation.
Only two high probability bounds exist, proposed in (Lei et al., 2021).
Unfortunately, they are of the slow order O (1 / √ n ) .
It is known that the high probability bound is beneﬁcial to understand the robustness of optimization algorithms (Bousquet et al., 2020; Klochkov & Zhivotovskiy, 2021) and is much more challenging to be derived (Bousquet et al., 2020;
Lei et al., 2021;
Lv et al., 2021).
In this paper, our goal is to provide the sharper high probability generalization bounds for minimax learning problems.
We leverage the lens of algorithmic stability, which is also served as an important tool in (Zhang et al., 2021a; Farnia & Ozdaglar, 2021; Lei et al., 2021).
Our contributions are summarized below.
In view of the coupling construction between the minimization variable and the maximization variable, minimax learning problems have many generalization measures (Lei et al., 2021; Farnia & Ozdaglar, 2021; Zhang et al., 2021a).
In this paper, we provide improved stability analyses for almost all existing generalization measures, based on which we establish sharper high probability generalization bounds.
These developed learning bounds can be employed to derive generalization bounds with fast rates for stable minimax learning algorithms.
The generalization performance of the ESP solution and gradient-based optimization algorithms stands a central place in the learning theory of minimax problems (Lei et al., 2021).
In this paper, we develop high probability generalization bounds with fast rates for ESP solution and several popular gradient-based optimization algorithms: GDA, SGDA, PPM, EG, and OGDA.
Overall, we provide a systematical analysis of sharper generalization bounds for minimax learning problems.
R ELATED W ORK
Algorithmic stability.
Algorithmic stability is a fundamental concept in learning theory (Bousquet & Elisseeff, 2002), which has a deep connection with learnability (Rakhlin et al., 2005;
ShalevShwartz & Ben-David, 2014; Shalev-Shwartz et al., 2010).
A training algorithm is stable if small changes in the training set lead to small differences in the output predictions of the trained model.
Different algorithmic stability measures have been developed, including uniform stability (Bousquet & Elisseeff, 2002; Feldman & Vondrak, 2018; 2019; Klochkov & Zhivotovskiy, 2021;
Hardt et al., 2016; Lei et al., 2020), uniform argument stability (Liu et al., 2017; Bassily et al., 2020), hypothesis stability (Bousquet & Elisseeff, 2002; Charles & Papailiopoulos, 2018), hypothesis set stability (Foster et al., 2019), on average stability (Shalev-Shwartz et al., 2010; Lei & Ying, 2020; Kuzborskij &
Lampert, 2018; Zhang et al., 2021b; Lei & Ying, 2021a), locally elastic stability (Deng et al., 2021), collective stability (London et al., 2016), and PAC-Bayesian stability (Li et al., 2020).
These stability measures have been extensively studied in the generalization analysis of the standard statistical learning theory setting (Chen et al., 2018; Zhang et al., 2021b).
Several stability measures have also been extended to minimax learning problems, for instance, weak stability, argument stability, and uniform stability (Farnia & Ozdaglar, 2021;
Zhang et al., 2021a;
Lei et al., 2021).
In related work (Farnia & Ozdaglar, 2021; Zhang et al., 2021a; Lei et al., 2021), they mostly focus on the expectation form of these stability measures since they are to derive bounds in expectation.
In this paper, we will focus on the last two measures, which are often used when establishing high probability generalization bounds (Feldman & Vondrak, 2018; 2019; Klochkov & Zhivotovskiy, 2021).
Convergence analysis.
Convergence analysis has been widely studied in different settings, including convex-concave learning (Nemirovski, 2005; Nedic & Ozdaglar, 2009; Mokhtari et al., 2020;
Cherukuri et al., 2017; Mokhtari et al., 2019; Balamurugan & Bach, 2016; Hsieh et al., 2019; Yan et al., 2020; Lin et al., 2020b; Wang & Li, 2020; Yoon & Ryu, 2021), nonconvex-concave learning (Raﬁque et al., 2018; Kong & Monteiro, 2019; Luo et al., 2020; Grnarova et al., 2017; Thekumparampil et al., 2019; Lu et al., 2020; Namkoong & Duchi, 2016; Sanjabi et al., 2018; Nouiehed et al., 2019; Lin et al., 2020a; Sinha et al., 2017; Chen et al., 2021), and nonconvex-nonconcave learning (Heusel et al., 2017; Balduzzi et al., 2018; Daskalakis & Panageas, 2018; Mertikopoulos et al., 2019;
Loizou et al., 2020; Yang et al., 2020; Liu et al., 2021a; Lin et al., 2018; Diakonikolas et al., 2021;
Wang et al., 2020; Loizou et al., 2021; Fiez & Ratliff, 2021).
There are so many studies on convergence.
Thus, considering the length limit, the references listed here are not complete.
Please refer to the related references concerning the above work.
We investigate the generalization performance of minimax problems instead of the convergence behavior.
Note that the convergence analysis also plays an essential role in this paper, formalized as strong PD empirical risk (please refer to Deﬁnition 1), which is deﬁned on the function value difference and referred to as optimization error or primal-dual gap in some convergence literature (Lei et al., 2021; Nemirovski, 2005; Mokhtari et al., 2019; 2020).
P RELIMINARIES
Let X and Y be two parameter spaces in R d .
Let P be a probability measure deﬁned on a sample space Z .
We deﬁne f : X ×Y ×Z (cid:55)→ R and consider the following minimax optimization problem
[Equation]
The above minimax objective represents an expectation of a cost function f
( x , y ; z ) for minimization variable x , maximization variable y and data variable z .
Unfortunately, we typically are not available to the underlying distribution P .
In practice, F is approximated by the corresponding empirical risk.
Let S
=
{ z 1 , ..., z n } be a dataset whose samples are independent drawn according to P , the empirical risk is deﬁned as
[Equation]
Let the output of a (randomized) algorithm
A on a dataset S be A ( S ) :
=
( A x ( S ) , A y ( S ))
∈
X ×Y .
Since A ( S ) is just an empirical approximated solution of the true minimax optimization problem, we are interested in studying how well A ( S ) generalizes to the unseen data.
As claimed in (Farnia & Ozdaglar, 2021; Lei et al., 2021), the coupling between the minimization variable and the maximization variable in (1) makes minimax problems have many different generalization performance measures.
These measures are collected in (Lei et al., 2021).
For better readability, we use their symbols.
Let E be the expectation with respect to (w.r.t.)
the randomness of algorithm A and the dataset S .
These generalization measures are listed below.
Deﬁnition 1.
(Lei et al., 2021)
There are four groups of generalization measures.
1 (Primal Measures
The primal population risk of a model x is deﬁned as R ( x ) =
sup y ∈Y F ( x , y ) , and the corresponding primal empirical risk is deﬁned as R S ( x )
[Equation]
sup y ∈Y F S ( x , y ) .
Then, when using empirical risk R S ( x ) to bound R ( x ) , we call this error of the model x
the primal generalization error.
While using optimal inf x ∈X R ( x ) to bound R ( x ) , we call this error of the model x the excess primal population risk.
2 (Plain Measure.)
When using F S ( x , y ) to bound F ( x , y ) , we call the this error of a model ( x , y ) the plain generalization error.
3 (Strong Measures.)
The strong primal-dual (PD) population risk of a model ( x , y ) is deﬁned as
[Equation]
and the corresponding strong PD empirical risk is deﬁned as
[Equation]
Then, the strong PD generalization error (cid:52) s ( x , y ) − (cid:52) sS ( x , y ) of the model ( x , y ) is deﬁned as
[Equation]
sup
[Equation]
4 (Weak Measures.)
The weak PD population risk of a (randomized) model ( x , y ) is deﬁned as
[Equation]
, and the corresponding weak PD empirical risk is deﬁned as
[Equation]
Then, the weak PD generalization error (cid:52) w ( x , y ) − (cid:52) wS
( x , y ) of the model ( x , y ) is deﬁned as (cid:16) sup
[Equation]
Remark 1.
We provide some discussions for the four groups of measures.
Primal Measures:)
In the context of GANs, the primal population risk R ( x ) represents a divergence measure between the learned and true distributions, and in the context of adversarial training it represents the learner’s risk under adversarial perturbations (Farnia & Ozdaglar, 2021).
One would be interested in the relationship between R ( x ) and its corresponding empirical risk R S ( x ) , and the relationship between
R ( x ) and its inﬁmum inf x (cid:48) ∈X R ( x (cid:48) ) .
Plain Measure:)
This generalization measure is a direct extension of the standard generalization error in the minimization optimization.
Strong Measures:) (cid:52) sS ( x , y ) is referred to as the primal-dual gap in the optimization literature.
(cid:52) s ( x , y ) is the primal-dual gap of the population risk.
(cid:52) s ( x , y ) − (cid:52) sS ( x
, y ) studies the difference between the population primal-dual gap and its empirical counterpart.
Weak Measures:)
The difference between the strong and weak measures is that weak measures take the expectation over the randomness of the dataset and the algorithm, for instance, sup y (cid:48) ∈Y F ( x , y (cid:48) ) − inf x (cid:48) ∈X F ( x (cid:48) , y ) in the strong measures and sup y (cid:48) ∈Y E
[ F ( x , y (cid:48) )] − inf x (cid:48) ∈X E [ F ( x (cid:48) , y )] in the weak measures.
Therefore, the upper bounds of weak measures hold in expectation, while the upper bounds of strong measures hold uniformly for any dataset.
Denote the L p norm of a random variable Z as (cid:107) Z (cid:107) p =
( E Z | Z | p ) 1 /p .
Let (cid:107)·(cid:107) be the Euclidean norm and (cid:104)· , ·(cid:105) be the inner product.
A differentiable function g : W (cid:55)→ R is called µ -strongly-convex in w if the following inequality holds for every w 1 , w 2 :
[Equation]
where ∇ is the gradient operator.
We say g is µ -strongly-concave if − g is µ -strongly-convex.
Deﬁnition 2.
Let g : X × Y (cid:55)→ R .
Assume that X and Y are convex feasible sets.
Then
g is µ -strongly-convex-strongly-concave ( µ -SC-SC) if g ( · , y ) is µ -strongly-convex for any y ∈ Y and g ( x , · ) is µ -strongly-concave for any x ∈ X . 2.
g is convex-concave (C-C) if g is 0 -SC-SC.
We then introduce the deﬁnition of algorithmic stability this paper used.
Algorithmic stability plays an important role in studying the generalization behavior of a learning algorithm.
Intuitively, an algorithm A : Z n (cid:55)→ ( X , Y ) is said to be stable if the output model ( A x ( S ) , A y ( S )) is insensitive to perturbations.
Let S (cid:48) be a neighboring dataset that differs at most one single example to S .
Deﬁnition 3 (Algorithmic Stability) .
Let A be a learning algorithm and (cid:15) > 0 .
We say A is (cid:15) -uniformly-stable if for any training datasets S, S (cid:48) ∈
Z n we have sup
z
[ f ( A x ( S ) , A y ( S ); z )
− f ( A x ( S (cid:48) ) , A y ( S (cid:48) )
[Equation]
z )] ≤ (cid:15).
We say A is (cid:15) -argument-stable if for any training datasets S, S (cid:48)
∈
Z n we have
[Equation]
A x ( S ) − A
[Equation]
( S
[Equation] [Equation]
+ (cid:107)
[Equation]
From Deﬁnition 3, one can see that the uniform stability measures the sensitivity of the function values, while the argument stability measures the sensitivity of the arguments.
We ﬁnally introduce two standard assumptions in minimax problems.
Assumption 1 implies f is
Lipschitz continuous w.r.t.
both x and y , while Assumption 2 implies f is smooth w.r.t.
( x , y ) .
Assumption 1 (Lipschitz continuity) .
Let L > 0 .
Assume that for any x ∈ X , y ∈ Y and z ∈ Z , f ( x , y ; z ) satisﬁes (cid:107)∇ x f ( x , y ; z ) (cid:107) ≤ L and (cid:107)∇ y f ( x , y ; z ) (cid:107) ≤ L.
Assumption 2 (Smoothness) .
Let β > 0 .
Assume that for any x 1 , x 2 ∈ X , y 1 , y 2 ∈ Y and z ∈ Z , f ( x , y ; z ) satisﬁes (cid:13)(cid:13)(cid:13)(cid:13)(cid:18) ∇ x
[Equation]
Under Assumption 1, the argument stability implies the uniform stability.
Therefore, the argument stability is the main stability measure that we will focus on.
M AIN R ESULTS
In this section, we provide sharper high probability bounds for the generalization measures of Deﬁnition 1, shown as follows.
Theorem 1.
Let A be a learning algorithm and (cid:15) > 0 .
Suppose | f ( x , y ; z ) | ≤ M for some M >and x ∈ X , y ∈ Y , z ∈ Z .
Fixed any η > 0 .
There exists an absolute positive constant C .
(a.)
If the algorithm A is (cid:15) -uniformly
stable, then for any δ > 0 , with probability at least 1 − δ ,
[Equation]
(b.) Assume that for all x , the function y
(cid:55)→ F ( x , y ) is µ -strongly-concave.
If the algorithm A is (cid:15) -argument stable and Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ ,
[Equation]
(c.)
Assume that for all x and y , the function F ( x , y ) is µ -SC-SC.
If the algorithm A is (cid:15) -argument stable and Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , (cid:52) s
[Equation]
A
[Equation]
S ) , A y ( S ))
≤
[Equation]
A x ( S ) , A y ( S ))
+
[Equation]
S
[Equation]
A x
[Equation]
S ) , A y
[Equation]
S ))
[Equation]
(d.) Assume that for all x and y , the function F ( x , y ) is µ -SC-SC.
If the algorithm A is (cid:15) -argument stable and Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , (cid:52) s
[Equation]
A
[Equation]
S
[Equation]
A y ( S )
[Equation]
S
[Equation]
S
[Equation]
(e.) Assume that for all x , the function y
(cid:55)→ F ( x , y ) is µ -strongly-concave.
If the algorithm A is (cid:15) -argument stable and Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ ,
R
[Equation]
x ( S )
[Equation]
≤
[Equation]
1 +
[Equation]
inf
[Equation] [Equation]
According to Deﬁnition 1 and Jensen’s inequality, we know that (cid:52) w ( x , y ) ≤ E
[ (cid:52) s ( x , y )] and (cid:52) wS ( x , y ) ≤ E [ (cid:52) sS ( x , y )] .
By this connection, we have (cid:52) w ( x , y ) − (cid:52) wS
( x , y ) ≤ E [ (cid:52) s ( x , y )]
+ | E
[ (cid:52) sS ( x , y )
] | .
We therefore obtain the following results for (cid:52) w ( x , y ) and (cid:52) w ( x , y ) − (cid:52) wS
( x , y ) .
Corollary 1.
Suppose the same conditions as Theorem 1 hold.
(f.)
If the assumptions of Part (c) in Theorem 1 hold, then with probability at least 1 − δ , (cid:52) w ( A x ( S ) , A y ( S )) ≤ (1 + η )
[Equation]
(cid:52)
[Equation]
A x
[Equation]
S
[Equation]
A y (
[Equation]
(g.)
If the assumptions of Part (d) in Theorem 1 hold, then with probability at least 1 − δ , (cid:52) w ( A x ( S ) , A y ( S )) − (cid:52) wS
( A x ( S ) , A y ( S )) ≤ | E (cid:52) sS
( A x ( S ) , A y ( S ))
| + (1 + η ) E (cid:52) sS ( A x ( S ) , A y ( S ))
+ C (1 + η ) (cid:16)
L
2 (1 + η
) nµη
[Equation]
Remark 2.
In Theorem 1, we have established a quantitative connection between the generalization measures and the stability measures.
The complete proof of Theorem 1 is provided in Appendix A.
Part (a) provides the relationship between the uniform stability and the plain generalization error of ( A x ( S ) , A y ( S )) .
If the uniform stability of algorithm A is of fast order O (1 /n ) , then
F ( A x ( S ) , A y ( S )) is bounded by (1 + η ) O (cid:0) F S ( A x ( S ) , A y ( S ))
+ log n log(1 /δ )
nη (cid:1) .
Usually for a well-trained model ( A x ( S ) , A y ( S )) over the training set, the empirical risk F S ( A x ( S ) , A y ( S )) is small or even zero (Lever et al., 2013;
Yang et al., 2019; Cortes et al., 2021).
If the empirical risk is of order O
(1 /n ) , then we can choose a proper constant for η and the plain generalization error will be of fast order O (cid:0) log n log(1 /δ )
n (cid:1) .
It is O
(1 /n ) when we hide the logarithmic term.
In the related work, (Lei et al., 2021) also establish the plain generalization error bound under the same assumptions, but their bound is of slow order O (cid:0) (cid:15) log n log(1 /δ )
+ Mn − 12 (cid:112) log(1 /δ )
(cid:1) .
Even if they get a sharper bound for stability measure (cid:15) , the inﬂuence of O (cid:0) n − 12 (cid:112) log(1 /δ )
(cid:1) can not disappear.
By comparison, we have completely removed the O (1 / √ n ) term.
Thus, our plain generalization error bound enables the fast O (1 /n ) rate when the empirical risk is small.
Part (b) provides the connection between the argument stability and the primal generalization error.
Similar to the analysis of Part (a), if both the argument stability of algorithm A and R S ( A x ( S )) are of order O (1 /n ) , then the primal generalization error implies a fast O (cid:0) 1 /n (cid:1) rate.
Considering that we assume the function f is well-behaved, i.e., Lipschitz continuity, smoothness, and the strong-concavity of its population risk F , and that R S ( A x ( S )) is data-dependent, thus it is reasonable to assume R S
( A x ( S )) is small for a well-trained model
A x ( S )
(Lever et al., 2013; Yang et al., 2019;
Cortes et al., 2021).
In (Lei et al., 2021), they also establish a bound for primal generalization error under the same assumptions.
However, their bound is O (cid:0) Lβµ − 1 (cid:15)logn log(1 /δ )
+
Mn − 12 (cid:112) log(1 /δ )
(cid:1) , limited to the O (1 / √ n ) order.
In contrast, we successfully removed the
O (1 / √ n ) term, which makes the fast rate possible.
(Farnia & Ozdaglar, 2021) studies the expected primal generalization error, i.e., bounding E S,
A [ R ( A x ( S ))] by E S,A [ R S (
A x ( S ))] .
They establish the connection between the stability measure and the expected error under the same assumptions as Part (b), which is then used to derive generalization bounds for (S)GDA, (S)GDmax, and PPM algorithms.
By comparison, our bound is derived in high probability.
Part (c) provides the relationship between the argument stability and the strong PD population risk.
If both the argument stability of algorithm A and the strong PD empirical risk are all of the order
O (1 /n ) , the strong PD population risk will be of the fast order O (1 /n ) .
Note that in our proof for the gradient-based optimization algorithms, the strong PD empirical risk mainly has a dependence on the iterative number T (see Lemma 8 of GDA, Lemma 11 of SGDA, etc.).
To obtain sharper generalization bounds, we require T to be associated with n , such as T
=
O ( n 2 ) for GDA, the strong PD empirical risk ﬁnally has a dependence on the sample size n .
To our best knowledge, this is the ﬁrst high probability strong PD population risk bound.
The expected version of this risk is studied for the ESP solution in (Zhang et al., 2021a).
However, the discussion there does not establish the connection between stability and generalization.
Their analysis is restricted to the speciﬁc ESP problem.
Under the same assumptions, they provide the upper bound of order O (1 /n ) .
Compared with their result, our result is presented in high probability.
Additionally, our strong PD population risk bound is applicable for any stable minimax optimization algorithms.
Part (d) provides the connection between the argument stability and the strong PD generalization error.
Similarly, if both the argument stability of algorithm A and the strong PD empirical risk are all of O (1 /n ) order, the strong PD generalization error will be of the fast order O (1 /n ) .
Although
Part (c) and Part (d) have a similar upper bound, they are different generalization measures (Lei et al., 2021).
To our best knowledge, this is also the ﬁrst high probability strong PD generalization error bound.
The expected version of this generalization error is studied in (Lei et al., 2021), that is E S,A
[ (cid:52) s
( A x ( S ) , A y ( S ))
− (cid:52) sS
( A x ( S ) , A y ( S ))] .
Under the same assumptions, their expected strong PD generalization error is bounded by (1 + β/µ ) L √ 2 (cid:15) , which can also be used to obtain
O (1 /n ) order rate when (cid:15) is of order O
(1 /n ) .
However, this bound is provided for the expected error, while our bound is high probabilistic and holds uniformly for any dataset.
Part (e) provides the relationship between the argument stability and the excess primal population risk.
Similar to the analysis of Part (a) and Part (b), if the argument stability of algorithm A , the strong PD empirical risk, and inf x ∈X R ( x ) are all of the order
O (1 /n ) , the excess primal population risk will also be of the fast order O (1 /n ) .
Meanwhile, in the minimization learning problems, assuming the optimal population risk F ∗ is small or even zero, i.e., F ∗ ≤ O (1 /n ) , can be found in (Lei & Ying, 2021a;
Zhang et al., 2017; Zhang & Zhou, 2019; Srebro et al., 2010; Lei & Ying, 2020).
Note that the optimal population risk F ∗ =
O (1 /n ) just to show that the improved bound can be got under low noise conditions.
F ∗ should be independent of n .
Similar to the assumption on F ∗ and considering that we assume the function f is well-behaved, it will also be reasonable to assume inf x ∈X R ( x ) is small.
High probability excess primal population risk bound is also studied for
SGDA in (Lei et al., 2021).
Their bound, however, is of slow order O (cid:0) ( β/µ ) n − 12 log n log 2 (1 /δ )
(cid:1) and is restricted to SGDA.
By comparison, our result in Part (e) enables O (1 /n ) bounds for stable minimax learning algorithms since we have completely removed the O (1 / √ n ) term.
We discuss a noteworthy difference between (Farnia & Ozdaglar, 2021; Lei et al., 2021) and ours.
In Part (a), Part (b), and Part (e), we study the upper bounds of F ( A x ( S ) , A y ( S )) , R ( A x ( S ))
(w.r.t.
R S ( A x ( S )) ), and R ( A x ( S ))
(w.r.t.
inf x ∈X R ( x ) ), respectively, while (Lei et al., 2021; Farnia &
Ozdaglar, 2021) study the upper bounds of F ( A x ( S ) , A y ( S ))
− F S ( A x ( S ) , A y ( S )) , R ( A x ( S )) −
R S ( A x ( S )) , and R ( A x ( S )) − inf x ∈X R ( x ) (or their expected forms).
One of our motivations to study such forms is that, in practice, we are often directly interested in the true risk, i.e., how the learned models behave on the testing data, such as F ( A x ( S ) , A y ( S )) , instead of the error between the true risk and empirical risk.
Note that in the above comparison between Theorem 1 and the results in (Lei et al., 2021; Farnia & Ozdaglar, 2021), we all take the right side of the generalization bound inequalities to compare, which is fair since our bounds can be written as F ( A x ( S ) , A y ( S ))
−
F S ( A x ( S ) , A y ( S )) ≤ ηF S ( A x ( S ) , A y ( S ))
+ C 1+ ηη
( Mn log(1 /δ )
+ (cid:15) log 2 n log 1 δ ) , etc.
Remark 3.
From Remark 2, one can see that compared with (Zhang et al., 2021a; Farnia &
Ozdaglar, 2021; Lei et al., 2021), we have established sharper high probability generalization bounds.
In the applications of Section 5, we will establish O (1 /n ) order bounds for two terms in Theorem 1: stability measures and strong PD empirical risk.
Hence, the strong PD population risk and the strong PD generalization error will be of the fast order O (1 /n ) when applying Theorem 1 to these applications.
These bounds are clearly of order O (1 /n ) and sharper than the results in (Zhang et al., 2021a; Farnia & Ozdaglar, 2021; Lei et al., 2021).
For the plain generalization error, the primal generalization error, and the excess primal population risk, to obtain O (1 /n ) order bounds for these applications, we need to assume the extra corresponding terms F ( A x ( S ) , A y ( S )) ,
R ( A x ( S )) , and inf x ∈X R ( x ) are of order O (1 /n ) , respectively.
The clear motivation is that in practice, learning algorithms achieve a small or even zero empirical risk, as discussed in Remark 2.
Remark 4.
This remark discusses η in Part (a), Part (b), and Part (e).
(1:) When establishing sharper generalization error bound (i.e., Pf − P n f ), the existence of η is common in the standard statistical learning theory.
Speciﬁcally, in the uniform localized convergence theory, the generalization error bound in (Bartlett et al., 2005) is of the form Pf ≤ ηη − 1 P n f + O ( ηr ∗ + η log(1 /δ ) n ) with η > 1 (see Theorem 3.3 and Theorem 4.1).
In the PAC-Bayesian theory, the generalization bounds in (Catoni, 2007)
(see Theorem 1.2.6), (Lever et al., 2013) (see Theorem 6), (Yang et al., 2019)
(see Proposition 3.1 and Theorem 4.3), etc., also have η .
For instance, the Catoni’s bound is of the form PQ ≤
[Equation]
) (cid:1) with η > 0 (Catoni, 2007).
In the algorithmic stability theory, Theorem 1.2 in (Klochkov & Zhivotovskiy, 2021) is of the form Pf ≤ (1 + η ) P n f + 1+ ηη O
(( (cid:15) log n + 1 n ) log( 1 δ )) with η > 0 .
In the recent Cortes’s deviation margin bounds (Cortes et al., 2021), they also imply a multiplier η .
The above bounds can be transformed into the form of empirical risk multiplied by 1 + η , similar to our results.
It is discussed in (Lever et al., 2013; Yang et al., 2019;
Cortes et al., 2021; Bartlett et al., 2005; Klochkov & Zhivotovskiy, 2021) that this type of generalization error bound can obtain a fast rate when the empirical risk is small.
Note that Part (e) also involves generalization error bounds due to the decomposition, see (31).
The above generalization error analysis thus holds for Part (e).
Furthermore, in (10), we show that F ( A x ( S ) , A y ( S ))
− F S ( A x ( S ) , A y ( S ))
≤ O (cid:0) ( MF ( A x ( S ) ,
A y ( S ))
log(1 /δ )
n ) 12 + (cid:15) log( 1 δ ) (cid:1) , where M means that | f ( x , y ; z ) | ≤ M, ∀ x , y , z .
Using the elementary inequality √ ab ≤ ηa + 1 η b for any a, b, η > 0 and by some rearrangements, the form of Part (a) appears.
This is the reason why η exists.
The corresponding bound in (Lei et al., 2021) is F ( A x ( S ) ,
A y ( S ))
−
F S ( A x ( S ) , A y ( S )) ≤ O (cid:0) (cid:15) log n log( 1 δ )
+ Mn − 12 log 1 / 2 ( 1 δ ) (cid:1) .
Focusing on the dominated term, it is clear that F ( A x ( S ) , A y ( S )) (cid:28) M since F ( A x ( S ) , A y ( S )) is data-dependent, which implies that our plain generalization error bound is sharper.
Similar analysis holds for Part (b) and Part (e).
[Table]
(cid:0) T 2 cµ 2 cµ
+3 /n
2 cµ +1 2 cµ +3 (cid:1)
V-WC-WC, Lip, S
Weak PD Generalization
O (cid:0) 1 / √ n (cid:1)
AGDA
NC-SC, PL, Lip, S (E.)
Excess Primal Risk
O (cid:0) n − cβ +
2 cβ +1 (cid:1)
Ours
ESP
SC-SC, Lip, LN
Plain Generalization
O (log n/n )
SC-SC, Lip, S, LN
Primal Generalization
O (log n/n )
SC-SC, Lip, S
Strong PD Risk
O (log n/n )
SC-SC, Lip, S
Strong PD Generalization
O (log n/n )
SC-SC, Lip, S, LN
Excess Primal Risk
O (log n/n )
GDA
SC-SC, Lip, LN
Plain Generalization
O ((log n ) 3 / 2 /n )
SC-SC, Lip, S, LN
Primal Generalization
O ((log n ) 3 / 2 /n )
SC-SC, Lip, S
Strong PD Risk
O ((log n ) 3 / 2 /n )
SC-SC, Lip, S
Strong PD Generalization
O ((log n ) 3 / 2 /n )
SC-SC, Lip, S, LN
Excess Primal Risk
O ((log n ) 3 / 2 /n )
SGDA
SC-SC, Lip, LN
Plain Generalization
O (log n/n )
SC-SC, Lip, S, LN
Primal Generalization
O (log n/n )
SC-SC, Lip, S
Strong PD Risk
O (log n/n )
SC-SC, Lip, S
Strong PD Generalization
O (log n/n )
SC-SC, Lip, S, LN
Excess Primal Risk
O (log n/n )
PPM
SC-SC, Lip, S, LN
Plain Generalization
O (log n/n )
SC-SC, Lip, S, LN
Primal Generalization
O (
[Table]
In summary, the above analyses from two different perspectives support our claim that Part (a), Part (b), and Part (e) provide sharper high probability generalization bounds.
Remark 5.
Different measures quantify different degrees of the generalization error.
Thus, deriving bounds of different generalization measures requires different assumptions (Lei et al., 2021;
Zhang et al., 2021a; Farnia & Ozdaglar, 2021).
Strong measures require stronger assumptions compared with the weak (Lei et al., 2021).
For instance, for the term sup y ∈Y F ( A x ( S ) , y ) in (cid:52) s ( A x ( S ) , A y ( S )) , one has to consider the fact that for different A x ( S ) , y is different, which makes the proof more challenging.
While in (cid:52) w ( A x ( S ) , A y ( S )) and (cid:52) w ( A x ( S ) , A y ( S )) − (cid:52) wS
( A x ( S ) , A y ( S )) , both the supremum over A x ( S ) and A y ( S ) are outside the expectation operator, thus one does not need to consider the coupling between A x ( S ) and y .
The upper bounds shown in Corollary 1 directly derived from Theorem 1 are sub-optimal since (cid:52) w ( A x ( S ) , A y ( S )) and (cid:52) w ( A x ( S ) , A y ( S ))
− (cid:52) wS
( A x ( S ) , A y ( S )) are pretty weak generalization measures (Lei et al., 2021).
We list Corollary 1 here to suggest that, when Theorem 1 is established, the fast order O (1 /n ) is easy to be achieved for (cid:52) w ( A x ( S ) , A y ( S )) and (cid:52) w ( A x ( S ) , A y ( S ))
− (cid:52) wS
( A x ( S ) , A y ( S )) .
On the other hand, the two weak generalization measures in (Zhang et al., 2021a) is studied for the speciﬁc ESP solution, while Corollary 1 is applicable for any stable minimax learning algorithms, it thus may be useful in some applications.
A PPLICATIONS
We now apply Theorem 1 to the ESP solution and several gradient-based optimization algorithms:
GDA, SGDA, PPM, EG, and OGDA.
Considering the length limit, we postpone the introductions and theorems of these applications to the Appendix.
Here, we list the generalization bounds of these optimization algorithms in Table 1.
5. D ESCRIPTIONS OF T ABLE
Table 1 gives almost all existing generalization bounds in minimax learning.
In Table 1, “LN” means the low noise conditions, i.e., the corresponding F ( A x ( S ) , A y ( S )) ,
R ( A x ( S )) , or inf x ∈X R ( x ) of these applications is of the order
O (1 /n ) .
For instance, for the ESP solution (ˆ x ∗ S , ˆ y ∗ S ) , we assume F (ˆ x ∗ S , ˆ y ∗ S ) , R (ˆ x ∗ S ) , and inf x ∈X R ( x ) are of the order O (1 /n ) for the plain generalization error, the primal generalization error, and the excess primal population risk, respectively.
For other learning algorithms, please refer to the Remarks in the Appendix.
In Table 1, we compare our results with (Zhang et al., 2021a; Farnia & Ozdaglar, 2021;
Lei et al., 2021) in the way described in the last paragraph of Remark 2.
(E.) denotes that the bound is derived in expectation, while (H.P.) denotes high probability.
Since our results are all established with high probability, we thus omit (H.P.) for brevity.
The descriptions of other notations are shown in Appendix H.
In Table 1, (Zhang et al., 2021a) and (Farnia & Ozdaglar, 2021) focus on the expected generalization measures.
We improve the learning bounds of the ESP solution in (Zhang et al., 2021a) to high probability guarantees.
Compared with (Farnia & Ozdaglar, 2021), we have provided high probability primal generalization error bounds for GDA, SGDA, and PPM.
Additionally, we also study other generalization measures.
(Lei et al., 2021) focus on SGDA and mainly provide guarantees for weak generalization measures, i.e., weak PD risk and weak PD generalization error.
In contrast, we have developed bounds for strong PD risk and strong PD generalization error.
Note that the two type of bounds don’t require the “LN” condition.
Moreover, although (Lei et al., 2021) provides two high probability bounds, however, in slow order.
Note that in addition to the classical GDA, SGDA, and
PPM, we also provide sharper high probability bounds for EG and OGDA in that their widespread use in training GANs (Mokhtari et al., 2019; Daskalakis et al., 2017; Liang & Stokes, 2019).
C ONCLUSION
In this paper, we provide a systematical analysis of sharper generalization bounds for minimax problems.
We ﬁrst establish sharper high probability bounds for almost all existing generalization measures via algorithmic stability and then apply these bounds to several important applications.
We believe that our research can provide in-depth insights into minimax learning problems.
For future work, it would be important to relax the assumptions in this paper.
Also, it would be interesting to investigate how well other theoretical tools perform on the generalization of minimax problems.
A CKNOWLEDGMENTS
We appreciate all the anonymous reviewers for their invaluable and constructive comments.
This work is supported in part by the National Natural Science Foundation of China (No.
62076234, No.61703396, No.
, Beijing Outstanding Young Scientist Program
NO.BJJWZYJH012019100020098, Intelligent Social Governance Platform, Major Innovation &
Planning Interdisciplinary Platform for the ”Double-First Class” initiative, Renmin University of
China, China Unicom Innovation Ecological Cooperation Plan, Public Computing Cloud of Renmin
University of China, Beijing Natural Science Foundation (No. 4222029).
A P ROOF OF T HEOREM
We now provide proofs of Theorem 1.
For better readability, we restate Theorem 1 below.
Theorem 2.
Let A be a learning algorithm and (cid:15) > 0 .
Suppose | f ( x , y ; z ) | ≤ M for some M >and x ∈ X , y ∈ Y , z ∈ Z .
Fixed any η > 0 .
There exists an absolute positive constant C .
(a.)
If the algorithm A is (cid:15) -uniformly
stable, then for any δ > 0 , with probability at least 1 − δ , F ( A x ( S ) , A y ( S )) ≤ (1 + η ) F S
[Equation]
A x
[Equation] [Equation]
(b.) Assume that for all x , the function y
(cid:55)→ F ( x , y ) is µ -strongly-concave.
If the algorithm A is (cid:15) -argument stable and Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , R
[Equation]
A x ( S ))
[Equation]
(c.)
Assume that for all x and y
, the function F ( x , y ) is µ -SC-SC.
If the algorithm A is (cid:15) -argument stable and Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ ,
[Equation]
(d.) Assume that for all x and y , the function F ( x , y ) is µ -SC-SC.
If the algorithm A is (cid:15) -argument stable and Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ ,
[Equation]
(e.) Assume that for all x , the function y
(cid:55)→ F ( x , y ) is µ -strongly-concave.
If the algorithm A is (cid:15) -argument stable and Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ ,
R
[Equation]
( S )
[Equation]
≤
[Equation]
1 + η
[Equation]
inf
[Equation]
x
[Equation]
(cid:16)
[Equation]
Remark 6.
To prove sharper high probability bounds than (Lei et al., 2021), the concentration inequality for a summation of weakly-dependent random variables proposed in (Bousquet et al., 2020)
(Lemma 2 in Appendix A) plays a key role in our analysis.
However, the direct use of this inequality will inevitably lead to a slow order bound since it contains a sampling error of slow order
O (1 / √ n ) .
We exploit the proof techniques of the recent breakthrough work (Klochkov & Zhivotovskiy, 2021) to make new constructions of g i ( S ) so that the parameter M in Lemma 2 is 0 .
However, the proof techniques of (Klochkov & Zhivotovskiy, 2021) can not be directly extended to minimax problems.
The coupling construction between the minimization variables and the maximization variables makes the proofs of minimax problems more difﬁcult than the minimization problem studied by (Klochkov & Zhivotovskiy, 2021).
We must proceed with novel decompositions for the generalization measures.
Note that different decompositions are required for different generalization measures.
A pretty technical decomposition is exploited in the proof of Part (c).
Moreover, the proof of minimax problems needs reﬁned analyses due to the minimax structure.
For instance, in proving the primal generalization error, we need to quantify the fact that for different
A x ( S ) , the optimal y is different in R ( A x ( S )) .
And the analysis of excess primal population risk in
Part (e) is different from the excess risk analysis in (Klochkov & Zhivotovskiy, 2021).
The reason is that the supremum operator in inf x ∈X R ( x ) makes the Bernstein condition used in (Klochkov & Zhivotovskiy, 2021) not applicable for excess primal population risk.
Additionally, (Klochkov & Zhivotovskiy, 2021) only study ERM and GD, while we study more optimization algorithms:
SGDA, PPM, EG, and OGDA.
Remark 7.
According to Deﬁnition 1 and Jensen’s inequality, we know that (cid:52) wS
( x , y ) ≤ E [ (cid:52) sS ( x , y )] .
Meanwhile, since we will provide the strong PD empirical risk bounds for several important optimization algorithms in Section 5, it thus implies that we also establish bounds with fast rates for (cid:52) wS
( x , y ) .
To begin the proof of Theorem 1, we ﬁrst introduce some key lemmas on concentration inequalities.
The ﬁrst lemma translates a moment bound into a high probability bound.
Lemma 1.
(Bousquet et al., 2020)
Let Z be a random variable with
[Equation]
for some a, b > 0 and for any p ≥ 2 .
Then for any δ ∈ (0 , 1) we have, with probability at least 1 − δ ,
[Equation]
where e is the base of the natural logarithm.
The second lemma establishes a concentration inequality for a summation of weakly-dependent random variables.
Lemma 2.
(Bousquet et al., 2020) Let S
=
{ z 1 , ..., z n } be a set of independent random variables each taking values in Z and M > 0 .
Denote [ n ] as the set { 1 , ..., n } .
Deﬁne S \{ z i } be set { z 1 , ...,
z i − 1 , z i +1 , ..., z n } .
Let g 1 , ..., g n be some functions
g i : Z n (cid:55)→ R such that the following inequalities hold for any i ∈ [ n ] ,
• (cid:12)(cid:12) E S \
{ z i } [ g i ( S )] (cid:12)(cid:12) ≤ M almost surely (a.s.), • E z i [ g i ( S )]
=
a.s., • for any j ∈ [ n ] with j (cid:54) =
i , and z (cid:48)(cid:48)
j ∈ Z (cid:12)(cid:12) g
i ( S ) − g i ( z 1 , ..., z j − 1 , z (cid:48)(cid:48) j , z j +1 , ..., z n ) (cid:12)(cid:12)
[Equation]
Then, for any p ≥
[Equation]
The following deﬁnition and lemma give the concentration inequality for non-negative weakly selfbounded functions.
Deﬁnition 4.
(Weakly Self-Bounded Function)
Assume that a, b > 0 .
A function f : Z n (cid:55)→ [0 , + ∞ ) is said to be ( a, b ) -weakly self-bounded if there exist functions
f i : Z n − 1 (cid:55)→ [0 , + ∞ ) that satisﬁes for all Z n ∈ Z n , n (cid:88) i =
( f ( Z n
[Equation]
− f i (
[Equation] [Equation]
≤ af ( Z
[Equation]
+ b.
Lemma 3.
(Klochkov & Zhivotovskiy, 2021) Suppose that z 1 , ..., z n are independent random variables and the function f : Z n (cid:55)→ [0 , + ∞ ) is ( a, b ) -weakly self-bounded and the corresponding function f
i satisfy f i ( Z n ) ≥ f ( Z n ) for i = 1 , ..., n and any Z n ∈ Z n .
Then, for any t > 0 ,
Pr ( E f ( z 1 , ..., z n ) ≥ f ( z 1 , ..., z n ) + t ) ≤ exp (cid:16) − t
[Equation]
E f (
[Equation] [Equation]
+
[Equation]
(cid:17) .
The following lemma is the classical Bernstein concentration inequality.
Lemma 4.
(Boucheron et al., 2013) Let z 1 , ...,
z n be i.i.d.
random variables and assume that E [ z i ] = µ .
Suppose | z i | ≤ c for any i .
Then for any δ ∈ (0 , 1) , with probability at least 1 − δ ,
[Equation]
where σ 2 is the variance of z i .
A. P ROOF OF P ART ( A )
We ﬁrst prove the plain generalization error bound.
Proof.
Let S
= { z 1 , ..., z n } be a set of independent random variables each taking values in Z and S (cid:48)
= { z (cid:48) 1 , ..., z (cid:48) n } be its independent copy.
For any i ∈ [ n ] , deﬁne S ( i ) = { z 1 , ..., z i − 1 , z (cid:48)
i , z i +1 , ..., z n } be a dataset by replacing the i -th sample in S with another i.i.d.
sample z (cid:48) i .
We ﬁrst have the following decomposition
[Equation]
According to the deﬁnition of uniform stability (Part 1 of Deﬁnition 3), we have nF
[Equation]
A x ( S ) , A y ( S ))
[Equation]
A
[Equation]
S )
[Equation]
S
[Equation]
n(cid:15) + n (cid:88)
[Equation]
=
g i ( S ) , where we have introduced
[Equation]
f ( A x ( S ( i ) ) , A y ( S ( i ) ); z i )] .
Thus, by a rearrangement, we have
[Equation]
Then, for any i = 1 , ..., n , we deﬁne h
i ( S ) =
g i ( S ) − E S \{ z i } [ g i ( S )] .
It is easy to verify that E
S \{ z i } [ h i ( S )]
= 0 and E z i
[ h i ( S )]
=
E z i [ g i ( S )]
− E z i E S \{ z i } [ g i ( S )]
= 0 − 0 = 0 .
Also, for any j ∈ [ n ] with j (cid:54) =
i , and z (cid:48)(cid:48)
j ∈ Z , we have the following inequality (cid:12)(cid:12) h
i ( S ) − h
i ( z 1 , ..., z j − 1 , z (cid:48)(cid:48) j , z j +1 , ..., z n ) (cid:12)(cid:12) ≤ (cid:12)(cid:12) g
i ( S ) − g i ( z 1 , ..., z j
[Equation]
1 , z
[Equation]
j
[Equation]
j
[Equation]
[Equation] [Equation]
n
[Equation]
For the ﬁrst term | g i ( S ) − g i ( z 1 , ..., z j − 1 , z (cid:48)(cid:48) j , z j +1 , ..., z n ) | , it can be bounded by 2 (cid:15) according to the deﬁnition of uniform stability.
Similar result holds for the second term (cid:12)(cid:12) E S \{ z i } [ g i ( S )] − E S \{ z i } [ g i ( z 1 , ..., z j − 1 , z (cid:48)(cid:48)
j , z j +1 , ..., z n )] (cid:12)(cid:12) according to the deﬁnition of uniform stability.
By a combination of the above analysis, we get |
h i ( S ) − h
i ( z 1 , ..., z j − 1 , z (cid:48)(cid:48) j , z j +1 , ..., z n ) | ≤ 4 (cid:15) .
We thus have veriﬁed that the three conditions in Lemma 2 are satisﬁed for h i ( S ) .
There will hold the following result for any p ≥
[Equation]
Furthermore, we can derive that
[Equation]
Due to the i.i.d.
property between S and S (cid:48) , we know that E S
[Equation]
F ( A x ( S
[Equation]
) , A y ( S (cid:48) ))
=
E
S F ( A x ( S ) , A y ( S )) .
Thus, combined (3) with (4), we have
[Equation]
A x ( S ) , A y
[Equation]
S )
[Equation]
(cid:13) p + (cid:13)(cid:13)(cid:13) n (cid:88) i = g
[Equation]
p = (cid:13)(cid:13)(cid:13)
[Equation] [Equation]
− n (cid:88) i =
[Equation]
g
[Equation]
(cid:13)(cid:13)(cid:13) p + (cid:13)(cid:13)(cid:13) n (cid:88) i
= h
i ( S ) (cid:13)(cid:13)(cid:13) p ≤ 2 n(cid:15) + 48 √ 2 (cid:15)pn (cid:100) log 2 n (cid:101) ≤ 50 √ 2 (cid:15)pn (cid:100) log 2 n (cid:101) .
According to Lemma 1, for any δ ∈ (0 , 1) , with probability at least 1 − δ/ 3 , we have
F ( A x ( S ) , A y ( S ))
− F S ( A x ( S ) , A y ( S ))
[Equation]
We now begin to bound the term E S (cid:48)
F S ( A x ( S (cid:48) ) , A y ( S (cid:48) ))
− E S F ( A x ( S ) , A y ( S )) .
There holds that
[Equation]
We ﬁrst consider the variance of E
S (cid:48)
f ( A x ( S (cid:48) ) , A y ( S (cid:48) ); z i ) .
By the Jensen’s inequality, we have
[Equation]
Then, by the Bernstein inequality in Lemma
[Equation]
we obtain the following inequality with probability at least 1 − δ/ 3 ,
[Equation] [Equation]
Combined (5) with (6), we ﬁnally obtain that with probability at least 1 − 2 δ/ 3 ,
[Equation]
In the following, we deﬁne q = q ( z 1 , ..., z n ) =
E Z
[( f ( A x ( S ) , A y ( S ); Z )) 2 ] and q i =
q i ( z 1 , ..., z n ) =
sup z i ∈Z q ( z 1 , ..., z n ) .
So there holds q i ≥ q for any i = 1 , .., n and any
{ z 1 , ..., z n } ∈ Z n .
Also, there holds that
[Equation]
where the ﬁrst inequality follows from the Jensen’s inequality and the deﬁnition of uniform stability, and where the second inequality also follows from the deﬁnition of uniform stability
[Equation]
From (8)
[Equation]
we know that q is (8 n(cid:15) 2 , 2 n(cid:15) 4 ) weakly self-bounded.
Thus, by Lemma
[Equation]
we obtain that with probability at least 1 − δ/ 3 ,
[Equation]
where the last inequality follows from that √ ab ≤ a + b 2 for all a, b > 0 .
Since E Z [( f ( A x ( S ) , A y ( S ); Z )) 2 ] ≤ MF ( A x ( S ) , A y ( S )) , we have
[Equation]
Substituting (9) into (7), we ﬁnally obtain that with probability at least 1 − δ ,
[Equation]
According to inequalities √ ab ≤ ηa + 1 η b and √ a + b ≤ √ a + √ b for any a, b, η > 0 , we have the following inequality with probability at least 1 − δ
[Equation]
which implies that
[Equation]
where C is an absolute constant.
The proof is complete.
A. P ROOF OF P ART ( B )
We then prove the primal generalization error bound.
Before presenting the proof, we ﬁrst introduce a lemma that quantiﬁes the sensitivity of the optimal y and x w.r.t the perturbation of x and y respectively.
Lemma 5.
(Zhang et al., 2021a)
Let f : X × Y (cid:55)→ R .
Assume that f is µ -strongly-convex-stronglyconcave.
Suppose that for any x , x (cid:48) ∈ X and y , y (cid:48)
∈ Y we have
[Equation]
and
[Equation]
Deﬁne x ∗ ( y ) =
arg min x ∈X h ( x , y ) and y ∗ ( x ) =
arg max y ∈Y h ( x , y ) for any y and x respectively.
Then, for any x , x (cid:48) ∈ X and y , y (cid:48) ∈ Y there holds that
[Equation]
and
[Equation]
The proof of Part (b) shares similar proof techniques with Part (a), but requires a novel decomposition and several important changes.
For instance, Lemma 5 should be needed to quantify the fact that for different A x ( S ) , the optimal y is different in R ( A x ( S )) .
Proof.
Let S
= { z 1 , ..., z n } be a set of independent random variables each taking values in Z and S (cid:48)
= { z (cid:48) 1 , ..., z (cid:48) n } be its independent copy.
For any i ∈ [ n ] , deﬁne S ( i ) = { z 1 , ..., z i − 1 , z (cid:48)
i , z i +1 , ..., z n } be a dataset by replacing the i -th sample in S with another i.i.d.
sample z (cid:48) i .
Denote y ∗ S = arg max y ∈Y F ( A x ( S ) , y ) and ˆ y ∗ S = arg max y ∈Y F S (
A x ( S ) , y ) .
We have the following decomposition
[Equation]
Firstly, we have
[Equation] [Equation]
where the second inequality follows from Lemma 5 with the fact that F is smooth and µ -SC-SC.
Secondly, we have
[Equation]
where the ﬁrst and the last inequalities follow from (12).
Substituting the above two results into (11)
[Equation]
we obtain that
[Equation]
where the last inequality follows from the facts that
[Equation]
x ( S ) , y ∗ S ; z i ) − (cid:80) ni
=1 f
( A x ( S ) , ˆ y ∗ S ; z i ) ≤ 0 and that we have introduced g
i ( S ) =
E z (cid:48)
i [ E Z [ f ( A x ( S ( i ) ) , y ∗ S ( i ) ; Z )] − f
( A x ( S ( i ) ) , y ∗ S ( i ) ; z i )] .
Now we get
[Equation]
For any i = 1 , ..., n , we deﬁne h
i ( S ) =
g i ( S ) − E S \{ z i } [ g i ( S )] .
We also get that E S \{ z i } [ h i ( S )]
= 0 and E z i
[ h i ( S )]
=
E z i [ g i ( S )]
− E z i E S \{ z i } [ g i ( S )]
= 0 − 0 = 0 .
Moreover, for any j ∈ [ n ] with j (cid:54) =
i , and z (cid:48)(cid:48)
j ∈ Z , we have the following inequality | h i ( S ) − h
i (
[Equation]
, ..., z
[Equation]
, z
[Equation]
j
[Equation]
z j
[Equation] [Equation]
z
[Equation]
g
[Equation] [Equation]
[Equation]
Denote S ( i ) j as the set collected by replacing the j -th element of S ( i ) with z (cid:48)(cid:48) j .
For the ﬁrst term | g i ( S ) − g i ( z 1 , ..., z j − 1 , z (cid:48)(cid:48) j , z j +1 , ..., z n ) | , we have
[Equation]
Furthermore, for any z , we have the following result which can help to bound the above inequality.
[Equation]
where the third inequality follows from Lemma 5.
Thus, we can bound the ﬁrst term by 2 (cid:16) βµ +1 (cid:17) L(cid:15) .
By a similar analysis, the second term | E S \{ z i } [ g i ( S
[Equation]
can also be bounded by 2 (cid:16) βµ + 1 (cid:17) L(cid:15) .
We now have veriﬁed that the three conditions in Lemma 2 are satisﬁed for h i ( S ) .
We obtain that for any p ≥ 2 , there holds
[Equation]
[Equation]
Thus, combined (13) with (16)
[Equation]
we derive that
[Equation] [Equation]
Then, according to Lemma
[Equation]
for any δ ∈ (0 , 1)
[Equation]
with probability at least 1 − δ/
[Equation]
we have
[Equation]
With a similar analysis to the proof of Part (a)
[Equation]
we now begin to bound the variance of E
S
[Equation]
) , y ∗ S
[Equation]
z i ) .
[Equation]
=
[Equation]
There holds that E S E S (cid:48) F S
A x ( S (cid:48) ) , y ∗ S (cid:48) ) =
E S (cid:48) F
( A x ( S (cid:48) ) , y ∗ S (cid:48) ) .
Then, by the Bernstein inequality in Lemma 4, we obtain that with probability at least 1 − δ/ 3 ,
[Equation]
Combined (18) with (19), we ﬁnally obtain that with probability at least 1 − 2 δ/ 3 ,
[Equation]
In the following, we deﬁne q = q ( z 1 , ..., z n ) =
E Z
[( f ( A x ( S ) , y ∗ S ; Z )) 2 ] and q i =
q i ( z 1 , ..., z n ) =
sup z i ∈Z q ( z 1 , ..., z n ) .
So there holds q i ≥ q for any i = 1 , .., n and any { z 1 , ..., z n } ∈ Z n .
Also, there holds that
[Equation]
where the ﬁrst inequality follows from the Jensen’s inequality and the second inequality follows from a similar analysis to (12) or (15).
Now, we know that q is (cid:16) 8 n (cid:16) βµ + 1 (cid:17) 2 L 2 (cid:15) 2 , 2 n (cid:16) βµ + 1 (cid:17) 4 L 4 (cid:15) 4 (cid:17) - weakly self-bounded.
Thus, by Lemma 3, we obtain the following inequality with probability at least 1 − δ/ 3 ,
[Equation]
where the last inequality follows from √ ab ≤ a + b 2 for all a, b > 0 .
Since E Z [( f ( A x ( S ) , y ∗ S ; Z )) 2 ] ≤ MF ( A x ( S ) , y ∗ S ) , we have
[Equation]
Plugging (21) into (20), we ﬁnally obtain that with probability at least 1 − δ ,
[Equation]
By the elementary inequalities √ ab ≤ ηa + 1 η b and √ a + b ≤ √ a + √ b for any a, b, η > 0 , we have the following inequality with probability at least 1 − δ
[Equation]
where C is an absolute constant.
The proof is complete.
A. P ROOF OF P ART (C)
We then prove the strong PD population risk bound.
To begin, we introduce the following concentration inequality, which is a moment version of the Bernstein inequality.
Lemma 6.
(Boucheron et al., 2013)
If z 1 , ..., z n are i.i.d., zero mean and | z i | ≤ M almost surely.
Then, for any p ≥ 2 ,
[Equation]
Proof.
Let S
= { z 1 , ..., z n } be a set of independent random variables each taking values in Z and S (cid:48)
= { z (cid:48) 1 , ..., z (cid:48) n } be its independent copy.
For any i ∈ [ n ] , deﬁne S ( i ) = { z 1 , ..., z i − 1 , z (cid:48)
i , z i +1 , ..., z n } be a dataset by replacing the i -th sample in S with another i.i.d.
sample z (cid:48) i .
Denote y ∗ S = arg max y ∈Y F ( A x ( S ) , y ) , ˆ y ∗ S = arg max y ∈Y F S (
A x ( S ) , y ) ,
x ∗ S = arg min x ∈X F ( x , A y ( S )) and
ˆ x ∗ S = arg
min x ∈X F S ( x , A y ( S )) .
The proof of Part (c) requires a pretty technical error decomposition, i.e., (cid:52) s
( A x ( S ) , A y ( S )) =
[Equation] [Equation]
−
[Equation]
+ F
[Equation]
( A x ( S )
[Equation]
ˆ y ∗ S )
− F S (ˆ x ∗ S , A y ( S )) .
Let’s ﬁrst consider the term F ( A x ( S ) , y ∗ S )
− F S
A x ( S ) , ˆ y ∗ S )
+ E S (cid:48)
F S ( A x ( S (cid:48)
[Equation]
y
[Equation]
S
[Equation]
E
S F ( A x ( S ) , y ∗ S ) .
It can be then decomposed into
[Equation]
which can be bounded by the proof techniques in the proof of Part (b).
According to (17)
[Equation]
we know that
[Equation]
For the second term
[Equation]
∈X F ( x (cid:48) , A y ( S )) , we have the following decomposition nF
[Equation]
=
[Equation]
−
[Equation]
+
[Equation]
=
[Equation]
It is clear that
[Equation] [Equation]
Denote g i ( S
[Equation]
=
[Equation]
A
[Equation]
( S
[Equation]
); z i )
[Equation] [Equation]
( x
[Equation]
S
[Equation]
) , A y ( S ( i ) ); Z )] (cid:3) .
By (24)
[Equation]
we now get
[Equation]
Furthermore, according to Lemma
[Equation]
we have
[Equation]
Similarly,
[Equation]
Thus
[Equation]
we obtain
[Equation]
where the last inequality follows from the deﬁnition of argument stability (Part 2 of Deﬁnition 3)
[Equation]
Furthermore, we deﬁne h
i ( S ) =
g i ( S ) − E S \{ z i } [ g i ( S )] .
For h i ( S ) , We have E S \{ z i } [ h i ( S )]
=and E z i
[ h i ( S )]
=
E z i [ g i ( S )]
[Equation]
E z i E S \{ z i } [ g i ( S )]
= 0 − 0 = 0 .
Moreover, for any j ∈ [ n ] with j (cid:54) =
i , and z (cid:48)(cid:48)
j ∈ Z , we get
[Equation]
where this inequality follows from a similar analysis to (14) and (15) of the proof of Part (b).
We thus can obtain that for any p ≥ 2 , there holds
[Equation]
Combined (25) with (26), we ﬁnally get the bound of the second term:
[Equation]
+ (cid:13)(cid:13)(cid:13) 1 n n (cid:88)
i =
h i ( S ) (cid:13)(cid:13)(cid:13) p ≤ 50 √ 2 (cid:16) 1 + β µ (cid:17) (cid:15)Lp (cid:100) log 2 n (cid:101) .
We then consider the third term − E S (cid:48)
F S ( A x ( S (cid:48) ) , y ∗ S (cid:48) )
+ E S F
( A x ( S ) , y ∗ S )
+ E S (cid:48)
F S ( x ∗ S (cid:48) , A y ( S (cid:48) ))
− E
S F ( x ∗ S , A y ( S )) .
It is clear that E S [ E S (cid:48)
F S ( x ∗ S (cid:48) , A y ( S (cid:48) ))]
[Equation]
S
[Equation]
S [
[Equation] [Equation]
(cid:48)
[Equation]
(cid:48)
[Equation]
] =
[Equation]
F ( A x ( S
[Equation]
, y ∗
[Equation]
Moreover
[Equation]
we having the following important property due to the strong convexity and strong concavity of F ,
[Equation]
where the ﬁrst inequality follows from Jensen’s inequality and the Lipschitz continuity of f (Assumption 1)
[Equation]
the second inequality follows from that ( a + b ) 2 ≤ 2 a
+ 2 b
and the third inequality follows from the property of strong convexity and strong concavity of F and the optimality condition, derived as follows,
[Equation]
It is clear that E [( Z − E Z ) 2 ] ≤ E
[ Z 2 ] .
Therefore, by the variance bound in (28) and applying the moment Bernstein inequality in Lemma 6 to the sum of independent random variables
[Equation]
); z i )+ E S F
[Equation]
A x
[Equation]
S ) , y
[Equation]
S ) − E S F ( x ∗ S , A y ( S )) , we have the following inequality for all p ≥ 2 ,
[Equation]
From Deﬁnition
[Equation]
we know that the last term F S ( A x ( S ) , ˆ y ∗ S ) − F S
(ˆ x ∗ S , A y ( S )) is actually the strong PD empirical risk (cid:52) sS ( A x ( S ) , A y ( S )) .
Based on the above analysis
[Equation]
we have derived that for each p ≥ 2 ,
[Equation]
where the last inequality holds since for any a, b, η > 0 , √ ab ≤ ηa + bη .
Taking p = 2 and using the Cauchy-Schwarz inequality
[Equation]
we obtain that
[Equation]
Since E S (cid:48)
[ F ( A x ( S (cid:48) ) , y ∗ S (cid:48) ) − F ( x ∗ S (cid:48) , A x ( S (cid:48) ))] =
E S [ F ( A x ( S ) , y ∗ S ) − F ( x ∗ S , A x ( S ))] , we ﬁnally get
[Equation]
Plugging this inequality into (29), we thus have that for each p ≥ 2 ,
[Equation]
According to Lemma 1, for any δ > 0 , with probability at least 1 − δ , there holds that
[Equation]
where C > 0 is an absolute constant.
The proof is complete.
A. P ROOF OF P ART ( D )
We now prove the strong PD generalization error bound.
Proof.
From (30) in the proof of Part (c), we know that for any δ > 0 , with probability at least 1 − δ , there holds that
[Equation]
Therefore, the proof is complete.
A. P ROOF OF P ART ( E )
We ﬁnally prove the excess primal population risk bound.
Proof.
Denote x ∗ =
arg min x ∈X R ( x ) and y ∗ S = arg max y ∈Y F
( A x ( S ) , y ) .
Firstly, we have the following decomposition
[Equation]
Consider the ﬁrst term R ( A x ( S ))
− R S ( A x ( S )) .
From (22) of the Part (b)
[Equation]
we know that with probability at least 1 − δ ,
[Equation]
n .
For the second term R S ( A x ( S ))
− F S ( x ∗ , A y ( S )) , we have R S (
A x ( S ))
− F S ( x ∗ , A y ( S )) ≤
R S ( A x ( S )) − inf x (cid:48) ∈Y F S ( x (cid:48) , A y ( S )) =
(cid:52) sS ( A x ( S ) ,
A y ( S )) .
Note that under Assumption 1, the argument stability implies the uniform stability.
Therefore, for the third term F S ( x ∗ , A y ( S ))
− F ( x ∗ , A y ( S )) , from (10) of Part (a), we know that with probability at least 1 − δ
[Equation]
n .
It is clear that F ( x ∗ , A y ( S ))
− R ( x ∗ ) ≤ 0 .
Since F ( x ∗ , A y ( S )) ≤ sup y (cid:48) ∈Y F ( x ∗ , y (cid:48) )
=
R ( x ∗ ) =
inf x (cid:48) ∈X R ( x ) , based on the above results, we have the following inequality with probability at least 1 − 2 δ
[Equation]
≤
[Equation]
βµ + 1 (cid:17) 2 L 2 (cid:15) 2 log(3 /δ ) (cid:17) log(3 /δ ) n + 4 M log(3 /δ )
3 n + (cid:115) (4 M inf x (cid:48) ∈X R ( x ) + 12 (cid:15) 2 + 32 n(cid:15) 2 log(3 /δ ))
log(3 /δ )
n + 50 √ 2 (cid:15)eLβ + µ µ (cid:100) log 2 n (cid:101) log(
e/δ )
+ (cid:52) sS
( A x ( S ) , A y ( S ))
+ 50 √ 2 e(cid:15) (cid:100)
log 2 n (cid:101) log(
e/δ ) ≤ (cid:118)(cid:117)(cid:117)(cid:116)(cid:16) 12 (cid:16) βµ + 1 (cid:17) 2 L 2 (cid:15)
+ 32 n (cid:16) βµ + 1 (cid:17) 2 L 2 (cid:15) 2 log(3 /δ )
(cid:17) log(3 /δ )
n + η 1 + η F ( A x ( S ) , y ∗ S )
+ 1 + η η 4 M log(3 /δ )
n + 4 M log(3 /δ )
3 n + (cid:115) ( 12 (cid:15) 2 + 32 n(cid:15) 2 log(3 /δ ))
log(3 /δ )
n + η 1 + η inf x (cid:48) ∈X R ( x ) + 1 + η η 4 M log(3 /δ ) n + 50 √ 2 (cid:15)eLβ + µ µ (cid:100) log 2 n (cid:101)
log(3 e/δ ) + (cid:52) sS
( A x ( S ) , A y ( S ))
+ 50 √ 2 e(cid:15) (cid:100)
log 2 n (cid:101) log(
e
[Equation]
where the last inequality follows from the elementary inequalities √ ab ≤ ηa + 1 η b and √ a + b ≤ √ a + √ b for any a, b > 0 .
Therefore, by a rearrangement, we have the following inequality with probability at least 1 − δ
[Equation]
that is
[Equation]
where C is an absolute constant.
The proof is complete.
Till here, the proof of Theorem 1 is complete.
B E MPIRICAL S ADDLE P OINT
Empirical saddle point (ESP) problem refers to problem (2), which is also known as sample average approximation (SAA) (Zhang et al., 2021a).
We denote (ˆ x ∗ S , ˆ y ∗ S ) as the ESP solution to (2), which is analogy to the ERM in stochastic optimization (Shalev-Shwartz et al., 2010).
We ﬁrst provide the main theorem of the ESP solution, as shown below.
Theorem 3.
Assume for all z , the function ( x , y ) (cid:55)→ f ( x , y ; z ) is µ -SC-SC.
Suppose | f ( x , y ; z ) | ≤
M for some M > 0 and x ∈ X , y ∈ Y , z ∈ Z .
Denote A x ( S ) =
ˆ x ∗ S and A y ( S )
=
ˆ y ∗ S for (ˆ x ∗ S , ˆ y ∗ S ) .
Fixed any η > 0 .
There exists an absolute positive constant C .
(a) If Assumption 1 holds, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(b) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(c) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(d) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(e) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
Proof.
To prove Theorem 3, we should derive the strong PD empirical risk bound and the stability bound of (ˆ x ∗ S , ˆ y ∗ S ) .
It is easy to verify that (cid:52) s S (ˆ x ∗ S , ˆ y ∗ S )
=
(Zhang et al., 2021a).
We then investigate the stability bound of (ˆ x ∗ S , ˆ y ∗ S ) .
Let S =
{ z 1 , ..., z n } be a set of independent random variables each taking values in Z .
For any i ∈ [ n ] , deﬁne S ( i ) =
{ z 1 , ..., z i − 1 , z (cid:48)
i , z i +1 , ..., z n } be a dataset by replacing the i -th sample in
S with another i.i.d.
sample z (cid:48) i .
We deﬁne
Then we have
[Equation]
where the ﬁrst inequality follows from the Lipschitz continuous assumption, and where the second inequality follows from the facts that the µ -SC-SC property of F S ( i ) and (ˆ x ∗ S ( i ) , ˆ y ∗ S ( i ) ) is the ESP solution of F S ( i ) .
Similarly, according to the µ -SC-SC property of F S , we have
[Equation]
Based on the above results
[Equation]
we have
[Equation]
where the last inequality uses the Caucy-Schwarz inequality.
Therefore
[Equation]
we have
[Equation]
Now, plugging this stability bound into Theorem
[Equation]
we obtain generalization bounds of the ESP solution.
The proof of Theorem 3 is complete.
Remark 8.
When conditions in Theorem 3 hold
[Equation]
we obtain that (a) If Assumption 1 holds and
F S (ˆ x
[Equation]
, ˆ y ∗ S )
=
O (cid:16) 1 n (cid:17) then δ
(ˆ x ∗ S , ˆ y ∗ S ) is of the order O (cid:16) log 2 n n log(1 /δ )
(cid:17) .
If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , the strong PD population risk and the strong PD generalization error of (ˆ x ∗ S , ˆ y ∗ S ) are all of the order O (cid:16) log 2 n n log(1 /δ )
(cid:17)
If Assumptions 1 and 2 hold and inf x ∈X R ( x ) =
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the excess primal population risk of (ˆ x ∗ S , ˆ y ∗ S ) is of the order O (cid:16) log 2 n n
[Table]
Remark 9.
(Zhang et al., 2021a) also studies the generalization bound of the ESP solution.
They provide O (1 /n ) order bounds for weak PD population risk and expected strong PD population risk.
Their proofs also show that the expected strong PD population risk is more difﬁcult to analyze than the former.
They have to consider the fact that different ˆ x ∗ S corresponds to different y , as discussed in Remark 5.
Moreover, the expectation operator in expected strong PD population risk also relaxes the difﬁculty of proof.
Speciﬁcally, deﬁne S ( i ) be a dataset by replacing the i -th sample in S with another i.i.d.
sample z (cid:48)
i and y ∗ ( x ) = arg max y ∈Y F ( x , y ) , there holds the following important property E (cid:2)
[Equation]
=n (cid:80) ni =1 E (cid:2) f
(ˆ x ∗ S ( i ) , y ∗ (ˆ x ∗ S ( i ) ); z i ) (cid:3) because (ˆ x ∗ S , y ∗ (ˆ x ∗ S )) and (ˆ x ∗ S ( i ) , y ∗ (ˆ x ∗ S ( i ) )) are identically distributed and the independence between z i and S ( i ) .
On the contrary, when there is no expectation operator, we do not have this property and the proof is much more challenging.
C G RADIENT D ESCENT A SCENT
We need some notations to state results on GDA.
Speciﬁcally, assume the initial point satisﬁes x 1 = 0 and y 1 = 0 .
Let { η t } be a sequence of positive step sizes.
At the t -th iteration, GDA updates
[Equation]
We denote the average of iterates by
[Equation]
Here, we ﬁrst provide an important lemma to connect the argument stability with the strong PD empirical risk, which will also be used in the remaining applications.
Lemma 7.
For any i ∈ [ n ] , deﬁne S ( i ) = { z 1 , ..., z i − 1 , z (cid:48)
i , z i +1 , ..., z n } .
Let ( x t , y t ) be the output produced by F S on dataset S in running a minimax learning algorithm.
Let ( x it , y it ) be the corresponding output produced by F S ( i ) on dataset S ( i ) , where F S ( i ) is empirical risk on dataset S ( i ) .
Suppose Assumption 1 holds.
Assume for all z , the function ( x , y ) (cid:55)→ f ( x , y ; z ) is µ -SC-SC.
For any S ( i ) and S , we have
[Equation]
Proof.
Deﬁne (ˆ x ∗ S ( i ) , ˆ y ∗ S ( i ) ) be the ESP solution on dataset S ( i ) and (ˆ x ∗ S , ˆ y ∗ S ) be the ESP solution on dataset S .
To prove the stability bound, we consider
[Equation]
it
[Equation]
[Table]
[Equation]
where the second inequality uses the result in (33), the third inequality uses the Caucy-Schwarz inequality, and the last inequality uses the strong convexity and strong concavity of F S ( i ) and
F S and the optimality condition (please refer to (32)).
As will see in the rest paper, we bound
F S ( i ) ( x it , ˆ y ∗ S ( i ) )
− F S ( i ) (ˆ x ∗ S ( i ) , ˆ y ∗ S ( i ) ) and F S ( x t , ˆ y ∗ S )
− F S (ˆ x ∗ S , y t ) with the same upper bound since they are all strong PD empirical risk.
Thus, for brevity, we derive the following inequality
[Equation]
The proof is complete.
Remark 10.
Lemma 7 provides the connection between the stability bound and the strong PD empirical risk.
The subscript t here represents not only the output of an iterative optimization algorithm, but any output of the empirical risk of any minimax learning algorithm.
Remark 11.
In studying the stability bound of gradient-based optimization algorithms, a popular approach is to use the property of smoothness to establish the nonexpansiveness of gradient mapping, proposed in the seminal work (Hardt et al., 2016).
(Farnia & Ozdaglar, 2021; Lei et al., 2021) extend this approach to the minimax problems and use it to analyze the stability bound of SGDA, GDA,
PPM, etc.
However, their stability bounds are often derived in expectation.
In (Lei et al., 2021), the authors also use the Chernoff bounds of Bernoulli variables to establish high probability stability bounds when they are to derive high probability generalization bounds.
Unfortunately, these stability bounds are often of slow order O (1 / √ n ) .
To derive sharper stability bounds, we established Lemma 7.
The following lemma shows the strong PD empirical risk of GDA.
Lemma 8.
Suppose Assumption 1 holds and F S ( · , · ) be µ -SC-SC with µ > 0 .
Let { x t , y t } be the sequence produced by (34) with η t = 1 µ ( t + t 0 ) .
Assume t 0 ≥ 0 .
Suppose sup x ∈X (cid:107) x (cid:107) ≤ R X and sup y ∈Y (cid:107)
y
(cid:107) ≤ R Y .
Then for (¯ x T , ¯ y T ) in (35) we have
[Equation]
If t 0 = 0 , then sup
[Equation]
Proof.
Firstly, we have
[Equation]
where the ﬁrst inequality holds because of Assumption 1.
By the strong convexity of F S ( · , y t ) , we have
[Equation]
Since η t = 1 µ ( t + t 0 )
[Equation]
we further get
[Equation]
Multiplying both sides by t + t 0 , we have
[Equation]
Since x 1 = 0 and (cid:80) Tt =1 t − 1 ≤ log( eT ) , by taking a summation of the above inequality from t =to T , we obtain
T
[Equation]
From the concavity of F S ( x , · ) we get
[Equation]
Since this inequality holds for any x
[Equation]
we get
[Equation]
This implies that
[Equation]
In a similar way
[Equation]
we have the following inequality
[Equation]
Combined the above two inequalities together we get
[Equation]
For optimization algorithm GDA, substituting the strong PD empirical risk bound of (¯ x T , ¯ y T ) into
Lemma
[Equation]
we get the following stability bound,
[Equation]
Furthermore, for any x ∈ X , y ∈ Y and z ∈ Z ,
[Equation]
which implies that
[Equation]
Till here, plugging (37), the stability bound in (36) and the strong PD empirical risk bound in Lemma 8 into Theorem 1, we obtain generalization bounds of GDA.
We now write the main theorem of GDA.
Theorem 4.
Assume for all z , the function ( x , y ) (cid:55)→ f ( x , y ; z ) is µ -SC-SC.
Suppose sup x ∈X (cid:107) x (cid:107) ≤
R X and sup y ∈Y (cid:107)
y (cid:107) ≤ R Y .
Let { x t , y t } be produced by (34) with η t = 1 µ ( t + t 0 ) .
Assume t 0 ≥ 0 .
Denote A x ( S ) =
¯ x T and A y ( S ) =
¯ y T for (¯ x T , ¯ y T ) in (35).
Let M = sup z ∈Z f (0 , 0; z )+ L ( R X
+
R Y ) .
Fixed any η > 0 .
There exists an absolute positive constant C .
(a) If Assumption 1 holds, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(b) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(c) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(d) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(e) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
Remark 12.
When conditions in Theorem 4 hold
[Equation]
we obtain that (a) If Assumption 1 holds and
F S (¯ x T , ¯ y T ) =
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the plain generalization error of (¯ x T , ¯ y T ) of GDA is of the order O (cid:16)(cid:16) 1 n + (cid:113) log TT (cid:17) log 2 n log(1 /δ )
(cid:17) .
(b) If Assumptions 1 and 2 hold and R S (¯ x T )
=
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the primal generalization error of (¯ x T , ¯ y T ) of GDA is of the order O (cid:16)(cid:16) 1 n + (cid:113) log TT (cid:17) log 2 n log(1 /δ )
(cid:17) .
(c)
If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , the strong PD population risk and the strong PD generalization error of (¯ x T , ¯ y T ) of GDA are all of the order
O (cid:16)(cid:16) 1 n + (cid:113) log TT (cid:17) log 2 n log(1 /δ )
(cid:17) .
(d) If Assumptions 1 and 2 hold and inf x ∈X R ( x ) =
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the excess primal population risk of (¯ x T , ¯ y T ) of GDA is of the order O (cid:16)(cid:16) 1 n + (cid:113) log TT (cid:17) log 2 n log(1 /δ )
(cid:17) .
For the above bounds, we can take
T =
O ( n 2 ) gradient evaluations to get bound of the order O (cid:16) log 3 / 2 n n log(1 /δ )
(cid:17) .
D S TOCHASTIC G RADIENT D ESCENT A SCENT
We need some notations to state results on SGDA.
Speciﬁcally, assume the initial point satisﬁes x 1 = 0 and y 1 = 0 .
Let { η t } be a sequence of positive step sizes.
At the t -th iteration, SGDA
[Table]
[Equation]
We denote the average of iterates by
[Equation]
Let’s ﬁrst introduce two concentration inequalities for martingales, which are required in deriving the strong PD empirical risk bound of SGDA.
Lemma 9.
(Boucheron et al., 2013) Let z 1 , ..., z n be a sequence of random variables such that z k may depend the previous variables
z 1 , ..., z k − 1 for all k = 1 , ..., n .
Consider a sequence of functionals ξ k ( z 1 , ..., z k ) , k = 1 , ..., n .
Assume | ξ k − E z k [ ξ k ] | ≤ b k for each k .
Let δ ∈ (0 , 1) .
With probability at least 1 − δ n (cid:88)
k
[Equation]
(cid:16)
[Equation]
Lemma 10.
(Tarres & Yao, 2014) Let { ξ k } k ∈ N be a martingale difference sequence in R d .
Suppose that almost surely (cid:107) ξ k (cid:107) ≤ D and (cid:80) tk =1 E [ (cid:107) ξ k (cid:107) 2 | ξ 1 , ..., ξ k − 1 ] ≤ σ 2 t .
Then, for any 0 < δ < 1 , the following inequality holds with probability at least 1 − δ
[Equation]
The following lemma shows the strong PD empirical risk bound of SGDA.
Lemma 11.
Suppose Assumption 1 holds and F S ( · , · ) be µ -SC-SC with µ > 0 .
Let { x t , y t } be the sequence produced by (38) with η t = 1 µ ( t + t 0 ) .
Assume t 0 ≥ 0 .
Suppose sup x ∈X (cid:107) x (cid:107) ≤ R X and sup y ∈Y (cid:107)
y
(cid:107) ≤ R Y .
Let δ > 0 .
Then for (¯ x T , ¯ y T ) in (39), with probability at least 1 − δ we have
[Equation]
If t 0 =
[Equation]
then with probability at least 1 − δ we have
[Equation]
Proof.
This proof follows from (Lei et al., 2021).
Firstly, we have
[Equation]
where the ﬁrst inequality holds because of Assumption 1.
By the strong convexity of F S ( ·
[Equation]
y t )
[Equation]
we have
[Equation]
Since η t = 1 µ ( t + t 0 ) , we further get
[Equation]
Multiplying both sides by t + t 0 , we have
[Equation]
Since x 1 = 0 and (cid:80) Tt =1 t − 1 ≤ log( eT )
[Equation]
by taking a summation of the above inequality from t =to T , we obtain
[Equation]
From the concavity of F S ( x , · ) we get
T (cid:88)
[Equation]
Since this inequality holds for any x , we get
T (cid:88)
[Equation]
By Schwarz’s inequality, we have
T (cid:88)
[Equation]
+
T (cid:88) t
= (cid:104) x t , ∇ x F S ( x t , y t ) − ∇ x f ( x t , y t ; z i t ) (cid:105) .
Denote ξ t = (cid:104) x t , ∇ x F S ( x t , y t ) − ∇ x f ( x t , y t ; z i t ) (cid:105) .
Since E i t [ (cid:104) x t , ∇ x F S ( x t , y t ) − ∇ x f ( x t , y t ; z i t ) (cid:105) ]
= 0 , so { ξ t | t = 1 , ..., T } is a martingale difference sequence.
By Schwarz’s inequality and Assumption 1, we know that |(cid:104) x t , ∇ x F S ( x t , y t ) − ∇ x f ( x t , y t ; z i t ) (cid:105)| ≤ 2 LR X .
Then, according to Lemma 9, we have the following inequality with probability at least 1 − δ/
T (cid:88) t =
[Equation]
Deﬁne ξ (cid:48) t = ∇ x f ( x t , y t ; z i t ) − ∇ x F S ( x t , y t ) .
Then we get (cid:107) ξ (cid:48) t (cid:107) ≤ 2 L and
T (cid:88) t =
E [ (cid:107) ξ (cid:48) t (cid:107) 2 | ξ (cid:48) 1 , ..., ξ (cid:48)
t − 1 ] ≤ 4 TL 2 .
Applying Lemma 10 to the martingale difference sequence { ξ (cid:48) t } , we have the following inequality with probability at least 1 − δ/
[Equation]
That is, with probability at least 1 − δ/
[Equation]
Combined with the above results
[Equation]
we ﬁnally have the following inequality with probability at least 1 − δ/
T
T (cid:88)
[Equation]
In a similar way
[Equation]
we have the following inequality with probability at least 1 − δ/
[Equation]
Combined the above two inequalities together we get the result with probability at least 1 − δ
[Equation]
Denote E
= µt 0 ( R 2 X + R 2 Y ) T
[Equation]
Now, plugging Lemma 11 to Lemma
[Equation]
we know that the argument stability bound of SGDA is
[Equation]
Furthermore, for any x ∈ X , y ∈ Y and z ∈ Z ,
[Equation]
Note that since SGDA is a randomized algorithm, thus we need the following variant of Theorem 1.
Theorem 5.
Let A be a randomized learning algorithm and (cid:15) > 0 .
Suppose | f ( x , y ; z ) | ≤ M for some M > 0 and x ∈ X , y ∈ Y , z ∈ Z .
Fixed any η > 0 .
There exists an absolute positive constant C .
(a
If
A has (cid:15) -uniform
stability
with
probability at least 1 − δ
(cid:48) for
some δ
(cid:48) ∈ (0 ,
1) over the randomness of A , i.e.,
[Equation]
And if the randomness of A is independent of the training set S .
Then for any δ > 0 , with probability at least 1 − δ (cid:48) − δ ,
[Equation]
(b.) Assume that for all x , the function y
(cid:55)→ F ( x , y ) is µ -strongly-concave.
Suppose Assumptions 1 and 2 hold.
If the algorithm A is (cid:15) -argument stable with probability at least 1 − δ (cid:48) for some δ (cid:48) ∈ (0 , 1) over the randomness of A , i.e.,
[Equation]
And if the randomness of A is independent of the training set S .
Then for any δ > 0 , with probability at least 1 − δ (cid:48) − δ ,
[Equation]
(c.)
Assume that for all x and y , the function F ( x , y ) is µ -SC-SC.
Suppose Assumptions 1 andhold.
If the algorithm A is (cid:15) -argument stable with probability at least 1 − δ (cid:48) for some δ (cid:48) ∈ (0 , 1) over the randomness of A , i.e.,
[Equation]
And if the randomness of A is independent of the training set S .
Then for any δ > 0 , with probability at least 1 − δ (cid:48) − δ , (cid:52) s ( A x ( S ) ,
A
[Equation]
(d.) Assume that for all x and y , the function F ( x , y ) is µ -SC-SC.
Suppose Assumptions 1 andhold.
If the algorithm A is (cid:15) -argument stable with probability at least 1 − δ (cid:48) for some δ (cid:48) ∈ (0 , 1) over the randomness of A , i.e.,
[Equation]
And if the randomness of A is independent of the training set S .
Then for any δ > 0 , with probability at least 1 − δ (cid:48) − δ ,
[Equation]
(e.) Assume that for all x , the function y
(cid:55)→ F ( x , y ) is µ -strongly-concave.
Suppose Assumptions 1 and 2 hold.
If the algorithm A is (cid:15) -argument stable with probability at least 1 − δ (cid:48) for some δ (cid:48) ∈ (0 , 1) over the randomness of A , i.e.,
[Equation]
And if the randomness of A is independent of the training set S .
Then for any δ > 0 , with probability at least 1 − δ (cid:48) − δ ,
[Equation]
Therefore, plugging (41), the stability bound in (40) and the strong PD empirical risk bound in
Lemma 11 into Theorem 5, we obtain generalization bounds of SGDA.
Now, we write the main theorem of SGDA as follows.
Theorem 6.
Assume for all z , the function ( x , y ) (cid:55)→ f ( x , y ; z ) is µ -SC-SC.
Suppose sup x ∈X (cid:107) x (cid:107) ≤
R X and sup y ∈Y (cid:107)
y (cid:107) ≤ R Y .
Let { x t , y t } be produced by (38) with η t = 1 / ( µ ( t + t 0 )) .
Assume t 0 ≥ 0 .
Denote A x ( S ) =
¯ x T and A y ( S ) =
¯ y T for ( ¯ x T , ¯ y T ) in (39).
Fixed any η > 0 .
Let
[Equation]
and B = 4 Lnµ +4 (cid:113) 1 µ √ E .
There exists an absolute positive constant C .
(a) If Assumption 1 holds, then for any δ > 0 , with probability at least 1 − 2 δ , we have
[Equation]
(b) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − 2 δ , we have
[Equation]
(c) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − 2 δ , we have
[Equation]
(d) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − 2 δ , we have
[Equation]
(e) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − 2 δ , we have
[Equation]
Remark 13.
When conditions in Theorem 6 hold, we obtain that (a)
If Assumption 1 holds and
F S (¯ x T , ¯ y T ) =
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the plain generalization error of (¯ x T , ¯ y T ) of SGDA is of the order O (cid:16)(cid:16) 1 n + (cid:113)
[Equation] [Equation]
(b) If Assumptions 1 and 2 hold and R S (¯ x T )
=
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the primal generalization error of (¯ x T , ¯ y T ) of SGDA is of the order
O (cid:16)(cid:16) 1 n + (cid:113) log(1 /δ )
T(cid:17) log 2 n log(1 /δ )
(cid:17) .
(c) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , the strong PD population risk and strong PD generalization error of (¯ x T , ¯ y T ) of SGDA are all of the order O (cid:16)(cid:16) 1 n + (cid:113) log(1 /δ )
T(cid:17) log 2 n log(1 /δ )
(cid:17) .
(d) If Assumptionsand 2 hold and inf x ∈X R ( x )
=
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the excess primal population risk of (¯ x T , ¯ y T ) of SGDA is of the order O (cid:16)(cid:16) 1 n + (cid:113) log(1 /δ )
T(cid:17) log 2 n log(1 /δ )
(cid:17) .
For the above bounds, we can take T
= O ( n 4 ) stochastic gradient evaluations to get bound of the order O (cid:16) log nn log 32 (1 /δ )
(cid:17) .
E P ROXIMAL P OINT M ETHOD
One of the classical algorithms studied for solving the minimax problem is the Proximal Point method (Rockafellar, 1976).
We denote the t -th iterate of PPM as ( x t , y t ) .
The averaged iterate is deﬁned as
[Equation]
Given stepsize parameter ν , the PPM generates the iterate { x t +1 , y t +1 } by
[Equation]
{ x t +1 , y t +1 } is the unique solution since the objective function of problem (43) is strongly convex in x and strongly concave in y .
From the discussion in (Mokhtari et al., 2019), the update of PPM can be written as
[Equation]
Assume that the initial point satisﬁes x 0 = 0 and y
= 0 .
We now begin to prove the strong PD empirical risk.
Firstly, two lemmas are introduced.
Lemma 12.
(Nemirovski, 2005)
Deﬁne vector v =
[ x , y ] ∈ R 2 d and the operator P : R 2 d (cid:55)→ R 2 d as
[Equation]
Consider (¯ x T , ¯ y T ) in (42).
Suppose the ESP solution exists.
Assume the function F S ( x , y ) is continuously differentiable in x and y .
Assume that F S ( x , y ) is a convex function of x for any y and is a concave function of y for any x .
Then for any v = [ x , y ] ∈ R 2 d , we have
[Equation]
Lemma 13.
(Mokhtari et al., 2019) Consider the sequence of iterates { v t } ∈ R 2 d generated by the following update
[Equation]
where P is a monotone and Lipschitz continuous operator, and ν is a positive constant.
Then for any v ∈ R 2 d and for each t ≥ 1 we have
[Equation]
The following lemma is the strong PD empirical risk bound of PPM.
Lemma 14.
Let { x t , y t } be the iterates generated by PPM in (44).
Assume ν is a positive constant.
Suppose the ESP solution exists.
Assume that F S ( x , y ) is a convex function of x for any y and is a concave function of y for any x .
Suppose sup x ∈X (cid:107) x (cid:107) ≤ R X and sup y ∈Y (cid:107)
y
(cid:107) ≤ R Y .
If
Assumption 2 holds, then for all T ≥ 1 , we have
[Equation]
Proof.
The update of the PPM in (44) can be written as
[Equation]
According to Lemma 1 in (Mokhtari et al., 2019), if F S ( x , y ) is convex-concave and Assumptionholds, then P ( v ) deﬁned in (45) is monotone and Lipschitz continuous.
According to Lemma 13, we have
[Equation]
Taking a summation of the above inequality from t = 0 to T − 1 , we obtain
[Equation] [Equation]
According to (46), we know that
[Equation]
Combined this result with Lemma 12, we can write
[Equation]
which implies that
[Equation]
The proof is complete
[Equation]
Combined Lemma 7 and Lemma 14, we know that the argument stability bound of PPM is
[Equation]
Furthermore, for any x ∈ X
[Equation]
y ∈ Y and z ∈ Z ,
[Equation]
Therefore, plugging (48), the stability bound in (47) and the strong PD empirical risk bound in
Lemma 14 into Theorem 1, we obtain generalization bounds of PPM, shown as below.
Theorem 7.
Assume for all z , the function ( x , y ) (cid:55)→ f
( x , y ; z ) is µ -SC-SC.
Suppose sup x ∈X (cid:107) x (cid:107) ≤
R X and sup y ∈Y (cid:107)
y (cid:107) ≤ R Y .
Let { x t , y t } be produced by (44).
Assume ν is a positive constant.
Denote A x ( S ) =
¯ x T and A y ( S ) =
¯ y T for (¯ x T , ¯ y T ) in (42).
Fixed any η > 0 .
Let
M = sup z ∈Z f (0 , 0; z )
+ L ( R X + R Y ) .
Let E
= R 2 X
+ R 2 Y 2 νT and B = 4 Lnµ
+ 4 (cid:113) 1 µ √ E .
There exists an absolute positive constant C .
(a) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(b) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(c) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(d) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
T
[Equation]
¯
[Equation]
T
[Equation]
(e) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
Remark 14.
When conditions in Theorem 7 hold, we obtain that (a)
If Assumption 1 and 2 hold and
F S (¯ x T , ¯ y T ) =
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the plain generalization error of (¯ x T , ¯ y T ) of PPM is of the order O (cid:16)(cid:16) 1 n + (cid:113) 1 T (cid:17) log 2 n log(1 /δ )
(cid:17) .
(b) If Assumptions 1 and 2 hold and R S (¯ x T )
=
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the primal generalization error of (¯ x T , ¯ y T ) of PPM is of the order O (cid:16)(cid:16) 1 n + (cid:113) 1 T (cid:17) log 2 n log(1 /δ )
(cid:17) .
(c) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , the strong
PD population risk and strong PD generalization error of (¯ x T , ¯ y T ) of PPM are all of the order
O (cid:16)(cid:16) 1 n + (cid:113) 1 T (cid:17) log 2 n log(1 /δ )
(cid:17) .
(d) If Assumptions 1 and 2 hold and inf x ∈X R ( x ) =
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the excess primal population risk of (¯ x T , ¯ y T ) of
PPM is of the order O (cid:16)(cid:16) 1 n + (cid:113) 1 T (cid:17) log 2 n log(1 /δ )
(cid:17) .
For the above bounds, we can take T
= O ( n 2 ) gradient evaluations to get bound of the order O (cid:16) log nn log(1 /δ )
(cid:17) .
F E XTRAGRADIENT M ETHOD
EG is a classical algorithm for solving minimax problems introduced by (Korpelevich, 1976).
We now introduce some notations.
Followed (Mokhtari et al., 2019), we consider the following update of EG: given stepsize parameter ν , we ﬁrst compute a set of mid-point iterates { x t + 12 , y t + 12 }
[Equation]
we then compute the next iterates { x t +1 , y t +1 }
[Equation]
Consider the averaged iterate
[Equation]
Assume that the initial point satisﬁes x 0 = x − 1 / 2 and y
= y − 1 / 2 .
We now show the strong PD empirical risk bound for (¯ x T , ¯ y T ) of EG.
Lemma 15.
(Mokhtari et al., 2019) Let { x t , y t } , { x t +1 / 2 , y t +1 / 2 } be the iterates generated by the
EG updates in (49) and (50).
Assume that the initial point satisﬁes x 0 = x − 1 / 2 and y
= y − 1 / 2 .
Suppose the ESP solution (ˆ x ∗ , ˆ y ∗ ) exists.
Assume that F S ( x , y ) is a convex function of x for any y and is a concave function of y for any x .
If Assumption 2 holds and the stepsize ν satisﬁes the condition ν
= c 2 β for any c ∈ (0 , 1) , then:
(a) the iterates { x t , y t
} , { x t +1 / 2 , y t +1 / 2 } stay within the compact convex set
[Equation]
(b for all T ≥ 1 , we have sup
[Equation]
T .
Combined Lemma 7 and Lemma 15, we know that the argument stability bound of EG is
[Equation]
Furthermore, for any x ∈ X , y ∈ Y and z ∈ Z ,
[Equation]
where the ﬁrst inequality follows from Assumption
[Equation]
the second inequality follows from CaucySchwarz inequality and the last inequality follows from (52).
Now, plugging (54), the stability bound in (53) and the strong PD empirical risk bound in Lemma 15 into Theorem
[Equation]
we obtain generalization bounds of EG.
The main theorem is shown below.
Theorem 8.
Assume for all z , the function ( x , y ) (cid:55)→ f ( x , y ; z ) is µ -SC-SC.
Let { x t , y t } and { x t +1 / 2 , y t +1 / 2 } be the iterates produced by (49)-(50).
Assume the stepsize ν satisﬁes the condition ν
= c 2 β for any c ∈ (0 , 1) .
Denote A x ( S ) =
¯ x T and A y ( S ) =
¯ y T for (¯ x T , ¯ y T ) in (51).
Denote the ESP solution as (ˆ x ∗ , ˆ y ∗ ) .
Consider the compact convex set in (52).
Fixed any η > 0 .
Let M = sup z ∈Z f
(ˆ x ∗ , ˆ y ∗ ; z ) + L (cid:114)(cid:16) 4 +
[Equation]
Let
B =
[Equation]
+
[Equation]
and E = 4 Lnµ + 4 (cid:113) 1 µ √ B .
There exists an absolute positive constant C .
(a) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(b) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(c) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(d) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(e) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
Remark 15.
When conditions in Theorem 8 hold
[Equation]
we obtain that (a) If Assumptions 1 and 2 hold and
F S ( ¯ x T , ¯ y T ) =
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the plain generalization error of (¯ x T , ¯ y T ) of EG is of the order O (cid:16)(cid:16) 1 n + (cid:113) 1 T (cid:17) log 2 n log(1 /δ )
(cid:17) .
(b) If Assumptionsand 2 hold and R S (¯ x T )
=
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the primal generalization error of (¯ x T , ¯ y T ) of EG is of the order O (cid:16)(cid:16) 1 n + (cid:113) 1 T (cid:17) log 2 n log(1 /δ )
(cid:17) .
(c) If
Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , the strong PD population risk and the strong PD generalization error of (¯ x T , ¯ y T ) of EG are all of the order O (cid:16)(cid:16) 1 n + (cid:113) 1 T (cid:17) log 2 n log(1 /δ )
(cid:17) .
(d) If Assumptions 1 and 2 hold and inf x ∈X R ( x ) =
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the excess primal population risk of (¯ x T , ¯ y T ) of EG is O (cid:16)(cid:16) 1 n + (cid:113) 1 T (cid:17) log 2 n log(1 /δ )
(cid:17) .
For the above bounds, we can take T
= O ( n 2 ) gradient evaluations to get bound of the order O (cid:16) log nn log(1 /δ )
(cid:17) .
G O PTIMISTIC G RADIENT D ESCENT A SCENT
OGDA is introduced by Popov (1980), as a variant of the EG method.
We introduce some notations to state the result of OGDA.
Given a stepsize parameter ν > 0 , OGDA do the following update for each t ≥
[Equation]
Assume that the initial point satisﬁes x 0 = x − 1 and y
= y − 1 .
Consider the averaged iterate ¯ x T
[Equation]
t
[Equation] [Equation]
t and
[Equation] [Equation]
We ﬁrst provide a lemma on the strong PD empirical risk of OGDA.
Lemma 16.
(Mokhtari et al., 2019) Let { x t , y t } be the iterates generated by the OGDA updates in (55).
Assume that the initial point satisﬁes x 0 = x − 1 and y
= y − 1 .
Suppose the ESP solution (ˆ x ∗ , ˆ y ∗ ) exists.
Assume that F S ( x , y ) is a convex function of x for any y and is a concave function of y for any x .
If Assumption 2 holds and the stepsize ν satisﬁes 0 < ν ≤ 14 β , then:
(a) the iterates { x t , y t } stay within the compact convex set
[Equation]
(b) for all T ≥ 1 , we have
[Equation]
Combined Lemma 7 and Lemma 16, we know that the argument stability bound of OGDA is
[Equation]
Moreover, similar to (54)
[Equation]
we have
[Equation]
Therefore, plugging (59)
[Equation]
the stability bound in (58) and the strong PD empirical risk bound in
Lemma 16 into Theorem 1, we obtain generalization bounds of OGDA
[Equation]
Theorem 9.
Assume for all z , the function ( x , y ) (cid:55)→ f ( x , y ; z ) is µ -SC-SC.
Let { x t , y t } be produced by (55).
Assume the stepsize parameter ν satisﬁes 0 < ν ≤ 14 β .
Denote A x ( S ) =
¯ x T and
A y ( S ) =
¯ y T for (¯ x T , ¯ y T ) in (56).
Denote the ESP solution as (ˆ x ∗ , ˆ y ∗ ) .
Consider the compact convex set in (57).
Fixed any η > 0 .
Let M = sup z ∈Z f
(ˆ x ∗ , ˆ y ∗ ; z ) + 2 L (cid:112) (cid:107) x
0 − ˆ x ∗ (cid:107) 2 + (cid:107) y 0 − ˆ y ∗ (cid:107) 2 .
Deﬁne B = (16 β + 12 ν )( (cid:107) x 0 − ˆ x ∗ (cid:107) 2 + (cid:107)
y
− ˆ y ∗ (cid:107) 2 ) T and E =
Lnµ + 4 (cid:113) 1 µ √ B .
There exists an absolute positive constant C .
(a) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(b) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(c) If Assumptions 1 and 2 hold, then for any δ > 0 , with
probability at least 1 − δ , we have
[Equation]
(d) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
(e) If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , we have
[Equation]
Remark 16.
When conditions in Theorem 9 hold, we obtain that (a) If Assumptions 1 and 2 hold and
F S (¯ x T , ¯ y T ) =
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the plain generalization error of (¯ x T , ¯ y T ) of OGDA is of the order O (cid:16)(cid:16) 1 n + (cid:113) 1 T (cid:17) log 2 n log(1 /δ )
(cid:17) .
(b) If Assumptionsand 2 hold and R S (¯ x T )
=
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the primal generalization error of (¯ x T , ¯ y T ) of OGDA is of the order O (cid:16)(cid:16) 1 n + (cid:113) 1 T (cid:17) log 2 n log(1 /δ )
(cid:17) .
(c)
If Assumptions 1 and 2 hold, then for any δ > 0 , with probability at least 1 − δ , the strong PD population risk and the strong PD generalization error of (¯ x T , ¯ y T ) of OGDA are all of the order
O (cid:16)(cid:16) 1 n + (cid:113) 1 T (cid:17) log 2 n log(1 /δ )
(cid:17) .
(d) If Assumptions 1 and 2 hold and inf x ∈X R ( x ) =
O (cid:16) 1 n (cid:17) , then for any δ > 0 , with probability at least 1 − δ , the excess primal population risk of OGDA is of the order O (cid:16)(cid:16) 1 n + (cid:113) 1 T (cid:17) log 2 n log(1 /δ )
(cid:17) .
For the above bounds, we can take T
= O ( n 2 ) gradient evaluations to get bound of the order O (cid:16) log nn log(1 /δ )
(cid:17) .
H A UXILIARY D ESCRIPTIONS OF T ABLE
In Table 1, Lip means Lipschitz continuity and S means smoothness.
(R)-ESP means the (regularized)-empirical risk saddle point (Zhang et al., 2021a).
C-SC means convex- µ -stronglyconcave, and NC-SC means nonconvex- µ -strongly-concave.
A function f ( x , y ) is called nonconvex-strongly-concave if f ( x , · ) is strongly-concave for every x .
Moreover, a function f ( x , y ) is µ -weakly-convex-weakly-concave (WC-WC) if f + µ 2 (cid:0) (cid:107) x (cid:107) 2 + (cid:107) y (cid:107) 2 (cid:1) is convex-concave.
V-WCWC is a variant of WC-WC, please refer to (Lei et al., 2021).
PL means the two-sided PL condition, which relaxes the convex-concavity requirement of the objective function (Yang et al., 2020) and is usually used to guarantee the linear convergence rate (Karimi et al., 2016; Yang et al., 2020).
AGDA algorithm is variant of GDA with alternating updates of the primal-dual variables.
c is a parameter in the step size, β is a parameter in Assumption 2 and k :
= β/µ .
I N UMERICAL E XPERIMENTS
In this section, we report preliminary experimental results to verify our theoretical results by performing numerical experiments on the simulated data.
We study how the generalization error would behave along the number of samples.
To this aim, we consider an isotropic Gaussian data vector
Z ∼ N ( 0 , I d × d ) with zero mean and identity covariance.
We will draw n independent samples from the underlying Gaussian distribution to form a training dataset S = { z 1 , ..., z n } .
We set the dimension d of Z as 50 .
Similar to the strongly-convex-strong-concave case of (Farnia & Ozdaglar, 2021), we consider the following minimax objective function
[Equation]
In the experiments, we set µ = 1 and constrain optimization variables x and y to satisfy (cid:107) x (cid:107) , (cid:107) y (cid:107) ≤ 100 which we enforced by projection.
For this minimax objective function, one can verify that
[Equation]
where E [ Z ] = 0 since the mean of the underlying Gaussian distribution is 0 , and where E S [ Z ] =n (cid:80) ni
=
z i .
For brevity, we call | x T ( E [ Z ] − E S
[ Z ]) | the “generalization error”.
We apply the above experimental settings to validate the theoretical results of GDA, SGDA, EG, and OGDA.
We evaluate the generalization error | x T ( E [ Z ] − E S
[ Z ]) | and apply these algorithms to S .
For GDA and SGDA, we consider the stepsize parameter as 1 /t .
We iterate GDA with ntimes and SGDA with n 4 times.
The generalization error of GDA and SGDA with different sizes of training data are reported in Figure 1.
And for EG and OGDA, we select the stepsize parameter as 0 .
We run EG and OGDA n 2 times.
Similarly, the generalization error of EG and OGDA with different sizes of training data are given in Figure 2.
From the two ﬁgures, we can see that the line of best ﬁt for the generalization error is log 3 / 2 n n 0 .
98 for GDA, log n n 0 .
98 for SGDA, log 0 .
99 n n 0 .for EG, and log n n 1 .for OGDA.
These results match the predictive rates of the plain generalization error and the primal generalization error in Table 1, i.e., log 3 / 2 n n for GDA, log nn for SGDA, log nn for EG, and log nn for
OGDA, which veriﬁes our theoretical ﬁndings.