Sampling without Replacement Leads to Faster Rates in Finite-Sum Minimax Optimization
Abstract
We analyze the convergence rates of stochastic gradient algorithms for smooth finite-sum minimax optimization and show that, for many such algorithms, sampling the data points without replacement leads to faster convergence compared to sampling with replacement.
For the smooth and strongly convex-strongly concave setting, we consider gradient descent ascent and the proximal point method, and present a unified analysis of two popular without-replacement sampling strategies, namely Random Reshuffling (RR), which shuffles the data every epoch, and Single
Shuffling or Shuffle Once (SO), which shuffles only at the beginning.
We obtain tight convergence rates for RR and SO and demonstrate that these strategies lead to faster convergence than uniform sampling.
Moving beyond convexity, we obtain similar results for smooth nonconvex-nonconcave objectives satisfying a two-sided
Polyak-Łojasiewicz inequality.
Finally, we demonstrate that our techniques are general enough to analyze the effect of data-ordering attacks , where an adversary manipulates the order in which data points are supplied to the optimizer.
Our analysis also recovers tight rates for the incremental gradient method, where the data points are not shuffled at all.
Introduction
The approximate solution of large-scale optimization problems using first-order stochastic gradient methods constitutes one of the foundations of classical machine learning.
However, emerging problems in machine learning go beyond pattern recognition and involve real-world decision making, where learning algorithms interact with unknown or even adversarial environments or are deployed in multi-agent settings.
Decision making in such environments often involves solving a minimax optimization problem of the form min
x max y F ( x , y ) , whose analysis has been a focus of research in mathematics, economics and theoretical computer science [38, 13, 10].
Recent examples of its applications in machine learning include adversarial learning [32, 52, 6], reinforcement learning [30, 58, 11, 42], imitation learning [14, 8, 24], and generative adversarial networks [18, 2].
In most of these applications, the objective F ( x , y ) has a finite-sum structure, i.e., F ( x , y )
{+= 1 /n (cid:80) ni =
+}
f i ( x , y ) where n denotes the size of the dataset and each component function f i denotes the objective associated with the i th data point.
The resultant problem is known as finite-sum minimax optimization : min x ∈ R d x max y ∈ R
d y n n (cid:88) i = f i ( x , y ) .
When n is large and the f i ’s are differentiable (which holds for most applications), approximate solutions to (1) are computed using stochastic gradient algorithms.
These algorithms typically sample an index i ∈ [ n ] at each iteration as per some specified sampling routine, and use the gradients of f i to compute the next iterate.
Among these methods, perhaps the simplest and most commonly used algorithm is Stochastic Gradient Descent Ascent (SGDA), a natural extension of Stochastic Gradient
Descent (SGD) for minimax optimization.
Similar to the stochastic optimization literature, analysis of stochastic minimax optimization often assumes that, at every iteration, the index i is sampled uniformly with replacement .
Analysis of the resulting algorithm closely parallels the analysis of SGD, and relies on the fact that uniform sampling leads to unbiased gradient estimates.
Recent works [20, 31] have also extended this paradigm to i.i.d uniform sampling of mini-batches.
While uniform sampling assumptions simplify theoretical analysis, practical implementations of these algorithms often deviate from this paradigm, and instead incorporate various heuristics, which are often empirically found to improve runtime.
A common and notable heuristic is to replace uniform sampling by procedures that perform multiple passes over the entire dataset, and in each such pass (called an epoch), sample the data points without replacement .
Thus, each data point is sampled exactly once in every epoch.
These procedures are generally implemented using one of the following approaches:
Random Reshuffling (RR): Uniformly sample a random permutation of [ n ] at the start of every epoch, and process the data points within that epoch as per the order specified by the permutation.
Single Shuffling or Shuffle Once (SO): Uniformly sample a random permutation at the beginning and reuse it across all epochs to order the data points.
Incremental Gradient (IG): Do not permute the data points at all and follow a fixed deterministic data ordering for every epoch.
Sampling without replacement is ubiquitous in both stochastic minimization [7, 48, 4] and stochastic minimax optimization [18, 2] as it often exhibits faster runtime than uniform sampling.
However, these empirical benefits come at the cost of limited theoretical understanding, due to the absence of provably unbiased gradient estimates.
It is well known in the optimization literature that SGD with replacement has a tight rate of O (1 /nK ) for smooth and strongly convex minimization [44, 25], where n is the number of component functions and K denotes the number of epochs.
On the contrary, recent works on SGD without replacement for smooth and strongly convex minimization [1, 36, 34, 39] show that both RR and SO achieve a non-asymptotic rate of ˜ O
(1 /nK 2 ) , once the number of epochs K is larger than a certain threshold
K 0 (usually polynomial in the condition number), and thereby converge faster than SGD with replacement.
These rates match the lower bound of Ω(1 /nK 2 ) for RR and SO established in prior works [43, 48], modulo logarithmic factors.
For RR, prior works have also established a similar ˜ O (1 /nK 2 ) rate for nonconvex objectives satisfying the Polyak-Łojasiewicz (PŁ) inequality [1, 34].
While the asymptotic behavior of IG has been known to the community for a long time in both smooth and non-smooth settings [5, 37], non -asymptotic ˜ O (1 /K 2 ) convergence rates have been established quite recently [34, 39, 22], and are complemented by a matching Ω(1 /K 2 ) lower bound [48].
1.1 Contributions
Although the empirical benefits of sampling without replacement have been substantiated for minimization, analysis of these methods for minimax optimization have received much less attention, despite being widely prevalent in many applications.
Our work aims to fill this gap by analyzing these methods for minimax optimization.
To this end, our main contributions are as follows:
Unified analysis of RR and SO for smooth strongly convex-strongly concave problems : We analyze RR and SO in conjunction with simultaneous Gradient Descent Ascent (GDA), calling the resulting algorithms GDA-RR and GDA-SO, respectively.
Assuming the components f i are smooth and F is strongly convex-strongly concave, we present a unified analysis of GDA-RR/SO and establish a convergence rate of ˜ O
(exp( − K/ 5 κ 2 ) + 1 /nK 2 ) for both (where κ is the condition number).
Comparing with lower bounds, we show that our rates are nearly tight , i.e., they differ from the lower bound only by an exponentially decaying term.
Moreover, when K ≥ 10 κ 2 log( n 1 / 2 K ) , the convergence rate matches the lower bounds for GDA-RR/SO, modulo logarithmic factors, and also converges provably faster than SGDA with replacement.
Under the same setting, we obtain similar guarantees for the RR and SO variants of the Proximal Point Method (PPM), named PPM-RR and PPM-SO respectively.
Our analysis for both GDA-RR/SO and PPM-RR/SO is general enough to cover smooth strongly monotone finite-sum variational inequalities, which covers minimization, minimax optimization, and multiplayer games.
RR for smooth two-sided PŁ objectives: We consider a class of nonconvex-nonconcave problems where the objective F satisfies a two-sided Polyak-Łojasiewicz inequality.
For such problems, we propose an algorithm that combines RR with two-timescale Alternating Gradient Descent Ascent (AGDA), which we call AGDA-RR.
We show that AGDA-RR has a nearly tight convergence rate of ˜ O
(exp( − K/ 365 κ 3 ) + 1 /nK 2 ) when the gradient variance is uniformly bounded.
When
K ≥ 730 κ 3 log( n 1 / 2 K ) , this rate matches the lower bound (modulo logarithmic factors) and improves on the best known rates of with-replacement algorithms for this class of problems.
Minimax optimization under data ordering attacks: Our techniques for analyzing RR/SO generalize to the analysis of finite-sum minimax optimization under data ordering attacks [51].
These attacks target the inherent randomness assumptions of stochastic gradient algorithms, significantly increasing training time and reducing model quality, only by manipulating the order in which the algorithm receives data points, without performing any data contamination.
To model these attacks, we propose the Adversarial Shuffling (AS) setup, where the data points are shuffled every epoch by a computationally unrestricted adversary.
In this setup, we show that GDA and PPM (now called GDA-AS and
PPM-AS) have a convergence rate of ˜ O
(exp( − K/ 5 κ 2 )+1 /K 2 ) for smooth strongly convex-strongly concave objectives, and AGDA (now called AGDA-AS) has a rate of ˜ O
(exp( − K/ 365 κ 3 ) + 1 /K 2 ) for two-sided PŁ objectives.
We note that, compared to RR and SO, the convergence rate worsens by a factor of 1 /n for large enough K .
When n is large (true for most applications), this slowdown significantly impacts convergence and thus, theoretically justifies the empirical observations in prior work [51].
We also establish that our analysis in the AS regime also applies to the Incremental
Gradient (IG) variants of these algorithms (namely GDA-IG, PPM-IG, and AGDA-IG), and use this to show that our obtained rates for GDA-RR and AGDA-RR are nearly tight.
To the best of our knowledge, our work is the first to: 1) analyze RR, SO, and IG for strongly monotone unconstrained variational inequalities, 2) analyze RR and IG for a class of nonconvex-nonconcave minimax problems, 3) provably demonstrate the advantages of sampling without replacement for both these settings and justify its empirical benefits in a wide variety of problems ranging from minimization, minimax optimization to smooth multiplayer games, 4) analyze sampling without replacement under data-ordering attacks.
Furthermore, unlike prior works on sampling without replacement for minimax optimization [55, 33], which are restricted to random reshuffling and require the component functions to be convex-concave, Lipschitz, and smooth, our analysis does not impose any restrictions on the components
f i other than smoothness, allowing them to be arbitrary nonconvex-nonconcave functions.
Notation and Preliminaries
We work with Euclidean spaces ( R d , ⟨ ., .
⟩ ) equipped with the standard inner product ⟨ x 1 , x 2 ⟩ and the induced norm | x | .
For any x ∈ R d x and y ∈ R d y , we denote z =
( x , y ) ∈
R d where d = d x + d y .
Moreover, for any z 1 = ( x 1 , y 1 ) ∈ R d and z 2 = ( x 2 , y 2 ) ∈ R d , ⟨ z 1 , z 2 ⟩ = ⟨ x 1 , x 2 ⟩ + ⟨ y 1 , y 2 ⟩ and | z 1 | 2 = | x 1 | 2 + | y 1 | 2 .
Whenever z = ( x , y ) is clear from the context, we write f ( x , y ) as f ( z ) .
We use S n to denote the set of all permutations of [ n ] = { 1 , . . . , n } .
For any matrix A , its operator norm is denoted by | A | = sup | x | =1 |
Ax | .
We use the O notation to characterize the dependence of our convergence rates on n and K , suppressing numerical and problem-specific constants such as κ, µ, σ , etc.
Additionally, we use the ˜ O notation to suppress logarithmic factors of n and K .
Our work studies finite-sum minimax optimization (1).
Solutions to (1) are known as global minimax points of F = 1 /n (cid:80) ni
=
f i , which we assume to always exist .
We also assume that the components f i are continuously differentiable, and hence, the same applies to F .
This allows us to define the gradient operators ω
i : R d → R d and ν : R d → R d as follows:
[Equation]
n
[Equation]
We also impose the following smoothness assumption on the components f i .
Assumption 1 (Component Smoothness) .
The component functions f
i are l -smooth, i.e., each gradient operator
ω
i is l -Lipschitz
[Equation]
Consequently, the operator ν is also l -Lipschitz, i.e., F is l -smooth.
Analysis for Strongly Convex-Strongly Concave Objectives
In this section, we analyze two very popular without-replacement algorithms for finite-sum minimax optimization, Gradient Descent Ascent (GDA) without replacement and Proximal Point Method (PPM) without replacement.
For each of these, we present a unified analysis of the Random
Reshuffling (RR) and Shuffle Once (SO) variants (called GDA-RR/SO and PPM-RR/SO respectively).
For a fixed K > 0 , GDA-RR/SO approximately solves (1) by iterating over the entire dataset for K epochs, and within each epoch, uses the operators ω
i to perform the following iterative update:
[Equation]
where τ k is a uniformly sampled random permutation of [ n ] and 0 < α < 1 /l is a constant step-size.
GDA-RR resamples τ k at the start of every epoch, whereas GDA-SO samples it only once in the beginning.
The details of both algorithms are presented in Algorithm 1.
The Proximal Point Method without replacement is a closely related algorithm which, instead of performing gradient descent-style updates within an epoch, solves the following implicit update equation for z ki : [Equation] ∈ [ n ] .
[Equation]
As before, τ k is resampled at every epoch for PPM-RR, and sampled once and fixed for all epochs for PPM-SO.
We present the details in Algorithm 2.
The l -smoothness of ω i along with the choice of α < 1 /l ensures that z ki is uniquely defined, since it is a fixed point of the contraction mapping ζ ( z ) =
z ki − 1 − αω τ k
( i ) ( z ) .
This method is actually a generalization of the (stochastic) proximal point method for minimization problems, and is popular for problems where (3) can be solved easily or in closed form.
We refer the readers to Rockafellar [47], Patrascu and Necoara [41] for a review of this method and its connections to the original proximal point method for minimization.
3.1 Setting
We analyze GDA-RR/SO and PPM-RR/SO for smooth finite-sum strongly convex-strongly concave (or SC-SC) objectives.
This allows us to formulate the minimax optimization problem for F as a root finding problem for the gradient operator ν , as described below.
Assumption 2 (Strong Convexity-Strong Concavity) .
The objective F is µ strongly convex-strongly concave (or SC-SC), i.e., F ( ., y ) is µ -strongly convex for any y ∈ R d y and − F ( x , . )
is µ -strongly convex for any x ∈ R d x .
Assumption 2 has the following consequences for the gradient operator ν :
Lemma 1.
Let F satisfy Assumptions 1 and 2.
Then, the gradient operator ν is µ -strongly monotone:
[Equation]
Furthermore, (1) admits a unique solution z ∗ , which is also the unique solution of ν ( z ∗ ) = 0 .
Lemma 1 allows us to recast (1) for SC-SC objectives as the following root finding problem :
Find z ∈ R d such that ν ( z )
= 1 /n n (cid:88) i = ω i ( z ) = 0 . Algorithm 1: GDA-RR/SO/AS
Input : Number of epochs K , step-size α > 0 , and initialization z
Initialize z 10 ← z
SO: Sample τ ∼ Uniform ( S n ) for k ∈ [ K ] do
RR: Sample τ k ∼ Uniform ( S n )
SO:
τ k ← τ
AS: Adversary chooses τ k ∈ S n for i ∈ [ n ] do z ki ← z ki − 1 − αω τ k ( i ) ( z ki − 1 ) end z k + ← z kn end Algorithm 2: PPM-RR/SO/AS
Input : Number of epochs K , step-size α > 0 , and initialization z
Initialize z 10 ← z
SO: Sample τ ∼ Uniform ( S n ) for k ∈ [ K ] do
RR: Sample τ k ∼ Uniform ( S n )
SO:
τ k ← τ
AS: Adversary chooses τ k ∈ S n for i ∈ [ n ] do
Solve the implicit update for z ki where, z ki =
z ki − 1 − αω τ k
( i ) ( z ki ) end
z k + ← z kn end Problem (4) is more general than SC-SC minimax optimization, and is a special case of strongly monotone variational inequalities [13] without constraints.
Notably, (4) includes the Nash Equilibrium problem for unconstrained multiplayer games with smooth strongly convex objectives [50] and is sometimes called a finite-sum unconstrained variational inequality in the literature [31].
We also highlight that smooth strongly convex optimization is a special case of (4).
However, unlike the optimization setting, ν is no longer restricted to be the gradient of a strongly convex function, which, as we shall see, has important consequences for the attainable convergence rates of our algorithms.
3.2 Analysis of RR/SO
We now state the expected last-iterate convergence guarantees for Algorithms 1 and 2 for solving (4), where the expecation is taken over the stochasticity of the sampled permutation(s).
Theorem 1 (Convergence of GDA-RR/SO and PPM-RR/SO) .
Consider Problem (4) for the µstrongly monotone operator
ν ( z ) = 1 / n (cid:80) ni
=1 ω i ( z ) where each ω i is l -Lipschitz, but not necessarily monotone.
Let z ∗ denote the unique root of ν .
Then, there exists a step-size α ≤ µ / 5 nl 2 for which both GDA-RR/SO and PPM-RR/SO satisfy the following for any K ≥ 1 :
[Equation]
where κ = l / µ is the condition number and σ 2 ∗ = 1 / n (cid:80) ni
=1 | ω i ( z ∗ ) | 2 is the gradient variance at z ∗ .
Proof.
We present an outline for GDA-RR/SO and defer the full proof to Appendix C.2 (for GDARR/SO) and Appendix D.2 (for PPM-RR/SO).
Furthermore, we recall that the updates of GDA-RR/SO are given by
[Equation]
We begin with the following key insight from earlier works on sampling without replacement for minimization [23, 1, 37, 22]: for small enough step-sizes, the epoch iterates z k 0 of GD without replacement approximately follow the trajectory of full-batch gradient descent.
To this end, we derive the following epoch-level update rule for GDA-RR/SO by linearizing ω τ k ( i ) ( z ki − 1 ) around z ∗ :
[Equation]
where | H k | ≤ 1 − nαµ/ 2 and r k
= (cid:80) n − 1 i
=
A τ k
( i ) (cid:80) i j =1 ω τ k ( j ) ( z ∗ ) with (cid:12) (cid:12)
A τ k ( i ) (cid:12) (cid:12) ≤ le 1 / 5 .
The term r k encapsulates the noise of the stochastic gradient updates accumulated over an entire epoch.
To ensure convergence, we control the influence of the noise term r k by using standard properties of without-replacement sample means to show that E [ | r k | 2 ] ≤ l 2 n 3 σ 2 ∗ / 4 for both RR and SO.
We then complete the proof by unrolling (5) for K epochs, substituting the upper bounds for | H k | and
E [ | r k | 2 ] wherever necessary, and setting α = min { µ/ 5 nl 2 , 2 log( | ν ( z 0 ) | n 1 / 2 K/µ ) /µnK } .
As we show in Appendix C, the update rule (5) resembles the linearized update rule of full batch
GDA with added noise.
In fact, for n = 1 , r k = 0 and thus, we recover the rates of full-batch
GDA.
Expressing GDA-RR/SO (and later AS) as noisy full-batch GDA in this fashion is a central component of our unified analysis, and relies on the fact that (cid:80) ni =1 ω τ k
( i ) ( z ∗ ) =
∀ τ k ∈ S n , which is specific to sampling without replacement.
Comparing to SGDA with replacement, we note that sampling the components i.i.d.
uniformly as u ( i ) ∼ Uniform ([ n ]) gives rise to an additional noise term α p k in the update rule, where p k = (cid:80) ni
=1 ω u ( i ) ( z ∗ ) vanishes only in expecation, and has a variance of E [ | p k
| 2 ] = nσ 2 ∗ .
Subsequently, the dominant noise term for SGDA updates is O ( α 2 nσ 2 ∗ ) whereas that of GDA-RR/SO is O
( α 4 n 3 σ 2 ∗ ) , which qualitatively demonstrates the implicit variance reduction of sampling without replacement.
As we shall see in the complete proof, this allows RR/SO to converge faster (for large enough K ) by carefully selecting α .
Comparison with lower bounds: Since smooth strongly convex minimization is a special case of (4), the Ω(1 /nK 2 ) lower bound established in prior works [48, 43] for smooth strongly convex minimization using GD with RR/SO also applies to GDA-RR/SO.
Comparing with this lower bound, we note that the convergence rate of GDA-RR/SO is nearly tight , i.e., it differs from the lower bound only by an exponentially decaying term.
In fact, for K ≥ 10 κ 2 log( n 1 / 2 K ) , the convergence rate becomes ˜ O
(1 /nK 2 ) , which precisely matches the lower bound, modulo logarithmic factors.
Comparison with uniform sampling: Similarly, the Ω(1 /nK ) lower bound of SGD with replacement for smooth and strongly convex functions [44] also applies to SGDA with replacement.
On the contrary, both GDA-RR and GDA-SO converge with a faster rate of ˜ O
(1 /nK 2 ) when
K ≥ 10 κ 2 log( n 1 / 2 K ) .
Thus, GDA-RR/SO provably outperform SGDA with replacement (modulo logarithmic factors) when K ≥ 10 κ 2 log( n 1 / 2 K ) .
As we show in Appendix C.2, the κ 2 dependence of this inequality cannot be improved for constant step-sizes.
A similar argument also applies to stochastic PPM.
To the best of our knowledge, the fastest known convergence rate for stochastic PPM is O (1 /nK ) for minimizing smooth strongly convex functions [41].
Hence, Theorem 1 suggests that PPM-RR/SO enjoy a faster ˜ O (1 /nK 2 ) convergence rate for both minimization and minimax optimization when K ≥ 10 κ 2 log( n 1 / 2 K ) .
3.3 Analysis in the Adversarial Shuffling Regime
We now consider without-replacement minimax optimization algorithms in an adversarial setting.
We focus on a novel class of training-time attacks known as data ordering attacks proposed by Shumailov et al.
These attacks differ from standard data-perturbation attacks [19] and exploit the fact that most implementations of stochastic gradient optimizers do not verify whether the permutation τ k is truly sampled at random.
Shumailov et al.
[51] propose three distinct attack strategies, namely, batch reordering , which changes the order in which mini-batches are supplied to the algorithm, batch reshuffling , which changes the order in which individual data points are supplied, and replacing which prevents certain data points from being observed by the algorithm by consistently replacing them with other data points in the training set.
We analyze the convergence of without-replacement GDA and PPM under batch reshuffling attacks.
To this end, we consider an adversarial modification of RR/SO where the permutations τ k , instead of being sampled by the algorithm, are now selected by an adversary using a strategy unknown to the algorithm.
We also the assume that, while choosing τ k , the adversary is computationally unrestricted and has complete knowledge of all the components ω
i , the minimax point z ∗ , and the iterates z ki observed so far.
We call this setup Adversarial Shuffling (AS) and obtain convergence rates of GDA and PPM (named GDA-AS and PPM-AS) when solving (4).
Thus, our analysis naturally holds for minimization, minimax optimization as well as finite-sum multiplayer games.
The details are stated in Algorithms 1 and 2, respectively.
Our last iterate convergence guarantees are deterministic and hold uniformly over any sequence of permutations τ 1 , . .
, τ K that the adversary can choose.
Theorem 2 (Convergence of GDA-AS and PPM-AS) .
Consider Problem (4) for the µ -strongly monotone operator
ν ( z ) = 1 / n (cid:80) ni
=1 ω i ( z ) where each ω i is l -Lipschitz, but not necessarily monotone.
Let z ∗ denote the unique root of ν .
Then, there exists a step-size α ≤ µ / 5 nl 2 for which both GDA-AS and PPM-AS satisfy the following for any K ≥ 1 :
[Equation]
where κ, σ 2 ∗ are as defined in Theorem 1 and τ 1 , . .
, τ K are the permutations chosen by the adversary.
Convergence rates of IG and comparison with lower bounds: We note that the Incremental
Gradient and the Incremental Proximal Point Methods, which do not shuffle the data, are a special case of GDA-AS/PPM-AS with τ 1 , . .
, τ K = id .
Thus, Theorem 2 also gives us convergence rates for
GDA-IG/PPM-IG.
Moreover, since GDA-AS generalizes GDA-IG and (4) covers minimization, the Ω(1 /K 2 ) lower bound of IG established in prior works [48] for smooth strongly convex minimization also applies to GDA-AS.
Thus, our obtained rate for GDA-AS is nearly tight and matches the lower bound (modulo logarithmic factors) for K ≥ 10 κ 2 log( K ) .
Effectiveness of batch reshuffling: When K ≥ 10 κ 2 log( K ) , ˜ O
(1 /K 2 ) becomes the dominant term in the convergence rate of AS.
This is worse than that of RR/SO by a factor of 1 /n and causes a significant slowdown in convergence, since, in many applications, the dataset size n is much larger than K .
Thus, our analysis justifies the effectiveness of batch reshuffling attacks in reducing model accuracy and increasing training time, which is empirically verified by Shumailov et al.
RR for Two-Sided PŁ Objectives
We now analyze RR for a class of smooth nonconvex-nonconcave problems where the objective F satisfies a two-sided Polyak Łojasiewicz inequality , first proposed in Yang et al.
We denote this function class as 2PŁ and formally state the assumption as follows.
Assumption 3 (Two-sided Polyak Łojasiewicz Inequality or 2PŁ condition) .
For any x ∈ R d x , y ∈ R d y , the sets arg max ˜ y F ( x , ˜ y ) and arg min ˜ x F (˜ x , y ) are non-empty.
Furthermore, there exist positive constants µ 1 , µ 2 such that F satisfies the following:
[Equation]
The 2PŁ condition is satisfied in several practical settings, including, but not limited to, robust least squares [12], imitation learning for linear quadratic regulators [14, 8], and various other problems in reinforcement learning and robust control [11, 8].
Clearly, any SC-SC function is 2PŁ.
However, 2PŁ functions need not be SC-SC, or even convex-concave.
We refer the readers to Yang et al.
[54] for a detailed discussion of the 2PŁ class and its applications.
Analysis of RR for 2PŁ objectives is challenging not only due to nonconvexity-nonconcavity, but also because F may not have a unique minimax point.
Indeed, as we demonstrate in Appendix E.1, it is possible to construct 2PŁ functions where the set of minimax points is an unbounded proper subset of R d .
Hence, the notion of gradient variance at the minimax point , which we used in our earlier analyses, is no longer meaningful.
To overcome this, we impose the following assumption.
Assumption 4 (Bounded Gradient Variance) .
There exists a positive constant σ such that the component gradient operators ω
i satisfy the following for any z ∈ R d :
[Equation]
In order to establish the provable benefits of RR for smooth finite-sum minimax optimization of 2PŁ objectives, we propose the Alternating Gradient Descent Ascent with Random Reshuffling (AGDARR) algorithm. AGDA-RR achieves near-optimal convergence guarantees for 2PŁ objectives by combining RR with alternating updates [17, 57, 3] and timescale separation [29, 15, 16], two ideas that have been very useful for improving convergence and stability in nonconvex-nonconcave minimax optimization. Within each epoch k ∈ [ K ] , AGDA-RR uniformly samples a random permutation τ k , makes one full pass over the dataset, and performs gradient descent (with RR) updates for the variable x using the permutation τ k .
This is followed by sampling another permutation π k and performing gradient ascent (with RR) updates for y using the permutation π k . The detailed procedure is stated in Algorithm 3. We also analyze a variant of AGDA-RR in the adversarial shuffling setting, which we call AGDA-AS.
The procedure, as described in Algorithm 4, is almost identical to AGDA-RR, except that the permutations τ k and π k are chosen by an adversary.
Algorithm
Input : Number of epochs K , step-sizes α, β > 0 , and initialization ( x 0 , y 0 )
Initialize ( x 10 , y 10 ) ← ( x 0 , y 0 ) for k ∈ [ K ] do
Sample a permutation τ k ∈ S n for i ∈ [ n ] do x ki ← x ki − 1 − α ∇ x f τ k ( i ) ( x ki − 1 , y k 0 ) end
Sample a permutation π k ∈ S n for i ∈ [ n ] do y ki ← y ki − 1 + β ∇ y f π k ( i ) ( x kn , y ki − 1 ) end ( x k + , y k + ) ← ( x kn , y kn ) end
Input : Number of epochs K , step-sizes α, β > 0 , and initialization ( x 0 , y 0 )
Initialize ( x 10 , y 10 ) ← ( x 0 , y 0 ) for k ∈ [ K ] do
Adversary chooses a permutation τ k ∈ S n for i ∈ [ n ] do x ki ← x ki − 1 − α ∇ x f τ k ( i ) ( x ki − 1 , y k 0 ) end
Adversary chooses a permutation π k ∈ S n for i ∈ [ n ] do y ki ← y ki − 1 + β ∇ y f π k ( i ) ( x kn , y ki − 1 ) end ( x k
+ , y k + ) ← ( x kn , y kn ) end
Before presenting a convergence analysis, we highlight that the absence of a unique minimax point prevents us from using the squared distance to the optimum as a Lyapunov function.
To this end, we use the Lyapunov function V λ : R d → R which was previously suggested by Yang et al.
We begin by first defining the best response function Φ : R d → R and its minimum Φ ∗ as follows,
[Equation]
Assumption 3 ensures that Φ is well defined and the existence of a global minimax point guarantees that Φ ∗ is finite.
Subsequently, for any λ > 0 , we define the Lyapunov function V λ as
[Equation]
By definition of Φ , V λ is non-negative for any λ > 0 and V λ
( z ) =
if and only if z is a minimax point of F .
Hence, we present our convergence proofs for AGDA-RR and AGDA-AS in terms of V λ .
Theorem 3 (Convergence of AGDA-RR/AS) .
Let Assumptions 1, 3, and 4 hold and let η = 73 l 2 / 2 µ 22 .
Then, there exists a step-size α ≤ 1 / 5 ηnl such that for β = ηα , AGDA-RR satisfies the following for λ = 1 / 10 and any K ≥ 1 :
[Equation]
where κ = max { l / µ 1 , l / µ 2 } and c > 0 is a constant independent of κ, µ 1 , µ 2 , σ 2 .
Under the same setting, AGDA-AS satisfies the following ( ˆ c > 0 is a constant independent of κ, µ 1 , µ 2 , σ 2 ):
[Equation]
where τ 1 , π 1 . .
, τ K , π K are the permutations chosen by the adversary.
Convergence to a Saddle Point: As demonstrated in Appendix E.4, the convergence guarantee of
Theorem 3, which is presented in terms of V λ , can be easily translated into an equivalent convergence guarantee in terms of dist ( z , Z ∗ ) 2 , where Z ∗ denotes the set of saddle points of F .
In particular,
Theorem 3 implies the following convergence guarantee for AGDA-RR:
[Equation] as well as the following convergence rate for AGDA-AS: [Equation] Comparison with lower bounds: Strongly convex minimization is a special case of 2PŁ minimax optimization, since minimizing the strongly convex function f is equivalent to minimax optimization of the 2PŁ function F ( x , y ) =
f ( x ) − ⟨ y , y ⟩ .
In fact, the x iterates of AGDA-RR for F are exactly 100 Epochs 0. D i s t a n c e t o M i n i m a x P o i n t
{+SGDAGDA-RRGDA-SOGDA-IG (a) GDA     100 Epochs 
+}
D i s t a n c e t
{+o M i n i m a x P o i n t SPPMPPM-RRPPM-SOPPM-IG (b [Figure] PPM     100 Epochs 
+}
D i s t a n c e t
o M i n i m a x P o i n t
[Figure]
IG
[Figure]
that of GD with RR for f .
Hence, the Ω(1 /nK 2 ) lower bound for strongly convex minimization using
GD with RR also applies to AGDA-RR.
Thus our convergence rate for AGDA-RR is nearly tight and matches the lower bound (modulo logarithmic factors) for K ≥ 730 κ 3 log( n 1 / 2 K ) .
Similarly, the
Incremental Gradient version of AGDA is a special case of AGDA-AS with τ 1 , π 1 , . .
, τ K , π K = id and hence, AGDA-AS is nearly tight and matches the Ω(1 /K 2 ) lower bound (modulo logarithmic factors) for K ≥ 730 κ 3 log( K ) .
Comparison with stochastic AGDA: Similarly, the Ω(1 /nK ) lower bound of SGD with replacement also holds for the Stochastic AGDA algorithm [54], which samples the component functions with replacement and performs two-timescale alternating updates similar to AGDA-RR.
Hence, Theoremdemonstrates that AGDA-RR provably outperforms stochastic AGDA when K ≥ 730 κ 3 log( n 1 / 2 K ) .
Bounded iterate assumption Assumption 4 is also used in analyzing RR for PŁ function minimization [34].
In this setting, an alternative bounded iterate assumption , which assumes that all the iterates z ki lie within a compact set, has also been used [1].
As shown in Appendix E.2, our proof of
Theorem 3 easily adapts to this assumption.
In the absence of either assumption, Li et al.
[28] use time-varying step-sizes to obtain asymptotic O (1 /K 2 ) rates for RR on PŁ (and more generally for
KŁ) minimization.
Experiments
We evaluate our theoretical results by benchmarking on finite-sum SC-SC quadratic minimax games.
This class of problems appears in several applications such as reinforcement learning [11], robust regression, [12] and online learning [26].
The objective F and the components f i are given by:
[Equation]
[-SGDAGDA-RRGDA-SOGDA-IG (a) GDA     100 Epochs 
-]
[-PPM     100 Epochs 
 D i s t a n c e t o M i n i m a x P o i n t Stochastic AGDA AGDA-RRAGDA-IG (c) AGDA-] where A and C are strictly positive definite.
We generate the components
f i randomly, such that (cid:80) ni
=1 u i =
(cid:80) ni
=1 v i =
and the expected singular values of B are larger than that of A and C .
This ensures that the bilinear coupling term x T
By is sufficiently strong, since a weak coupling practically reduces to quadratic minimization, which has already been investigated in prior works.
Finally, to investigate how the presence of nonconvex-nonconcave components impacts convergence, a few randomly chosen f i ’s are allowed to be nonconvex-nonconcave quadratics.
For each algorithm analyzed in the text, we benchmark sampling without replacement against uniform sampling by running each method for 100 epochs using constant step-sizes that are selected independently for each method via grid search.
Further details regarding the setup is discussed in Appendix G.
We present our results in Figure 2, where we plot the relative distance of the epoch iterates from the minimax point, defined as | z k 0 − z ∗ | 2 / | z 0 − z ∗ | 2 , averaged over 50 independent runs.
In agreement with our theoretical findings, sampling without replacement consistently outperforms uniform sampling 100 Epochs 0. D i s t a n c e t
{+o M i n i m a x P o i n t SGDAGDA-RRGDA-SOGDA-IG (a) GDA     100 Epochs 
 D i s t a n c e t o M i n i m a x P o i n t+}
SPPMPPM-RRPPM-SOPPM-IG
{+[Figure] 100 Epochs 
 D i s t a n c e t o M i n i m a x P o i n t+}
[Figure]
across all three setups.
Furthermore, to demonstrate that our observations are not particular to one specific instance, we repeat the experiment for 20 independently sampled quadratic games, and for each instance, perform 5 independent runs of each algorithm and plot the average relative distance of the epoch iterates from the minimax point.
The results, presented in Figure 3, substantiates the superior convergence of sampling without replacement across multiple instances.
Conclusion
We derived near optimal convergence rates for several without-replacement stochastic gradient algorithms for finite-sum minimax optimization, and demonstrated that they converge faster than algorithms that use uniform sampling.
We considered two problem classes, strongly convex-strongly concave problems (generalized to unconstrained strongly monotone variational inequalities) and nonconvex-nonconcave problems with two-sided PŁ objectives.
We also formally defined adversarial shuffling , where an attacker can control the order in which data points are supplied to the optimizer, and analyzed minimax optimization in this regime.
Interesting future directions include the analysis of inexact proximal point methods, more general function
Acknowledgements and Disclosure of Funding
Michael Muehlebach and Bernhard Schölkopf thank the German Research Foundation and the Branco
Weiss Fellowship, administered by ETH Zurich, for the generous support.