Sampling without Replacement Leads to Faster Rates in Finite-Sum Minimax Optimization
Anonymous Author(s)
AffiliationAddress email
Abstract
We analyze the convergence rates of stochastic gradient algorithms for smoothfinite-sum minimax optimization and show that, for many such algorithms, sampling the data points without replacement leads to faster convergence compared tosampling with replacement.
For the smooth and strongly convex-strongly concavesetting, we consider gradient descent ascent and the proximal point method, andpresent a unified analysis of two popular without-replacement sampling strategies,namely Random Reshuffling (RR), which shuffles the data every epoch, and Single
Shuffling or Shuffle Once (SO), which shuffles only at the beginning.
We obtaintight convergence rates for RR and SO and demonstrate that these strategies lead tofaster convergence than uniform sampling.
Moving beyond convexity, we obtainsimilar results for smooth nonconvex-nonconcave objectives satisfying a two-sided
Polyak-Łojasiewicz inequality.
Finally, we demonstrate that our techniques aregeneral enough to analyze the effect of data-ordering attacks , where an adversarymanipulates the order in which data points are supplied to the optimizer.
Ouranalysis also recovers tight rates for the incremental gradient method, where thedata points are not shuffled at all.
Introduction
The approximate solution of large-scale optimization problems using first-order stochastic gradientmethods constitutes one of the foundations of classical machine learning.
However, emergingproblems in machine learning go beyond pattern recognition and involve real-world decision making,where learning algorithms interact with unknown or even adversarial environments or are deployedin multi-agent settings.
Decision making in such environments often involves solving a minimaxoptimization problem of the form min
x max y F ( x , y ) , whose analysis has been a focus of researchin mathematics, economics and theoretical computer science [33, 12, 9].
Recent examples of itsapplications in machine learning include adversarial learning [29, 43, 6], reinforcement learning [27,47, 10, 36], imitation learning [13, 8, 23] and generative adversarial networks [18, 2].
In most of theseapplications, the objective F ( x , y ) has a finite-sum structure, i.e.,
[Equation]
where n denotes the size of the dataset and each component function f i denotes the objectiveassociated with the i th data point.
The resultant problem is known as finite-sum minimax optimization :
[Equation]
When n is large and the f i ’s are differentiable (which holds for most applications), approximatesolutions to (1) are computed using stochastic gradient algorithms.
These algorithms typically samplean index i ∈ [ n ] at each iteration as per some specified sampling routine, and use the gradients of f ito compute the next iterate.
Among these methods, perhaps the simplest and most commonly used algorithm is Stochastic Gradient Descent Ascent (SGDA), a natural extension of Stochastic Gradient
Descent (SGD) for minimax optimization.
Similar to the stochastic optimization literature, analysis of stochastic minimax optimization oftenassumes that, at every iteration, the index i is sampled uniformly with replacement .
Analysis of theresulting algorithm closely parallels the analysis of SGD, and relies on the fact that uniform samplingleads to unbiased gradient estimates.
Recent works [20, 28] have also extended this paradigm toi.i.d uniform sampling of mini-batches.
While uniform sampling assumptions simplify theoreticalanalysis, practical implementations of these algorithms often deviate from this paradigm, and insteadincorporate various heuristics, which are often empirically found to improve runtime.
A commonand notable heuristic is to replace uniform sampling by procedures that perform multiple passesover the entire dataset, and in each such pass (called an epoch), sample the data points withoutreplacement .
Thus, each data point is sampled exactly once in every epoch.
These procedures aregenerally implemented using one of the following approaches:
Random Reshuffling (RR): Uniformly sample a random permutation of [ n ] at the start of everyepoch, and process the data points within that epoch as per the order specified by the permutation.
Single Shuffling or Shuffle Once (SO): Uniformly sample a random permutation at the beginningand reuse it across all epochs to order the data points.
Incremental Gradient (IG): Do not permute the data points at all and follow a fixed deterministicdata ordering
for every epoch.
Sampling without replacement is ubiquitous in both stochastic minimization [7, 40, 4] and stochasticminimax optimization [18, 2] as it often exhibits faster runtime than uniform sampling.
However,these empirical benefits come at the cost of limited theoretical understanding, due to the absence ofprovably unbiased gradient estimates.
It is well known in the optimization literature that SGD with replacement has a tight rate of Θ(1 /nK )for smooth and strongly convex minimization [38, 24], where n is the number of component functionsand K denotes the number of epochs.
On the contrary, recent works on SGD without replacementfor smooth and strongly convex minimization [1, 32, 31, 34] show that both RR and SO achieve anon-asymptotic rate of ˜ O
(1 /nK 2 ) , once the number of epochs K is larger than a certain threshold
K 0 (usually polynomial in the condition number), and thereby converge faster than SGD withreplacement.
These rates match the lower bound of Ω(1 /nK 2 ) for RR and SO established in priorworks [37, 40], modulo logarithmic factors.
For RR, prior works have also established a similar˜ O (1 /nK 2 ) rate for nonconvex objectives satisfying the Polyak-Łojasiewicz (PŁ) inequality [1, 31].
While the asymptotic behavior of IG has been known to the community for a long time in both smoothand non-smooth settings [5, 16], non-asymptotic ˜ O (1 /nK 2 ) convergence rates have been establishedquite recently [31, 34, 21], and are complemented by a matching Ω(1 /nK 2 ) lower bound [40].
1.1 Contributions
Although the empirical benefits of sampling without replacement have been substantiated for minimization, analysis of these methods for minimax optimization have received much less attention,despite being widely prevalent in many applications.
Our work aims to fill this gap by analyzingthese methods for minimax optimization.
To this end, our main contributions are as follows:
Unified analysis of RR and SO for strongly convex-strongly concave problems: We analyze RR and SO in conjunction with simultaneous Gradient Descent Ascent (GDA), calling the resultingalgorithms GDA-RR and GDA-SO, respectively.
Assuming the components f i are smooth and Fis strongly convex-strongly concave, we present a unified analysis of GDA-RR/SO and establish aconvergence rate of ˜ O
(exp( − K/ 5 κ 2 ) + σ 2 ∗ /nK 2 ) for both. Here, κ denotes the condition numberand σ 2 ∗ is the gradient variance at the minimax point.
Comparing with lower bounds, we show thatour rates are nearly tight , i.e., they differ from the lower bound only by an exponentially decayingterm.
We also show that when K ≥ Ω( κ 2 ) , the convergence rate matches the lower bounds for GDA-RR/SO, modulo logarithmic factors, and also converges provably faster than SGDA withreplacement.
Under the same setting, we obtain similar guarantees for the RR and SO variants of the Proximal Point Method (PPM), named PPM-RR and PPM-SO respectively.
Our analysis for both GDA-RR/SO and PPM-RR/SO is general enough to cover smooth strongly monotone finite-sum variational inequalities, which encompasses minimization, minimax optimization, and multiplayergames.
RR for two-sided PŁ objectives: We consider a class of nonconvex-nonconcave problems wherethe objective F satisfies a two-sided Polyak-Łojasiewicz inequality.
For such problems, we propose an algorithm that combines RR with two-timescale Alternating Gradient Descent Ascent(AGDA), which we call AGDA-RR.
We show that AGDA-RR has a nearly tight convergence rateof ˜ O
(exp( − K/ 365 κ 3 ) + σ 2 /nK 2 ) when the gradient variance is uniformly bounded by σ 2 .
When
K ≥ Ω( κ 3 ) , this rate matches the lower bound (modulo logarithmic factors) and improves on thebest known rates of with-replacement algorithms for this class of problems.
Minimax optimization under data ordering attacks: Our techniques for analyzing RR generalizeto the analysis of finite-sum minimax optimization under data ordering attacks [42].
These attackstarget the inherent randomness assumptions of stochastic gradient algorithms, significantly increasingtraining time and reducing model quality, only by manipulating the order in which the algorithmreceives data points, without performing any data contamination.
To model these attacks, wepropose the Adversarial Shuffling (AS) setup, where the data points are shuffled every epoch by acomputationally unrestricted adversary.
In this setup, we show that GDA and PPM (now called GDAAS and PPM-AS) have a convergence rate of ˜ O
(exp( − K/ 5 κ 2 )+ σ 2 ∗ /K 2 ) for smooth strongly convexstrongly concave objectives, and AGDA (now called AGDA-AS) has a rate of ˜ O
(exp( − K/ 365 κ 3 ) +σ 2 /K 2 ) for two-sided PŁ objectives.
We note that, compared to RR and SO, the convergence rateworsens by a factor of 1 /n for large enough K .
When n is large (true for most applications), thisslowdown significantly impacts convergence and provides a theoretical justification for the empiricalobservations in prior work [42].
We also establish that our analysis in the AS regime also applies to the
Incremental Gradient (IG) variants of these algorithms (namely GDA-IG, PPM-IG, and AGDA-IG),and use this to show that our obtained rates of GDA-AS and AGDA-AS are nearly tight.
To the best of our knowledge, our work is the first to: 1) analyze RR, SO, and IG for stronglymonotone unconstrained VIs, 2) analyze RR and IG for a class of nonconvex-nonconcave minimaxproblems, 3) provably demonstrate the advantages of sampling without replacement for both thesesettings and justify its empirical benefits in a wide variety of problems ranging from minimization,minimax optimization to smooth multiplayer games, 4) analyze sampling without replacementunder data-ordering attacks.
Furthermore, unlike prior works on random reshuffling for minimaxoptimization [45, 30] which required the component functions to be convex, Lipschitz, and smooth,our analysis does not impose any restrictions on the components
f i other than smoothness, therebyallowing them to be arbitrary nonconvex-nonconcave functions.
Notation and Preliminaries
We work with Euclidean spaces ( R d , ⟨ ., .
⟩ ) equipped with the standard inner product ⟨ x 1 , x 2 ⟩ andthe induced norm | x | .
For any x ∈ R d x and y ∈ R d y , we denote the concatenation z
= ( x , y ) ∈
R dwhere d = d x + d y , and equip R d with the standard inner product and norm, i.e., for any z 1 =( x 1 , y 1 ) ∈ R d and z {+
+} = ( x 2 , y 2 ) ∈ R d , ⟨ z 1 , z 2 ⟩ = ⟨ x 1 , x 2 ⟩ + ⟨ y 1 , y 2 ⟩ and | z 1 | 2 = | x 1 | 2 + | y 1 | 2 .
Whenever z = ( x , y ) is clear from the context, we interchange f ( x , y ) and f ( z ) .
We use S n todenote the set of all permutations of [ n ] = { 1 , . . . , n } .
For any matrix A , its operator norm orspectral norm is denoted by | A | = sup | x | =1 |
Ax | .
Our work studies finite-sum minimax optimization (1).
Solutions to (1) are known as global minimaxpoints of F = 1 /n (cid:80) ni
=
f i , which we assume to always exist .
We also assume that the componentsf i are continuously differentiable, and hence, the same applies to F .
This allows us to define thegradient operators ω
i : R d → R d and ν : R d → R d as follows:
[Equation]
We also impose the following smoothness assumption on the components f i .
Assumption 1 (Component Smoothness) .
The component functions f
i are l -smooth, i.e., each gradient operator
ω
i is l -Lipschitz
[Equation]
Consequently, the operator ν is also l -Lipschitz, i.e., F is l -smooth.
Algorithm 1: GDA-RR/SO/AS
Input : Number of epochs K , step-size α > 0 , and initialization z
Initialize z 10 ← z
SO: Sample τ ∼ Uniform ( S n ) for k ∈ [ K ] do
RR: Sample τ k ∼ Uniform ( S n )
SO:
τ k ← τ
AS: Adversary chooses τ k ∈ S n for i ∈ [ n ] do z ki ← z ki − 1 − αω τ k
( i ) ( z ki − 1 ) end
z k + ← z kn end
Algorithm 2: PPM-RR/SO/AS
Input : Number of epochs K , step-size α > 0 , and initialization z
Initialize z 10 ← z
SO: Sample τ ∼ Uniform ( S n ) for k ∈ [ K ] do
RR: Sample τ k ∼ Uniform ( S n )
SO:
τ k ← τ
AS: Adversary chooses τ k ∈ S n for i ∈ [ n ] do
Solve the implicit update for z ki where, z ki =
z ki − 1 − αω τ k
( i ) ( z ki ) end
z
k
+ ← z kn end
Analysis for Strongly Convex-Strongly Concave Objectives
In this section, we analyze two very popular without-replacement algorithms for finite-sum minimaxoptimization, Gradient Descent Ascent (GDA) without replacement and Proximal Point Method(PPM) without replacement.
For each of these, we present a unified analysis of the Random
Reshuffling (RR) and Shuffle Once (SO) variants (called GDA-RR/SO and PPM-RR/SO respectively).
For a fixed K > 0 , GDA-RR/SO approximately solves (1) by iterating over the entire dataset for Kepochs, and within each epoch, uses the operators ω
i to perform the following iterative update:
[Equation]
where τ k is a uniformly sampled random permutation of [ n ] and 0 < α < 1 /l is a constant step-size.
GDA-RR resamples τ k at the start of every epoch, whereas GDA-SO samples it only once in thebeginning.
The details of both algorithms are presented in Algorithm 1.
The Proximal Point Methodwithout replacement is a closely related algorithm which, instead of performing gradient descent-styleupdates within an epoch, solves the following implicit update equation for z ki :
[Equation]
As before, τ k is resampled at every epoch for PPM-RR, and sampled once and fixed for all epochsfor PPM-SO.
We present the details in Algorithm 2.
The l -smoothness of ω i along with the choiceof α < 1 /l ensures that z ki is uniquely defined, since it is a fixed point of the contraction mappingζ ( z ) =
z ki − 1 − αω τ k
( i ) ( z ) .
This method is actually a generalization of the (stochastic) proximal pointmethod for minimization problems, and is popular for problems where (3) can be solved easily or inclosed form.
We refer the readers to Rockafellar [39], Patrascu and Necoara [35] for a review of thismethod and its connections to the original proximal point method for minimization.
3.1 Setting
We analyze GDA-RR/SO and PPM-RR/SO for smooth finite-sum strongly convex-strongly concave(or SC-SC) objectives.
This allows us to formulate the minimax optimization problem for F as a rootfinding problem for the gradient operator ν , as described below.
Assumption 2 (Strong Convexity-Strong Concavity) .
The objective F is µ strongly convex-stronglyconcave (or SC-SC), i.e., F ( ., y ) is µ -strongly convex for any y ∈ R d y and − F ( x , . )
is µ -stronglyconvex for any x ∈ R d x .
Assumption 2 has the following consequences for the gradient operator ν :
Lemma 1.
Let F satisfy Assumptions 1 and 2.
Then, the gradient operator ν is µ -strongly monotone:
[Equation]
Furthermore, (1) admits a unique solution z ∗ which is also the unique solution of ν ( z ∗ ) = 0 .
Lemma 1 allows us to recast (1) for SC-SC objectives as the following root finding problem :
Find z ∈ R d such that ν ( z )
= 1 /n n
[Equation]
Problem (4) is more general than SC-SC minimax optimization, and is a special case of stronglymonotone variational inequalities [12] without constraints.
Notably, (4) includes the Nash Equilibriumproblem for unconstrained multiplayer games with smooth strongly convex objectives [41] and issometimes called a finite-sum unconstrained variational inequality in the literature [28].
We alsohighlight that smooth strongly convex optimization is a special case of (4).
However, unlike theoptimization setting, ν is no longer restricted to be the gradient of a strongly convex function, which,as we shall see, has important consequences for the attainable convergence rates of our algorithms.
3.2 Analysis of RR/SO
We now state the last-iterate convergence guarantees for Algorithms 1 and 2 for solving (4), wherethe expecation is taken over the stochasticity of the sampled permutation(s).
Theorem 1 (Convergence of GDA-RR/SO and PPM-RR/SO) .
Consider Problem 4 for the µ -strongly monotone operator ν
( z ) = 1 / n (cid:80) ni
=1 ω i ( z ) where each ω i is l -Lipschitz, but not necessarily monotone.
Let z ∗ denote the unique root of ν .
Then, there exists a step-size α ≤ µ / 5 nl 2 for which both GDA-RR/SO and PPM-RR/SO satisfy the following for any K ≥ 1 :
[Equation]
where κ = l / µ is the condition number and σ 2 ∗ = 1 / n (cid:80) ni
=1 | ω i ( z ∗ ) | 2 is the gradient variance at z ∗ .
Proof.
We present an outline for GDA-RR/SO and defer the full proof to the appendix.
Furthermore,we recall that the updates of GDA-RR/SO are given by z ki =
z ki − 1 − αω τ k
( i ) ( z ki − 1 ) .
We begin with the following key insight from earlier works on sampling without replacement forminimization [22, 1, 16, 21]: for small enough step-sizes, the epoch iterates z k 0 of GD withoutreplacement approximately follow the trajectory of full-batch gradient descent.
To this end, we derivethe following epoch-level update rule for GDA-RR/SO by linearizing ω τ k ( i ) ( z ki − 1 ) around z ∗ :
[Equation]
where | H k | ≤ 1 − nαµ/ 2 and r k
= (cid:80) n − 1 i
=
A τ k
( i ) (cid:80) ij =1 ω τ k ( j ) ( z ∗ ) with (cid:12)(cid:12)
A τ k ( i ) (cid:12)(cid:12) ≤ le 1 / 5 .
Theterm r k encapsulates the noise of the stochastic gradient updates accumulated over an entire epoch.
{+Clearly, for n = 1 , r k = 
+}
and we recover the rates of full-batch GDA.
To ensure convergence,we control the influence of the noise term r k by using standard properties of without-replacementsample means to show that E [ | r k | 2 ] ≤ l 2 n 3 σ 2 ∗ / 4 for both RR and SO.
We then complete the proof byunrolling (5) for K epochs, substituting the upper bounds for | H k | and E [ | r k | 2 ] wherever necessary,and setting α = min { µ/ 5 nl 2 , 2 log( | ν ( z 0 ) | n 1 / 2 K/µ ) /µnK } .
We highlight that a key step in deriving (5) is the fact that (cid:80) ni
=1 ω τ k
( i ) ( z ∗ ) =
∀ τ k ∈ S n , whichis specific to sampling without replacement.
Comparing to SGDA with replacement, we note thatsampling the components i.i.d.
uniformly as u ( i ) ∼ Uniform ([ n ]) gives rise to an additional noiseterm of α p k in the update rule, where p k = (cid:80) ni
=1 ω u ( i ) ( z ∗ ) vanishes only in expecation, and has avariance of E [ | p k
| 2 ] = nσ 2 ∗ .
As a result, the dominant noise term for SGDA updates is O ( α 2 nσ 2 ∗ )whereas that of GDA-RR/SO is O
( α 4 n 3 σ 2 ∗ ) , which qualitatively demonstrates the implicit variancereduction of sampling without replacement.
As we shall see in the complete proof, this allows us toobtain faster convergence for RR/SO (for large enough K ) by carefully selecting α .
Comparison with lower bounds: Since smooth strongly convex minimization is a univariate specialcase of (4), the Ω(1 /nK 2 ) lower bound established in prior works [40, 37] for smooth stronglyconvex minimization using GD with RR/SO also applies to GDA-RR/SO, where the lower boundsuppresses problem-specific parameters like κ, µ, σ 2 ∗ .
We note, by comaprison with this lower bound, that the convergence rate of GDA-RR/SO is nearly tight , i.e., it differs from the lower bound only byan exponentially decaying term.
In fact, for K ≥ Ω( κ 2 ) , the rate obtained for GDA-RR/SO becomes˜ O (1 /nK 2 ) , which precisely matches the lower bound, modulo logarithmic factors.
Comparison with uniform sampling: Similarly, the Ω(1 /nK ) lower bound of SGD with replacement for smooth and strongly convex functions [38] also applies to SGDA with replacement.
On thecontrary, both GDA-RR and GDA-SO converge with a faster rate of ˜ O
(1 /nK 2 ) when K ≥ Ω( κ 2 ) .
Thus, GDA-RR/SO provably outperform SGDA with replacement (modulo logarithmic factors)when K ≥ Ω( κ 2 ) .
As we shall demonstrate in the appendix, this Ω( κ 2 ) requirement cannot beimproved for constant step-sizes.
A similar argument also applies to stochastic PPM.
To the best ofour knowledge, the fastest known convergence rate for stochastic PPM is O (1 /nK ) for minimizingsmooth strongly convex functions [35].
Hence, Theorem 1 suggests that PPM-RR/SO enjoy a faster˜ O (1 /nK 2 ) convergence rate for both minimization and minimax optimization when K ≥ Ω( κ 2 ) .
3.3 Analysis in the Adversarial Shuffling Regime
We now consider without-replacement minimax optimization algorithms in an adversarial setting.
Wefocus on a novel class of training-time attacks known as data ordering attacks proposed by Shumailovet al.
These attacks differ from standard data-perturbation attacks [19] and exploit the fact thatmost implementations of stochastic gradient optimizers do not verify whether the permutation τ kis truly sampled at random.
Shumailov et al.
[42] propose three distinct attack strategies, namely,batch reordering , which changes the order in which mini-batches are supplied to the algorithm, batchreshuffling , which changes the order in which individual data points are supplied, and replacingwhich prevents certain data points from being observed by the algorithm by consistently replacingthem with other data points in the training set.
We analyze the convergence of without-replacement GDA and PPM under batch reshuffling attacks.
To this end, we consider an adversarial modification of RR/SO where the permutations τ k , instead ofbeing sampled by the algorithm, are now selected by an adversary using a strategy unknown to thealgorithm.
We also the assume that, while choosing τ k , the adversary is computationally unrestrictedand has complete knowledge of all the components
ω
i , the minimax point z ∗ , and the iterates z kiobserved so far.
We call this setup Adversarial Shuffling (AS) and obtain convergence rates of GDAand PPM (named GDA-AS and PPM-AS) when solving (4).
Thus, our analysis naturally holds forminimization, minimax optimization as well as finite-sum multiplayer games.
The details are statedin Algorithms 1 and 2, respectively.
Our last iterate convergence guarantees are deterministic andhold uniformly over any sequence of permutations τ 1 , . .
, τ K that the adversary can choose.
Theorem 2 (Convergence of GDA-AS and PPM-AS) .
Consider Problem 4 for the µ -strongly monotone operator ν
( z ) = 1 / n (cid:80) ni
=1 ω i ( z ) where each ω i is l -Lipschitz, but not necessarily monotone.
Let z ∗ denote the unique root of ν .
Then, there exists a step-size α ≤ µ / 5 nl 2 for which both GDA-AS and PPM-AS satisfy the following for any K ≥ 1 :
[Equation]
where κ = l / µ is the condition number, σ 2 ∗ = 1 / n (cid:80) ni =1 | ω i ( z ∗ ) | 2 is the gradient variance at z ∗ , andτ 1 , τ 2 , . . .
, τ K denotes any sequence of permutations chosen by the adversary.
Convergence rates of IG and comparison with lower bounds: We note that the Incremental
Gradient and the Incremental Proximal Point Methods, which do not shuffle the data, are a specialcase of GDA-AS/PPM-AS with τ 1 , . .
, τ K = id .
Thus, Theorem 2 also gives us convergence ratesfor GDA-IG/PPM-IG.
Moreover, since GDA-AS generalizes GDA-IG and (4) covers minimization,the Ω(1 /K 2 ) lower bound of IG for smooth strongly convex minimization [40] also applies to GDA-AS.
Thus, our obtained rate for GDA-AS is nearly tight and matches the lower bound (modulologarithmic factors) for K ≥ Ω( κ 2 ) .
Effectiveness of batch reshuffling: When K ≥ Ω( κ 2 ) , ˜ O
(1 /K 2 ) becomes the dominant term in theconvergence rate of AS.
This is worse than that of RR/SO by a factor of 1 /n and causes a significantslowdown in convergence, since, in many applications, the dataset size n is much larger than K .
Thus, our analysis justifies the effectiveness of batch reshuffling attacks in reducing model accuracyand increasing training time, which is empirically verified by Shumailov et al.
RR for Two-Sided PŁ Objectives
We now analyze RR for a class of smooth nonconvex-nonconcave problems where the objective Fsatisfies a two-sided Polyak Łojasiewicz inequality , first proposed in Yang et al.
We denote thisfunction class as 2PŁ and formally state the assumption as follows.
Assumption 3 (Two-sided Polyak Łojasiewicz Inequality or 2PŁ condition) .
For any x ∈ R d x , y ∈ 249 R d y , the sets arg max ˜ y F ( x , ˜ y ) and arg min ˜ x F (˜ x , y ) are non-empty.
Furthermore, there existpositive constants µ 1 , µ 2 such that F satisfies the following:
[Equation]
The 2PŁ condition is satisfied in several practical settings, including, but not limited to, robust leastsquares [11], imitation learning for linear quadratic requlators [13, 8], and various other problems inreinforcement learning and robust control [10, 8].
Clearly, any SC-SC function is 2PŁ.
However, 2PŁfunctions need not be SC-SC, or even convex-concave.
We refer the readers to Yang et al.
[44] for adetailed discussion of the 2PŁ class and its applications.
Analysis of RR for 2PŁ objectives is challenging not only due to nonconvexity-nonconcavity, but alsobecause F may not have a unique minimax point.
Indeed, as we demonstrate in the appendix, it ispossible to construct 2PŁ functions where the set of minimax points is unbounded.
Hence, the notionof gradient variance at the optima , which we used in our earlier analyses, is no longer meaningful.
To overcome this, we impose the following assumption.
Assumption 4 (Bounded gradient variance) .
There exists a positive constant σ such that the component gradient operators ω
i satisfy the following for any z ∈ R d : 1 /n n (cid:88) i = | ω
i ( z ) − ν
( z ) | 2 ≤ σ 2 .
Input : Number of epochs K , step-sizes α, β > 0 , and initialization ( x 0 , y 0 )
Initialize ( x 10 , y 10 ) ← ( x 0 , y 0 ) for k ∈ [ K ] do
Sample a permutation τ k ∈ S n for i ∈ [ n ] do x ki ← x ki − 1 − α ∇ x f τ k ( i ) ( x ki − 1 , y k 0 ) end
Sample a permutation π k ∈ S n for i ∈ [ n ] do y k i ←
y k
i − 1 + β ∇ y f π k ( i ) ( x k n , y k
i − 1 ) end ( x k
+ , y k + ) ← ( x kn , y kn ) end
Algorithm
Input : Number of epochs K , step-sizes α, β > 0 , and initialization ( x 0 , y 0 )
Initialize ( x 10 , y 10 ) ← ( x 0 , y 0 ) for k ∈ [ K ] do
Adversary chooses a permutation τ k ∈ S n for i ∈ [ n ] do x ki ← x ki − 1 − α ∇ x f τ k ( i ) ( x ki − 1 , y k 0 ) end
Adversary chooses a permutation π k ∈ S n for i ∈ [ n ] do y k
i ← y k
i − 1 + β ∇ y f π k ( i ) ( x k n , y k
i − 1 ) end ( x k
+ , y k + ) ← ( x kn , y kn ) end
4.1 Analysis of AGDA-RR and AGDA-AS
In order to establish the provable benefits of RR for smooth finite-sum minimax optimization of 2PŁobjectives, we propose the Alternating Gradient Descent Ascent with Random Reshuffling (AGDARR) algorithm.
AGDA-RR achieves near-optimal convergence guarantees for 2PŁ objectives bycombining RR with alternating updates [17, 46, 3] and timescale separation [26, 14, 15], two ideasthat have been very useful for improving convergence and stability in nonconvex-nonconcave minimaxoptimization.
Within each epoch k ∈ [ K ] , AGDA-RR uniformly samples a random permutation τ k ,makes one full pass over the dataset, and performs gradient descent (with RR) updates for the variablex using the permutation τ k .
This is followed by sampling another permutation π k and performinggradient ascent (with RR) updates for y using the permutation π k .
The detailed procedure is stated in Algorithm 3.
We also analyze a variant of AGDA-RR in the adversarial shuffling setting, whichwe call AGDA-AS.
The procedure, as described in Algorithm 4, is almost identical to AGDA-RR,except that the permutations τ k and π k are chosen by an adversary.
Before presenting a convergence analysis, we highlight that the absence of a unique minimax pointprevents us from using the squared distance to the optimum as a Lyapunov function.
To this end, weuse the Lyapunov function V λ : R d → R used by Yang et al.
[44] for their analysis.
We begin by firstdefining the best response function Φ : R d → R and its minimum Φ ∗ as follows,
[Equation]
Assumption 3 ensures that Φ is well defined and the existence of a global minimax point guarantees that Φ ∗ is finite.
Subsequently, for any λ > 0 , we define the Lyapunov function V λ as
[Equation]
By definition of Φ , V λ is non-negative for any λ > 0 and V λ
( z ) =
if and only if z is a minimaxpoint of F .
Hence, we present our convergence proofs for AGDA-RR and AGDA-AS in terms of V λ .
Theorem 3 (Convergence of AGDA-RR/AS) .
Let Assumptions 1, 3, and 4 hold and let η = 73 l 2 / 2 µ 22 .
Then, there exists a step-size α ≤ 1 / 5 ηnl such that for β = ηα , AGDA-RR satisfies the following for λ = 1 / 10 and any K ≥ 1 :
[Equation]
where κ = max { l / µ 1 , l / µ 2 } .
Under the same setting, AGDA-AS satisfies the following:
[Equation]
where τ 1 , π 1 . .
, τ K , π K denotes any sequence of permutations chosen by the adversary.
Comparison with lower bounds: Strongly convex minimization is a special case of 2PŁ minimaxoptimization, since minimizing the strongly convex function f is equivalent to minimax optimizationof the 2PŁ function F ( x , y ) =
f ( x ) − | y | 2 .
In fact, the x iterates of AGDA-RR for F are exactlythat of GD with RR for f .
Hence, the Ω(1 /nK 2 ) lower bound for strongly convex minimizationusing GD with RR also applies to AGDA-RR.
Thus our convergence rate for AGDA-RR is nearlytight and matches the lower bound (modulo logarithmic factors) for K ≥ Ω( κ 3 ) .
Similarly, the
Incremental Gradient version of AGDA is a special case of AGDA-AS with τ 1 , π 1 , . .
, τ K , π K = idand hence, AGDA-AS is nearly tight and matches the Ω(1 /K 2 ) lower bound (modulo logarithmicfactors) for K ≥ Ω( κ 3 ) .
Comparison with stochastic AGDA: Similarly, the Ω(1 /nK ) lower bound of SGD with replacementalso holds for the Stochastic AGDA algorithm [44], which samples the component functions withreplacement and performs two-timescale alternating updates similar to AGDA-RR.
Hence, Theorem3 demonstrates that AGDA-RR provably outperforms stochastic AGDA when K ≥ Ω( κ 3 ) .
Bounded iterate assumption Assumption 4 is also used in the analysis of RR for minimizing PŁfunctions [31].
For minimization, an alternative bounded iterate assumption , which assumes all theiterates z ki lie within a compact set, has also been used to analyze RR [1].
As shown in the appendix,our proof of Theorem 3 is easily adaptable to this setting.
To the best of our knowledge, there isno result in the literature, which, even for minimization, analyzes RR for the PŁ class without anyassumption on the gradient variance or on the boundedness of the iterates.
Experiments
We validate our theoretical results by benchmarking on finite-sum SC-SC quadratic minimax games.
This class of problems appears in several applications such as reinforcement learning [10], robustregression, [11] and online learning [25].
The objective F and the components f i are given by:
[Equation]
100 Epochs 0.
D i s t a n c e t
o M i n i m a x P o i n t
SGDAGDA-RRGDA-SOGDA-IG (a) GDA     100 Epochs
D i s t a n c e t
o M i n i m a x P o i n t
SPPMPPM-RRPPM-SOPPM-IG (b)
PPM     100 Epochs
D i s t a n c e t
o M i n i m a x P o i n t
Stochastic AGDA AGDA-RRAGDA-IG (c) AGDA where A and C are strictly positive definite.
We generate the components
f i randomly, such that 305 (cid:80) ni
=1 u i =
(cid:80) ni
=1 v i =
and the expected singular values of B are larger than that of A and C .
This ensures that the bilinear coupling term x T
By is sufficiently strong, since a weak couplingpractically reduces to quadratic minimization, which has already been investigated in prior works.
Finally, to investigate how the presence of nonconvex-nonconcave components impacts convergence,a few randomly chosen f i ’s are allowed to be nonconvex-nonconcave quadratics.
For each algorithmanalyzed in the text, we benchmark sampling without replacement against uniform sampling byrunning each method for 100 epochs using constant step-sizes that are selected independently foreach method via grid search.
Further details regarding the setup is discussed in the appendix.
We present our results in Figure 2 where we plot the relative distance of the epoch iterates from theminimax point, defined as | z k 0 − z ∗ | 2 / | z 0 − z ∗ | 2 , averaged over 50 independent runs.
In agreement withour theoretical findings, sampling without replacement consistently outperforms uniform samplingacross all three setups.
Furthermore, to demonstrate that our observations are not particular to onespecific instance, we repeat the experiment for 20 independently sampled quadratic games, and foreach instance, perform 5 independent runs of each algorithm and plot the average relative distanceof the epoch iterates from the minimax point.
The results, presented in Figure 3, substantiates thesuperior convergence of sampling without replacement across multiple instances.
[Figure]
Conclusion
We derived near optimal convergence rates for several without-replacement stochastic gradientalgorithms for finite-sum minimax optimization, and demonstrated that they converge faster thanalgorithms that use uniform sampling.
We considered two problem classes, strongly convex-stronglyconcave problems (generalized to unconstrained strongly monotone variational inequalities) andnonconvex-nonconcave problems with two-sided PŁ objectives.
We also formally defined adversarialshuffling , where an attacker can control the order in which data points are supplied to the optimizer,and analyzed minimax optimization in this regime.
Interesting future directions include the analysisof inexact proximal point methods, more general function classes, and time-varying step-sizes.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
Yes]
(b) Did you describe the limitations of your work?
[Yes] Limitations such as structuralassumptions on the function class and noise variance are clearly stated wherever
appropriate.
(c) Did you discuss any potential negative societal impacts of your work?
[No] The work
is purely of a theoretical nature and analyzes existing algorithms that are already widelyprevalent.(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Assumptions
are stated in Sections 2, 3 and 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] A proof sketch of the
main result is presented in Section 3.2 while full proofs are deferred to the appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] They are
included in the supplementary material
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] .Key
details are specified in Section 5 and further details are
specified in the appendix.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] All results are reported with 95% confidence intervals.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] .These details are provided in the
appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]