Sampling without Replacement Leads to Faster Rates in Finite-Sum Minimax Optimization
Anonymous Author(s)
AffiliationAddress email
Abstract
We analyze the convergence rates of stochastic gradient algorithms for smoothfinite-sum minimax optimization and show that, for many such algorithms, sampling the data points without replacement leads to faster convergence compared tosampling with replacement.
For the smooth and strongly convex-strongly concavesetting, we consider gradient descent ascent and the proximal point method, andpresent a unified analysis of two popular without-replacement sampling strategies,namely Random Reshuffling (RR), which shuffles the data every epoch, and Single
Shuffling or Shuffle Once (SO), which shuffles only at the beginning.
We obtaintight convergence rates for RR and SO and demonstrate that these strategies lead tofaster convergence than uniform sampling.
Moving beyond convexity, we obtainsimilar results for smooth nonconvex-nonconcave objectives satisfying a two-sided
Polyak-Łojasiewicz inequality.
Finally, we demonstrate that our techniques aregeneral enough to analyze the effect of data-ordering attacks , where an adversarymanipulates the order in which data points are supplied to the optimizer.
Ouranalysis also recovers tight rates for the incremental gradient method, where thedata points are not shuffled at all.
Introduction
The approximate solution of large-scale optimization problems using first-order stochastic gradientmethods constitutes one of the foundations of classical machine learning.
However, emergingproblems in machine learning go beyond pattern recognition and involve real-world decision making,where learning algorithms interact with unknown or even adversarial environments or are deployedin multi-agent settings.
Decision making in such environments often involves solving a minimaxoptimization problem of the form min
x max y F ( x , y ) , whose analysis has been a focus of researchin mathematics, economics and theoretical computer science [34, 12, 9].
Recent examples of itsapplications in machine learning include adversarial learning [29, 44, 6], reinforcement learning [27,48, 10, 37], imitation learning [13, 8, 22] and generative adversarial networks [17, 2].
In most of theseapplications, the objective F ( x , y ) has a finite-sum structure, i.e.,
[Equation]
where n denotes the size of the dataset and each component function f i denotes the objectiveassociated with the i th data point.
The resultant problem is known as finite-sum minimax optimization :
[Equation]
When n is large and the f i ’s are differentiable (which holds for most applications), approximatesolutions to (1) are computed using stochastic gradient algorithms.
These algorithms typically samplean index i ∈ [ n ] at each iteration as per some specified sampling routine, and use the gradients of f ito compute the next iterate.
Among these methods, perhaps the simplest and most commonly used algorithm is Stochastic Gradient Descent Ascent (SGDA), a natural extension of Stochastic Gradient
Descent (SGD) for minimax optimization.
Similar to the stochastic optimization literature, analysis of stochastic minimax optimization oftenassumes that, at every iteration, the index i is sampled uniformly with replacement .
Analysis of theresulting algorithm closely parallels the analysis of SGD, and relies on the fact that uniform samplingleads to unbiased gradient estimates.
Recent works [19, 28] have also extended this paradigm toi.i.d uniform sampling of mini-batches.
While uniform sampling assumptions simplify theoreticalanalysis, practical implementations of these algorithms often deviate from this paradigm, and insteadincorporate various heuristics, which are often empirically found to improve runtime.
A commonand notable heuristic is to replace uniform sampling by procedures that perform multiple passesover the entire dataset, and in each such pass (called an epoch), sample the data points withoutreplacement .
Thus, each data point is sampled exactly once in every epoch.
These procedures aregenerally implemented using one of the following approaches:
Random Reshuffling (RR): Uniformly sample a random permutation of [ n ] at the start of everyepoch, and process the data points within that epoch as per the order specified by the permutation.
Single Shuffling or Shuffle Once (SO): Uniformly sample a random permutation at the beginningand reuse it across all epochs to order the data points.
Incremental Gradient (IG): Do not permute the data points at all and follow a fixed deterministic data ordering
for every epoch.
Sampling without replacement is ubiquitous in both stochastic minimization [7, 41, 4] and stochasticminimax optimization [17, 2] as it often exhibits faster runtime than uniform sampling.
However,these empirical benefits come at the cost of limited theoretical understanding, due to the absence ofprovably unbiased gradient estimates.
It is well known in the optimization literature that SGD with replacement has a tight rate of O (1 /nK )for smooth and strongly convex minimization [39, 23], where n is the number of component functionsand K denotes the number of epochs.
On the contrary, recent works on SGD without replacementfor smooth and strongly convex minimization [1, 32, 31, 35] show that both RR and SO achieve anon-asymptotic rate of ˜ O
(1 /nK 2 ) , once the number of epochs K is larger than a certain threshold
K 0 (usually polynomial in the condition number), and thereby converge faster than SGD withreplacement.
These rates match the lower bound of Ω(1 /nK 2 ) for RR and SO established in priorworks [38, 41], modulo logarithmic factors.
For RR, prior works have also established a similar˜ O (1 /nK 2 ) rate for nonconvex objectives satisfying the Polyak-Łojasiewicz (PŁ) inequality [1, 31].
While the asymptotic behavior of IG has been known to the community for a long time in both smoothand non-smooth settings [5, 33], non-asymptotic ˜ O (1 /K 2 ) convergence rates have been establishedquite recently [31, 35, 20], and are complemented by a matching Ω(1 /K 2 ) lower bound [41].
1.1 Contributions
Although the empirical benefits of sampling without replacement have been substantiated for minimization, analysis of these methods for minimax optimization have received much less attention,despite being widely prevalent in many applications.
Our work aims to fill this gap by analyzingthese methods for minimax optimization.
To this end, our main contributions are as follows:
Unified analysis of RR and SO for smooth strongly convex-strongly concave problems: Weanalyze RR and SO in conjunction with simultaneous Gradient Descent Ascent (GDA), callingthe resulting algorithms GDA-RR and GDA-SO, respectively.
Assuming the components f i aresmooth and F is strongly convex-strongly concave, we present a unified analysis of GDA-RR/SOand establish a convergence rate of ˜ O
(exp( − K/ 5 κ 2 ) + 1 /nK 2 ) for both (where κ is the conditionnumber).
Comparing with lower bounds, we show that our rates are nearly tight , i.e., they differ fromthe lower bound only by an exponentially decaying term.
Moreover, when K ≥ 10 κ 2 log( n 1 / 2 K ) ,the convergence rate matches the lower bounds for GDA-RR/SO, modulo logarithmic factors, andalso converges provably faster than SGDA with replacement.
Under the same setting, we obtainsimilar guarantees for the RR and SO variants of the Proximal Point Method (PPM), named PPM-RRand PPM-SO respectively.
Our analysis for both GDA-RR/SO and PPM-RR/SO is general enoughto cover smooth strongly monotone finite-sum variational inequalities, which covers minimization,minimax optimization, and multiplayer games.
RR for smooth two-sided PŁ objectives: We consider a class of nonconvex-nonconcave problemswhere the objective F satisfies a two-sided Polyak-Łojasiewicz inequality.
For such problems, wepropose an algorithm that combines RR with two-timescale Alternating Gradient Descent Ascent(AGDA), which we call AGDA-RR.
We show that AGDA-RR has a nearly tight convergencerate of ˜ O
(exp( − K/ 365 κ 3 ) + 1 /nK 2 ) when the gradient variance is uniformly bounded.
When
K ≥ 730 κ 3 log( n 1 / 2 K ) , this rate matches the lower bound (modulo logarithmic factors) andimproves on the best known rates of with-replacement algorithms for this class of problems.
Minimax optimization under data ordering attacks: Our techniques for analyzing RR/SO generalize to the analysis of finite-sum minimax optimization under data ordering attacks [43].
These attackstarget the inherent randomness assumptions of stochastic gradient algorithms, significantly increasingtraining time and reducing model quality, only by manipulating the order in which the algorithmreceives data points, without performing any data contamination.
To model these attacks, we proposethe Adversarial Shuffling (AS) setup, where the data points are shuffled every epoch by a computationally unrestricted adversary.
In this setup, we show that GDA and PPM (now called GDA-AS and PPM-AS) have a convergence rate of ˜ O
(exp( − K/ 5 κ 2 )+1 /K 2 ) for smooth strongly convex-stronglyconcave objectives, and AGDA (now called AGDA-AS) has a rate of ˜ O
(exp( − K/ 365 κ 3 ) + 1 /K 2 )for two-sided PŁ objectives.
We note that, compared to RR and SO, the convergence rate worsensby a factor of 1 /n for large enough K .
When n is large (true for most applications), this slowdownsignificantly impacts convergence and thus, theoretically justifies the empirical observations in priorwork [43].
We also establish that our analysis in the AS regime also applies to the Incremental
Gradient (IG) variants of these algorithms (namely GDA-IG, PPM-IG, and AGDA-IG), and use thisto show that our obtained rates for GDA-RR and AGDA-RR are nearly tight.
To the best of our knowledge, our work is the first to: 1) analyze RR, SO, and IG for stronglymonotone unconstrained VIs, 2) analyze RR and IG for a class of nonconvex-nonconcave minimaxproblems, 3) provably demonstrate the advantages of sampling without replacement for both thesesettings and justify its empirical benefits in a wide variety of problems ranging from minimization,minimax optimization to smooth multiplayer games, 4) analyze sampling without replacement underdata-ordering attacks.
Furthermore, unlike prior works on sampling without replacement for minimaxoptimization [46, 30] which are restricted to random reshuffling, and require the component functionsto be convex-concave, Lipschitz, and smooth, our analysis does not impose any restrictions onthe components
f i other than smoothness, allowing them to be arbitrary nonconvex-nonconcavefunctions.
Notation and Preliminaries
We work with Euclidean spaces ( R d , ⟨ ., .
⟩ ) equipped with the standard inner product ⟨ x 1 , x 2 ⟩ and theinduced norm | x | .
For any x ∈ R d x and y ∈ R d y , we denote z =
( x , y ) ∈
R d where d = d x + d y . Moreover, for any z 1 = ( x 1 , y 1 ) ∈ R d and z [-
-]2 = ( x 2 , y 2 ) ∈ R d , ⟨ z 1 , z 2 ⟩ = ⟨ x 1 , x 2 ⟩ + ⟨ y 1 , y 2 ⟩ and| z 1 | 2 = | x 1 | 2 + | y 1 | 2 .
Whenever z = ( x , y ) is clear from the context, we write f ( x , y ) as f ( z ) .
Weuse S n to denote the set of all permutations of [ n ] = { 1 , . . . , n } .
For any matrix A , its operator normis denoted by | A | = sup | x | =1 |
Ax | .
We use the O notation to characterize the dependence of ourconvergence rate on n and K , suppressing numerical and problem-specific constants such as κ, µ, σ ,etc.
Additionally, we use the ˜ O notation to suppress logarithmic factors of n and K .
Our work studies finite-sum minimax optimization (1).
Solutions to (1) are known as global minimaxpoints of F = 1 /n (cid:80) ni
=
f i , which we assume to always exist .
We also assume that the componentsf i are continuously differentiable, and hence, the same applies to F .
This allows us to define thegradient operators ω
i : R d → R d and ν : R d → R d as follows:
[Equation]
We also impose the following smoothness assumption on the components f i .
Assumption 1 (Component Smoothness) .
The component functions f
i are l -smooth, i.e., each gradient operator
ω
i is [Equation] -Lipschitz
[Equation]
Consequently, the operator ν is also l -Lipschitz, i.e., F is l -smooth.
Algorithm 1: GDA-RR/SO/AS
Input : Number of epochs K , step-size α > 0 , and initialization z
Initialize z 10 ← z
SO: Sample τ ∼ Uniform ( S n ) for k ∈ [ K ] do
RR: Sample τ k ∼ Uniform ( S n )
SO:
τ k ← τ
AS: Adversary chooses τ k ∈ S n for i ∈ [ n ] do z ki ← z ki − 1 − αω τ k
( i ) ( z ki − 1 ) end
z k + ← z kn end
Algorithm 2: PPM-RR/SO/AS
Input : Number of epochs K , step-size α > 0 , and initialization z
Initialize z 10 ← z
SO: Sample τ ∼ Uniform ( S n ) for k ∈ [ K ] do
RR: Sample τ k ∼ Uniform ( S n )
SO:
τ k ← τ
AS: Adversary chooses τ k ∈ S n for i ∈ [ n ] do
Solve the implicit update for z ki where, z ki =
z ki − 1 − αω τ k
( i ) ( z ki ) end
z
k
+ ← z kn end
Analysis for Strongly Convex-Strongly Concave Objectives
In this section, we analyze two very popular without-replacement algorithms for finite-sum minimaxoptimization, Gradient Descent Ascent (GDA) without replacement and Proximal Point Method(PPM) without replacement.
For each of these, we present a unified analysis of the Random
Reshuffling (RR) and Shuffle Once (SO) variants (called GDA-RR/SO and PPM-RR/SO respectively).
For a fixed K > 0 , GDA-RR/SO approximately solves (1) by iterating over the entire dataset for Kepochs, and within each epoch, uses the operators ω
i to perform the following iterative update:
[Equation]
where τ k is a uniformly sampled random permutation of [ n ] and 0 < α < 1 /l is a constant step-size.
GDA-RR resamples τ k at the start of every epoch, whereas GDA-SO samples it only once in thebeginning.
The details of both algorithms are presented in Algorithm 1.
The Proximal Point Methodwithout replacement is a closely related algorithm which, instead of performing gradient descent-styleupdates within an epoch, solves the following implicit update equation for z ki :
[Equation]
As before, τ k is resampled at every epoch for PPM-RR, and sampled once and fixed for all epochsfor PPM-SO.
We present the details in Algorithm 2.
The l -smoothness of ω i along with the choiceof α < 1 /l ensures that z ki is uniquely defined, since it is a fixed point of the contraction mappingζ ( z ) =
z ki − 1 − αω τ k
( i ) ( z ) .
This method is actually a generalization of the (stochastic) proximal pointmethod for minimization problems, and is popular for problems where (3) can be solved easily or inclosed form.
We refer the readers to Rockafellar [40], Patrascu and Necoara [36] for a review of thismethod and its connections to the original proximal point method for minimization.
3.1 Setting
We analyze GDA-RR/SO and PPM-RR/SO for smooth finite-sum strongly convex-strongly concave(or SC-SC) objectives.
This allows us to formulate the minimax optimization problem for F as a rootfinding problem for the gradient operator ν , as described below.
Assumption 2 (Strong Convexity-Strong Concavity) .
The objective F is µ strongly convex-stronglyconcave (or SC-SC), i.e., F ( ., y ) is µ -strongly convex for any y ∈ R d y and − F ( x , . )
is µ -stronglyconvex for any x ∈ R d x .
Assumption 2 has the following consequences for the gradient operator ν :
Lemma 1.
Let F satisfy Assumptions 1 and 2.
Then, the gradient operator ν is µ -strongly monotone:
[Equation]
Furthermore, (1) admits a unique solution z ∗ which is also the unique solution of ν ( z ∗ ) = 0 .
Lemma 1 allows us to recast (1) for SC-SC objectives as the following root finding problem :
Find z ∈ R d such that ν ( z )
= 1 /n
[Equation]
Problem (4) is more general than SC-SC minimax optimization, and is a special case of stronglymonotone variational inequalities [12] without constraints.
Notably, (4) includes the Nash Equilibriumproblem for unconstrained multiplayer games with smooth strongly convex objectives [42] and issometimes called a finite-sum unconstrained variational inequality in the literature [28].
We alsohighlight that smooth strongly convex optimization is a special case of (4).
However, unlike theoptimization setting, ν is no longer restricted to be the gradient of a strongly convex function, which,as we shall see, has important consequences for the attainable convergence rates of our algorithms.
3.2 Analysis of RR/SO
We now state the last-iterate convergence guarantees for Algorithms 1 and 2 for solving (4), wherethe expecation is taken over the stochasticity of the sampled permutation(s).
Theorem 1 (Convergence of GDA-RR/SO and PPM-RR/SO) .
Consider Problem (4) for the µstrongly monotone operator
ν ( z ) = 1 / n (cid:80) ni
=1 ω i ( z ) where each ω i is l -Lipschitz, but not necessarily monotone.
Let z ∗ denote the unique root of ν .
Then, there exists a step-size α ≤ µ / 5 nl 2 for which both GDA-RR/SO and PPM-RR/SO satisfy the following for any K ≥ 1 :
[Equation]
+ 1 / nK 2 ) , where κ = l / µ is the condition number and σ 2 ∗ = 1 / n (cid:80) ni
=1 | ω i ( z ∗ ) | 2 is the gradient variance at z ∗ .
Proof.
We present an outline for GDA-RR/SO and defer the full proof to Appendix C.2 (for GDARR/SO) and Appendix D.2 (for PPM-RR/SO).
Furthermore, we recall that the updates of GDA-RR/SOare given by z ki =
z ki − 1 − αω τ k
( i ) ( z ki − 1 ) .
We begin with the following key insight from earlier works on sampling without replacement forminimization [21, 1, 33, 20]: for small enough step-sizes, the epoch iterates z k 0 of GD withoutreplacement approximately follow the trajectory of full-batch gradient descent.
To this end, we derivethe following epoch-level update rule for GDA-RR/SO by linearizing ω τ k ( i ) ( z ki − 1 ) around z ∗ :
[Equation]
where | H k | ≤ 1 − nαµ/ 2 and r k
= (cid:80) n − 1 i
=
A τ k
( i ) (cid:80) ij =1 ω τ k ( j ) ( z ∗ ) with (cid:12)(cid:12)
A τ k ( i ) (cid:12)(cid:12) ≤ le 1 / 5 .
Theterm r k encapsulates the noise of the stochastic gradient updates accumulated over an entire epoch.
[-Clearly, for n = 1 , r k = 
-]
To ensure convergence, we control the influence of the noise term r k by using standard propertiesof without-replacement sample means to show that E [ | r k | 2 ] ≤ l 2 n 3 σ 2 ∗ / 4 for both RR and SO.
Wethen complete the proof by unrolling (5) for K epochs, substituting the upper bounds for | H k | and E [ | r k | 2 ] wherever necessary, and setting α = min { µ/ 5 nl 2 , 2 log( | ν ( z 0 ) | n 1 / 2 K/µ ) /µnK } .
As we show in Appendix C, the update rule (5) resembles the linearized update rule of full batch
GDA with added noise. In fact, for n = 1 , r k = 0 and thus, we recover the rates of full-batch GDA. Expressing GDA-RR/SO (and later AS) as noisy full-batch GDA in this fashion is a centralcomponent of our unified analysis, and relies on the fact that (cid:80) ni =1 ω τ k
( i ) ( z ∗ ) =
∀ τ k ∈ S n , whichis specific to sampling without replacement.
Comparing to SGDA with replacement, we note thatsampling the components i.i.d.
uniformly as u ( i ) ∼ Uniform ([ n ]) gives rise to an additional noiseterm α p k in the update rule, where p k = (cid:80) ni
=1 ω u ( i ) ( z ∗ ) vanishes only in expecation, and has avariance of E [ | p k
| 2 ] = nσ 2 ∗ .
Subsequently, the dominant noise term for SGDA updates is O ( α 2 nσ 2 ∗ )whereas that of GDA-RR/SO is O
( α 4 n 3 σ 2 ∗ ) , which qualitatively demonstrates the implicit variancereduction of sampling without replacement.
As we shall see in the complete proof, this allows RR/SOto converge faster (for large enough K ) by carefully selecting α .
Comparison with lower bounds: Since smooth strongly convex minimization is a special caseof (4), the Ω(1 /nK 2 ) lower bound established in prior works [41, 38] for smooth strongly convex minimization using GD with RR/SO also applies to GDA-RR/SO.
Comparing with this lower bound,we note that the convergence rate of GDA-RR/SO is nearly tight , i.e., it differs from the lower boundonly by an exponentially decaying term.
In fact, for K ≥ 10 κ 2 log( n 1 / 2 K ) , the convergence ratebecomes ˜ O (1 /nK 2 ) , which precisely matches the lower bound, modulo logarithmic factors.
Comparison with uniform sampling: Similarly, the Ω(1 /nK ) lower bound of SGD with replacement for smooth and strongly convex functions [39] also applies to SGDA with replacement.
On the contrary, both GDA-RR and GDA-SO converge with a faster rate of ˜ O
(1 /nK 2 ) when K ≥ 10 κ 2 log( n 1 / 2 K ) .
Thus, GDA-RR/SO provably outperform SGDA with replacement (modulologarithmic factors) when K ≥ 10 κ 2 log( n 1 / 2 K ) .
As we show in Appendix C.2, the κ 2 dependenceof this inequality cannot be improved for constant step-sizes.
A similar argument also applies tostochastic PPM.
To the best of our knowledge, the fastest known convergence rate for stochastic PPMis O (1 /nK ) for minimizing smooth strongly convex functions [36].
Hence, Theorem 1 suggeststhat PPM-RR/SO enjoy a faster ˜ O (1 /nK 2 ) convergence rate for both minimization and minimaxoptimization when K ≥ 10 κ 2 log( n 1 / 2 K ) .
3.3 Analysis in the Adversarial Shuffling Regime
We now consider without-replacement minimax optimization algorithms in an adversarial setting.
Wefocus on a novel class of training-time attacks known as data ordering attacks proposed by Shumailovet al.
These attacks differ from standard data-perturbation attacks [18] and exploit the fact thatmost implementations of stochastic gradient optimizers do not verify whether the permutation τ kis truly sampled at random.
Shumailov et al.
[43] propose three distinct attack strategies, namely,batch reordering , which changes the order in which mini-batches are supplied to the algorithm, batchreshuffling , which changes the order in which individual data points are supplied, and replacingwhich prevents certain data points from being observed by the algorithm by consistently replacingthem with other data points in the training set.
We analyze the convergence of without-replacement GDA and PPM under batch reshuffling attacks.
To this end, we consider an adversarial modification of RR/SO where the permutations τ k , instead ofbeing sampled by the algorithm, are now selected by an adversary using a strategy unknown to thealgorithm.
We also the assume that, while choosing τ k , the adversary is computationally unrestrictedand has complete knowledge of all the components
ω
i , the minimax point z ∗ , and the iterates z kiobserved so far.
We call this setup Adversarial Shuffling (AS) and obtain convergence rates of GDAand PPM (named GDA-AS and PPM-AS) when solving (4).
Thus, our analysis naturally holds forminimization, minimax optimization as well as finite-sum multiplayer games.
The details are statedin Algorithms 1 and 2, respectively.
Our last iterate convergence guarantees are deterministic andhold uniformly over any sequence of permutations τ 1 , . .
, τ K that the adversary can choose.
Theorem 2 (Convergence of GDA-AS and PPM-AS) .
Consider Problem (4) for the µ -strongly monotone operator
ν ( z ) = 1 / n (cid:80) ni
=1 ω i ( z ) where each ω i is l -Lipschitz, but not necessarily monotone.
Let z ∗ denote the unique root of ν .
Then, there exists a step-size α ≤ µ / 5 nl 2 for which both GDA-AS and PPM-AS satisfy the following for any K ≥ 1 :
[Equation]
µ 2 K= ˜ O ( e − K / 5 κ 2 + 1 / K 2 ) , where κ, σ 2 ∗ are as defined in Theorem 1 and τ 1 , . . .
, τ K are the permutations chosen by the adversary.
Convergence rates of IG and comparison with lower bounds: We note that the Incremental
Gradient and the Incremental Proximal Point Methods, which do not shuffle the data, are a specialcase of GDA-AS/PPM-AS with τ 1 , . .
, τ K = id .
Thus, Theorem 2 also gives us convergence rates for GDA-IG/PPM-IG.
Moreover, since GDA-AS generalizes GDA-IG and (4) covers minimization, theΩ(1 /K 2 ) lower bound of IG establihsed in prior works [41] for smooth strongly convex minimizationalso applies to GDA-AS.
Thus, our obtained rate for GDA-AS is nearly tight and matches the lowerbound (modulo logarithmic factors) for K ≥ 10 κ 2 log( K ) .
Effectiveness of batch reshuffling: When K ≥ 10 κ 2 log( K ) , ˜ O
(1 /K 2 ) becomes the dominantterm in the convergence rate of AS.
This is worse than that of RR/SO by a factor of 1 /n and causes asignificant slowdown in convergence, since, in many applications, the dataset size n is much largerthan K .
Thus, our analysis justifies the effectiveness of batch reshuffling attacks in reducing modelaccuracy and increasing training time, which is empirically verified by Shumailov et al.
RR for Two-Sided PŁ Objectives
We now analyze RR for a class of smooth nonconvex-nonconcave problems where the objective Fsatisfies a two-sided Polyak Łojasiewicz inequality , first proposed in Yang et al.
We denote thisfunction class as 2PŁ and formally state the assumption as follows.
Assumption 3 (Two-sided Polyak Łojasiewicz Inequality or 2PŁ condition) .
For any x ∈ R d x , y ∈ 252 R d y , the sets arg max ˜ y F ( x , ˜ y ) and arg min ˜ x F (˜ x , y ) are non-empty.
Furthermore, there existpositive constants µ 1 , µ 2 such that F satisfies the following:
[Equation]
The 2PŁ condition is satisfied in several practical settings, including, but not limited to, robust leastsquares [11], imitation learning for linear quadratic regulators [13, 8], and various other problems inreinforcement learning and robust control [10, 8].
Clearly, any SC-SC function is 2PŁ.
However, 2PŁfunctions need not be SC-SC, or even convex-concave.
We refer the readers to Yang et al.
[45] for adetailed discussion of the 2PŁ class and its applications.
Analysis of RR for 2PŁ objectives is challenging not only due to nonconvexity-nonconcavity, butalso because F may not have a unique minimax point.
Indeed, as we demonstrate in Appendix E.1, itis possible to construct 2PŁ functions where the set of minimax points is an unbounded proper subsetof R d .
Hence, the notion of gradient variance at the minimax point , which we used in our earlieranalyses, is no longer meaningful.
To overcome this, we impose the following assumption.
Assumption 4 (Bounded gradient variance) .
There exists a positive constant σ such that the component gradient operators ω
i satisfy the following for any z ∈ R d : 1 /n [Equation] (cid:88)
[Equation]
| [Equation]
Input : Number of epochs K , step-sizes α, β > 0 , and initialization ( x 0 , y 0 )
Initialize ( x 10 , y 10 ) ← ( x 0 , y 0 ) for k ∈ [ K ] do
Sample a permutation τ k ∈ S n for i ∈ [ n ] do x ki ← x ki − 1 − α ∇ x f τ k ( i ) ( x ki − 1 , y k 0 ) end
Sample a permutation π k ∈ S n for i ∈ [ n ] do y k i ←
y k
i − 1 + β ∇ y f π k ( i ) ( x k n , y k
i − 1 ) end ( x k
+ , y k + ) ← ( x kn , y kn ) end
Algorithm 4: AGDA-AS
Input : Number of epochs K , step-sizes α, β > 0 , and initialization ( x 0 , y 0 )
Initialize ( x 10 , y 10 ) ← ( x 0 , y 0 ) for k ∈ [ K ] do
Adversary chooses a permutation τ k ∈ S n for i ∈ [ n ] do x ki ← x ki − 1 − α ∇ x f τ k ( i ) ( x ki − 1 , y k 0 ) end
Adversary chooses a permutation π k ∈ S n for i ∈ [ n ] do y k
i ← y k
i − 1 + β ∇ y f π k ( i ) ( x k n , y k
i − 1 ) end ( x k
+ , y k + ) ← ( x kn , y kn ) end
4.1 Analysis of AGDA-RR and AGDA-AS
In order to establish the provable benefits of RR for smooth finite-sum minimax optimization of 2PŁobjectives, we propose the Alternating Gradient Descent Ascent with Random Reshuffling (AGDARR) algorithm.
AGDA-RR achieves near-optimal convergence guarantees for 2PŁ objectives bycombining RR with alternating updates [16, 47, 3] and timescale separation [26, 14, 15], two ideasthat have been very useful for improving convergence and stability in nonconvex-nonconcave minimaxoptimization.
Within each epoch k ∈ [ K ] , AGDA-RR uniformly samples a random permutation τ k ,makes one full pass over the dataset, and performs gradient descent (with RR) updates for the variablex using the permutation τ k .
This is followed by sampling another permutation π k and performinggradient ascent (with RR) updates for y using the permutation π k .
The detailed procedure is stated in Algorithm 3.
We also analyze a variant of AGDA-RR in the adversarial shuffling setting, whichwe call AGDA-AS.
The procedure, as described in Algorithm 4, is almost identical to AGDA-RR,except that the permutations τ k and π k are chosen by an adversary.
Before presenting a convergence analysis, we highlight that the absence of a unique minimax pointprevents us from using the squared distance to the optimum as a Lyapunov function.
To this end, weuse the Lyapunov function V λ : R d → R used by Yang et al.
[45] for their analysis.
We begin by firstdefining the best response function Φ : R d → R and its minimum Φ ∗ as follows,
[Equation]
Assumption 3 ensures that Φ is well defined and the existence of a global minimax point guarantees that Φ ∗ is finite.
Subsequently, for any λ > 0 , we define the Lyapunov function V λ as
[Equation]
By definition of Φ , V λ is non-negative for any λ > 0 and V λ
( z ) =
if and only if z is a minimaxpoint of F .
Hence, we present our convergence proofs for AGDA-RR and AGDA-AS in terms of V λ .
Theorem 3 (Convergence of AGDA-RR/AS) .
Let Assumptions 1, 3, and 4 hold and let η = 73 l 2 / 2 µ 22 .
Then, there exists a step-size α ≤ 1 / 5 ηnl such that for β = ηα , AGDA-RR satisfies the following for λ = 1 / 10 and any K ≥ 1 :
[Equation]
where κ = max { l / µ 1 , l / µ 2 } and c > 0 is a constant independent of κ, µ 1 , µ 2 , σ 2 .
Under the same setting, AGDA-AS satisfies the following ( ˆ c > 0 is a constant independent of κ, µ 1 , µ 2 , σ 2 ):
[Equation]
where τ 1 , π 1 . .
, τ K , π K are the permutations chosen by the adversary.
Comparison with lower bounds: Strongly convex minimization is a special case of 2PŁ minimaxoptimization, since minimizing the strongly convex function f is equivalent to minimax optimizationof the 2PŁ function F ( x , y ) =
f ( x ) −| y | 2 .
In fact, the x iterates of AGDA-RR for F are exactly thatof GD with RR for f .
Hence, the Ω(1 /nK 2 ) lower bound for strongly convex minimization using GD with RR also applies to AGDA-RR.
Thus our convergence rate for AGDA-RR is nearly tight andmatches the lower bound (modulo logarithmic factors) for K ≥ 730 κ 3 log( n 1 / 2 K ) .
Similarly, the
Incremental Gradient version of AGDA is a special case of AGDA-AS with τ 1 , π 1 , . .
, τ K , π K = idand hence, AGDA-AS is nearly tight and matches the Ω(1 /K 2 ) lower bound (modulo logarithmicfactors) for K ≥ 730 κ 3 log( K ) .
Comparison with stochastic AGDA: Similarly, the Ω(1 /nK ) lower bound of SGD with replacementalso holds for the Stochastic AGDA algorithm [45], which samples the component functions withreplacement and performs two-timescale alternating updates similar to AGDA-RR.
Hence, Theorem 3demonstrates that AGDA-RR provably outperforms stochastic AGDA when K ≥ 730 κ 3 log( n 1 / 2 K ) .
Bounded iterate assumption Assumption 4 is also used in analyzing RR for PŁ function minimization [31].
In this setting, an alternative bounded iterate assumption , which assumes that all theiterates z ki lie within a compact set, has also been used [1].
As shown in Appendix E.2, our proof of Theorem 3 easily adapts to this assumption.
In the absence of either assumption, Li et al. [25] usetime-varying step-sizes to obtain asymptotic O (1 /K 2 ) rates for RR on PŁ (and more generally for KŁ) minimization.
Experiments
We validate our theoretical results by benchmarking on finite-sum SC-SC quadratic minimax games.
This class of problems appears in several applications such as reinforcement learning [10], robustregression, [11] and online learning [24].
The objective F and the components f i are given by:
[Equation]
100 Epochs 0.
D i s t a n c e t
o M i n i m a x P o i n t
SGDAGDA-RRGDA-SOGDA-IG (a) GDA     100 Epochs
D i s t a n c e t
o M i n i m a x P o i n t
SPPMPPM-RRPPM-SOPPM-IG (b)
PPM     100 Epochs
D i s t a n c e t
o M i n i m a x P o i n t
Stochastic AGDA AGDA-RRAGDA-IG (c) AGDA where A and C are strictly positive definite.
We generate the components
f i randomly, such that 308 (cid:80) ni
=1 u i =
(cid:80) ni
=1 v i =
and the expected singular values of B are larger than that of A and C .
This ensures that the bilinear coupling term x T
By is sufficiently strong, since a weak couplingpractically reduces to quadratic minimization, which has already been investigated in prior works.
Finally, to investigate how the presence of nonconvex-nonconcave components impacts convergence,a few randomly chosen f i ’s are allowed to be nonconvex-nonconcave quadratics.
For each algorithmanalyzed in the text, we benchmark sampling without replacement against uniform sampling byrunning each method for 100 epochs using constant step-sizes that are selected independently foreach method via grid search.
Further details regarding the setup is discussed in Appendix G.
We present our results in Figure 2 where we plot the relative distance of the epoch iterates from theminimax point, defined as | z k 0 − z ∗ | 2 / | z 0 − z ∗ | 2 , averaged over 50 independent runs.
In agreement withour theoretical findings, sampling without replacement consistently outperforms uniform samplingacross all three setups.
Furthermore, to demonstrate that our observations are not particular to onespecific instance, we repeat the experiment for 20 independently sampled quadratic games, and foreach instance, perform 5 independent runs of each algorithm and plot the average relative distanceof the epoch iterates from the minimax point.
The results, presented in Figure 3, substantiates thesuperior convergence of sampling without replacement across multiple instances.
[Figure]
Conclusion
We derived near optimal convergence rates for several without-replacement stochastic gradientalgorithms for finite-sum minimax optimization, and demonstrated that they converge faster thanalgorithms that use uniform sampling.
We considered two problem classes, strongly convex-stronglyconcave problems (generalized to unconstrained strongly monotone variational inequalities) andnonconvex-nonconcave problems with two-sided PŁ objectives.
We also formally defined adversarialshuffling , where an attacker can control the order in which data points are supplied to the optimizer,and analyzed minimax optimization in this regime.
Interesting future directions include the analysisof inexact proximal point methods, more general function classes, and time-varying step-sizes.
Checklist
For all authors...(
a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
Yes]
(b) Did you describe the limitations of your work?
[Yes] Limitations such as structuralassumptions on the function class and noise variance are clearly stated wherever
appropriate.
(c) Did you discuss any potential negative societal impacts of your work?
[No] The work
is purely of a theoretical nature and analyzes existing algorithms that are already widelyprevalent.(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Assumptions
are stated in Sections 2, 3 and 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] A proof sketch of the
main result is presented in Section 3.2 while full proofs are deferred to the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] They are
included in the supplementary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] .Key
details are specified in Section 5 and further details are
specified in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All results are reported with 95% confidence intervals.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] .These details are provided in the
appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]