Paper: Stochastic Extragradient with Flip-Flop Shuffling & Anchoring: Provable Improvements (ICLR-2024)

Reviewer comment (Novelty):
The paper does not explain why they use the gradient norm as the convergence measure and fails to mention a line of research studying that. In convex-concave minimax problems, the standard convergence measure is the duality gap $\max_y f(\bar x,y) - \min_x f(x,\bar y)$ or minty solution $F(x)^\top (\bar x - x)$ in the sense of variational inequality. This is natural since the optimal point of interest is the Nash equilibrium or the saddle point. There does exist a line of research studying monotone inclusion or gradient norm convergence in the same setting, e.g., Halpern iteration [arXiv:2002.08872, arXiv:2203.09436] or anchoring [arXiv:1905.10899, arXiv:2102.07922, arXiv:2106.02326], but the paper does not compare with them. Specifically, the algorithm in [arXiv:2203.09436] achieves $E\Vert F z_k\Vert^2 \leq 1/k^2$ (see Theorem 4.1) with less assumptions (no Hessian Lipschitzness). The idea of combining extragradient with anchoring is also not new [arXiv:2102.07922].

Author response:
1. **Gradient norm as an optimality measure, and Nash equilibria (W1, W3):**

    We would like to first point out that for (unconstrained) convex-concave problems, a point $(\boldsymbol{x}^∗, \boldsymbol{y}^∗)$ is a first-order stationary point if and only if it is a global Nash equilibrium, since $f(\boldsymbol{x}, \boldsymbol{y}^*) \geq f(\boldsymbol{x}^*, \boldsymbol{y}^*) + \langle \nabla_{\boldsymbol{x}} f(\boldsymbol{x}^*, \boldsymbol{y}^*), \boldsymbol{x} - \boldsymbol{x}^* \rangle$ and $f(\boldsymbol{x}^*, \boldsymbol{y}) \leq f(\boldsymbol{x}^*, \boldsymbol{y}^*) + \langle \nabla_{\boldsymbol{y}} f(\boldsymbol{x}^*, \boldsymbol{y}^*), \boldsymbol{y} - \boldsymbol{y}^* \rangle$. 

    While we agree that the squared gradient norm is not an adequate measure of convergence for *constrained* problems (as also pointed out in Section 1.1 of (Gorbunov et al., 2022b)), taking the squared gradient norm as an optimality measure is not unnatural in *unconstrained* problems. We would also like to note that Gorbunov et al. (2022b) study the convergence of EG in the unconstrained case, using the squared gradient norm as the optimality measure.

    We would like to also note that the convex and compact domain assumption in von Neumann’s minimax theorem is a *sufficient* condition for the existence of a Nash equilibrium, but not a *necessary* condition. For example, consider a simple bilinear problem with $f (x, y) = xy$. This has a unique first-order stationary point $(0, 0)$, which is a global Nash equilibrium. For this reason, we believe that Assumption 2 is natural, as it is analogous to assuming the existence of a minimum in convex minimization.

2. **Existing results on anchoring (W1):**

    We appreciate the reviewer for bringing up the existing results that adopt similar ideas of anchoring, which was not explained well. First of all, we have clarified in footnote 1 that our anchoring and the Halpern-based anchoring are different in their formulations. In addition, their motivations differ as the former (ours) is for the *second-order matching* that makes SEG converge in the convex-concave cases, while the latter is for acceleration in rate. Therefore, we would like to claim that our anchoring approach is new, in the sense that we adopt the idea for making SEG converge in the first place. The  (Cai et al., 2022) [arXiv:2203.09436] paper you kindly mentioned introduces the Halpern-based anchoring to SEG for acceleration, but it requires increasing the batch size after each epoch, on top of the requirement that the variance is uniformly bounded over the domain, which are quite restrictive compared to our setting. This is now stated in Section 2.

Reviewer comment (Theory):
The paper keeps mentioning that GDA and stochastic extragradient do not converge for convex-concave minimax problems and claims that one important contribution is to fix the non-convergence problem. However, for convex-concave minimax problems, it is easy to show that the average iterate of (stochastic) GDA converges with the rate $1/k^{1/2}$ measured by the duality gap, without additional smoothness or Hessian Lipschitzness assumptions (even though the last-iterate does diverge). I am also confused by the statement that previous works (Mishchenko, 2020b & Diakonikolas, 2021) on stochastic extragradient only guarantee convergence to a neighborhood. To my best understanding, these works are able to show $E\Vert F(x)\Vert\leq\epsilon$ (Theorem 4.4 of Diakonikolas, 2021) or $E[F(x)^\top (\bar x - x)]\leq\epsilon$ (Theorem 3 of Mishchenko, 2020b). This paper can be misleading and overclaiming.

Author response:
3. **Stochastic GDA (W2):**

    We are not sure which paper the reviewer is referring to, and we would appreciate it if the reviewer could give us a pointer; nevertheless, we suspect that the mentioned convergence guarantee might not apply to our unconstrained convex-concave setting. This is because, in the *unconstrained* $\min_x \max_y f (x, y) = xy$ case, the duality gap $\max_y f (\bar{x}, y) − \min_x f (x, \bar{y})$ is always $\infty$ unless $(\bar{x}, \bar{y}) = (0, 0)$. Hence, duality gap results should require the assumption of bounded domain. Nevertheless, we would be happy to learn about the reference that discusses the ergodic convergence of the stochastic GDA.

4. **Statement regarding convergence to neighborhood of an optimum (W2)**

    The reviewer is correct that both (Theorem 4.4, Diakonikolas et al., 2021) and (Theorem 3, Mishchenko et al., 2020b) show that they can find an $\epsilon$-stationary point. However, what we originally meant by “convergence to neighborhood” was that the analyses do not imply that the methods will reach an optimum as $k \to \infty$, in the sense that the optimality measure goes all the way to zero as $k \to \infty$. (Here $k$ denotes the number of iterations.) In particular, by taking a close look at the proof of (Theorem 4.4, Diakonikolas et al., 2021), one can see that we need to choose the number of samples per iteration that is greater than $\mathcal{O}(1/\epsilon)$ to guarantee finding an $\epsilon$-stationary point, and to guarantee reaching an optimum one needs $n \to \infty$ in the proof. In addition, (Theorem 3, Mishchenko et al., 2020b) presents a rate bound for a constant step size $\mathcal{O}(1/\sqrt{t})$ for a fixed budget of $t$ iterations, hence cannot handle the $t \to \infty$ case. Nevertheless, we learned that this is subtle and may confuse the readers. So, we decided to emphasize more on our other advantages over existing literature (especially those brought up by the reviewers) in unconstrained cases, and do not argue on the aspect of the convergence to neighborhood.

    We also would like to note that, as now explicitly stated in our Section 2, Diakonikolas et al. (2021) assume the variance to be uniformly bounded over the domain, which is more restrictive than our setting. In addition, Mishchenko et al. (2020b) require the domain to be bounded (as they take the supremum over the entire domain in the final bound) and the stochastic component to be monotone almost surely.

Reviewer comment (Theory):
The paper seems to claim that the bounded domain assumption is strong and thus studies the unconstrained setting. However, according to the well-known von Neumann's minimax theorem, when the objective is convex-concave, the convex and compact domain assumption is necessary to ensure the existence of the Nash equilibrium. When the domain is $R^d$, Assumption 2 is questionable, and it's not clear what solution the algorithm finds when $\Vert Fz\Vert\to 0$.

Author response:
1. **Gradient norm as an optimality measure, and Nash equilibria (W1, W3):**

    We would like to first point out that for (unconstrained) convex-concave problems, a point $(\boldsymbol{x}^∗, \boldsymbol{y}^∗)$ is a first-order stationary point if and only if it is a global Nash equilibrium, since $f(\boldsymbol{x}, \boldsymbol{y}^*) \geq f(\boldsymbol{x}^*, \boldsymbol{y}^*) + \langle \nabla_{\boldsymbol{x}} f(\boldsymbol{x}^*, \boldsymbol{y}^*), \boldsymbol{x} - \boldsymbol{x}^* \rangle$ and $f(\boldsymbol{x}^*, \boldsymbol{y}) \leq f(\boldsymbol{x}^*, \boldsymbol{y}^*) + \langle \nabla_{\boldsymbol{y}} f(\boldsymbol{x}^*, \boldsymbol{y}^*), \boldsymbol{y} - \boldsymbol{y}^* \rangle$. 

    While we agree that the squared gradient norm is not an adequate measure of convergence for *constrained* problems (as also pointed out in Section 1.1 of (Gorbunov et al., 2022b)), taking the squared gradient norm as an optimality measure is not unnatural in *unconstrained* problems. We would also like to note that Gorbunov et al. (2022b) study the convergence of EG in the unconstrained case, using the squared gradient norm as the optimality measure.

    We would like to also note that the convex and compact domain assumption in von Neumann’s minimax theorem is a *sufficient* condition for the existence of a Nash equilibrium, but not a *necessary* condition. For example, consider a simple bilinear problem with $f (x, y) = xy$. This has a unique first-order stationary point $(0, 0)$, which is a global Nash equilibrium. For this reason, we believe that Assumption 2 is natural, as it is analogous to assuming the existence of a minimum in convex minimization.

Reviewer comment (Evaluation):
The convergence of SEG-RR and SEG-FF is strictly worse than SGDA-RR in the strongly monotone setting. Could the authors explain why?

Author response:
6. **Convergence rates of SEG-RR and SEG-FF (W5):**

    It is indeed true that our convergence rates $\mathcal{O} ( \frac{1}{n K^{2-3\epsilon}} )$ for SEG-RR and SEG-FF are slightly slower than the known $\tilde{\mathcal{O}} ( \frac{1}{nK^2} )$ of SGDA-RR. We believe that it is an artifact of our attempt on establishing a unified analysis (Theorem 6.2), and it is not a serious limitation since $\epsilon$ is a number that we can choose arbitrarily small. Also, our main focus is to show the effectiveness of SEG-FFA, which attains a strictly improved rate over SGDA and other SEG variants on both strongly-monotone and monotone problems, with provable gaps (recall Theorems 4.2 and 5.6 in particular).

Reviewer comment (Theory):
Proposition 5.2 says the reason why anchoring is also required to match (13). Then what changes after anchoring is introduced? How does each term in (13) match? Its LHS is of order $\eta_1\eta_2/n^2$ while the RHS is of order $\eta^2$. Does this mean $\eta_1 = \eta n$ also holds? Isn't $\eta_2=2\eta_1$ still true? Moreover, $z_0-\eta n F(z_0 - \eta n F(z_0))$ in Proposition 5.3 is just one step of EG from $z_0$. Does it suggest that $N$ steps of SEG-FFA are equivalent to one step of EG, up to a small error, where $N$ can be multiple of $n$? Is this specifically for $N=2n$?

Author response:
7. **On the questions regarding Section 5 (W6):**

    Proposition 5.3 states that after introducing the anchoring step with $\theta =1$, choosing $\beta = \eta$ and $\alpha = \eta/2$ makes the LHS and RHS terms in (13) (which is now (15) in the revised manuscript) to match with "$\eta_1 = \eta_2 = n\eta$". This is not easy to verify because, in case of filp-flop sampling ($N = 2n$), the sum $\sum_{0 \leq i < j \leq N-1} D\boldsymbol{T}_j (\boldsymbol{z}_0) \boldsymbol{T}_i (\boldsymbol{z}_0)$ on the RHS contains both of the sums in the LHS of (13). So, we would like to refer the reviewer to Lemma C.4 for more exact calculations.

    Here, the reviewer is correct on observing that the overall update made in the FFA epoch (consisting of $2n$ steps and the anchoring) is equivalent to a single step of EG plus some small noise. This is in fact the key idea in many existing analysis of shuffling-based methods: aggregate the stochastic updates over an epoch and write them as a single deterministic update plus small noise. For SEG-FFA we have $N = 2n$ because one epoch contains $2n$ updates, but we chose this general notation $N$ because other methods may have different values of $N$; for example, we have $N = n$ for SEG-RR. We have further clarified this in our revised Section 5.

Reviewer comment (Experiments):
The paper should also include SGDA-US/RR with anchoring and SEG-US/RR with anchoring, at least in experiments, to show that both ingredients of the algorithms are necessary. What if anchoring is the key step that makes everything converge? I do not fully believe that FF is something that should be done in practice, given that its computation is twice as much as the standard method in every epoch.

Author response:
8. **Anchoring but without FF (W7):** 

    According to our analyses demonstrated in Section 5, adding anchoring on US/RR will not show a decisive improvement, because anchoring by itself cannot achieve second-order matching. Nevertheless, we performed an ablation study testing the effect of anchoring applied to SEG-US and SEG-RR; please see Appendix H. We have not considered applying anchoring to SGDA-US and SGDA-RR, since it is obvious that they diverge for a simple bilinear finite-sum problem when $f_1(x, y) = f_2(x, y) = \dots = f_n(x, y) = xy$.

    As for the computational burden caused by flip-flop shuffling, one can simply reduce the number of epochs by half. So, in our experiments, we make comparisons between RR and FF schemes with respect to the number of passes through the components.

Reviewer comment (Theory):
(Minor) Could the author comment on the dependence on the condition number when the operator is strongly monotone and compare it with existing works? It could be more clear if the author explicitly listed all the necessary assumptions required for each theorem. Are Assumptions 1(1'), 2, 3, and 4 necessary for every theorem? The paper should also clearly state that the benefit of shuffling-based methods over uniform sampling only holds when $K>1$, i.e., the multi-pass case.

Author response:
9. **Three minor comments (W8):**

    We agree that it is best to include the dependency on the condition number, but for the rebuttal period we had to focus on addressing other major issues. We will make this clear in our next revision.

    In the initial submission, some theorems did not explicitly include the assumptions used. As correctly guessed by the reviewer, Assumptions 1–4 are used in most of the theorems. We have modified the statements accordingly.

    We added "when $K$ is large enough" below Theorem 4.1.

Reviewer comment (Theory):
There are a few inconsistencies in the paper. In my opinion, some parts and claims are not totally clear based on prior works. For example, the authors claim that there is an example that SEG-US and SEG-RR diverge. However, the example that they select is a simple bilinear problem for which we know now that the SEG-US converges to the solution with a proper selection of step-sizes (see Hsieh et al. 2020). In particular, the authors claim that SEG-US diverges in expectation for any positive step-size selection. This comes in contradiction with existing known theoretical results. Why is that the case? Can the authors provide a detailed comparison with prior work?

Author response:
You are correct that the initial version was confusing, and we have now revised the paper (in footnote 2) so that it is clear that we focus on a *same-sample* version of the SEG (i.e., use the same component function for both extrapolation and update steps), which we believe is a more efficient and natural implementation of EG in without-replacement sampling scenarios. Hsieh et al. (2020) consider different independent samples for the two steps within one update of the SEG, so there is no contradiction between (Hsieh et al., 2020) and our results.

    A noteworthy remark regarding the comparison of our algorithm against (Hsieh et al., 2020) is that the existing paper shows convergence to an optimum for a convex-concave problem but does not have an explicit rate; in contrast, we prove convergence with a rate.

Reviewer comment (Theory):
In several parts of the paper, the authors claim that EG is a good approximation of PP. This was already disproved in Gorbunov et al. 2022b, where the authors emphasize a significant difference between EG and the Proximal Point method.

Author response:
The claim by Gorbunov et al. (2022b) on the “significant” difference between EG and PP is based on the observation that PP update, as an operator, is always nonexpansive, while EG update may be expansive. In contrast, we look at the two algorithms from a different angle; our motivation was that a method that differs from EG by at most $\mathcal{O}(\eta^3)$ would lead to convergence in convex-concave settings, where $\eta$ is the step size. This was based on the observation, in (Mokhtari et al., 2020), that the update equations of the convergent methods—namely EG and PP (and even OGDA)—differ from each other by at most $\mathcal{O}(\eta^3)$ per epoch. Importantly, the updates of divergent methods such as GDA and various SEGs without FFA differ from the convergent methods by at least $\mathcal{O}(\eta^2)$, leading to our conjecture that the $\mathcal{O}(\eta^2)$ error is the main cause of nonconvergence. We believe this concern arose because of our unclear writing of Section 5, and we put efforts on improving the presentation of that section.

Reviewer comment (Writing):
Section 5 had the potential to be very informative. In the first paragraph of this section, the authors mentioned that the section is where the underlying cause for non-convergence of SEG-RR is investigated, and the motivation of using anchoring is explained. However, the section is not clear, and several important parts are missing. The presentation of the "SECOND-ORDER MATCHING" part is not robust, which makes the understanding and importance of the results in section 5 unclear. For example, there is no proper definition of what second-order means in EG update rules, and the results are not positioned in the literature.

Author response:
By *second-order matching* we mean choosing proper step sizes, sampling scheme, and anchoring scheme so that our without-replacement SEG can *deterministically* match the update equation of a convergent algorithm (EG or PP) up to the $\mathcal{O}(\eta^2)$ terms (*i.e.*, *second-order* terms in the Taylor expansion). We added the term “Taylor expansion” and improved corresponding sentences in the revision to make the definition of second-order matching clear.

    We have also improved Section 5 to make our argument “$\mathcal{O}(\eta^3)$  for the approximation error turns out to be the key to the exact convergence to an optimum under the monotone setting” clearer. We provide a brief summary of Section 5 below for your understanding.

    We show in Section 5.1 that the second-order matching is not achievable for both SEG-RR and SEG-FF. SEG-RR suffers a difference of at least $\mathcal{O}(\eta^2)$ from the convergent methods. For SEG-FF, as we show in Proposition 5.2, the best we can do is to choose specific step sizes and get an $\mathcal{O}(\eta_1^3)$ approximation of an instance of EG+ with $\eta_2 = 2\eta_1$. However, unfortunately, EG+ with $\eta_2 = 2\eta_1$ fails to converge already on bilinear problems. On the contrary, as we show in Proposition 5.3, SEG-FFA can approximate the standard EG with an error of $\mathcal{O}(\eta^3)$. In Section 5.2, we show that this second-order matching property of SEG-FFA indeed enables us to establish improved convergence rates.

Reviewer comment (Theory):
On theoretical statements: The authors wrote the following in their work: "$O(\eta^3)$ for the approximation error turns out to be the key to the exact convergence to an optimum under the monotone setting". This sentence, unfortunately, was not adequately explained in detail in the main paper. Why is this statement necessary?

Author response:
By *second-order matching* we mean choosing proper step sizes, sampling scheme, and anchoring scheme so that our without-replacement SEG can *deterministically* match the update equation of a convergent algorithm (EG or PP) up to the $\mathcal{O}(\eta^2)$ terms (*i.e.*, *second-order* terms in the Taylor expansion). We added the term “Taylor expansion” and improved corresponding sentences in the revision to make the definition of second-order matching clear.

    We have also improved Section 5 to make our argument “$\mathcal{O}(\eta^3)$  for the approximation error turns out to be the key to the exact convergence to an optimum under the monotone setting” clearer. We provide a brief summary of Section 5 below for your understanding.

    We show in Section 5.1 that the second-order matching is not achievable for both SEG-RR and SEG-FF. SEG-RR suffers a difference of at least $\mathcal{O}(\eta^2)$ from the convergent methods. For SEG-FF, as we show in Proposition 5.2, the best we can do is to choose specific step sizes and get an $\mathcal{O}(\eta_1^3)$ approximation of an instance of EG+ with $\eta_2 = 2\eta_1$. However, unfortunately, EG+ with $\eta_2 = 2\eta_1$ fails to converge already on bilinear problems. On the contrary, as we show in Proposition 5.3, SEG-FFA can approximate the standard EG with an error of $\mathcal{O}(\eta^3)$. In Section 5.2, we show that this second-order matching property of SEG-FFA indeed enables us to establish improved convergence rates.

Reviewer comment (Theory):
On theory: Statement of the theorems do not include all necessary assumption. I imagine that Assumptions 3 and 4 need to hold for every theorem, but this is not precisely stated.

Author response:
In the initial submission, statements of some theorems did not explicitly include the assumptions used. As correctly guessed by the reviewer, Assumptions 3 and 4 are used in most of the convergence proofs. We have modified the statements accordingly.

Reviewer comment (Theory):
Important comment on Assumptions: The authors claim that Assumptions 3 and 4 are reasonable. They pointed out that the work of Golowich et al., 2020 uses Assumption 3(ii) to argue that this is a reasonable assumption. However, Assumption 3(ii) is never used in any other paper on analysis for SEG (a first-order method). For example, Gorbunov et al., 2022b remove Assumption 3(ii) and still have nice convergence guarantees for SEG-US. In addition, Assumption 4 is also not reasonable to have. Of course, Assumption 4 is more relaxed than bounded variance, but as explained in Gorbunov et al., 2022a this is not a necessary condition to show convergence. SEG-US does not require any bound on the stochastic gradient to guarantee convergence (see Gorbunov et al., 2022a).

Author response:
For our comments for Assumption 3, we invite the reviewer to check the general comment.

    In regard of Assumption 4, the analyses of Gorbunov et al. (2022a) does not require any bound on the variance, but they do require the stochastic gradients to be sufficiently regulated in the following sense: assuming for each $i$ that the operator *$\boldsymbol{F}_i$* is $\mu_i$-strongly monotone (allowing $\mu_i < 0)$, it should hold that $\sum_{i=1}^{n} \mu_i ({\boldsymbol{1}_{{\mu_i \geq 0}}} + 4 \cdot \boldsymbol{1}_{ \mu_i  \textless 0 }) > 0$ where $\boldsymbol{1}$ denotes the indicator function (see eq. (9) in their paper). Thus, this is close to assuming strong monotonicity of the majority of component functions $\boldsymbol{F}_i$, and we believe that this already should provide sufficient level of “contraction” to show convergence without bounded variance assumptions. In stark contrast, we do *not* make any (strong) monotonicity assumptions on $\boldsymbol{F}_i$; our monotonicity assumptions only apply to their mean $\boldsymbol{F}$. Thus, we believe that our paper and (Gorbunov et al., 2022a) are not directly comparable because of different regularity conditions on the stochastic gradients. We now briefly mention this in footnote 3.

Reviewer comment (Experiments):
On Experiments: I believe algorithms in the experiments are compared with wrong step-size selection, which led to unfair comparison. That is, even if in Appendix A, all pseudocodes use $\alpha_t$ and $\beta_t$ to denote the extrapolation and update steps, respectively, in the experiments, $\eta_k = \alpha_k = \beta_k$ is used for all methods. However, for the bilinear example, it is known that using the same step-size leads to no convergence of SEG-US. See Hsieh et al., 2020.

Author response:
We hope now that your concern on our choice of same step size $\alpha_k = \beta_k$ in the experiment has been somewhat alleviated, considering that we focus on the same-sample case. Nevertheless, we learned that a numerical comparison with a convergent independent-sample SEG in (Hsieh et al., 2020) is necessary, so we added an experiment in which we discovered that SEG-FFA outperforms the method in (Hsieh et al., 2020). We also conducted an experiment on strongly monotone problems in which SEG-FFA again outpaced all other baseline methods.

Reviewer comment (Experiments):
The paper needs further experiments for completeness. For example, what is the practical benefit of the proposed method compared to other RR algorithms for solving strongly monotone problems?

Author response:
We hope now that your concern on our choice of same step size $\alpha_k = \beta_k$ in the experiment has been somewhat alleviated, considering that we focus on the same-sample case. Nevertheless, we learned that a numerical comparison with a convergent independent-sample SEG in (Hsieh et al., 2020) is necessary, so we added an experiment in which we discovered that SEG-FFA outperforms the method in (Hsieh et al., 2020). We also conducted an experiment on strongly monotone problems in which SEG-FFA again outpaced all other baseline methods.