Paper: NAG-GS: Semi-Implicit, Accelerated and Robust Stochastic Optimizer (ICLR-2024)

Reviewer comment (Writing):
some of the writings are sloppy. Example: page 4, line 2, two "either" appeared

Author response:
In addition, we have carefully proofread the manuscript and fixed typos.

Reviewer comment (Theory):
The theoretical analysis is bounded to the quadratic case. This is not mentioned accurately in the contribution. Specifically, the second main contribution is: **We analyze the properties of the proposed method both theoretically and empirically;**.

Author response:
Thank you for your comment, we have clarified our theoretical contribution for the quadratic case in the main text.

Reviewer comment (Novelty):
The introduction is not really an introduction of this work. By just reading the introduction, it is not possible to get an accurate idea of what differences this work has with any other work in "stochastic optimization algorithms".

Author response:
We place in the Introduction section the main contributions and the organization of the manuscript that provides step-by-step plan on how we analyze the proposed dmethod theoretically and empirically.

Reviewer comment (Novelty):
Gauss-Seidel splitting used here is not a novel idea in discretizing ODEs for acceleration as it was previously discussed in [Luo & Chen (2021)].

Author response:
Thank you for your valuable comment. It's acknowledged that deterministic Gauss-Seidel splitting in the context of accelerating SDEs was previously discussed by Luo & Chen (2021). In our work, we introduce novel contributions, particularly in the quadratic case:
Extension of Deterministic Results: We provide a detailed analysis of the optimal step size computation, which exceeds the limit proposed by Luo and Chen (refer to Lemma 1 in our paper). Additionally, we identify a scenario (μ > L/2) where we can freely choose the step size, offering a new perspective.

Stochastic Setting under Gaussian Noise: We extend our approach to the stochastic setting under a Gaussian assumption for the gradient noise, introducing a family of accelerated SDEs (Equation 11). We prove the asymptotic convergence of NAG-GS with the same optimal step size as in the deterministic case. We provide optimal choice for the step size leading to the highest possible contraction rate when aiming at minimizing a strongly convex quadratic form. Furthermore, our numerical experiments demonstrate the expected robustness concerning large step sizes across a diverse range of problems.

Additionally, in Appendix 1.2, we explore a fully implicit version. Our findings reveal that, in the stochastic case for quadratic minimization, we can select any step size, mitigating the noise effect. This implies that the spreading of particles can be arbitrarily close to the arg min, showcasing a notable result.

References:
Alvarez, F., Attouch, H. An Inertial Proximal Method for Maximal Monotone Operators via Discretization of a Nonlinear Oscillator with Damping. Set-Valued Analysis 9, 3–11 (2001).

Reviewer comment (Novelty):
The discussion of NAG-GS with other similar methods is insufficient. For example, is NAG-GS faster than Polyak's momentum method for solving quadratic objectives? How does it compare with other variants of NAG, such as Triple momentum method [1, 2], and ITEM [3]? It is not clear what is the key benefit of NAG-GS in its original setting.

Author response:
1. **Comparison with Other Methods:**
   - We recognize the importance of comparing NAG-GS with existing methods like Polyak's momentum method and Triple momentum method. To this end, We have checked the mentioned references and updated Section 4. Both ITEM and Triple momentum do not study the robustness w.r.t, the large step size and consider only deterministic setup. The incorporting these methods in the ODE-inspired setup is the interesting future work. This comparison highlights the unique advantages of NAG-GS, particularly in its original setting.
We added additional figures 15 and 16 to the section "Numerical tests for quadratic case" in supplementary materials, which illustrates the fastest asymptotic convergence rate of the NAG-GS for the considered quadratic problems. The figures are also available here [15](https://freeimage.host/i/JoRuOKv) and [16](https://freeimage.host/i/JoRuwiJ).
To illustrate the comparison the NAG-GS with Polyak’s momentum metod we also have updated Figure 1.

Reviewer comment (Evaluation):
The improvement in the neural network experiments seems marginal. No deviation statistics is provided in the empirical results.

Author response:
2. **Experimental Improvement and Deviation Statistics:**
   - We acknowledge that the improvement in neural network experiments appeared marginal and lacked deviation statistics. To address this, we have now included a set of experiments of NAG-GS and AdamW on the ViT training problem on food-101 dataset with the best hyperparameters found with the hyperparameter search on the portion of data. The number of experiments per method is 5. Mean values and standard errors of the evaluation loss are presented. . The results are included in the section Additional experiments with ViT in Appendix. [Link for eval accuracy graph](https://freeimage.host/i/JoRAykN) [Link for eval loss graph](https://freeimage.host/i/JoRAp7p)

Reviewer comment (Presentation):
A minor point: As one of the key features of NAG-GS, the derived optimal learning rate should be mentioned and discussed in the main text.

Author response:
3. **Optimal Learning Rate Discussion:**
   - Based on your suggestion, we have included Remark 2 in the main text, which explicitly discusses the derived optimal learning rate for NAG-GS. This addition emphasizes one of the key features of NAG-GS and its relevance in the context of gradient methods.

Reviewer comment (Theory):
The analysis for general f (replacing Ax with grad f for the method) is non-trivial, since the "lifted" spectrum needs to be bounded effectively.

Author response:
We acknowledge that extending analysis to more general functions introduces additional complexities and is the question for the further research.

Reviewer comment (Theory):
The algorithm requires prior knowledge of μ, which is very limiting.

Author response:
Regarding the requirement of prior knowledge of $\mu$, we approached this by aligning the weight decay parameter used in standard optimizers (like SGD, Adam, etc.) with the strongly convex constant $\mu$. Consequently, in our experiments, we set the constant for weight decay in competitor algorithms and for the $\mu$ in NAG-GS correspondingly.

Reviewer comment (Writing):
There are some typos that need to be fixed.

Author response:
**Typographical Corrections**: We corrected the typo you pointed out and carefully reviewed the manuscript for any other errors.

Reviewer comment (Writing):
Please explain the definition of A-stable in the Background section, as readers unfamiliar with it would benefit from an explanation after you have already described the discretization process.

Author response:
**Definition of A-stability**: We added a high-level definition and reference in the main text and detailed insights in Appendix 1.3.

Reviewer comment (Writing):
Clarify what \(\mathcal{R}(\lambda)\) represents below equation (8).

Author response:
**Clarification of $\mathcal{R}(\lambda)$**: We now specify that $\mathcal{R}(\lambda)$ refers to the real part of $\lambda$.

Reviewer comment (Writing):
The sentence “However, this requires to either solve a linear system either.” should be corrected to “However, this requires to either solve a linear system or.”

Author response:
**Typographical Corrections**: We corrected the typo you pointed out and carefully reviewed the manuscript for any other errors.

Reviewer comment (Experiments):
Please mention what the baseline SGD-MW is before using it in Table 1. Is there a reason Adam or a variant of it was not considered as a baseline for ResNet20?

Author response:
**SGD-MW**: We clarified that this stands for Stochastic Gradient Descent with Momentum and Weight decay, and added this explanation before its first use in the text.  
**Baseline Algorithms**: The choice of SGD-MW as a baseline for ResNet-20 was due to its comparable memory footprint to our NAG-GS. In contrast, Adam requires two state vectors, instead of one in case of NAG-GS ad SGD-MW.

Reviewer comment (Experiments):
What is the accelerated gradient descent baseline in Figure 1?

Author response:
Additionally, the accelerated gradient descent in Figure 1 references the Nesterov accelerated method variant presented in Su et al. (2014).

Reviewer comment (Theory):
Can the authors provide directions for future work, such as convergence analysis?

Author response:
Regarding future work, we have expanded Section 5 of our manuscript to outline potential directions. These include extending our convergence analysis to broader classes of functions and investigating the applicability of our approach to different machine learning models and problem domains, especially for gradient noise with unbounded variance.