Paper: The Effects of Overparameterization on Sharpness-aware Minimization: An Empirical and Theoretical Analysis (ICLR-2024)

Reviewer comment (Novelty):
First, the theoretical results do not seem to be that new or involved, and the proofs are mostly standard. That can be fine if new insights are uncovered in the paper, but the overall message of the paper is not surprising, and the flatness of SAM solutions is well-studied.

Author response:
**New insights?** 

> First, the theoretical results do not seem to be that new or involved, and the proofs are mostly standard. That can be fine if new insights are uncovered in the paper, but the overall message of the paper is not surprising.

We would like to respectfully disagree with the reviewer’s assessment for the following reasons.

First, our theoretical results in Sections 3 and 4 have been neither previously explored nor presented explicitly in prior works. In Section 3, we show that stochastic SAM can achieve a linear convergence rate for overparameterized models, whereas no previous work has focused on the effect of overparameterization, all remaining sublinear convergence rates [1-3]. Notably, this is followed by our experiments showing that these improved convergence properties in theory indeed translate to realistic and practical settings, which has not been shown explicitly in any fashion. In Section 4, we show that SAM finds a flatter solution with bounded eigenvalues of Hessian moments through linear stability analysis. Although the flatness of the SAM solution has been discussed in [10] as pointed out by the reviewer, our results further characterize the eigenvalue divergence of the Hessian moments, which is again substantiated by empirical evaluations.

Also, we would like to stress the significance of our findings in experiments. In Section 6, we discover a strong trend that the generalization benefit of SAM tends to increase with more parameters, and also, that the optimal $\rho$ tends to increase as well as a result of an extensive hyperparameter search. In Section 7, we demonstrate for the first time that sparsity can be a remedy to the computational overhead induced by overparameterization without sacrificing the improved generalization benefit from SAM. We believe that all of these empirical findings can render non-trivial practical values, and can be considered a valuable initial exploration in the literature, as recognized by the reviewer `jLHw` (“*the experiments relating scaling of models to usefulness of SAM are, to my knowledge, novel, and I find them interesting*”) and the reviewer `9tdL` (“*an extensive numerical section verifies the practical utility of the theoretical results*”). Additionally, our results can contribute to new perspectives on SAM's potential in the current landscape of large-scale and efficient training [4, 5].

We sincerely hope that our theoretical results are considered as one of the many contributions in this work, and we provide non-trivial, if not significant, new empirical findings that can provide valuable insights into understanding SAM and render positive avenues for future work.

Reviewer comment (Novelty):
How does the results of the paper compare to linearization study of https://arxiv.org/pdf/2302.09693.pdf?

Author response:
**Comparison to [10]**

>How does the results of the paper compare to linearization study of [10]?

We would like to first acknowledge the reviewer for pointing at [10]; it appears to be very related to our work (Section 4 in particular), and admittedly it is our mistake. We believe it is due to the unfortunate concurrency, but we will certainly update our manuscript to properly cite this work.

First, we find that Section 3 of [10] conducts a linear stability analysis for mini-batch SAM and micro-batch SAM, referred to as SAM and mSAM respectively. As far as we understand, however, this is motivated to show that the proposed micro-batch SAM converges to flatter minima compared to mini-batch SAM.

Clearly, Section 4 of our paper takes a similar approach to analyzing the linear stability of stochastic SAM. However, the main difference lies in revealing that the maximum eigenvalues of the 2nd-4th moments of the stochastic Hessian matrix are bounded. We corroborated these results with empirical evaluations and found that the minima of SAM indeed have more uniformly distributed Hessian moments (see Figure 2c) as predicted in the aforementioned bounds. We would like to note that this is recognized as a strength of our work by the reviewer `jLHw` ("*in particular Equation 7 breaking down the stability condition into necessary conditions on different moments provides a lot of intuition about how SAM might shape network training*").