From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper proposes a quantity R that is related to a condition number of a matrix, and showed theoretically and empirically that Adam tends to converge a solution with small R (uniform eigenvalues) while vanilla SGD tends to converges to that with large R (non-uniform eigenvalues). The authors discuss that this tendency has correlation with the convergence speed and statistical performance. 

**Strength:** The findings regarding the evolution of R is interesting (especially, the theoretical justification on the linear 2 layer neural network is novel). The bias with respect to R induced by the choice of optimizer would reveal some interesting property of deep learning. The theoretical result is solid and the writing is good.    
**Weakness:** Unfortunately, this tendency is not necessarily universal, that is, an opposite phenomenon occurs depending on the setting. This is pointed out by all reviewers including those who are positive on this paper. For example, in the image dataset, we see that Adam does not have smaller R than SGD which is opposite to the theoretical result although the two layer linear network in the theoretical analysis looks more suited to the image tasks than the language tasks. Moreover, theoretical connection to the convergence speed and statistical performance is not strong. It is not presented in a convincing way how the quantity R around the solution affects properties of the resultant network and the training dynamics.    

In summary, although looking at R is an interesting viewpoint, its universality and its connection to practical performances are rather weak. If there is more consistent finding, the paper would be much stronger.  
For these reasons, the AC cannot recommend acceptance of this paper unfortunately.  

justification_for_why_not_higher_score: Although looking at the quantity R is interesting, its universality and connection to properties of the network training are not clearly presented. More investigation would be required to obtain more clear characterization of the role of R.  
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper2142/Reviewer_ZKz6
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The manuscript introduces a new statistic metric to analyze the local trajectory of SGD methods like SGDm and various adaptive methods.
The paper argues the positive correlation between the uniformity of diagonal Hessian and fast convergence, and that adaptive methods such as Adam bias the trajectories towards regions where the metric is small (corresponding to a faster training/convergence).
strength_and_weaknesses: ## Strength
* The manuscript is generally well-written and well-structured: the main message is clear, with some well-supported empirical evidence.
* The studied problem and the insights therein are quite crucial to the optimization community. 

## Weaknesses
The reviewer is generally happy with the manuscript. However, the authors must address the following questions/comments
1. Some arguments in the manuscript need more justifications. For example, 
    * The authors argue to use the diagonal entries of Hessian, instead of the corresponding eigenvalues, to measure the local curvature; however, no empirical/theoretical justifications can be found. Even though in Appendix E, the authors demonstrate that loss Hessian approaches diagonal during training for Adam and SGDm, it is insufficient to support the arguments like "we believe the diagonal of Hessian is more relevant than the spectrum" and diagonal entries are better.
    * If the reviewer remembers the related work correctly, some existing studies consider using $\lambda_{max} / \lambda_{10}$ (where $\lambda_{10}$ is the 10-th sorted eigenvalue of Hessian, and 10 can be replaced by other values?). In order to justify the novelty of the proposed metric, the manuscript is required to compare these metrics in the same experimental setting, otherwise, it is hard to justify the significance of the submission.
2. The generality of the proposed metric needs to be evaluated on more optimizers, e.g., SGD, other more recent adaptive methods, or even SAM and look-ahead.
3. The explanation for the inconsistency behavior on the first several layers and the latter layers is expected.
4. The manuscript mainly studies the training behaviors of different optimizers. As Hessian is also closely related to the generalization performance, it is necessary to validate the proposed metric on both training and (in-distribution) test data, through some comments/remarks.


clarity,_quality,_novelty_and_reproducibility: check the comments above.
summary_of_the_review: The main concern of the review lies in (1) the missing experiments to justify the novelty and significance of the proposed metric and observations, and (2) missing justifications on some arguments/statments.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2142/Reviewer_m6my
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper tries to explain why the adaptive methods converge faster than (S)GD.

 Specifically, this paper first defines a measure called $R_{med}^{OPT}$ which can be viewed as a stable version of the condition number. This paper then empirically observes that over tasks where Adam achieves empirical success, Adam and SGD follow quite different training trajectories -- the $R_{med}^{OPT}$ of Adam is significantly smaller than that of SGD.

The experiments in this paper show: (1). With the same loss, the parameter provided by Adam has different underlying geometry from that provided by SGD. Switching to SGDM respectively will lead to different convergence speeds. (2). With the same loss, Adam has smaller $R_{med}^{OPT}$ than SGD. (3). With the same iteration number, Adam has smaller $R_{med}^{OPT}$ than SGD. (4). Switching SGD to Adam leads to smaller $R_{med}^{OPT}$ compared to SGD which is not switched. (5). Adding small regularization, the forementioned observation continues to hold. But using large regularization, the observation reverses, while Adam still converges faster than SGD. (6). Over image tasks where Adam does not have an advantage in terms of convergence over SGD, the $R_{med}^{OPT}$ of Adam is no smaller than SGD.

Theoretically, this paper proves that over a two-layer linear network, $R_{med}^{OPT}$ of large-batch SGD is larger than that of Adam when the input dimension is large.


strength_and_weaknesses: Strengths:
1. To the best of my knowledge, this is the first work explaining the empirical advantage of adaptive optimizers over SGD from the perspective that the trajectories of Adam and SGD have different geometric properties. I find this perspective interesting.

2. This paper conducts extensive experiments to explore the training trajectory of Adam and SGD. The experiment, which compares the $R_{med}^{OPT}$ between Adam and SGD while controlling the loss, rules out the possibility that Adam and SGD follow the same trajectory but a smaller loss leads to smaller  $R_{med}^{OPT}$.

3. The theoretical result is solid.

Weakness:
1. (Please also regard this as a question) Theorem 1 cannot explain why the $R_{med}^{OPT}$ of Adam is no smaller than SGD over image tasks. Based on the experiments, I expect a theory showing that the relationship between the $R_{med}^{OPT}$ of Adam and SGD changes with tasks. To me, a two-layer linear neural network looks more like the neural networks used in image tasks than those used in language tasks (e.g., the four-layer network used for MNIST classification).

2. (Please also regard this as a question) Why does smaller $R_{med}^{OPT}$ (or condition number) lead to faster convergence for deep neural networks? To the best of my knowledge, such a conclusion has only been derived for strongly convex objectives. 

3. I would expect a brief proof idea after Theorem 1 given the proof in the appendix is quite lengthy.
clarity,_quality,_novelty_and_reproducibility: About clarity: Overall, the writing of this paper is good, and it is not hard to get the main message of this paper. Experiment settings are clearly explained. However, I suggest that the authors can put Section 3.3 into Section 5. I would expect the main theoretical result to be directly after the notations.


About novelty: To the best of my knowledge, the perspective provided by this paper is novel.
summary_of_the_review: Overall, I think that this paper provides a new perspective on Adam's empirical advantage in terms of convergence. Such a perspective may be of interest to the optimization community. Still, the paper can be further polished. Please refer to the questions in the "Strength And Weaknesses" part for details.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2142/Reviewer_djbz
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper studies the impact of adaptive optimization on local deep network geometry. The main discovery is that, adaptive
methods such as Adam bias the trajectories towards regions where the diagonal elements of Hessian are more uniform (compare to SGD type algorithms), and moreover such uniform diagonal Hessian tends to imply faster convergence. The experiments in this paper mostly use transformer architectures and language modeling datasets. In addition, the author also provides a theoretical justification of this phenomenon, by considering linear 2-layer neural networks with Gaussian data.
strength_and_weaknesses: Strength:

-This paper's main discovery is supported by large-scale experiments on big models with real data.

-This topic is important for understanding the success and convergence speed of deep learning optimization algorithms. 

Weakness/Questions/Comments:

-The paper defines the $R$ statistics based on the diagonal elements of the Hessian matrix computed at each iteration. While this connects to the adaptive algorithms and is easy to compute, one may wonder how different is this statistic from that defined via singular values of the Hessian. If they are very different, does that imply that the convergence of adaptive algorithms is not very much correlated with the spectrum of Hessian?

-In most of the language model experiments, the results show that the Adam trajectory tends to have Hessians with more uniform diagonal elements, when compared to SGD with momentum. But this does not necessarily hold for the first several layers. An interesting question is if the comparison fails to hold on shallow models (e.g., shallow MLP on MNIST). 

-The paper also presents a result on ResNet with CIFAR-10, which shows a different picture, i.e., SGD-M trajectory is more uniform than Adam's and yields faster convergence. Since most other experiments are focused on language models, I am wondering if the main claim of this paper applies to only language model/tasks. Put in another way, will the claim be consistent in standard image classification tasks? 
clarity,_quality,_novelty_and_reproducibility: Good
summary_of_the_review: See above
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold