Paper: ADOPT: Modified Adam Can Converge with the Optimal Rate with Any Hyperparameters (ICLR-2024)

Reviewer comment (Theory):
For relaxing the condition of gradient assumption, it is misleading to claim that "the convergence is established without the bounded stochastic gradient assumption". In fact, in the paper, the authors replace the uniformly bounded gradients with bounded second-order moments, which still bounds the gradients of the expected function. Further, [1] shows the convergence of Adam without the assumption of gradients uniformly bounded or second-order moments bounded.

Author response:
**Our assumptions seem stronger than existing analyses on Adam**

As you mentioned, our Assumption 4 requires the true gradient $\nabla f$ to be bounded although the stochastic gradient $g$ does not need to be bounded.
Hence, the difference between Assumptions 4 and 5 is that Assumptions 4 does not assume the gradient noise $\|| g - \nabla f \||$ to be bounded; instead, it assumes the variance of the stochastic gradient, i.e., $\mathbb{E} [ \|| g - \nabla f \||^2 ]$, to be bounded, which is weaker than the bounded noise assumption.

Although Li et al. (2023) do not assume the true gradient $\nabla f$ to be bounded, they instead assume that the gradient noise $\|| g - \nabla f \||$ is bounded, which is not assumed in our analysis.
They also provide a result where the bounded noise assumption is relaxed to a sub-Gaussian noise norm assumption, but the convergence rate gets a little worse in that case.

Therefore, our assumptions are not quite stronger in total compared to existing analyses on Adam (e.g., Li et al. (2023)), and our result of ADOPT's optimal convergence without depending hyperparameters is much stronger than existing results on Adam, which require problem-dependent tuning of hyperparameters.
We also discuss the relation to existing analyses in detail in the general response and Appendix A in the revised paper, so please also refer to them.

Reviewer comment (Theory):
In practice, people will use random shuffle instead of random sampling. However, the convergence results only hold for random sampling. In fact, there is a counter-example for the proposed algorithm that can not converge when we use a random shuffle. Let $\beta1 = \beta2 = 0$, and $f_1(x) = 1.9x$ and $f_2(x) = f_3(x) = -x$. With the constraint that $x \in [-1,1]$ the optimal solution should be 1 instead of -1. Thus, although the algorithm can converge in the random sampling setting, it does not deal the case in the practical case.

Author response:
**Counter example in the case of random shuffling**

Thank you for pointing it out.
We did not recognize that there is a counter example where ADOPT cannot converge when applying without-replacement sampling (a.k.a. random shuffling) instead of with-replacement sampling.
Since random shuffling violates Assumption 2, it is out of scope of our theory; hence the existence of such a counter example does not contradict to our theoretical findings.
However, as you mentioned, practitioners often use random shuffling, so we have come to think that the existence of a counter example in the random shuffling setting should be mentioned in the paper.
Therefore, we have added a discussion on it in Appendix B in the revised version.

We would like to emphasize again that the non-convergent issue in such a counter example can be easily avoided by using with-replacement sampling.
Moreover, the difference between with- and without-replacement sampling becomes negligible when the dataset size is large enough, so it does not affect the practical performance very much.
In fact, our experiments except for the toy example are performed using without-replacement sampling, but divergent behaviors are not observed.

Reviewer comment (Theory):
The theoretical contribution is not very significant. Even though the bounded gradient assumption is relaxed, the paper requires a bound on the gradient norm squared. For example, the bounded gradient assumption is also not required in the recent work by Yushun Zhang et al. [1].

Author response:
**On the assumptions of our analysis compared to existing ones**

As you mentioned, our Assumption 4 is stronger than the growth condition adopted in Zhang et al. (2022).
Zhang et al. (2022) instead assume that the stochastic gradient $g$ is Lipschitz continuous, which is a stronger assumption than ours, in which only the true gradient $\nabla f$ is assumed to be Lipschitz.
Moreover, Zhang et al. (2022) focus on the finite-sum problems, whereas we deal with general nonconvex problems.
Therefore, our assumptions are not quite stronger in total compared to existing analyses on Adam, and our result of ADOPT's optimal convergence without depending hyperparameters is much stronger than existing results on Adam, which require problem-dependent tuning of hyperparameters.
We also discuss the relation to existing analyses in detail in the general response and Appendix A in the revised paper, so please also refer to them.

Reviewer comment (Novelty):
The proof technique seems to be quite similar to that of Zhou et al. [2].

Author response:
**Proof technique seems to be similar to that of Zhou et al. (2019)**

Zhou et al. (2019) only consider a single convex optimization problem in their analysis, but our analysis deals with general smooth nonconvex optimization problems; hence the scope of our analysis is much broader than theirs.
Moreover, techniques used for the proofs are also very different, so we think that our theoretical contribution is not minor compared to Zhou et al. (2019).
About the theoretical contribution, we have added a detailed discussion in the general response and Appendix A in the revised paper, so please also refer to it.

Reviewer comment (Evaluation):
The experimental gain seems to be marginal. For the results in Table 1 and Table 2, it is better to add standard deviations to clearly contrast the results of different algorithms.

Author response:
**Experimental gain seems marginal and error bars are missing in Tables 1 and 2**

In the revised version, we have added error bars in Tables 1 and 2, which may help to show that the experimental gain is not so marginal.
In addition, to clearly show the superiority of ADOPT over Adam (and AMSGrad), we have revised the toy experiment.
Please see the general response for a detailed experimental settings.
This experiment clearly shows that Adam tends to fail to converge when the gradient noise is large, even if $\beta_2$ is chosen to be close to 1 (e.g., 0.999), whereas ADOPT can always converge to the solution without careful tuning of hyperparameters.

Reviewer comment (Theory):
The major issue is in the theoretical analysis. I feel that both (5) and (14) are vacuous. From assumption 4, we know $||\nabla f||^2\le G^2$. However, (14) basically says
$$
\min_{t=1,\cdots,T}||\nabla f(\theta_t)||^2\le O\left(G^2\frac{\alpha L}{\epsilon}\right)
$$
In practice, the learning rate $\alpha$ is not too small, $\epsilon$ has the order of 1e-8, the Lipschitz constant $L$ is very large, so $\frac{\alpha L}{\epsilon}$ is often far greater than $1$. So, the upper bound in (14) is even larger than $G^2$, which is correct by assumption. Therefore, the theoretical analysis is not meaningful.

Author response:
**Convergence bounds are too loose**

We agree that the bounds in the original submission were too loose to be meaningful, so we have revised the theorems to derive tighter bounds.
We believe that this will address your main concern.

Reviewer comment (Experiments):
As ADOPT contains two changes compared to Adam, the authors should consider adding some ablation studies in section 5 to see which change has the most significant impact on the performance. Such studies are very informative for the future improvement of adaptive stepsize algorithms.

Author response:
**Ablation study on each algorithmic modification**

Thank you for an insightful suggestion. 
We have added an ablation study on algorithmic modifications from Adam to ADOPT in Section 5.
As can be seen in Figure 2 in the revised paper, ADOPT fails to converge if either of the two modifications (i.e., decorrelation between $g_t$ and $v$, and change of order of momentum and scaling operation) is not applied.
This result aligns with the theoretical finding, and helps it to be more convincing.

Reviewer comment (Evaluation):
In Figure 1 and 3, ADOPT has larger variances. Does this mean ADOPT is less robust than Adam?

Author response:
**Large variance of empirical results**

In Figure 1 of the original submission, ADOPT seems to have high variance when $\sigma = 10$ compared to Adam and AMSGrad, but we do not think that it is not because ADOPT is less robust.
First of all, Adam does not even converge to the correct solution, so the variance comparison is less meaningful.
As for AMSGrad, its variance seems lower than ADOPT, but this is because $v_t$ of AMSGrad is non-decreasing in terms of $t$, so the effective learning rate tends to be lower than ADOPT.
If the learning rate of ADOPT is set to a lower value, the variance of ADOPT will also decrease.

In addition, ADOPT's high variance in Figure 3 of the original submission is just due to the nature of deep reinforcement learning (RL).
In deep RL, the variance tends to be larger as the agent obtains high reward, so this phenomenon is quite natural.

Moreover, in the revision during the rebuttal period, we have added error bars in Tables 1 and 2, and we observe that the variance of ADOPT is not higher than baselines (e.g., AdamW).

Reviewer comment (Experiments):
Did authors consider adding the bias correction step in the algorithm?

Author response:
**On the bias correction technique**

In fact, we have tried to add the bias correction technique of Adam to our ADOPT, and it empirically worked well.
However, it did not bring performance improvement (also did not harm the performance) in practice, so we exclude it from our implementation for simplicity.
Our hypothesis on why the bias correction is less meaningful for ADOPT is that $v_t$ is initialized to $1$ for ADOPT to prevent it from being too small at the first parameter update, so the effect of bias correction is limited compared to Adam.
Moreover, as for the momentum $m_t$, initialization with $0$ works like a warm-up of the learning rate, so it may have a positive effect for stable training especially in the early phase of optimization.
Although there might be a better way to introduce a technique like the bias correction for ADOPT, we leave it for future work.
We have added a discussion on the bias correction in Appendix C in the revised version.