Paper: Revisiting High-Resolution ODEs for Faster Convergence Rates (ICLR-2024)

Reviewer comment (Novelty):
This work follows the line of research on understanding accelerated methods via (high-resolution) ODE. Given the vast literature on this topic, I am afraid the contribution of this work is not significant enough. Although generalization and unification are developed, the results derived here are expected and the techniques are quite standard.

Author response:
**There is a vast literature**: The *vast literature* on this subject justifies the relevance of our work. We believe that our contribution is both significant and timely, as it unifies various approaches within this extensive body of literature, explaining the connections between them, all while achieving improved convergence rates. 

**Rates are expected**: We respectfully disagree. Please observe that proving these rates through a continuous-time analysis is a non-trivial task, and it was not clear whether it was possible. It is important to note that the rates presented in the earlier works [1,2] are known to be suboptimal in discrete and continuous-time settings, whereas in many cases we achieve the optimal rates. To further highlight this, we have enhanced the discussion about comparisons with prior work in Section 5 with Table 2 in our revision to compare our rates with existing results.

Reviewer comment (Novelty):
The theoretical improvements are kind of minor to me, for example, improving the constant from $1/2$ to $2/3$. The theoretical understanding based on this new high-resolution ODE does not provide any new insight on acceleration. Neither does it lead to any novel algorithms with more attractive practical performance.

Author response:
**Improvements are in the constants**: We agree that our rates are linear, just as in the prior work. However, please note that our improvements are in the *exponents* and not the *constants*. This forms a significant enhancement in comparison with the prior work. 
For example, Theorem 4.1 proves a convergence rate of order $C_{GM}(1-\sqrt{ 1/\kappa})^k)$, and the best previous result in this setting was $5L||x_0-x^*||^2_2(1+\sqrt{ 1/\kappa}/9)^{-k}$. Suppose $L=10,\mu=1,\kappa=10$ and $k=50$, then $C_{GM}(1-\sqrt{ 1/\kappa})^k \approx 2.35\times 10^{-7}\times ||x_0-x^*||_2^2$ while $5L||x_0-x^*||_2^2((1+\sqrt{ 1/\kappa}/9)^{-k}) \approx 8.8939||x_0-x^*||_2^2$.
Similarly, for QHM, we prove a rate of order $O((1-\sqrt{\frac{1}{3\kappa}})^k)$ while the previous best rate was $O((1+\frac{1}{40\sqrt{\kappa}})^{-k})$. If $\kappa=10$ and $k=50$, then $(1-\sqrt{\frac{1}{3\kappa}})^k \approx 4.19 \cdot 10^{-5}$ while $(1+\frac{1}{40\sqrt{\kappa}})^{-k} \approx 0.6745$.
Finally, we would like to draw your attention to the ability of (GM2-ODE) to recover other high-resolution ODEs. 
**Leading to other ODEs** 
As shown in Table 1, (GM2-ODE) and its SIE discretization recover the TM method and its ODE through different choices of coefficients. In addition, the current existing rates on (HR_TM) in the paper are not comparable to the algorithm's convergence rate $(\mathcal O\left((1-\sqrt{\tfrac{\mu}{L}}\right)^{2k}$ which should correspond to $\mathcal{O}\left(e^{-2\sqrt{\mu}t}\right)$ in continuous time). This observation suggests a new high-resolution ODE for the TM method. Setting the same coefficients that recover the TM method in (GM2-ODE) reads
    $$\ddot{X}_t+\sqrt{\mu}\left(\frac{3-\sqrt{\tfrac{\mu}{L}}}{1-\sqrt{\tfrac{\mu}{L}}}\right)\dot X_t +\frac{1}{\sqrt{L}}\nabla^2 f(X_t)\dot X_t+\left(\frac{2}{1-\sqrt{\tfrac{\mu}{L}}}+\sqrt{\frac{\mu}{L}}\right)\nabla f(X_t)=0.$$
Surprisingly, if the step-size $\tfrac{1}{\sqrt{L}}\rightarrow 0$ the recent ODE reduces to$$ \ddot{X}_t+3\sqrt{\mu}\dot X_t+2\nabla f(X_t)=0$$
which is the low-resolution ODE corresponding to the TM method [3]. We will include this discussion in the revision of our paper. .

Reviewer comment (Novelty):
As I mentioned in Weakness, I am afraid the contribution of this work is significant enough given many existing works on the same topic using almost the same analysis techniques. Could the authors justify this point? Is there any particular novelty in technical and algorithmic developments I'm missing?

Author response:
1. We would like to emphasize that our unified model (GM2-ODE) has two attractive features: \
**A)** Not only does it unify the ODEs in continuous time, but also existing algorithms through SIE discretization. Surprisingly, all these algorithms are found to be well-known methods (NAG,HB,TMM, etc) and they are all recovered through the same routine. \
**B)** The corresponding *unifying* Lyapunov function leads to better convergence rates than the previous results obtained by continuous time analysis. To further emphasize this, we added Table 2 to compare our rates with existing results, which shows precisely the improvements. \
It is true that we use well-established techniques like Lyapunov analysis and Grönwall lemma. Nevertheless, we leverage these tools to achieve a family of Lyapunov functions whose form is not seen before in the literature. Crucially, our analysis is NOT a trivial extension of another work, the form of the unified Lyapunov function requires different steps in the analysis. For example, we use a limit analysis after (55) in the appendix which is different from all of the prior works. This is just one example of the many analytical and algebraic complexities that arise from the generality of the Lyapunov function we addressed in our analysis.

Reviewer comment (Experiments):
The work is focused on the theoretical analysis of existing momentum-based algorithms. I'm wondering if the understanding can help develop some new approaches leading to stronger practical performance? For example, does the best possible rate improve empirical performances in practice?

Author response:
2. Yes, we have actually demonstrated that the best possible rate setting works well in practice in our numerical experiments. Please note that the best theoretical rate coefficients lead to NAG. In Figure 2, we show that NAG outperforms other methods like QHM on a binary classification task also in practice. We will clarify this. 
We managed to recover a large number of existing algorithms through our analysis. Although we did not design a new algorithm, we believe that our analysis has the potential to lead to the discovery of new methods as well.

Reviewer comment (Theory):
In Figure 2, it is observed that NAG is the algorithm with fastest convergence against QHM. I'm curious if the viewpoint of the ODE developed here can provide some explanation to this phenomenon.

Author response:
3. Yes! The best possible rate through Corollary 4.1.1 occurs when $a=1/4$ and $s=3/4L$. Substituting these values into the residual rate gives $f(x_{k})-f(x^*)\leq C_{QHM}(1-\sqrt{\mu/3L})^k$, which is slower than the NAG rate $f(x_{k})-f(x^*)\leq C_{QHM}(1-\sqrt{\mu/L})^k$. 
We will clarify this in the paper.

Reviewer comment (Novelty):
The main concern is the incremental contributions.

Author response:
- **Use of IQCs**: It is true that we leveraged IQCs, but please note we use it only in one of the three Lyapunov functions we propose in this paper. We introduced two more Lyapunov functions which cannot be derived from the IQC theorem: One is in continuous time (Theorem 3.2), and the other one is in discrete time (equation (7)). The latter is the first Lyapunov function that uniformly captures the discrete time behavior of the algorithms for different choices of parameters.

Reviewer comment (Novelty):
The techniques used are well known (Lyapunov analysis, Grönwall lemma, etc), e.g. Thm. 3.2. is an instance of Gronwall lemma.

Author response:
- **Use of well-known techniques**: It is true that we use well-established techniques like Lyapunov analysis and Grönwall lemma. Nevertheless, we leverage these tools to achieve a family of Lyapunov functions whose form is not seen before in the literature. Crucially, our analysis is NOT a trivial extension of another work, the form of the unified Lyapunov function requires different steps in the analysis. For example, we use a limit analysis after (55) in the appendix which is different from all of the prior works. This is just one example of the many analytical and algebraic complexities that arise from the generality of the Lyapunov function we addressed in our analysis.

Reviewer comment (Novelty):
*GM2-ODE.* In terms of structuring, it is surprising that the proposed GM2-ODE is stated in the introduction. Moreover, this ODE is not derived but rather considers a set of some HR-ODEs that exist and aims to unify them in the sense that these can be seen as instances of the GM2-ODE. This is fairly straightforward to do given several ODEs as the terms that appear have already known interpretations; there's no discussion or further development if this ODE is general enough to lead to other useful methods. Also, it is very similar to the existing ODE in Zhang et al. (2021), see eq. GM-ODE in the main part. Considering all, this is a fairly limited contribution stated as central/main.

Author response:
- **Comparison with GM-ODE**: We discussed the differences between GM-ODE and GM2-ODE in detail in pages 8-9 in our submission. The main drawback of GM-ODE is its inconsistency in terms of ODE and algorithm recovery. Moreover, GM2-ODE is more capable of recovering algorithms and leads to better convergence guarantees. Please note that the Lyapunov function corresponding to GM-ODE does not achieve the optimal rates. Also note that discretization of GM-ODE using SIE method does not exactly recover the NAG algorithm. Finally, we would like to draw your attention to the ability of GM2-ODE to recover other high-resolution ODEs. 
Our unified model (GM^2-ODE) has the following two attractive features: 
1. Not only does it unify the ODEs in continuous time, but also existing algorithms through SIE discretization. Surprisingly, all these algorithms are found to be well-known methods (NAG,HB,TMM, etc) and they are all recovered through the same routine. 
 2. The corresponding *unifying* Lyapunov function leads to better convergence rates than the previous results obtained by continuous time analysis. To further emphasize this, we added Table 2 to compare our rates with existing results, which shows precisely the improvements.    
- **Less condition on the convergence of the QHM algorithm**: Please note that the previous rate on QHM requires $L/\mu\geq 9$ [6], while our result in corollary 4.1.1 is free from such condition.
- **Does GM2-ODE lead to new methods?**: As shown in Table1, (GM2-ODE) and its SIE discretization recover the TM method and its ODE through different choices of coefficients. In addition, the current existing rates on (HR_TM) in the paper are not comparable to the algorithm's convergence rate ($\mathcal O\left((1-\sqrt{\tfrac{\mu}{L}}\right)^{2k}$ which should correspond to $\mathcal{O}\left(e^{-2\sqrt{\mu}t}\right)$ in continuous time). This observation suggests a new high-resolution ODE for the TM method. Setting the same coefficients that recover the TM method in (GM2-ODE) reads

$$\ddot{X}_t+\sqrt{\mu}\left(\frac{3-\sqrt{\tfrac{\mu}{L}}}{1-\sqrt{\tfrac{\mu}{L}}}\right)\dot X_t +\frac{1}{\sqrt{L}}\nabla^2 f(X_t)\dot X_t+\left(\frac{2}{1-\sqrt{\tfrac{\mu}{L}}}+\sqrt{\frac{\mu}{L}}\right)\nabla f(X_t)=0.$$

Surprisingly, if the step-size $\tfrac{1}{\sqrt{L}}\rightarrow 0$ the above ODE reduces to $$ \ddot{X}_t+3\sqrt{\mu}\dot X_t+2\nabla f(X_t)=0$$
which is the low-resolution ODE corresponding to the TM method [3]. This discussion will be added to the paper.

Reviewer comment (Presentation):
The paper reads well, and I enjoyed reading it. However, content-wise, it is not on point regarding the actual focus of this paper / exact contributions / motivation for these contributions, etc. It often focuses on general optimization comments that are enjoyable to read but perhaps more suitable for a textbook, etc., and due to that, it is not concise in bringing the reader to the actual contributions and their motivation.

Author response:
- **On the Introduction**: We believe that the first two paragraphs provide insight to the reader as to why the phenomenon of acceleration in discrete time analysis is considered mysterious and explain why other approaches, including continuous-time analysis, are studied for these methods. But we can shorten these two sections to save space and extend our discussions on the continuous-time perspective and why we study this specific perspective.

Reviewer comment (Novelty):
Abstract — A large part focuses on general comments about HR-ODEs or the Lyapunov function, which is an intermediate step of proving convergence that many methods can be seen as a discretization of the ODE, which is often the case.

Author response:
- **On the Abstract**: Thank you for bringing this up. We will revise our abstract to be more specific about the strongly convex problem setting, discretization techniques used, and the improvements achieved in the rate.

Reviewer comment (Theory):
Abstract — It leaves very unclearly what precisely the "improved convergence guarantees compared to prior art" are -- it would be helpful to state precisely if the constants or the order is improved and by what factor; what is the precise advantage of this unifying HR-ODE (is it more interpretable, etc), etc.

Author response:
- **On the Abstract**: Thank you for bringing this up. We will revise our abstract to be more specific about the strongly convex problem setting, discretization techniques used, and the improvements achieved in the rate.

Reviewer comment (Writing):
Abstract — It does not even mention the setting, e.g., that the results are for strongly convex functions.

Author response:
- **On the Abstract**: Thank you for bringing this up. We will revise our abstract to be more specific about the strongly convex problem setting, discretization techniques used, and the improvements achieved in the rate.

Reviewer comment (Writing):
Abstract — That discrete methods can be viewed as discretizations of ODEs is well known. If keeping this sentence, it is worth mentioning the type of discretization.

Author response:
1. The Semi-Implicit Euler (SIE) is a symplectic discretizer, meaning that it preserves the energy of Hamiltonian systems after discretization. Therefore, it was used in previous work to show the acceleration of the NAG algorithm [5]. Being the simplest symplectic discretization, it is well-suited for our analysis, as you pointed out, ensuring consistency in our work. This means that after discretization, we recover algorithms using the same coefficients in the corresponding ODE. A more detailed discussion on this subject can be found in subsection 5.2 in our submission.

Reviewer comment (Presentation):
Introduction — The first two paragraphs that refer to (discrete) optimization methods generally are very enjoyable to read. Still, the motivation for using continuous-time analyses is rushed, which is more relevant to this paper. The paper would benefit from reconsidering the content vs. the page limit and prioritizing better.

Author response:
- **On the Introduction**: We believe that the first two paragraphs provide insight to the reader as to why the phenomenon of acceleration in discrete time analysis is considered mysterious and explain why other approaches, including continuous-time analysis, are studied for these methods. But we can shorten these two sections to save space and extend our discussions on the continuous-time perspective and why we study this specific perspective.

Reviewer comment (Theory):
Thm. 3.1. states that \( f \) is strongly convex, but the proof relies on Thm. 6.4 in (Fazlyab et al., 2018), which uses the assumption that \( f \) is also \( L \) smooth. This assumption should be stated.

Author response:
- **Missing smoothness assumption in Thm 3.1**: We would like to thank you for your careful reading of our paper, including the appendix. However, while there is a smoothness assumption in Theorem A.1, we intentionally discard it in Theorem 3.1. Please note that the smoothness assumption is crucial for the discrete-time analysis of first-order methods, but its effect is negligible in the continuous-time analysis, as stated in [1, the last sentence in the introduction]. The coefficient $\sigma$ is defined as a nonnegative constant of the matrix $M^{(3)},$ which is the only matrix that relies on the smoothness assumption. Therefore, $\sigma$ is defined in Theorem A.1, and we explicitly set $\sigma=0$ in Appendix B.2 below equation (31). This choice effectively removes any influence of the smoothness constant $L$ and the smoothness assumption from the theorem.

Reviewer comment (Writing):
The proof in App. B.2. also mentions \( \sigma \), which is not defined in the paper.

Author response:
- **Missing smoothness assumption in Thm 3.1**: We would like to thank you for your careful reading of our paper, including the appendix. However, while there is a smoothness assumption in Theorem A.1, we intentionally discard it in Theorem 3.1. Please note that the smoothness assumption is crucial for the discrete-time analysis of first-order methods, but its effect is negligible in the continuous-time analysis, as stated in [1, the last sentence in the introduction]. The coefficient $\sigma$ is defined as a nonnegative constant of the matrix $M^{(3)},$ which is the only matrix that relies on the smoothness assumption. Therefore, $\sigma$ is defined in Theorem A.1, and we explicitly set $\sigma=0$ in Appendix B.2 below equation (31). This choice effectively removes any influence of the smoothness constant $L$ and the smoothness assumption from the theorem.

Reviewer comment (Writing):
The curly F notation, used in the main part and Thm. A.1 was not introduced.

Author response:
- **Unclear notations**: The curly $F$ notation denotes the set of $\mu$-strongly convex and $L$-smooth functions. We will clarify this in the revision to avoid ambiguity. Other minor comments and typos will be fixed.