Paper: Advancing the Lower Bounds: an Accelerated, Stochastic, Second-order Method with Optimal Adaptation to Inexactness (ICLR-2024)

Reviewer comment (Writing):
Though the text has a good flow in terms of the context, there is still room for improvements: e.g. (a) In page 2 there are two loose sentences in the middle of the page. They can be integrated with the previous paragraph or just in a new independent paragraph.

Author response:
**Response:** Thank you for suggestions and pointing out several inaccuracies. We agree with comments (a, c, d, e, f , g) and will change it in the revised version. Regarding comment (b), we maintain the expression "E[g(x,ξ)]" exactly as it is below Assumption 1.2

Reviewer comment (Writing):
Though the text has a good flow in terms of the context, there is still room for improvements: e.g. (b) Below assumption 1.2: $E[g(x,\xi)]$ or $E[F(x,\xi)]$?

Author response:
**Response:** Thank you for suggestions and pointing out several inaccuracies. We agree with comments (a, c, d, e, f , g) and will change it in the revised version. Regarding comment (b), we maintain the expression "E[g(x,ξ)]" exactly as it is below Assumption 1.2

Reviewer comment (Writing):
Though the text has a good flow in terms of the context, there is still room for improvements: e.g. (d) Function $\psi_t(x)$ above algorithm 1 is not defined (I think it is in the algorithm so you should refer to that or simply introduce it).

Author response:
**Response:** Thank you for suggestions and pointing out several inaccuracies. We agree with comments (a, c, d, e, f , g) and will change it in the revised version. Regarding comment (b), we maintain the expression "E[g(x,ξ)]" exactly as it is below Assumption 1.2

Reviewer comment (Presentation):
Though the text has a good flow in terms of the context, there is still room for improvements: e.g. (f) I suggest larger font size on Figures 1,2.

Author response:
**Response:** Thank you for suggestions and pointing out several inaccuracies. We agree with comments (a, c, d, e, f , g) and will change it in the revised version. Regarding comment (b), we maintain the expression "E[g(x,ξ)]" exactly as it is below Assumption 1.2

Reviewer comment (Presentation):
I think contributions 3 & 1 should be integrated.

Author response:
**Response:**  Contribution 1 signifies an enhanced convergence rate, whereas Contribution 3 underscores the successful integration of an inexact subproblem solution—a novel aspect not previously addressed in inexact second-order and high-order convex methods. We appreciate your thoughtful input and will consider how best to integrate these contributions for an improved presentation of our research.

Reviewer comment (Evaluation):
Table 1 might lead to misunderstanding. Your result is based on the asssumption of unbiased gradients. The rate by Agafonov does not have this assumption. Thus, it seems like an unfair comparison.

Author response:
**Response:** Certainly, we acknowledge that a direct comparison of the convergence of these methods is not feasible, as elucidated in the paragraph following Assumption 2.3. To provide a comprehensive overview of the current landscape in the realm of inexact/stochastic second-order methods, we present Table 1. This table includes a column outlining assumptions related to inexactness, explicitly noting that the unbiasedness of the gradient was not assumed in the work [1]. We believe that Table 1's clear articulation of assumptions minimizes any potential misunderstandings regarding the comparative context.

Reviewer comment (Writing):
The title of the section “Lower Bound” is very generic and also the description is vague. For example, the first sentence of this section is vague.

Author response:
**Response:** Could you kindly offer additional insights on why the title and the first sentence might be considered vague? We value your perspective and would appreciate any suggestions you may have for improvement. While the current wording seems acceptable to us, we are open to refining it based on your feedback. Thank you.

Reviewer comment (Writing):
Is the $\boldsymbol \Phi_x$ defined at the beginning of Section 2 used? Same question holds for $\boldsymbol\Phi_x$ defined under Assumption 5.1.

Author response:
**Response:** Formally, its utilization is confined to the Appendix. Nevertheless, we believe it is crucial to introduce the exact Taylor polynomial $\Phi_x(y)$, as it provides valuable insights into the construction of high-order methods.

Reviewer comment (Novelty):
Lemma 2.1 seems like a special case of a similar lemma in Agafonov et al 2020. Is there a reason you did not cite their work when you present the Lemma? Same question holds for Lemma 5.2.

Author response:
**Response:**  
Thank you, we agree with your comment, and we will add a direct citation near Lemmas 2.1 and 5.1 in revised version.

Reviewer comment (Theory):
According to your investigation and results, does it make sense to analyze inexactness in Hessian (or higher order information of the objective function) when gradients are inexact? This question mainly concerns the $O(1/\sqrt{T})$ convergence rate related to the inexactness of the gradients which dominates the convergence rate as $T\rightarrow \infty$.

Author response:
**Response:** From a theoretical perspective, we agree that the rate of $O(\frac{1}{\sqrt{T}})$ is achieved when $T \to \infty$. However, in practice, we are not interested in this case, as we have either predefined number of iterations or a desired accuracy to achieve. In that case the convergence rate of the method depends on the relationship between constants $M,\sigma_1,\sigma_2$ and can be dominated by either term in rates (9-10).

We think that there are several reasons why stochastic high-order methods are important:
1. Please have a look at experiments. Although all methods achieve $O(\frac{1}{\sqrt{T}})$ while $T \to \infty$, the superiority of second-order methods is clear (Figure 2). Moreover, even the power in Hessian inexactness convergence term plays an important role in the behavior of the methods (Figure 2 (c, d)). 
2. Next, imagine the case when we can control inexactness in gradients and Hessians. Then, with proper choice of $\sigma_1 = O(\varepsilon^{5/6})$, $\sigma_2  =O(\varepsilon^{1/3})$ (see Corollary 3.3) the method achieves convergence rate $T = O(\varepsilon ^{-1/3})$. With proper choice of gradient inexactness gradient-type methods can not be faster than $O(\varepsilon^{-1/2})$. One particular case, where the inexactness can be controlled is finite-sum optimization, where we can choose batch sizes. Then, in the case of the parallelized computations, we care more about the total number of iterations over the total stochastic derivatives computations.
3. Also consider the following scenario: a distributed centralized network where the server stores a data sample $f_1(x)$ with a Hessian $\nabla^2 f_1(x)$ that offers a "good" approximation of the exact Hessian of the objective function:
$$ ||\nabla^2 f_1(x) - \nabla^2 f(x) || \leq \beta.$$
This assumption, known as beta-similarity (Hessian-similarity) [1], can be inferred from equation (5 with $H(x) = \nabla^2 f_1(x)$ and $\delta_2 = \beta$. By leveraging this approximation, we can avoid exchanging Hessians, resulting in significant communication savings. This allows running on a server a second-order method with only gradient communications. 

[1] Zhang, Yuchen, and Xiao Lin. "DiSCO: Distributed optimization for self-concordant empirical loss." International conference on machine learning. PMLR, 2015.

Reviewer comment (Theory):
The paper lacks an intuitive explanation of the proposed algorithm. It is suggested to explain the motivation and impact of parameters such as $\alpha_t$, the choice of the model $\omega_x^{M,\bar{\delta}}$, and the technique of estimating sequence on the algorithm's performance and convergence.

Author response:
**Response:** Kindly examine the common commentary titled "On intuition behind the algorithm." 

Furthermore, we have included a discussion on the intuition behind our method to Appendix B.

Reviewer comment (Presentation):
The figures in the paper are too small, making it difficult for readers to interpret the results.

Author response:
**Response:** Thank you for your feedback regarding the figures in our paper. We acknowledge your concern about their size impacting the ease of interpretation for readers. In the revised version, we will try to address this by resizing the figures for better visibility and comprehension. However, it's essential to note that we are constrained to 9 pages, and any adjustments to figure sizes may necessitate the removal of other elements from the work to maintain brevity. Alternatively, we are open to adding figures from Section 7 to the Appendix, presenting them with increased size as shown in Figures 3 and 4. Your further suggestions on this ma tter are appreciated.

Additionally, we have increased the font size for Figures 1 and 2. To provide further clarity and accessibility, we have added a larger version of these graphics to Appendix C (Figures 5 and 6).

Reviewer comment (Reproducibility):
The algorithm of step 4 also includes another optimization problem.

Author response:
**The subproblem in line 4 of Algorithm 1 admits a closed form solution**, 
so it cannot be considered as a separate optimization problem. We apologize for not presenting it explicitly in the paper. Allow us to rectify that here.

First, note that the problem in line 4 is convex. Let us write optimality condition
$$0 = \nabla \psi_t(y_t) = (\lambda_{t}  + \kappa_2^{t} + \kappa_3^{t} ||y_{t} - x_0||)(y_{t} - x_0) + \sum \limits_{j=0}^{t-1} \frac{\alpha_j}{A_j} g(x_{j+1}).$$
Thus,
$$(\lambda_{t}  + \kappa_2^{t} + \kappa_3^{t} ||y_{t} - x_0||)||y_{t} - x_0||  =  || \sum \limits_{j=0}^{t-1}\frac{\alpha_j}{A_j} g(x_{j+1})||.$$
Let us denote $r_{t} = ||y_{t} - x_0||$ and $S_t = || \sum \limits_{j=0}^{t-1}\frac{\alpha_j}{A_j} g(x_{j+1})||$. Then, we get
$$\kappa_3^{t} r_{t}^2 + (\lambda_{t}  + \kappa_2^{t})r_{t} - S_t = 0.$$
Next, we get the solution of quadratic equation
$$r_{t} = \frac{\sqrt{(\lambda_{t}  + \kappa_2^{t})^2 + 4 \kappa_3^{t}S_t } - (\lambda_{t} + \kappa_2^{t} )}{2\kappa_3^{t}}.$$
Finally, we get explicit solution

$$y_{t} = x_0 - \frac{\sum \limits_{j=0}^{t-1}\frac{\alpha_j}{A_j} g(x_{j+1}) }{\lambda_t + \bar{\kappa}_2^t + \bar{\kappa}_3^t r_t}.$$

We will add this derivation to the revised version of the paper.


Furthermore, we would like to point out that *solving minimization subproblems are in the heart of  algorithm design*. For example, even the iconic gradient descent scheme stems directly from minimizing quadratically regularized 1st order Taylor approximation sub-problem:
$$x_{k+1} = \arg \min_x  (f(x_k) + \langle \nabla f(x_k), x- x_k\rangle + \frac{1}{2\gamma}||x - x_k||^2). $$
 Gradient descent allows for closed form solution as a subproblem in line 4 in Algorithm 1 . Subproblems like line 4 in Algorithm 1 naturally arise from acceleration techniques. See line 1 of (4.8) from [1], lines 6-7 of Algorithm 1 from [2].  Furthermore, the absence of a closed-form solution for a subproblem should not be viewed as a deterrent when considering a specific optimization method, such as Mirror-Prox [3], Frank-Wolfe methods [4, 5], or Trust-Region methods [5].

**Practical performance**


As we have established, the practical performance of subproblem 4 is not an issue. We use the explicit solution in our implementation of the algorithm. We provide numerical evidence for the overall algorithm in section 7. The proposed method works well in practice, one can see the behavior on Figures 1, 2.


*Having outlined the specifics of the subproblem in line 4 of Algorithm 1 along with its explicit solution, we wonder whether this might be the reason for the pronounced strong rejection.*

[1] Nesterov, Yu. "Accelerating the cubic regularization of Newton’s method on convex problems." Mathematical Programming 112.1 (2008): 159-181.

[2] Allen-Zhu, Zeyuan, and Lorenzo Orecchia. "Linear coupling: An ultimate unification of gradient and mirror descent." arXiv preprint arXiv:1407.1537 (2014).

[3] Nemirovskij, Arkadij Semenovič, and David Borisovich Yudin. "Problem complexity and method efficiency in optimization." (1983).

[4] Jaggi, Martin. "Revisiting Frank-Wolfe: Projection-free sparse convex optimization." International conference on machine learning. PMLR, 2013.

[5] Nocedal, Jorge, and Stephen J. Wright, eds. Numerical optimization. New York, NY: Springer New York, 1999.

Reviewer comment (Writing):
How does the algorithm work in step 4 of the proposed algorithm?

Author response:
**The subproblem in line 4 of Algorithm 1 admits a closed form solution**, 
so it cannot be considered as a separate optimization problem. We apologize for not presenting it explicitly in the paper. Allow us to rectify that here.

First, note that the problem in line 4 is convex. Let us write optimality condition
$$0 = \nabla \psi_t(y_t) = (\lambda_{t}  + \kappa_2^{t} + \kappa_3^{t} ||y_{t} - x_0||)(y_{t} - x_0) + \sum \limits_{j=0}^{t-1} \frac{\alpha_j}{A_j} g(x_{j+1}).$$
Thus,
$$(\lambda_{t}  + \kappa_2^{t} + \kappa_3^{t} ||y_{t} - x_0||)||y_{t} - x_0||  =  || \sum \limits_{j=0}^{t-1}\frac{\alpha_j}{A_j} g(x_{j+1})||.$$
Let us denote $r_{t} = ||y_{t} - x_0||$ and $S_t = || \sum \limits_{j=0}^{t-1}\frac{\alpha_j}{A_j} g(x_{j+1})||$. Then, we get
$$\kappa_3^{t} r_{t}^2 + (\lambda_{t}  + \kappa_2^{t})r_{t} - S_t = 0.$$
Next, we get the solution of quadratic equation
$$r_{t} = \frac{\sqrt{(\lambda_{t}  + \kappa_2^{t})^2 + 4 \kappa_3^{t}S_t } - (\lambda_{t} + \kappa_2^{t} )}{2\kappa_3^{t}}.$$
Finally, we get explicit solution

$$y_{t} = x_0 - \frac{\sum \limits_{j=0}^{t-1}\frac{\alpha_j}{A_j} g(x_{j+1}) }{\lambda_t + \bar{\kappa}_2^t + \bar{\kappa}_3^t r_t}.$$

We will add this derivation to the revised version of the paper.


Furthermore, we would like to point out that *solving minimization subproblems are in the heart of  algorithm design*. For example, even the iconic gradient descent scheme stems directly from minimizing quadratically regularized 1st order Taylor approximation sub-problem:
$$x_{k+1} = \arg \min_x  (f(x_k) + \langle \nabla f(x_k), x- x_k\rangle + \frac{1}{2\gamma}||x - x_k||^2). $$
 Gradient descent allows for closed form solution as a subproblem in line 4 in Algorithm 1 . Subproblems like line 4 in Algorithm 1 naturally arise from acceleration techniques. See line 1 of (4.8) from [1], lines 6-7 of Algorithm 1 from [2].  Furthermore, the absence of a closed-form solution for a subproblem should not be viewed as a deterrent when considering a specific optimization method, such as Mirror-Prox [3], Frank-Wolfe methods [4, 5], or Trust-Region methods [5].

**Practical performance**


As we have established, the practical performance of subproblem 4 is not an issue. We use the explicit solution in our implementation of the algorithm. We provide numerical evidence for the overall algorithm in section 7. The proposed method works well in practice, one can see the behavior on Figures 1, 2.


*Having outlined the specifics of the subproblem in line 4 of Algorithm 1 along with its explicit solution, we wonder whether this might be the reason for the pronounced strong rejection.*

[1] Nesterov, Yu. "Accelerating the cubic regularization of Newton’s method on convex problems." Mathematical Programming 112.1 (2008): 159-181.

[2] Allen-Zhu, Zeyuan, and Lorenzo Orecchia. "Linear coupling: An ultimate unification of gradient and mirror descent." arXiv preprint arXiv:1407.1537 (2014).

[3] Nemirovskij, Arkadij Semenovič, and David Borisovich Yudin. "Problem complexity and method efficiency in optimization." (1983).

[4] Jaggi, Martin. "Revisiting Frank-Wolfe: Projection-free sparse convex optimization." International conference on machine learning. PMLR, 2013.

[5] Nocedal, Jorge, and Stephen J. Wright, eds. Numerical optimization. New York, NY: Springer New York, 1999.

Reviewer comment (Experiments):
Does it work in practice?

Author response:
**The subproblem in line 4 of Algorithm 1 admits a closed form solution**, 
so it cannot be considered as a separate optimization problem. We apologize for not presenting it explicitly in the paper. Allow us to rectify that here.

First, note that the problem in line 4 is convex. Let us write optimality condition
$$0 = \nabla \psi_t(y_t) = (\lambda_{t}  + \kappa_2^{t} + \kappa_3^{t} ||y_{t} - x_0||)(y_{t} - x_0) + \sum \limits_{j=0}^{t-1} \frac{\alpha_j}{A_j} g(x_{j+1}).$$
Thus,
$$(\lambda_{t}  + \kappa_2^{t} + \kappa_3^{t} ||y_{t} - x_0||)||y_{t} - x_0||  =  || \sum \limits_{j=0}^{t-1}\frac{\alpha_j}{A_j} g(x_{j+1})||.$$
Let us denote $r_{t} = ||y_{t} - x_0||$ and $S_t = || \sum \limits_{j=0}^{t-1}\frac{\alpha_j}{A_j} g(x_{j+1})||$. Then, we get
$$\kappa_3^{t} r_{t}^2 + (\lambda_{t}  + \kappa_2^{t})r_{t} - S_t = 0.$$
Next, we get the solution of quadratic equation
$$r_{t} = \frac{\sqrt{(\lambda_{t}  + \kappa_2^{t})^2 + 4 \kappa_3^{t}S_t } - (\lambda_{t} + \kappa_2^{t} )}{2\kappa_3^{t}}.$$
Finally, we get explicit solution

$$y_{t} = x_0 - \frac{\sum \limits_{j=0}^{t-1}\frac{\alpha_j}{A_j} g(x_{j+1}) }{\lambda_t + \bar{\kappa}_2^t + \bar{\kappa}_3^t r_t}.$$

We will add this derivation to the revised version of the paper.


Furthermore, we would like to point out that *solving minimization subproblems are in the heart of  algorithm design*. For example, even the iconic gradient descent scheme stems directly from minimizing quadratically regularized 1st order Taylor approximation sub-problem:
$$x_{k+1} = \arg \min_x  (f(x_k) + \langle \nabla f(x_k), x- x_k\rangle + \frac{1}{2\gamma}||x - x_k||^2). $$
 Gradient descent allows for closed form solution as a subproblem in line 4 in Algorithm 1 . Subproblems like line 4 in Algorithm 1 naturally arise from acceleration techniques. See line 1 of (4.8) from [1], lines 6-7 of Algorithm 1 from [2].  Furthermore, the absence of a closed-form solution for a subproblem should not be viewed as a deterrent when considering a specific optimization method, such as Mirror-Prox [3], Frank-Wolfe methods [4, 5], or Trust-Region methods [5].

**Practical performance**


As we have established, the practical performance of subproblem 4 is not an issue. We use the explicit solution in our implementation of the algorithm. We provide numerical evidence for the overall algorithm in section 7. The proposed method works well in practice, one can see the behavior on Figures 1, 2.


*Having outlined the specifics of the subproblem in line 4 of Algorithm 1 along with its explicit solution, we wonder whether this might be the reason for the pronounced strong rejection.*

[1] Nesterov, Yu. "Accelerating the cubic regularization of Newton’s method on convex problems." Mathematical Programming 112.1 (2008): 159-181.

[2] Allen-Zhu, Zeyuan, and Lorenzo Orecchia. "Linear coupling: An ultimate unification of gradient and mirror descent." arXiv preprint arXiv:1407.1537 (2014).

[3] Nemirovskij, Arkadij Semenovič, and David Borisovich Yudin. "Problem complexity and method efficiency in optimization." (1983).

[4] Jaggi, Martin. "Revisiting Frank-Wolfe: Projection-free sparse convex optimization." International conference on machine learning. PMLR, 2013.

[5] Nocedal, Jorge, and Stephen J. Wright, eds. Numerical optimization. New York, NY: Springer New York, 1999.

Reviewer comment (Theory):
The assumptions (2.2 and 2.3) of stochacity and inexactness differ but seem highly related, as they lead to two quite similar convergence rates and proof for your algorithm. In this paper the way of discussing the stochasticity and inexactness settings seems a bit nesting. Maybe it could be better if their highlevel relations are set forth in a proper manner.

Author response:
**Response:** 
We appreciate your observation regarding the nesting of Assumptions (2.2 and 2.3) and their apparent similarities in leading to analogous convergence rates and proof for our algorithms. While we recognize their close relationship, it's crucial to emphasize a fundamental distinction between the two.  Assumption 2.2 imposes a bound on the expected norm of the difference between the Hessian and the stochastic Hessian, whereas Assumption 2.3 pertains to the difference along the direction $y - x$. Consequently, 2.3 is necessary only for iterations of the Algorithm, and the value of $\delta_2^{x, y}$ can be significantly smaller than the norm of the difference between the Hessian and its approximation. This delineation underscores why we present both assumptions and provide separate theoretical results for each. If you believe that presenting both assumptions is unnecessary, we welcome further discussion on this matter.

Reviewer comment (Writing):
Clarification: O(1/T^2) -> \Omega(1/T^2) in P2 line 2; bounded variance stochastic Hessian -> stochastic Hessian with bounded variance; formatting issues such as sequence numbers in the algorithm list.

Author response:
**Response:**  Thank you for bringing that to our attention; we will make the necessary changes to the paper in the revised version.

Reviewer comment (Theory):
The gloabal convergent (accelerated, cubic newton type) second-order methods, while they have accelerated global convergence, they usually can be proven to have superlinear local convergence rate. Is the parallel local characteristic worth to be mentioned and investigated in your stochastic/inexact setting?

Author response:
**Response:** Thank you for your insightful commentary. 

While local convergence seems to be an interesting theoretical direction, it lies beyond the scope of this paper, which primarily concentrates on global convergence without assumptions on initial point. Local convergence normally requires some very restrictive assumptions on starting point and batch size (gradient accuracy) which are hard to satisfy in practice. 

Illustratively, consider the local superlinear convergence from [1], which requires $f(x_0) - f(x^*) \leq \frac{\mu^3}{L_2^2}$. In most practical scenarios desired accuracy $\varepsilon \lesssim  \frac{\mu^3}{L_2^2}$. 

In summary, while the local convergence of stochastic/inexact accelerated second-order/high-order methods remains an open question, we believe that exploring this aspect may not make the proposed scheme much more practical. Consequently, we maintain our focus on global performance within the defined scope of this work.

[1] Kovalev, Dmitry, Konstantin Mishchenko, and Peter Richtárik. "Stochastic Newton and cubic Newton methods with simple local linear-quadratic rates." arXiv preprint arXiv:1912.01597 (2019).

Reviewer comment (Presentation):
The open question mentioned, "what's the optimal trade-off between inexactness in gradients and the Hessian?", where in the article is intuitively investigated?

Author response:
**Response:** We establish the lower bound $\Omega(\frac{\sigma_1 R}{\sqrt{T}} + \frac{\sigma_2 R^2}{T^2} + \frac{L_2 R^{3}}{T^{7/2}})$, which shows that the best possible tradeoff between inexactness is $\Omega(\frac{\sigma_1 R}{\sqrt{T}} + \frac{\sigma_2 R^2}{T^2})$, which is achieved by proposed algorithm.

Reviewer comment (Theory):
Regarding Algorithm 3, you mentioned the 'Restarted Accelerated Stochastic Tensor Method'. Could you further elaborate on the specific mechanism and necessity of this 'restarting'? Under what circumstances should a restart be performed?

Author response:
**Response:** Restarts serve as a technique to adapt a method designed for convex problems to those that are strongly convex. Typically, restarts ensure the preservation of optimality, meaning that an optimal method for convex functions can be extended to become an optimal method for strongly convex ones. The restart occurs after a predetermined number of iterations, as outlined in Algorithm 3 and (15).

Our presentation of the restarted method is motivated not only by its applicability to strongly-convex problems but also by a compelling observation. Specifically, the overall number of stochastic Hessian computations, $O(\frac{\sigma_2}{\mu^{1/3}} )\log \frac{1}{\varepsilon}$ scales linearly with the desired accuracy $\varepsilon$, as elucidated in the paragraph following Theorem 6.2. This noteworthy insight enables the use of constant batch sizes for stochastic Hessian, independent of $\varepsilon$. This stands in contrast to first-order derivatives, where the batch size is dependent on $\varepsilon$.