Paper: Limited-Memory Greedy Quasi-Newton Method with Non-asymptotic Superlinear Convergence Rate (ICLR-2024)

Reviewer comment (Reproducibility):
The version of LG-BFGS which has super-linear convergence uses a correction strategy that requires Hessian-vector products. The naive cost of these computations are $O(d^2)$, which would make LG-BFGS more expensive than Greedy BFGS in practice. However, this computational cost is not addressed anywhere in the text.

Author response:
**W1, W2, Q3**: We agree with the Reviewer that the weighted norm $\phi_t=\lVert x_{t+1}-x_t\rVert_{\nabla^2f(x_t)}$ in the correction strategy requires the computation $O(d^2)$. However, we would like to clarify that this weighted norm can be replaced by $\hat{\phi_t}=\lVert x_{t+1}-x_{t}\rVert_{LI}$, which provides the same correction guarantees and only requires the computation $\mathcal{O}(d)$. 

Specifically, from $\mu I\preceq\nabla^2 f(x_t)\preceq LI$ of Assumption 1, we have $\hat{\phi}_t\ge\phi_t$ and $(1+\hat{\phi}_tC_M)B_t\succeq (1+\phi_tC_M)B_t\succeq B_t$, and thus $\hat{\phi}_t$ provides the same correction guarantees as $\phi_t$. Since $LI$ is a diagonal matrix, $\hat{\phi}_t$ only requires the computation $O(d)$. Additionally, we note that this modification has an impact. It leads to a larger error between the Hessian approximation $\hat{B}_t$ and the actual Hessian $\nabla^2f(x_t)$, subsequently reducing the update progress of algorithm. We will clarify this point and include this modification in the final upload. Thank you for pointing this out.

Reviewer comment (Theory):
Related to the previous point, the authors repeatedly miss-represent the computational complexity of LG-BFGS as being comparable to L-BFGS, which it is formally not.

Author response:
**W1, W2, Q3**: We agree with the Reviewer that the weighted norm $\phi_t=\lVert x_{t+1}-x_t\rVert_{\nabla^2f(x_t)}$ in the correction strategy requires the computation $O(d^2)$. However, we would like to clarify that this weighted norm can be replaced by $\hat{\phi_t}=\lVert x_{t+1}-x_{t}\rVert_{LI}$, which provides the same correction guarantees and only requires the computation $\mathcal{O}(d)$. 

Specifically, from $\mu I\preceq\nabla^2 f(x_t)\preceq LI$ of Assumption 1, we have $\hat{\phi}_t\ge\phi_t$ and $(1+\hat{\phi}_tC_M)B_t\succeq (1+\phi_tC_M)B_t\succeq B_t$, and thus $\hat{\phi}_t$ provides the same correction guarantees as $\phi_t$. Since $LI$ is a diagonal matrix, $\hat{\phi}_t$ only requires the computation $O(d)$. Additionally, we note that this modification has an impact. It leads to a larger error between the Hessian approximation $\hat{B}_t$ and the actual Hessian $\nabla^2f(x_t)$, subsequently reducing the update progress of algorithm. We will clarify this point and include this modification in the final upload. Thank you for pointing this out.

Reviewer comment (Experiments):
The role of the correction strategy is not studied empirically despite the necessity of this trick for deriving convergence guarantees. Experimental results appear to be shown only for the algorithm without correction, which does not match the analysis in the paper.

Author response:
**W4, Q8.1**: Thank you for the comment. We would like to clarify that we do not apply the correction strategy in all experiments, following the existing greedy BFGS methods (Rodomanov & Nesterov 2021a; Lin et al. 2021). Specifically, the application of the correction strategy is to guarantee the error matrix positive semi-definite, i.e., $\hat{B_t}-\nabla^2f(x_{t+1})\succeq 0$. However, in practical implementation, the error matrix is positive definite without correction and thus, there is no need to apply the correction strategy, as recommended by (Rodomanov & Nesterov 2021a; Lin et al. 2021). To further address the Reviewer's concern, we will provide an ablation study comparing the performance of LG-BFGS with and without the correction factor in the final upload.

Reviewer comment (Theory):
Remark 1: The computational cost is of a different order for L-BFGS and LG-BFGS: $O(\tau^2 d + \tau^4)$ vs $O(4 \tau d)$, so I would not say that they are comparable. If $\tau \leq 4$ is very small, then they are comparable, but not in an asymptotic sense. Moreover, if you are going to use big-O notation, then the complexity of L-BFGS should be written as $O(\tau d)$ since the constant $4$ does not contribute to the asymptotic growth. The first comment also applies to the discussion in Section 5.

Author response:
**Q2**: We agree with the Reviewer and we will endeavor to be consistent with the use of complexity notation, to omit constant factors that are not problem-dependent. We thank the reviewer for helping us clarify the algorithm's required computational effort.

Reviewer comment (Theory):
Comparison to BFGS: Do you think the slower rate of BFGS compared to G-BFGS is an artifact of the analysis by Rodomanov and Nesterov, or is it because of the different update strategy used for the curvature matrix? The per-iteration convergence of L-BFGS and LG-BFGS suggests it is the latter.

Author response:
**Q6**: We agree with the Reviewer that the rate improvement of Greedy BFGS compared to BFGS is because of the greedy update strategy used for the curvature matrix. Specifically, the greedy update strategy of G-BFGS selects the variable variation $s_t$ as the basis vector that maximizes the weighted norm ratio [cf. (6)]. It optimizes the update progress of the Hessian approximation [cf. (41) in the proof of Proposition 2] and improves the convergence rate. We will clarify this point in the revised paper. Thank you for the comment.

Reviewer comment (Experiments):
Experimental Comparison: It doesn't make sense to compare G-FBGS to LG-BFGS with any choice of $\tau$ as a linear function of $d$. In this setting, LG-BFGS is asymptotically more expensive than G-BFGS and both theoretically and experimentally slower than G-BFGS. It only makes sense to use LG-BFGS when $\tau$ is an absolute constant or a slowly growing sub-linear function of $d$. For example, choosing $\tau \in \{5, 10, 25, 50\}$ would be appropriate for MNIST and Protein, while somewhat smaller choices would be suitable for Connect-4.

Author response:
**Q7**: Thank you for the comment. If $\tau$ is small, the performance of LG-BFGS will degrade close to L-BFGS, though the computation of LG-BFGS decreases. The reason we explored this range of $\tau$ is to illuminate the performance tradeoffs between these extremes. In general, we agree selecting $\tau$ as an absolute constant makes sense, but doing so would not reveal how performance varies with respect to the interplay between memory size and feature dimension. We will clarify this selection of memory parameter and conduct this additional numerical study in the final upload, if afforded the opportunity.

Reviewer comment (Presentation):
It would be advisable to present the contributions as bullet points to provide a clearer view.

Author response:
**W1**: Thank you for the suggestion. We will revise the presentation of our contributions in the final upload to be itemized bullets as follows:

1. We propose a Limited-memory Greedy (LG)-BFGS method, which is a novel synthesis of greedy basis selection and displacement aggregation. It maximizes the update progress per iteration to improve the convergence, while maintaining the limited memory setting. 

2. We furnish a clear and explicit expression for the non-asymptotic convergence of the LG-BFGS, which not only delineates the direct influence of memory size through the concept of relative condition number $\beta_\tau$ [Def. 1], but also identifies roles played by other factors (e.g., problem properties, initial conditions, etc.). This is the first work that characterizes the explicit effect of memory size on the convergence rate.

3. The LG-BFGS presents an non-asymptotic superlinear rate with the boundness assumption on $\beta_\tau$. This is the first work that shows the possibility of achieving superlinear rates in limited-memory methods and identifies the exact condition / assumption under which this could be possible. The result recovers that of the full-memory method when the memory size increases to the feature dimension. This establishes a clear path from limited-memory methods to full-memory ones, and provides an explicit theoretical explanation behind the performance change along the reduction of memory size

In summary, this paper answers three unknown questions: (i) Is it possible for limited-memory methods to achieve explicit superlinear rates? (ii) If so, which condition does it require? (iii) What is the explicit effect of memory size on the convergence rate?

Reviewer comment (Experiments):
There is missing citations and work comparison with [1].

Author response:
**W4**: Thank you for the suggestion. We will add this citation, and are grateful for the Reviewer to bringing this innovative work to our attention. We have reproduced the manner in which it will be discussed in the paper here for convenience:

Accelerations of standard BFGS have been considered in various forms; however, Nesterov acceleration is one approach that has been shown to enhance the rate of convergence of standard BFGS after a threshold number of iterations, under suitable choice of step-size and error-bound tolerance [1]. However, we note that, similar to standard BFGS, accelerated variants require full memory to achieve these rates.

Reviewer comment (Writing):
The title of the paper may lead one to believe that the proposed method is applicable to Quasi-Newton methods, including SR1. However, upon reading the paper, it becomes clear that the authors only discuss BFGS. To avoid any ambiguity, it would be better to modify the title accordingly.

Author response:
**W5**: We agree with the Reviewer that the title may give the impresion that our results hold for the entire Broyden class of Quasi-Newton methods, or SR1, whereas our developments are only for BFGS. However, similar techniques could be applied to these alternative approaches. We will be more careful to caveat these points of distinction in the abstract and introduction so as to clarify the interpretation. Thank you for the suggestion.

Reviewer comment (Theory):
The paper claims an explicity dependence on memory. This dependence is hidden in a parameter $C_{\beta}$, and while looking at its definition, I can understand its relationship with the memory.

Author response:
We would like to thank the Reviewer for reviewing our manuscript and for the constructive feedback. We are glad to see the Reviewer appreciated the presentation and the results of the paper. In what follows, we provide our response and actions to your comments.

**Response to Weakness**: The Reviewer is correct that the minimal relative condition number $\beta_\tau$ (or the bound $C_\beta$) embeds the role of the memory size $\tau$ in the convergence rate. Specifically, a larger $\tau$ decreases the minimal relative condition number $\beta_{\tau}$, reduces the contraction factor $1-\mu/(\beta_{\tau}dL)$, and improves the convergence rate, but increases the storage memory and computational cost; hence, yielding an inevitable trade-off between these factors. 

On the other hand, we agree with the Reviewer that interesting topics for future research include deriving explicit dependency of $\beta_\tau$ (or $C_\beta$) on the memory size $\tau$. We consider this work as the first stride to explore the possibility of achieving superlinear rates with limited-memory BFGS methods and analyze the inherent role of the memory size on the convergence rate. We will clarify this point in the final upload. Thank you for the comment.

Reviewer comment (Theory):
This relationship is still not that explicit because it is not an explicit function of the memory.

Author response:
We would like to thank the Reviewer for reviewing our manuscript and for the constructive feedback. We are glad to see the Reviewer appreciated the presentation and the results of the paper. In what follows, we provide our response and actions to your comments.

**Response to Weakness**: The Reviewer is correct that the minimal relative condition number $\beta_\tau$ (or the bound $C_\beta$) embeds the role of the memory size $\tau$ in the convergence rate. Specifically, a larger $\tau$ decreases the minimal relative condition number $\beta_{\tau}$, reduces the contraction factor $1-\mu/(\beta_{\tau}dL)$, and improves the convergence rate, but increases the storage memory and computational cost; hence, yielding an inevitable trade-off between these factors. 

On the other hand, we agree with the Reviewer that interesting topics for future research include deriving explicit dependency of $\beta_\tau$ (or $C_\beta$) on the memory size $\tau$. We consider this work as the first stride to explore the possibility of achieving superlinear rates with limited-memory BFGS methods and analyze the inherent role of the memory size on the convergence rate. We will clarify this point in the final upload. Thank you for the comment.