Paper: Efficient Continual Finite-Sum Minimization (ICLR-2024)

Reviewer comment (Theory):
The main weakness of this paper is the strong convexity assumption of f, which is considered an old-school assumption because nowadays deep neural networks are typically used to fit data, and these models are almost never strongly convex.

Author response:
Thank you very much for all the work and the positive feedback on our work.

We just want to mention that the primary scope of our work was to formally frame and investigate the problem of efficient updates of models upon the arrival of new data. We agree with the reviewer that going beyond the strong convexity assumption is a very important and interesting research direction. Investigating the strong convex case serves as an important first step towards this direction while it already tackles several challenges and reveals surprising properties (especially with respect to lower bounds).

We would also like to remark that we have added additional experimental evaluations for NNs (see Appendix K and our response to Reviewer jqUD and Reviewer s2LS) revealing interesting results for SIOPT-Grad. 

We hope that our work will instantiate a research direction towards investigating efficient update methods with favorable empirical and theoretical performance.

Reviewer comment (Experiments):
Variance reduced techniques, such as those used in the proposed method, are not very useful for neural networks, making the method potentially impractical for real-world models.

Author response:
Thank you very much for all the work and the positive feedback on our work.

We just want to mention that the primary scope of our work was to formally frame and investigate the problem of efficient updates of models upon the arrival of new data. We agree with the reviewer that going beyond the strong convexity assumption is a very important and interesting research direction. Investigating the strong convex case serves as an important first step towards this direction while it already tackles several challenges and reveals surprising properties (especially with respect to lower bounds).

We would also like to remark that we have added additional experimental evaluations for NNs (see Appendix K and our response to Reviewer jqUD and Reviewer s2LS) revealing interesting results for SIOPT-Grad. 

We hope that our work will instantiate a research direction towards investigating efficient update methods with favorable empirical and theoretical performance.

Reviewer comment (Writing):
Additional "the" in the second line of the first paragraph of section 3.1.

Author response:
Thank you for your work and the positive feedback on our paper. We will incorporate all the minor issues and thank you for spotting them.

Reviewer comment (Writing):
The VR is never defined. Suggestion: "variance reduction(VR)" and then use VR afterwards.

Author response:
Thank you for your work and the positive feedback on our paper. We will incorporate all the minor issues and thank you for spotting them.

Reviewer comment (Evaluation):
The solution proposed in this paper, although is better in the number of FOs, may result in much added overall running time. Is there any practical regime where the number of FOs is expensive, compared to the efficiency of the algorithm measured by the running time?

Author response:
Concerning your question: We believe that considering *projection-free methods* that even with higher FO complexity may result in better running time due to the lack of a projection step, is a very interesting research direction.

Reviewer comment (Experiments):
Experiments are only conducted in simple regression settings. Dataset statistics (e.g. the number of points) are not shown. It is unclear how exactly you execute the baselines.

Author response:
- *Concerning our experimental evaluations*

  We first want to remark that our work is mainly theoretical. Thus in our experiments we chose settings matching the requirements of our theoretical results. We chose ridge regression at which the exact optimal solution can be analytically computed so as to be able to precisely track the suboptimality of each baseline.

  In our experimental evaluation, we reveal a new data point at each stage $i$ that appear in the $x$-axis of our plots. All the baselines are implemented according to the presented algorithms in the respective papers and the parameters used are presented in Appendix I.

  We remark that in Appendix K we have added additional experimental evaluations for $2$-layer Neural Networks in the continual learning example on the MNIST data set that we described above, which illustrate the advantages of SIOPT-Grad.

Reviewer comment (Writing):
Commas are missing in many places.

Author response:
- *Questions* 

     - Thank you for the suggestion. We will fix the missing commas.

Reviewer comment (Writing):
Page 2: “for Problem 1 using \( O(1/\epsilon) \).” - “FOs” is missing.

Author response:
- *Questions* 

     - Thank you for spotting the typo.