Paper: FORKS: Fast Second-Order Online Kernel Learning using Incremental Sketching (ICLR-2024)

Reviewer comment (Writing):
In the discussion of the work of Calandriello et al. it says ‘rendering a significant cost of updates’. I didn’t follow this? Why is this? Isn’t the goal of that line of work exactly to make the cost scale linearly in the number of data points seen?

Author response:
Thank you to the reviewers for their valuable suggestions on TISVD. It is important to clarify that our article primarily focuses on enhancing existing second-order online kernel learning methods. Our emphasis is on both the regret guarantee of the algorithm and its performance, particularly on large-scale datasets.  
In reference to Calandriello's work, which involves maintaining an approximate kernel matrix with a size of $B$ through online sampling, it is noteworthy that its feature mapping dimension is also $B$. This characteristic results in a quadratic order of second-order updates, influencing the update cost. Additionally, it is essential to highlight that once the approximate kernel matrix reaches the specified budget, its feature mapping remains unchanged, rendering it susceptible to distribution drift.

Reviewer comment (Writing):
In abstract and intro it is not clear what ‘the budget’ is. Relatedly I don’t understand why the work of Calandriello has a quadratic scaling. Can this be made more concrete?

Author response:
The concept of a "budget" is widely employed in online kernel learning. Given the escalating size of the kernel matrix over time, leading to uncontrollable time complexity, setting a sufficient budget becomes imperative to constrain the matrix size. Reference [1] presents a table detailing update times, and we believe that Table 1 effectively illustrates the comparison of update times and regrets across various online kernel learning algorithms.  
[1]  Jing Lu, Steven CH Hoi, Jialei Wang, Peilin Zhao, and Zhi-Yong Liu. Large scale online kernel learning. Journal of Machine Learning Research, 17(47):1, 2016.

Reviewer comment (Theory):
In Theorem 1, the coherence is undefined. This has multiple different definitions in the literature so needs to be defined.

Author response:
Yes, we will make modifications to Theorem 1. Thank you for your suggestions.

Reviewer comment (Theory):
Assumption 3 is badly written: (1) it is written as a definition while it is an assumption about S_p, and (2) it contains a big typo — the equation as presented seems nonsensical. The reviewer assumes that the event on which the probability is computed should be $\|  B_iA_i  - B_I S_p S_p^T A_i \|^2 > Constant \| A_i \|^2 \|B_I \|^2$ with the right notations.

Author response:
Regarding assumption 3, which is a typo, we will modify it as follows:

$\mathrm{Pr} (  \|  B_i  A_i -  B_i {S_p} { S_p}^{\top}  A_i   \|_{F}^2 > 2\| || B_i ||_F^2 || A_i ||_F^2 / (\delta_i s_p)  ) \leq \delta_i$

Reviewer comment (Theory):
Revisit Assumption 3, confirm the typo, and propose a clean version of it.

Author response:
Regarding assumption 3, which is a typo, we will modify it as follows:

$\mathrm{Pr} (  \|  B_i  A_i -  B_i {S_p} { S_p}^{\top}  A_i   \|_{F}^2 > 2\| || B_i ||_F^2 || A_i ||_F^2 / (\delta_i s_p)  ) \leq \delta_i$

Reviewer comment (Theory):
This paper does not seems to consider the approximation error caused by truncated SVD.

Author response:
We agree that the the approximation error caused by truncated SVD should be considered in the theory, We will make it the focus of the modification.