Paper: The optimality of kernel classifiers in Sobolev space (ICLR-2024)

Reviewer comment (Writing):
Page 3: "Therefore, the framework of spectra algorithm" -> spectral algorithm. Similarly for the first sentence in Page 5.

Author response:
For Questions 1,2 and 4, we will correct them and re-organize the proofs in the final version.

Reviewer comment (Writing):
Page 5: "bounded domain with smooth boundary $\mathcal{X} \subseteq R^d$" should be $\mathcal{X} \subseteq \mathbb{R}^d$ here?

Author response:
For Questions 1,2 and 4, we will correct them and re-organize the proofs in the final version.

Reviewer comment (Writing):
I do not understand Assumption 3 and the notations therein, could the authors clarify this?

Author response:
Question 3: I do not understand Assumption 3 and the notations therein, could the authors clarify this?

Answer: Assumption 3 is corresponding to how a RKHS is compactly embedded into $L^{\infty}$-space. This directly implies that all the functions in $[\mathcal{H}]^{\alpha}$ are $\mu$-a.e bounded for $\alpha>\alpha_0$.

Reviewer comment (Presentation):
The proof of Theorem 1-2, Corollary 2, and the appendix should be re-organized for better readability.

Author response:
For Questions 1,2 and 4, we will correct them and re-organize the proofs in the final version.

Reviewer comment (Theory):
I do not understand the statement of "width m and the sample size n are sufficiently large" in Corollary 1: should Corollary 1 be understood as asymptotic statements as $m,n \to \infty$? I am a bit confused since in many DNN and NTK literature, one must have $m \gg n$ for the NTK to accurately approximate the DNN behavior. Also, can the results in Corollary 1 be compared to existing results in NTK kernel literature?

Author response:
Answer to Question 5: Generally speaking, [1] studying the neural network with NTK can be divided into the following steps. In the first step, we want to verify the time-varying Neural network kernel function uniformly converges to NTK kernel, where a large enough wide $m$ corresponding to sample size $n$ is needed,  such as $m \geq \operatorname{poly}\left(n, || y ||, \ln (1 / \delta)\right)$. Therefore, if the training process of the neural network uniformly converges to NTK kernel regression, we need the spectral properties of NTK, in other words, the eigenvalue decay rate. The last step is to analyze the kernel regression, where the spectral algorithm is considered because we can study not only ridge regression but also gradient descent and other training process.

NTK kernel provides a perspective of kernel methods to study the neural network. Although it is based on a wide neural network in which we need a large enough $m$ corresponding to $n$, we can study the generalization properties and understand the benefits of over-parameterization in the neural network. 

-----------------

## Rebuttal 2 (ID: e8pFk7TFYc)
Thank you for your comment and reminder.

Your understanding is correct. We have updated the revised version of the paper and included the clear condition of the width for Proposition 1 $m \geq poly(n, \lambda_{min}^{-1},  \ln (1 / \delta), \ln(1/\epsilon) )$, where $\lambda_{min}$ is the minimum eigenvalue of the kernel matrix, and for Corollary 1  $m \geq \text{poly}\left(n, \lambda_{min}^{-1}, \ln (1 / \delta) \right)$.

Reviewer comment (Theory):
I do not understand the connection between the min kernel discussed above Equation (17) and the NTK.

Author response:
Question 6:  I do not understand the connection between the min kernel discussed above Equation (17) and the NTK.

Answer:  The min kernel is an example to illustrate how to estimate $s$. The reason why we use the min kernel is that we know the eigenvalue and explicit eigenfunction of the min kernel on $\mathbb{R}$. Therefore, we can calculate the theoretical smoothness $s$ for given $f^*$ and the optimal rate. Then we compare them with our numeric results. After showing the way to estimate $s$ by using the min kernel, we then estimate $s$ of the real data with respect to the NTK kernels.

[1]Yicheng Li, Zixiong Yu, Guhan Chen, and Qian Lin. Statistical optimality of deep wide neural networks. arXiv preprint arXiv:2305.02657, 2023.

Reviewer comment (Experiments):
Question: Could the results be generalized to multi-class classification? There is also curiosity about other loss functions: the Brier score should be trivial and the log-loss should require additional modifications.

Author response:
Answer to Question (2): We think kernel methods with squared loss can be extended to multi-class classification for we can consider our label a $(s-1)$-dimensional vector, where $s$ is the number of classes. However, the minimax rate for multi-class classification is not clear by far, we only know that [2] obtained a minimax lower bound for VC-class, but it is unknown for RKHS. I think it is a good question for further work. 
    
For the loss function, we think that there is no clear evidence that cross-entropy loss can work better than the squared loss function. In contrast, analyzing the classification upper bound of logistic regression in statistics [3] requires adding some additional unnatural conditions.

Reviewer comment (Theory):
Major comment: Before Corollary 1 — While NTKs are dot-product kernels on the sphere, they are not in general dot-product kernels outside of the sphere. For example, from Eq. (15) we have $K_{ntk}(0, 0) = 1$, which means that $K_{ntk}$ cannot be a dot-product kernel. However, it is unclear why it would be needed to be a dot-product kernel for Corollary 1.

Author response:
Answer to Question (3): We apologize for the mistakes in the formulas of the neural network and the NTK. We consider the neural network without bias on all layers on $\mathbb{S}^{d-1}$. More specifically, we consider the neural network with $x，x'\in\mathbb{S}^{d-1}$:
    \begin{align}
    h^{1}(x) & = \sqrt{\frac{2}{m}} \sigma (Ax),\quad h^{l}(x)  = \sqrt{\frac{2}{m}}\sigma(W^{l-1}h^{l-1}(x)),\quad l=2,...,L \\\\
    f(x;\theta) & = W^{L}h^{L}(x)\quad \text{and} \quad \hat{y} = \operatorname{sign}(f(x;\theta)),
  \end{align}
  where $L\geq 2$, and the corresponding NTK
    \begin{equation}
K_{ntk}(x,x') = \sum_{r=0}^L \kappa^{(r)}_1(\bar{u}) \prod_s^{L-1}  \kappa_0(\kappa^{(s)}_1(\bar{u})) 
  \end{equation}

where $\bar{u} = \langle x,x'\rangle $. In this case, the NTK is the dot-product kernel. Assuming $x\in\mathbb{S}^{d-1}$, the embedding index of inner product kernels is $\alpha_0=1/\beta$. We will correct the formulas of the neural network and the NTK in the text and include more details on the embedding index in the appendix.

Reviewer comment (Theory):
Major comment: In Theorem 2, there seems to be an implicit assumption that $\alpha_0 = 1/\beta$, as suggested by the text before Eq. (14) in the appendix. This should be stated explicitly in the theorem. It is also unclear whether this holds in the setting of Corollary 1 for general domains. In the special case of the sphere, this could be fixed for very similar network architectures, since the RKHS of the NTK is known to be a Sobolev space (see, for example, the proof of Theorem G.5 in https://arxiv.org/abs/2305.14077).

Author response:
Answer to Question (4): For Sobolev kernels and dot-product kernels on the sphere, we can prove that $\alpha_0 = \frac{1}{\beta}$. That means $\alpha_0 = \frac{1}{\beta}$ for the NTK on the sphere. We will give more details in the appendix.

Reviewer comment (Theory):
Major comment: There is insufficient conviction in the proposed smoothness estimator. For example, if the target function $f^*: \mathbb{R}^2 \to \mathbb{R}$ has different degrees $s_1$ and $s_2$ of (Sobolev) smoothness in dimensions $1$ and $2$, eigenvalues would decay at different rates depending on the eigenfunction refinement direction. This would yield a plot as in Figure 1(a) with eigenvalues lying between two lines corresponding to $s_1$ and $s_2$, and linear regression would then estimate a smoothness between $s_1$ and $s_2$, though in interpolation space terms the function would still only lie in the space with lower smoothness. This could be checked by plotting the $p_j$ on the image datasets.

Author response:
Answer to Question (5):  The misspecified spectral algorithms (assuming $f^*_{\rho}\in[\mathcal{H}]^s$) have been studied since 2009 (e.g., [4,5,6,7,8]). However, to the best of our knowledge, there is barely any result on the estimation of the smoothness $s$. This paper is the first to propose the $s$ estimation problem and the more accurate $s$ estimation will be our future works.

There are three cases you have mentioned, the mixed smoothness case, the manifold case and the well-separated case. For the mixed smoothness case,  suppose $f^*_\rho= \sum_{j=1}^{\infty}f_je_j(x)$ and $f_j^2 = j^{-s_1\beta-1}$ if $j$ is odd and $f_j^2 = j^{-s_2\beta-1}$ if $j$ is even. Let's say $s_1>s_2$. Then for the kernel $K$ with eigenvalue decay rate (EDR) $\beta$, $f^*_\rho\in[\mathcal{H}]^{s_2}$ instead of $[\mathcal{H}]^{s_1}$ or $[\mathcal{H}]^{s}$ for some $s\in(s_2,s_1)$, meaning that the generalization error only depends on the 'upper line'. Thus, our theorem shows that we should estimate the slope of the 'upper line'. As for the $s$ estimation, in this case, we can estimate $s$ like your idea, performing linear regression on top of $\hat{p}_j$. The estimation indeed becomes more susceptible to noise in this case. However, our experiments show that our estimation becomes more accurate as the sample size $n$ increases. We will try to find more accurate $s$ estimation methods in the near future.

We have to admit that our theories need more effort to support the rest two cases. For the manifold case, so far we can not put the information on the submanifold into our theories. However, we speculate that compared with the data with $\mu$ supported on the original manifold, the data with $\mu$ supported on the submanifold has larger $s$ since the information on $f^*$ is more concentrated, meaning that the projection of $f^*$ might be on the top eigenfunction (the low-frequency functions). Thus, the estimation of $s$ might be used in the manifold case.

Reviewer comment (Evaluation):
Potential fix for the smoothness estimator issue: Perform linear regression on $\tilde p_j := \sup_{k \geq j} p_k$ to filter out the “upper line” in the plot, though this could be more noise-sensitive. A heuristic might be to automatically set the threshold so that only $p_j$ values larger than the maximum in the right half of the plot are included.

Author response:
Answer to Question (5):  The misspecified spectral algorithms (assuming $f^*_{\rho}\in[\mathcal{H}]^s$) have been studied since 2009 (e.g., [4,5,6,7,8]). However, to the best of our knowledge, there is barely any result on the estimation of the smoothness $s$. This paper is the first to propose the $s$ estimation problem and the more accurate $s$ estimation will be our future works.

There are three cases you have mentioned, the mixed smoothness case, the manifold case and the well-separated case. For the mixed smoothness case,  suppose $f^*_\rho= \sum_{j=1}^{\infty}f_je_j(x)$ and $f_j^2 = j^{-s_1\beta-1}$ if $j$ is odd and $f_j^2 = j^{-s_2\beta-1}$ if $j$ is even. Let's say $s_1>s_2$. Then for the kernel $K$ with eigenvalue decay rate (EDR) $\beta$, $f^*_\rho\in[\mathcal{H}]^{s_2}$ instead of $[\mathcal{H}]^{s_1}$ or $[\mathcal{H}]^{s}$ for some $s\in(s_2,s_1)$, meaning that the generalization error only depends on the 'upper line'. Thus, our theorem shows that we should estimate the slope of the 'upper line'. As for the $s$ estimation, in this case, we can estimate $s$ like your idea, performing linear regression on top of $\hat{p}_j$. The estimation indeed becomes more susceptible to noise in this case. However, our experiments show that our estimation becomes more accurate as the sample size $n$ increases. We will try to find more accurate $s$ estimation methods in the near future.

We have to admit that our theories need more effort to support the rest two cases. For the manifold case, so far we can not put the information on the submanifold into our theories. However, we speculate that compared with the data with $\mu$ supported on the original manifold, the data with $\mu$ supported on the submanifold has larger $s$ since the information on $f^*$ is more concentrated, meaning that the projection of $f^*$ might be on the top eigenfunction (the low-frequency functions). Thus, the estimation of $s$ might be used in the manifold case.

Reviewer comment (Theory):
Alternative idea for the estimator: Stay closer to the definition of interpolation spaces and work with (approximations of) the interpolation space norms of the regression function, possibly finding an “elbow” in $s \mapsto y^\top (K + \lambda I)^{-1} K^s (K + \lambda I)^{-1} y$.

Author response:
$s$ estimation: Suppose $f_j=j^{-r}$ and $\lambda_j=j^{-\beta}$. By the definition of the norm of the interpolation space, 
\begin{equation}
\sum_{j=1}^{\infty} \frac{f_j^2}{\lambda_j^s} = \sum_{j=1}^{\infty} \frac{j^{-2r}}{j^{-s\beta}} = \sum_{j=1}^{\infty} j^{s\beta-2r}<\infty
\end{equation}
That means if $||f^*||_{[\mathcal{H}]^s}<\infty$, we need $s\beta-2r< -1$, i.e., $s<\frac{2r-1}{\beta}$. $s$ can be arbitrarily close to $\frac{2r-1}{\beta}$.

For the simulation, $\beta$ is taken from theory since it can be exactly calculated. For the real data, since $d$ is large and the $\beta$ is near 1 no matter from theory or the estimation with data. Thus, $\beta$ is taken from theory in our paper.

Reviewer comment (Novelty):
Not very clear how the arrived results compare against the existing work on NTK. Would be interesting to understand that to appreciate the impact of the contributions.

Author response:
The most relevant result is Theorem 3.1 in [1], showing that if $f\in H_{ntk}$, then the upper bound of the generalization error rate is $n^{-\frac{\beta}{2(\beta+1)}}$ ($\kappa=0$ for our case in [1]).  However, we emphasize that $[H_{ntk}]^1=H_{ntk}$ and the assumption $f\in H_{ntk}$ is too strong. The experiments in Section 5 of our paper show that this assumption fails on the real data ($s<1$). However, our assumption $f\in[H_{ntk}]^s$  is so weak that even for small $s$, we show the upper bound of the generalization error $n^{-\frac{s\beta}{2(s\beta+1)}}$ and it matches the minimax lower bound. Our results are more general and more practical since the smoothness of the real data can be estimated (Section 5).

Reviewer comment (Experiments):
It is not very clear about the impact of this work from an application front. The experiments to real datasets are restricted to estimation of the smoothness. Some insights would be helpful in this regard. How to enforce learning estimators constrained with a requirement on the smoothness.

Author response:
As mentioned before, we can estimate the smoothness of the real data with respect to the kernel. That means that given the real data, we can do the kernel selection for the real data with larger smoothness. For example, [2] showed that the decay rate of NTK with any depth is equal to $\beta=d/(d-1)$. Then we can test the smoothness of the real data with respect to NTKs with different depths and then find the best NTK. For the kernels with different $\beta$, we can try to find the kernel with the highest $s\beta$.