Paper: On the Positive Definiteness of the Neural Tangent Kernel (ICLR-2024)

Reviewer comment (Theory):
In recent years, many works have pointed out the limitations of NTK and infinite-width neural networks. To apply the NTK theory, it may be necessary to assume that the width should be \( m = \Omega(n^4) \), which is too impractical. Therefore, this paper may only have a limited broader impact on the deep learning theory community.

Author response:
It is also true, as the reviewer points out, that there are several shortcomings of considering infinitely wide networks in which the NTK is constant. This is therefore a valid critique, but in our defense, the NTK continues to be a fundamental tool in understanding wide neural networks.

Reviewer comment (Novelty):
The contribution made in this paper is incremental. Compared to the previous work (Du et al. 2019), it only improves the condition a little. Besides, I agree proving the positive definiteness of the NTK is an interesting question but I don't think it's significant enough for ICLR.

Author response:
(1) There have been several works which have attempted to establish the positivity of the NTK. However, none have so far been able to completely eradicate unnecessary and quite restrictive hypothesis such as that the data points lie in a sphere (which is never the case in practice) or that the activation function is analytic (which for example fails for the ReLu). Our paper completely solves the question of establishing the positivity of the NTK by making no unnecessary hypothesis and proving such positivity in full generality. %Without this contribution the literature would be incomplete and at an unsatisfactory stage.

Reviewer comment (Theory):
Theorem 1 is wrong. If two data samples are the same, then the NTK is not positive definite. I think the authors missed the condition that data samples are different.

Author response:
(2) It was stated in the preamble to Theorem 1, when defining positive definite Kernel, that we will always consider a data set without repeated points. To further emphasize it, we have now abstracted this into Definition 1 and make reference to it in the statement of Theorem 1.

Reviewer comment (Theory):
In the proof of Theorem 4, it says ''with the $z_i$ pairwise distinct and $\beta \neq 0$, in the view of Theorem 3...". However, Theorem 3 requires totally non-aligned where being pairwise distinct is not sufficient.

Author response:
(4) In the scope of Theorem 4, meaning with biases (equivalently $\beta \neq 0$) the $z_i$ being all distinct is equivalent to $z=(z_1,\ldots , z_N)$ being totally non-aligned with the bias vector (as was stated in Lemma 1). Hence, we can invoke Theorem 3 at this stage.

Reviewer comment (Theory):
I didn't follow the proof of Proposition 1. ''under the previous circumstances, $\sigma$ must be a constant.'' I am unclear about the mentioned circumstances and how to see that $\sigma$ is a constant.

Author response:
The words "under the previous circumstances" simply refer to the hypothesis of the Theorem which have already been used. At this stage, the fact that $\sigma$ is constant follows immediately from the formula $\sigma(x)=-\sigma(0) \sum_{i=1}^{N-1}\frac{u_i}{u_N}$ written in the line immediately above (in the paper).