Paper: Graph Neural Tangent Kernel and Graph Neural Network Gaussian Processes for Node Classification/ Regression (ICLR-2024)

Reviewer comment (Novelty):
The novelty and technical contribution is limited. GNTK has been studied in graph-level tasks, and also extended to node-level ones in some published papers that are neglected in this work, e.g. [1-2] (though they did not explicitly claim that as a contribution). While considering other architectures might be novel, there is no discussion on how they might help theoretical understandings or become practically useful (incorporating such discussions could greatly strengthen the paper).

Author response:
The GNTK formula in [1] is based on the paper "Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels".
[1] is explicitly calculating a two layer GNTK closed form expression in Lemma 2. We compare their expression to ours. Their expression:

Lemma 2: The explicit form of GNTK feature map for a two-layer GNN $\phi_{gnn}(x)$ with average aggregation and ReLU activation in node regression is:
$$
\phi_{gnn}(x_i) =c \sum_{j \in \mathcal N_i  \cup\{i\}}  \left[\mathbf{X}^{\top} a_j     \cdot \mathbb{I_{+}}    \left(w^{(k)^{\top}} \mathbf{X}^{\top} a_j \right), w^{(k)^{\top}} \mathbf{X}^{\top} a_j \cdot \mathbb{I_{+}}\left(w^{(k)^{\top}} \mathbf{X}^{\top} a_j \right), \ldots\right]
$$
where $c=O(\tilde{d}^{-1})$ is a constant proportional to the inverse of node degree, $\mathbf X\in \mathbb R^{n\times d}$ is node features, $a_i\in \mathbb R^{n}$ denotes adjacency vector of node $i$, $w^{(k)} \sim \mathcal{N}(\mathbf{0}, I_d)$ is a random Gaussian vector in $\mathbb R^d$, two components in the brackets repeat infinitely many times with $k$ ranging from $1$ to $\infty$, and $\mathbb{I}_{+}$ is an indicator function that outputs $1$ if the input is positive otherwise $0$.

Their NTK is $\Theta_{i,j} = NTK(x_{i}, x_{j}) = \phi_{gnn}(x_i)^{T} \phi_{gnn}(x_j)$

Now our NTK expression:
$$
\Theta^{L} =  A \left( \Lambda^{L-1} +  \left( \dot{\Lambda}^{L-1}  \odot \Theta^{L-1} \right) \right)  A^{T}
$$
$$
\Lambda^{L} = \sigma_{w}^{2} \mathbb{E} \left[ \sigma(u) \sigma(u^{T}) \right] +\sigma_{b}^{2}
$$
$$
\dot{\Lambda}^{L} = \sigma_{w}^{2}\mathbb{E} \left[ \dot{\sigma}(u) \dot{\sigma}(u^{T}) \right] +\sigma_{b}^{2} \text{ with } u \sim N(0, A\Lambda^{L-1}A^{T})
$$
$$
 \Theta^{1} = \Lambda^{0} = \sigma_{w}^{2} X^{T}X + \sigma_{b}^{2}
$$

Both have to be the same but our derivation is much cleaner and simpler to comprehend, dervied differently and we can make statements about psd of the kernel.
Proof of Lemma 2 has to take into account that it is a **two layer** GNN whereas our formula is recursive and for any number of layers.

Paper [1] requires the GNTK only, to prove some of their claims. They don't do extensive evaluation on any datasets.

The paper [2] Graph Neural Tangent Kernel: Convergence on Large Graphs is only concerned with the empirical NTK and **not** with the infinite width limit closed form expression.

Their are no simulations (on NTK Theory) on any paper mentioned, or to the best of our knowledge any paper existing, validating NTK Theory. We included simulations.

We added a Discussion Section explaining how these closed form expression might help in the future. Theoretical section at the end of GNN NTK proof, comments on the positive definiteness and the effect of the Graph Scaling to the Kernel matrix and its theoreticl implications.

Reviewer comment (Theory):
Theorem 3.2 seems incorrect. $A^\top$ should be outside the bracket in $\mathbf \Theta$; otherwise the kernel matrix is not symmetric and thus invalid. (This could be a typo rather than a technical flaw.)

Author response:
Was a typo is fixed

Reviewer comment (Experiments):
The method seems not scalable or efficient, though this is a common limitation for applying NTKs of other types of neural networks. Additionally, the datasets in this paper are not “large” (or even “medium”) as claimed in Section 4.1. For large datasets, the authors can consider those in [3].

Author response:
We removed any statements about the size of the dataset from the paper as your remark is correct.

Reviewer comment (Writing):
Many formatting issues and nonstandard notations impair the readability. Examples include abuse of capital letters in writing, unnumbered equations, line wraps in theorems (3.2, 3.3, 3.5), inconsistent precision for the Crocodile dataset, and many informal expressions throughout the paper.

Author response:
Changed most to small letters (Neural Net/ Graph Neural Net is left with capital letters). Added numbers in equation in the main part of the paper. The GAT*NTK spans 4 lines for readability. It is impossible to put it in one line. We tried reformulated large parts improving readability and using more scientific language.

Reviewer comment (Evaluation):
How exactly is the efficiency of KRR-based methods compared with standard GNNs?

Author response:
Added runtime and memory analysis in Appendix.