Paper: General Graph Random Features (ICLR-2024)

Reviewer comment (Novelty):
The proposed method is an extension of previous works, in particular, Choromansky (2023).

Author response:
**Difference from Choromanski (2023)**: The reviewer correctly notes that this work extends Choromanski's paper (https://arxiv.org/abs/2305.00156, ICML 2023, oral presentation). Whilst this previous work presents a random feature mechanism for unbiased approximation of the $d$-regularised Laplace kernel, our extension allows unbiased approximation of *any* function of a weighted adjacency matrix. It is therefore significantly more general; it contains Choromanski's scheme as a special case. Moreover, unlike its predecessor, our method also allows for implicit kernel learning via a neural modulation function. Hence, our work is substantially broader in scope.

Reviewer comment (Evaluation):
The experiments do not clearly show the price that one has to pay in terms of compute time when requesting better accuracy. Experiments show only relations between the number of walks and the accuracy/error. Linking accuracy and execution time (e.g., in standard machines) would have given a better sense of the relevance of the proposed methods.

Author response:
**Accuracy vs compute**: Our algorithm has the convenient feature that walkers are independent and straightforwardly distributed across machines. This means that the run time to construct features is directly proportional to the number of walkers, so the plots in e.g. Fig 2 would look identical if the $x$-axis variable were replaced with simulation time. Moreover, from Alg. 1 it is clear that the time-complexity to construct features is subquadratic so scales better than exact methods (cubic). That said, we agree that adding some wall-clock times to compare the exact and approximated speeds might be interesting for practitioners. We are measuring these and will add the results to the appendix.

Reviewer comment (Theory):
The kernels apply only to weighted graphs without node and edge attributes.

Author response:
**Classes of kernels being approximated**: The reviewer is correct that our algorithm can approximate arbitrary functions of a weighted adjacency matrix, which is not the same as any kernel. This is the sense in which they are 'universal', as we state in the first sentence of the abstract. We chose this name to disambiguate from 'graph random features' (https://arxiv.org/abs/2305.00156, ICML 2023) which can only approximate the $d$-regularised Laplacian function. We intended to be unambiguous and welcome suggestions to make our contribution clearer -- does the reviewer think 'general graph random features' would avoid confusion? We would be happy to make any reasonable changes to the name.

Reviewer comment (Theory):
The use of the term 'universal' is therefore misleading; as the authors say, the provided random features can approximate only kernels from a specific family.

Author response:
**Classes of kernels being approximated**: The reviewer is correct that our algorithm can approximate arbitrary functions of a weighted adjacency matrix, which is not the same as any kernel. This is the sense in which they are 'universal', as we state in the first sentence of the abstract. We chose this name to disambiguate from 'graph random features' (https://arxiv.org/abs/2305.00156, ICML 2023) which can only approximate the $d$-regularised Laplacian function. We intended to be unambiguous and welcome suggestions to make our contribution clearer -- does the reviewer think 'general graph random features' would avoid confusion? We would be happy to make any reasonable changes to the name.

Reviewer comment (Theory):
Alg. 1: shouldn't the normalization be $1/\sqrt{m}$ instead of $1/m$?

Author response:
**Normalisation factor**: the $\frac{1}{m}$ normalisation factor is actually correct since we are interested in the distribution over subwalks of the average walk. We invite the reviewer to read Sec. A.2 for more detail. This might seem strange compared to e.g. random Fourier features (https://people.eecs.berkeley.edu/~brecht/papers/07.rah.rec.nips.pdf, NIPS 2007) which are normalised by the square root of the feature length, but upon careful inspection this is indeed the right factor.

Reviewer comment (Presentation):
Fig. 3: please specify if the extremely narrow shaded area around the lines refers to +/- one standard deviation.

Author response:
**Shaded area and standard deviation**: The shaded area is indeed the standard deviation on the estimate of the mean, i.e. the standard deviation of one trial divided by the square root of the number of trials. We will clarify this in the manuscript. We will also clarify bracket notation for standard deviation on the last digit.

Reviewer comment (Reproducibility):
Sec. 3.4: it is claimed that the modulation function learned on one data set can be effectively applied to other graphs, referring to Tab. 3. However, I don't see specified how the table has been generated. Please provide more details.

Author response:
**Table 3 and kernel learning**: To produce Table 3, the modulation function was trained for better quality approximation on the smallest graph ('on the small Erdős-Rényi graph ($N = 20$) with $m = 16$ walks, we minimise the loss'). We then freeze the modulation function and test the quality of kernel approximation on the other bigger graphs. Even though $f$ was only trained on to be optimal on the smallest graph (top row of table), it also improves approximation quality for all the remaining graphs (remaining rows), even when $N$ is bigger by a factor of $60$. We will amend the text to make this important point clearer.

Reviewer comment (Theory):
I suggest discussing the impact of the fact that the sum in Eq. 2 is not always well-defined - an aspect that is only briefly mentioned in the paper.

Author response:
**When the kernel is not defined**: The reviewer is correct to observe that if the kernel diverges then we cannot hope to approximate it using finite random features. In the literature, this is avoided by multiplying the weighted adjacency matrix by a regulariser $\sigma^2$ which downscales its spectral radius. We will add further discussion to the manuscript.

Reviewer comment (Evaluation):
I suggest discussing the pros and cons of symmetric vs asymmetric approaches.

Author response:
**Symmetric vs asymmetric approaches**: We sincerely thank the reviewer for their interesting question. Eq. 4 encodes is the condition for an unbiased estimator; symmetric functions is the special case when we constrain $f_1 = f_2$. This has the benefit of requiring half as many parameters in the neural modulation functions, and empirically during training functions tend to become close to symmetric anyway (see Fig. 6). A detailed study of the effect of different choices among the possible modulation function pairs, including whether the symmetric case is best, is an open and exciting research direction. We will add this discussion and look forward to tackling these questions fully in the future.

Reviewer comment (Experiments):
The authors highlight several times in the paper that one major issue regarding in terms of computational complexity is to inverse the matrix K(W) (where W is the weighted adjacency matrix of the graph). However, in the numerical experiments presented in the paper, the authors do not propose an example of application of their method to inverse such matrix K. It might be relevant to add such experiment in the paper or at least to explain how the method can be used to efficiently compute the inverse.

Author response:
**Efficiency gains and the kernel inverse**: The reviewer correctly notes that the large cost of kernel inversion is a big motivating factor for constructing random feature mechanisms. In particular, in the presence of explicit random features for a kernel $K$ so that $K \simeq \phi^\top \phi$, we can avoid needing to compute its inverse $K^{-1}$. This is well-established in the random features literature. However, graph kernel methods have the additional challenge that even *computing* $K$ scales cubically with the number of nodes $N$. Our predominant focus is on overcoming this (using our sub-quadratic algorithm), with the understanding that the other benefits of explicitly-manifested features also follow. We will make this clearer in the manuscript.

Reviewer comment (Experiments):
The authors did a good job in providing several experiments to illustrate their method. Nevertheless, I think it would have been good to propose an experiment on real data that tackles one of the main applications mentioned in the introduction (such as community detection or recommender systems). In particular, it is still an ongoing research question to find the right representation of a graph to use spectral methods for community detection. Indeed, using the adjacency matrix is known to be suboptimal in particular when dealing with sparse graphs. That's why people have proposed to use the non-backtracking matrix or the Bethe-Hessian. Since the authors show the interesting possibility to learn the modularity function to automatically selects a graph representation well suited for some downstream task, I think that a very interesting experiment would be to use their method to automatically select a good kernel for community detection. In particular, does the algorithm converge to a representation close to the non backtracking matrix?

Author response:
**Bethe Hessian and community detection**: The reviewer rightly identifies one of the novel contributions of the paper as a scalable new technique for implicit kernel learning in random feature space. Though this only constitutes one short section (the paper also introduces an unbiased mechanism for kernel estimation and learns the modulation function to improve kernel approximation quality), we sincerely thank them for their interest in this exciting research direction. We agree that there is plenty of scope for interesting work, including their excellent suggestion to apply it to community detection problems and see whether the implicitly learned kernel approximates any popular choices from the literature. We are excited to explore this aspect to u-GRFs further, but for now must defer it as future work.

Reviewer comment (Experiments):
The authors do not discuss really how some parameters of the method are chosen in practice (such as the parameter $p_{halt}$), and how these choices influence the performance of the algorithm.

Author response:
**Choice of parameters**: When approximating a fixed graph kernel, the only free choice of parameter is $p_\text{halt}$. It encodes a simple trade-off between speed and approximation quality; if $p_\text{halt}$ is lower, walks last longer and sample more subwalks so give a better approximation of graph kernels. In practice, any small value works well (as also reported in https://arxiv.org/abs/2305.00156 which introduced the GRF mechanism -- ICML 2023) so we typically take $p_\text{halt} \sim 0.1$. We will add this discussion to the manuscript.

Reviewer comment (Experiments):
Real world graphs are known to be sparse. It would be interesting to comment of the influence on the algorithm when dealing with sparse graphs. A possibility would be to conduct simulated experiments by varying the sparsity level of the graph and see the impact of the performance.

Author response:
**Sparsity and performance**: We thank the reviewer for their interesting suggestions. We have included experiments on both synthetic and real-world graphs which naturally have a range of sparsities, and in all cases the performance is similar (see e.g. the plots in Fig. 2 or the clustering errors in Table 2 -- they do not change much between graphs). That said, we agree that it would be interesting to systematically investigate how approximation error changes with sparsity so we are running this extra experiment and will add the results to the Appendix.

Reviewer comment (Writing):
In Eq.(9), I think a notation is missing since $\alpha$ is not appearing in the mapping $x\mapsto w^{\top} \psi_{K}(x)$. I think the correction would be $x\mapsto w^{\top} \psi_{K_{\alpha}}(x)$.

Author response:
**Additional comments and typos**: We thank the reviewer for their observation about Eq. 9 and agree that their suggestion would make the notation clearer, so we have added an $\alpha$ subscript. We have also added the word 'for' to the sentence at the start of Sec. 3.1. In Eq. 14, $\boldsymbol{\phi}_j$ actually refers to the $N \times N$ random feature matrix used to approximate the particular time evolution operator $\exp(\mathbf{W}(t - \tau_j))$ (i.e. the diffusion kernel corresponding to one particular timestep indexed by $j$) rather than $j$ indexing a row. We will clarify this notation and thank the reviewer for flagging this.

Reviewer comment (Writing):
Between Eq.(13) and Eq.(14), I think there is an issue with the notations. Should $\Phi$ depend on $\tau_j$? Based on what I could understand, $\Phi$ is a rectangular matrix of size $n\times N$ where the $j$-th row is $\Phi_j = ((t-\tau_j) \phi(i))_{i=1}^N$.

Author response:
**Additional comments and typos**: We thank the reviewer for their observation about Eq. 9 and agree that their suggestion would make the notation clearer, so we have added an $\alpha$ subscript. We have also added the word 'for' to the sentence at the start of Sec. 3.1. In Eq. 14, $\boldsymbol{\phi}_j$ actually refers to the $N \times N$ random feature matrix used to approximate the particular time evolution operator $\exp(\mathbf{W}(t - \tau_j))$ (i.e. the diffusion kernel corresponding to one particular timestep indexed by $j$) rather than $j$ indexing a row. We will clarify this notation and thank the reviewer for flagging this.

Reviewer comment (Experiments):
The experiments are carried over small graphs. It would be nice to include some experiments on larger graphs, e.g. N=10,000. For example, fix the Erdos-Renyi model but vary the number of nodes. I wonder if and how the approximation error would scale with N, while keeping everything else fixed.

Author response:
1. **Graph size**: We consider graph node clustering with $N=3300$ and node attribute prediction with $N=4350$ which are already sufficiently large that exact methods, penalised by cubic time-complexity scaling, become slow (for clustering, by a factor of $> 5$ compared to u-GRFs). Nonetheless, we agree that it would be interesting to continue to push the methods to yet larger graphs. **Therefore, following the reviewer's suggestion, we also ran the node attribute prediction experiment on the 'cycloidal' mesh which has $21384$ nodes**. In this regime exact kernel evaluation is extremely slow, but once again u-GRFs perform well and the learned kernel (trained on the cylinder graph) does best. Specifically, the normalised differences in prediction error compared to the learned kernel are: $0.065(4)$, $0.129(6)$ and $0.011(1)$ for the $1$-regularised Laplacian, $2$-regularised Laplacian and diffusion kernels respectively. Indeed, throughout the work we see that performance is similar across graph size (see e.g. Fig. 2, where the plots are essentially identical even as $N$ differs by a factor of $60$). However, we will also add a plot to the appendix explicitly comparing number of $N$ and the kernel approximation error; we thank the reviewer for their excellent suggestion.

As a further update, we have now uploaded new versions of the main and supplementary pdfs. Additions and changes are indicated in red. The reviewer might be particularly interested to see Sec. A.6 of the Appendix, which includes a new plot of wall-clock time and approximation error vs the number of nodes in an Erdős-Rényi graph. As suggested, we scale graphs to as big as 12800 nodes where exact computation becomes prohibitively slow. Our method continues to perform well in this regime, with the relative Frobenius norm error between the true and approximated Gram matrices remaining very small. We have also added discussion comparing symmetric and asymmetric modulation functions and clarified the standard deviation notation. We warmly thank the reviewer for suggesting these interesting additions and important clarifications.

Reviewer comment (Evaluation):
For experiments on node clustering, Table 2 shows the difference between u-GRFs and the exact kernel. It would be nice to have another table that shows the actual clustering accuracy when compared against the ground-truth information, given the ground-truth number of clusters in each graph.

Author response:
2. **Clustering experiment and metric**: We are unsure what the reviewer means by 'groundtruth clusters' for these graphs, but will be happy to respond if they clarify. The pairwise clustering metric in Eq. 15 is a standard way to capture the quality of clustering (the accuracy becomes low if lots of pairs that should be in the same cluster are in different clusters or vice versa), but our scheme would perform well under any reasonable way of comparing the clusterings with the exact and approximated kernels.

Reviewer comment (Presentation):
If I understood correctly, the numbers in the parenthesis in Table 3 and Table 4 are standard deviations. Why are they integers?

Author response:
4. **Standard deviation notation**: the bracketed number gives the standard deviation of the final digit of a long number more compactly, e.g. $0.123(4)$ means $0.123 \pm 0.004$. We will clarify this notation in the manuscript.

As a further update, we have now uploaded new versions of the main and supplementary pdfs. Additions and changes are indicated in red. The reviewer might be particularly interested to see Sec. A.6 of the Appendix, which includes a new plot of wall-clock time and approximation error vs the number of nodes in an Erdős-Rényi graph. As suggested, we scale graphs to as big as 12800 nodes where exact computation becomes prohibitively slow. Our method continues to perform well in this regime, with the relative Frobenius norm error between the true and approximated Gram matrices remaining very small. We have also added discussion comparing symmetric and asymmetric modulation functions and clarified the standard deviation notation. We warmly thank the reviewer for suggesting these interesting additions and important clarifications.