Paper: LINK PREDICTION USING NEUMANN EIGENVALUES (ICLR-2024)

Reviewer comment (Presentation):
The language and presentation of the paper could be further improved, particularly in figures and visual representations. (e.g. Figure 2).

Author response:
We highly appreciate your time and effort in reviewing our paper and providing excellent questions and feedback to improve our work further. We updated the paper to improve on the presentation issues.

Reviewer comment (Theory):
While the paper is designed for link prediction tasks, the motivation behind applying LLwLC in link prediction is not well-explained, especially given that the propagation process appears to be a learnable spectral graph filter. It would be helpful to clarify why LLwLC is particularly suited for link prediction tasks.

Author response:
Theoretically, we propose a new spectral GNN that can be applied to many problem settings. Essentially, it is an eigenbasis capable of encoding the linear constraints, which improves the expressivity.  We focus on the link prediction task, which mainly its success depends on counting the sub-structures and triangles, which can be resolved by discriminating between the k-regular graphs, (what LLwLC is capable of doing it). We expect that LLwLC performs well in other tasks, but we did not conduct such experiments.

Reviewer comment (Theory):
Though the authors have demonstrated that the expressiveness LLwLC propagation is superior to MPNNs on k-regular graphs, they are still a limited subset of all the 1-WL isomorphism graphs.

Author response:
The sufficient conditions under which LLwLC can solve graph isomorphism entails that LLwLC is a universal approximator of functions defined on graphs [1]. Given that we can encode any subgraph into our eigenbasis, we can examine whether there exists a specific substructure collection that can completely characterise each graph. By the reconstruction conjecture we know that if we have all the $n − 1$ vertex-deleted subgraphs, then we can reconstruct the graph. Consequently, if the reconstruction conjecture holds and the substructure collection contains all graphs of size $k = n-1$ then LLwLC can distinguish all non-isomorphic graphs of size $n$ and is therefore universal. Based on [2], almost every graph has reconstruction number three. This is consistent with our experimental results, where we observed that certain small substructures such as Neumann constraints, significantly improve the results.   
    

[1] Chen, Zhengdao and Villar, Soledad and Chen, Lei and Bruna, Joan, "On the Equivalence between Graph Isomorphism Testing and Function Approximation with GNNs.", NeurIPS, 2019.  
[2] Bollobas, Bela, "Almost Every Graph Has Reconstruction Number Three.", Journal of Graph Theory, 1990.

Reviewer comment (Experiments):
What motivated the application of LLwLC in link prediction tasks? How does the model's performance compare in other node-wise downstream tasks, such as node classification?

Author response:
Theoretically, we propose a new spectral GNN that can be applied to many problem settings. Essentially, it is an eigenbasis capable of encoding the linear constraints, which improves the expressivity.  We focus on the link prediction task, which mainly its success depends on counting the sub-structures and triangles, which can be resolved by discriminating between the k-regular graphs, (what LLwLC is capable of doing it). We expect that LLwLC performs well in other tasks, but we did not conduct such experiments.

Reviewer comment (Theory):
Can you proof the superiority of LLwLC over MPNNs on a wider range of 1-WL isomorphism graphs?

Author response:
The sufficient conditions under which LLwLC can solve graph isomorphism entails that LLwLC is a universal approximator of functions defined on graphs [1]. Given that we can encode any subgraph into our eigenbasis, we can examine whether there exists a specific substructure collection that can completely characterise each graph. By the reconstruction conjecture we know that if we have all the $n − 1$ vertex-deleted subgraphs, then we can reconstruct the graph. Consequently, if the reconstruction conjecture holds and the substructure collection contains all graphs of size $k = n-1$ then LLwLC can distinguish all non-isomorphic graphs of size $n$ and is therefore universal. Based on [2], almost every graph has reconstruction number three. This is consistent with our experimental results, where we observed that certain small substructures such as Neumann constraints, significantly improve the results.   
    

[1] Chen, Zhengdao and Villar, Soledad and Chen, Lei and Bruna, Joan, "On the Equivalence between Graph Isomorphism Testing and Function Approximation with GNNs.", NeurIPS, 2019.  
[2] Bollobas, Bela, "Almost Every Graph Has Reconstruction Number Three.", Journal of Graph Theory, 1990.

Reviewer comment (Theory):
Can you provide information about the computational complexity of the proposed LLwLC algorithm?

Author response:
The time complexity of extracting the subgraphs is at most O(n) (though as  
we discussed we can do it with only three subgraphs). The time complexity of our method, given the sparsity of graph Laplacian matrix, is $\mathcal{O}(\kappa E)+ \mathcal{O}(k^2n)$ for the outer loop (Lanczos algorithm) and the QR factorization, respectively, where $\kappa \ll n$ is the number of computed eigenvectors and $k \ll n$ is the number of linear constraints. Thus, we are linear w.r.t number of nodes.

Reviewer comment (Writing):
The paper is not very clear. I had to repeat many sentences several times in order to understand what the authors aimed to state. The language also needs improvements.

Author response:
We highly appreciate your time in reviewing our paper and providing constructive feedback to improve our work further. We updated the paper to improve on the presentation issues.

Reviewer comment (Novelty):
The new algorithm seems to be a relative minor extension, especially the parts related to Lanczos and its theoretical analysis. What is the novel theoretical analysis? Most results are standard from what I know in the subject. Please elaborate more on these two fronts.

Author response:
We provide a new spectral GNN which encodes the subgraph structures into the eigenbasis. We encode relations between a "set of nodes/edges": we write each subgraph as a linear constraint, formulate it as an eigenvalue problem with linear constraints, and compute the eigenvectors accordingly. Encoding link representation (the same as the first constraint in Neumann constraints in Eq. 3 of the paper) in graph-structured data gives the power to count triangles and substructures. The sufficient conditions under which LLwLC can solve graph isomorphism entails that LLwLC is a universal approximator of functions defined on graphs [1]. Given that we can encode any subgraph into our eigenbasis, we can examine whether there exists a specific substructure collection that can completely characterise each graph. By the reconstruction conjecture[2], we know that if we have all the $n − 1$ vertex deleted subgraphs, then we can reconstruct the graph. Consequently, if the reconstruction conjecture holds and the substructure collection contains all graphs of size $k = n-1$, then LLwLC can distinguish all non-isomorphic graphs of size $n$ and is therefore universal. Based on [3], almost every graph has reconstruction number three. This is consistent with our experimental results, where we observed that certain small substructures such as Neumann constraints (or in the other ablation study with only ten vertex-deleted subgraphs) significantly improve the results. While, in theory, LLwLC can be applied to many problem settings, we focus on the challenging link prediction task. 

[1] Chen, Zhengdao and Villar, Soledad and Chen, Lei and Bruna, Joan, "On the Equivalence between Graph Isomorphism Testing and Function Approximation with GNNs.", NeurIPS, 2019.  
[2] Ulam, Stanislaw M, "A Collection of Mathematical Problems", 1960.  
[3] Bollobas, Bela, "Almost Every Graph Has Reconstruction Number Three.", Journal of Graph Theory, 1990.

Reviewer comment (Theory):
"Proposition. If we start simple Lanczos with ν ∈ N (C⊤), then qj ∈ N (C⊤) for all j." This is true only in theory as in practice round-off errors re-introduce the deflated direction. This is well-known in numerical linear algebra and Lanczos is very rarely used without some form of restart precisely for this reason.

Author response:
Thank you for the comment. As you mentioned in Q1, the round-off errors might raise numerical issues, which must be addressed. Actually, the numerical round-off errors are studied in sections 4.2 and 4.3and  we prove that our proposed LLwLC has a convergence (numerical round-off errors are analysed here).

In section 4.2 (section 3.3 in the revision), we theoretically analyze the effect of numerical roundoff errors, which lead to a perturbed tridiagonal matrix. Theorem 1 shows the error bounds of the computed perturbed tridiagonal matrix.  
In section 4.3 (section 3.3 in the revision), we utilize Greenbaum’s results, which mention that for our computed perturbed Lanczos algorithm exists, an exact Lanczos algorithm but for a different matrix.  
Also, Theorem 2 cognize the upper bound of the low-rank approximator of the Lanczos algorithm.

So, based on these three facts, we conclude that LLwLC has convergence, and the effect of practical round-off errors in the LLwLC is accurately investigated.

Reviewer comment (Presentation):
In Figure 2, purple box: you mention "svd(T)" but it is clear you mean "eig(T)".

Author response:
In this step, we decompose the tridiagonal matrix $\mathbf{T}$ using eigendecomposition.

Reviewer comment (Theory):
The statement about the orthogonal projector is wrong. C(C^TC)^{-1} holds for the case of independent rows in C; instead, in your case you have linearly independent columns thus it should be (C^TC)^{-1}C^T. You can verify then that P=I-C(C^TC)^{-1}C^T is what you want, and not P=I-CC(C^TC)^{-1} which the math indicate in your paper (the multiplication CC is not even defined).

Author response:
Thank you for the detailed correction.

Reviewer comment (Writing):
The keyword 'Neumann' keeps appearing only to be explained on page 7.

Author response:
We revised the introduction to clarify that the only necessary constraint for LLwLC is that matrix C be full column rank. So, we can encode any linear constraint in our matrix. One possibility which is quite useful is the Neumann constraints, in which the first constraint is the link representation between the links in one-hop and two-hop away nodes (link representation), and the second constraint is the one-hop away subgraph.

Reviewer comment (Writing):
The writing of this paper is poor. For example, I cannot understand the motivation of this article from the introduction section. On the contrary, there is too much content about existing work in the introduction. Still, I cannot see the connection between these existing methods and the proposed method from this section.

Author response:
In the introduction, we summarized the MPNN expressivity limitations, mainly their inability to count cycles and the three main directions in the literature to address this problem. Then, we propose our novel spectral GCN, which explicitly utilizes the subgraph structures in eigenspaces to mitigate the existing limitations of GNN expressivity specifically for the task of link prediction where we can accurately show the effectiveness of our LLwLC (as discussed in Q2). We could shorten the discussion of the limitations if the reviewer thinks it would improve the clarity.

Reviewer comment (Novelty):
What is the relationship between LLwLC and link prediction? LLwLC is only designed as a matrix factorization method for GNN.

Author response:
We provide a new spectral GNN which encodes the subgraph structures into the eigenbasis. We encode relations between a "set of nodes/edges": we write each subgraph as a linear constraint, formulate it as an eigenvalue problem with linear constraints and compute the eigenvectors accordingly. While, in theory, LLwLC can be applied to many problem settings, we focus on the challenging link prediction task, which mainly its success depends on counting the sub-structures and triangles, which can be resolved by discriminating between the k-regular graphs (what LLwLC is capable of doing it). (Also, the expressivity power of LLwLC is discussed in Q4).

Reviewer comment (Reproducibility):
This paper states that LLwLC has improved the efficiency of encoding link representations, but no specific form of link representation is mentioned in Section 4.

Author response:
The first constraint of the Neumann constraint is the link representation constraint.

Reviewer comment (Evaluation):
The conclusion of this article is not clear and requires a comparison between different methods, such as NBFNet and Seal, in terms of their link representation approach and complexity.

Author response:
The main difference between our work and SEAL & NBFNet is that we consider the relation between "a set of nodes (an induced subgraph encoded as a linear constraint) or link representations between a group of nodes", which leads to more expressivity compared to MPNNs (Specifically, encoding link representation (the same as the first constraint in Neumann constraints in Eq. 3 of the paper) in graph-structured data gives the power to count triangles and substructures). The time complexity of LLwLC, given the sparsity of the graph Laplacian matrix, is in average linear w.r.t number of nodes. The more important comparison is concerning the expressivity of LLwLC.  The sufficient conditions under which LLwLC can solve graph isomorphism entails that LLwLC is a universal approximator of functions defined on graphs [1]. Given that we can encode any subgraph into our eigenbasis, we can examine whether there exists a specific substructure collection that can completely characterise each graph. By the reconstruction conjecture[2], we know that we can reconstruct the graph if we have all the $n − 1$ vertex deleted subgraphs. Consequently, if the reconstruction conjecture holds and the substructure collection contains all graphs of size $k = n-1$, then LLwLC can distinguish all non-isomorphic graphs of size $n$ and is therefore universal. Based on [3], almost every graph has reconstruction number three. This is consistent with our experimental results, where we observed that certain small substructures such as Neumann constraints (or in the other ablation study with only 10 vertex-deleted subgraphs) significantly improve the results. 

[1] Chen, Zhengdao and Villar, Soledad and Chen, Lei and Bruna, Joan, "On the Equivalence between Graph Isomorphism Testing and Function Approximation with GNNs.", NeurIPS, 2019.    
[2] Ulam, Stanislaw M, "A Collection of Mathematical Problems", 1960.      
[3] Bollobas, Bela, "Almost Every Graph Has Reconstruction Number Three.", Journal of Graph Theory, 1990.

Reviewer comment (Experiments):
It is better to compare more methods on Ogbl-Collab. Currently, only LanczosNet is compared.

Author response:
Table 1 of the original paper actually contains the comparison to other methods. We just separated LanczosNet and LLwLC to show the effect of our proposed method in direct comparison to LanczosNet.

Reviewer comment (Experiments):
The benefit of introducing the proposed Neumann feature is debatable for link prediction. It shows that the node automorphism issue is addressed, but the model still depends on instance features for more expressiveness. Particularly, SubGNNs with simple features of low computation cost (e.g. zero-one labeling or DRNL) are already good enough, and the ablation study does not exactly separate the contribution between them.

Author response:
We introduce a novel eigenbasis that addresses the significant limitations of MPNNs. So, our introduction mainly focuses on the limitations of MPNNs in general and how our proposed eigenbasis can address them. We do experiments on LP tasks because their success depends on counting the sub-structures and triangles, which can be resolved by encoding linear constraints into our eigenbasis and discriminating between the k-regular graphs. Also, as discussed in the introduction, the major limitation of previous subgraph-based link prediction is just leveraging oversimplified pairwise node representation features. DRNL, as you mentioned, strictly provides the distance relation "just" between the query nodes that we want to predict the link for and the other individual nodes. No more information is provided by the DRNL, specifically regarding the relation between "other nodes in the graph with each other" or "any set of nodes" (any linear constraints between nodes/edges), which leads to more expressivity (This is what DRNL is missing).
On the other hand, LLwLC, given encoding "relation between a set of nodes/edges" in the input graph by encoding it as a linear constraint to the eigenbasis, can address this issue with time complexity linear to the number of nodes. (However, I agree with the reviewer that, given SEAL utilizing DRNL, it can provide oversimplified distance node representation. However, as I mentioned, it does not encode the relation between the graph's set of nodes/edges. So, LLwLC has stronger expressivity power than any labeling method by providing this information in its constraint matrix).

Reviewer comment (Evaluation):
LLwLC is claimed to be lightweight and efficient, however, neither the theoretical complexity nor the wall-clock time is provided. Meanwhile, the proposed model is subgraph-based, which still suffers from the computation overhead of subgraph extraction and raises concerns over its scalability on larger graphs (other OGB LP benchmark datasets).

Author response:
The time complexity of our method, given the sparsity of graph Laplacian matrix, is $\mathcal{O}(\kappa E)+ \mathcal{O}(k^2n)$ for the outer loop (Lanczos) and the QR factorization, respectively, where $\kappa \ll n$ is the number of computed eigenvectors and $k \ll n$ is the number of linear constraints.

Reviewer comment (Theory):
Is the proposed Neumann features only beneficial for the node automorphism issue? Can it also address the limitation of MPNN in counting sub-structures and triangles (other than the expressiveness inherited from subgraph-based models)?

Author response:
We encode relations between a "set of nodes/edges": we write each subgraph as a linear constraint and formulate it as an eigenvalue problem with linear constraints, and compute the eigenvectors accordingly. Encoding link representation (the same as the first constraint in Neumann constraints in Eq. 3 of the paper) in graph-structured data gives the power to count triangles and substructures.  The sufficient conditions under which LLwLC can solve graph isomorphism entails that LLwLC is a universal approximator of functions defined on graphs [1]. Given that we can encode any subgraph into our eigenbasis, we can examine whether a specific substructure collection can completely characterise each graph. By the reconstruction conjecture[2], we know that if we have all the $n − 1$ vertex deleted subgraphs, we can reconstruct the graph. Consequently, if the reconstruction conjecture holds and the substructure collection contains all graphs of size $k = n-1$, then LLwLC can distinguish all non-isomorphic graphs of size $n$ and is therefore universal. Based on [3], almost every graph has reconstruction number three. This is consistent with our experimental results, where we observed that certain small substructures such as Neumann constraints (or in the other ablation study with only ten vertex-deleted subgraphs) significantly improve the results. 

[1] Chen, Zhengdao and Villar, Soledad and Chen, Lei and Bruna, Joan, "On the Equivalence between Graph Isomorphism Testing and Function Approximation with GNNs." NeurIPS, 2019.  
[2] Ulam, Stanislaw M, "A Collection of Mathematical Problems", 1960.  
[3] Bollobas, Bela, "Almost Every Graph Has Reconstruction Number Three.", Journal of Graph Theory, 1990.

Reviewer comment (Theory):
SEAL and BUDDY are not necessarily limited to 2-hop neighborhoods (the former is limited by scalability, the latter can be applied to arbitrary $k$ order). Can the Neumann basis be applied to more than 2-hop?

Author response:
LLwLC is not limited to 2-hop neighborhoods (with an average time complexity of O(n)). In theory, the only requirement that matrix C must satisfy is to be full column rank.

Reviewer comment (Presentation):
It would be great to have a detailed example of Fig. 1 to show the construction of linear constraints.

Author response:
The first Neumann constraint is the relation between nodes in 1-hop-away nodes and two-hop-away nodes $\Sigma_{y \in S, y\sim x} (f(x) - f(y)) = 0$ (edge representation between them). We represent it in the first row. So, if we multiply the first constraint with the eigenvector ($C^Tf = 0; (Left: -0.40 + 0.61 + 0.03 - 0.25 \approx 0$,  Right: $-0.39 - 0.20 + 2 \times (0.29) \approx 0$)). 


The second Neumann constraint is 1-hop-away nodes of the subgraph which is encoded by the degree of nodes in the subgraph $\Sigma f(x)d_x = 0$. We show it in the second row. So, if we multiply the second row with the eigenvector in the third row ($C^Tf=0$; Left: $2 \times 0.24 - 2\times 0.46 - 2\times 0.40 + 2 \times 0.61 = 0$ Right: $-2 \times 0.39 + 2 \times 0.23 + 2\times 0.36 - 2 \times 0.20 = 0$).

When we compute the eigenvectors by projecting to the null space of the constraints, we force the eigenvectors to be aware of the relation between a set of nodes/edges explicitly. So, the third row is one of the computed eigenvectors for the graphs which satisfy the constraints explicitly.

Reviewer comment (Theory):
What is the complexity of the proposed method to extract a bag of subgraphs, obtain $\delta S$, and construct $C$ for each link?

Author response:
Theoretically, regarding the time complexity, as we discussed in question 1,  based on the reconstruction conjecture [1] we know that if we have all the $n-1$ vertex deleted subgraphs, then we can reconstruct the graph. So, worst-case time complexity is $\mathcal{O}(n)$, but as we discussed, theoretically[2] and experimentally, we can do it with a much less number of constraints.  
    

[1]  Ulam, Stanislaw M, "A Collection of Mathematical Problems", 1960.  
[2] Bollobas, Bela, "Almost Every Graph Has Reconstruction Number Three." Journal of Graph Theory, 1990.

Reviewer comment (Writing):
In Sec. 2 Related Work, the embedding methods are (not) inductive?

Author response:
* Q5.1: Thanks for the correction. Of course, they cannot be applied to the inductive setting.

Reviewer comment (Theory):
The iteration of sample Lanczos from [1] starts from $k=0$ while $j=1$ in LanczosNet. The inconsistency causes confusion about the iteration of the Lanczos Algorithm in introduced Sec 3, especially the coefficients of $q_{j-1}$, $q_{j+1}$.

Author response:
* Q5.2: We will change the iteration indexing such that both start at 0.

Reviewer comment (Theory):
What does it mean that “SEAL does not encode the pairwise node representation”? SEAL converts the link prediction as subgraph classification, which encodes the induced subgraph of two queried nodes with their distance labels. A similar framework is also adopted by LLwLC.

Author response:
* Q5.3 & W1: As mentioned in the answer to Q1, SEAL does not utilize the relation between a set of nodes/edges, whereas the goal of LLwLC is to encode the relation between a set of nodes into the eigenbasis  (SEAL ignores the pairwise relation between nodes). Also, regarding the effect of DRNL, as discussed in W1, please notice that the experiments with the LanczosNet in Tables 2 and 3 (specifically, Collab results) also utilize the DRNL features. However, their results are drastically worse than our novel LLwLC.