Paper: Pure Message Passing Can Estimate Common Neighbor for Link Prediction (ICLR-2024)

Reviewer comment (Theory):
The assumption of Gaussian distribution in the initialization of input vectors and weight matrices is reasonable for non-attributed benchmarks but it seems not to be correspondent for attributed benchmarks.

Author response:
We value your observation regarding the use of a Gaussian distribution in initializing input vectors and weight matrices. It's important to clarify that our empirical studies, specifically in Section 3.1, utilized the Gaussian distribution solely for sampling input vectors. In our theoretical and empirical analyses, the primary assumption was the zero-mean characteristic of the input vector distribution, rather than specifying a particular distribution type.

For both attributed and non-attributed benchmarks, our approach involves sampling random vectors and propagating them through the graph to estimate structural features. In the case of attributed benchmarks, while node attributes are also propagated, they are not fused with the random vectors. Instead, these attributes serve as node representations of individual nodes, similar to vanilla GNN practices. Consequently, the assumption of a Gaussian distribution does not adversely impact MPLP’s performance on attributed benchmarks.

Reviewer comment (Experiments):
The research focuses on the capability of MPNN for estimating common neighbors which is a critical heuristic for link prediction. In other words, MPLP injects the CN heuristic into MPNN. So, what about the link prediction performance when we directly regard CN as the node attribute? For example, the first elements of the input vectors could be substituted with CN value.

Author response:
Your query about incorporating CN heuristic directly into node attributes is thought-provoking. MPLP effectively operates as an estimator, where structural features, including CNs, are derived through message-passing. Directly injecting these structural heuristics as node attributes could indeed be a viable approach, potentially yielding performance on par with MPLP, particularly when estimation errors are minimal.

Nevertheless, this direct injection method lacks the efficiency of MPLP in several respects. For online inference tasks, MPLP offers superior time and space efficiency. MPLP can be seen as a way to estimate **link-level** features with a **node-level** complexity. MPLP allows preprocessing the propagated random vectors and node attributes. Then MPLP stores them at **node-level** with space complexity $O(N)$, where $N$ is the number of nodes. When an inference for a batch of links is requested, MPLP can generate the **link-level** features from the **node-level** features with an inner product operation. This opeartion can easily be parallelized on GPU.

However, to inject heuristic, there may exist two options: (1) precompute the CNs for **each link** and access them in constant time, which requires $O(N^2)$ space complexity since any node pair can be requested for a link prediction; (2) compute the **link-level** features on the fly, which requires storing the entire graph structure in memory and computing the heuristics for each requested link in real time, which is both time and space consuming for large graphs. Therefore, MPLP is more efficient than directly injecting heuristics as link features.

Reviewer comment (Presentation):
Why are the results of SAGE on some benchmarks (Computers/Photo) different between Table 2 and Table 4?

Author response:
We apologize for any confusion caused by the differing results of SAGE in Tables 2 and 4. This discrepancy arises from the distinct hyperparameter settings used in each table. In Table 2, SAGE's results are based on a range of hyperparameters, including varying the number of layers. Conversely, for simplicity and consistency, we set the number of layers to 2 in Table 4. We have updated Table 4's caption in the revised manuscript to clarify this distinction.

Reviewer comment (Experiments):
I am curious about when we concatenate the attribute vector with random feats, would the results be better than those with only attributes or only random feats? Since it is hard to evaluate whether attribute vectors are more important than random feats before conducting the experiments, the concatenation might achieve the better one automatically.

Author response:
Your suggestion about concatenating attribute vectors with random features is intriguing. While vanilla GNNs may benefit from such a combination, resulting in potentially enhanced performance, there is a caveat. Incorporating random features into the MLP component of GNNs might impede the model's ability to generalize, particularly when new nodes are introduced. Nonetheless, this idea represents a promising variant of MPLP worth exploring, and we plan to investigate this further in future work.

Reviewer comment (Writing):
What does the author mean by `MPLP holds its own’? What proves that?

Author response:
Our intention with the phrase "MPLP holds its own" was to convey that the design of MPLP should gurantee its strong capability of performing link prediction task. This assertion is supported by our experimental findings, where MPLP consistently outperforms or matches other models in diverse datasets, demonstrating its strong capability in link prediction.

Reviewer comment (Evaluation):
The ablation study shows that some of the proposed components do not work well.

Author response:
Thank you for pointing out the results of our ablation study. In Appendix Section D.2, we provide a detailed discussion on the contribution and effectiveness of various components within the MPLP framework. It is important to note that the performance of these components can vary depending on the specific characteristics of the dataset.

- **Shortcut Removal**: This component has proven to be particularly influential. Its primary role is to align the structural distribution of positive links between the training and testing datasets. This alignment is crucial for ensuring the generalizability and robustness of MPLP.

- **One-hot Hubs in Larger Graphs**: In the context of larger graphs, such as those in the Collab dataset, the one-hot hubs component becomes essential. It effectively reduces the estimation variance of structural features, thereby enhancing the overall accuracy of the model.

- **Norm Rescaling**: This component has been identified as a significant improvement in MPLP, especially noted in its positive impact on datasets like Yeast, Physics, Photo, and Collab. Norm rescaling contributes to better utilization of the structural features, leading to more accurate predictions.

Each component of MPLP has been designed to address specific challenges in link prediction. The findings from our ablation study highlight the nuanced role these components play in different scenarios, contributing to the overall effectiveness of MPLP.

Reviewer comment (Theory):
Although the theoretical analysis in Theorem 2 is appreciated, it is unclear how the CN is derived from the product operation and used in the GNNs.

Author response:
Thank you for bringing this to our attention. In our MPLP framework, the representation of a link involves capturing its structural neighborhood features. One of these features is the Common Neighbor (CN), which is quantified using Distance Encoding [1]. This approach involves counting the number of nodes at various distances from a target link, including CNs.

Traditionally, CNs can be counted using one-hot vectors, but to enhance scalability, we employ quasi-orthogonal (QO) vectors in MPLP. During message passing, nodes accumulate information in the form of linear combinations of these QO vectors. When we apply the inner product operation on these vectors, it effectively isolates and aggregates the norms of QO vectors corresponding to CNs, while nullifying contributions from non-common neighbors. This method allows MPLP to accurately estimate the count of CNs for each target link.

Reviewer comment (Presentation):
Figure 3 lacks a caption explaining what the colors stand for, making it difficult to interpret.

Author response:
We regret the confusion caused by the lack of a detailed caption for Figure 3 and appreciate your feedback on this aspect. The colors in the figure are intended to represent nodes at varying distances from the target link. Specifically, red nodes indicate the target nodes themselves, while green nodes denote their common neighbors. We have included a detailed explanation of these color codings in the caption of the revised version of the figure to enhance clarity and aid in the interpretation of the visual representation.

Reviewer comment (Reproducibility):
In Theorem 3, a walk operation is introduced to count the number of length-l walks between nodes u and v, but it is unclear how this operation is conducted in practice.

Author response:
Thank you for your question regarding the practical application of the walk operation described in Theorem 3. In MPLP, we approach this operation in two steps to ensure efficiency. The initial step involves standard message-passing on the quasi-orthogonal (QO) vectors, akin to the process in other GNNs. The second step is the inner product operation, whose computational cost is linear with respect to the dimension of the QO vectors. Notably, MPLP does not enumerate all possible walks to count them. Instead, it utilizes the message-passing of QO vectors to efficiently estimate the number of walks, thus achieving linear time complexity.

Reviewer comment (Evaluation):
The proposed method only achieves marginal improvement (not significant improvement according to the mean and the standard deviation) compared to the baselines in many cases.

Author response:
We apologize for any oversight in the paper. Our analysis across 13 benchmark datasets indicates that MPLP achieves significant improvements over baseline models on 7 datasets, including PB, C.ele, E.coli, CS, Computers, Photo, and Collab. In the remaining datasets, MPLP demonstrates more stable performance, as evidenced by smaller standard deviations. This consistency underlines MPLP's superiority in terms of both performance and stability when compared with the baselines.

Reviewer comment (Evaluation):
The experiments, as shown in Table 1 and 2 only consider results for Hits@50. This seems relevant for the subsequent evaluations of inference, dimensions, and batch sizes. But, is still limited since the current SOTA models run experiments with Hits@100, Hits@20, and MRR.

Author response:
We are grateful for your observation regarding the scope of evaluation metrics in our experiments, particularly in Tables 1 and 2 where we focused solely on Hits@50. Recognizing the importance of a broader metric range for a comprehensive evaluation, we have expanded our analysis to include Hits@20 and Hits@100. These additional metrics provide a more complete picture of our model's performance. We have incorporated these results in the appendix of our revised manuscript, specifically in Appendix Section D.5.
This addition not only addresses your concern but also strengthens our evaluation by offering a more nuanced understanding of our model's capabilities across different metric thresholds.

Reviewer comment (Experiments):
The test for the estimation capabilities seems limited by testing just GCN and SAGE. The inner product calculation seems like it should be extended to other types of popular GNNs such as: GIN and GAT. The results from which would lend more credence to the claims made about expressiveness and the effects of implementing quasi-orthogonal vectors into link prediction models.

Author response:
Your question regarding the comparison of MPLP with GAT and the potential inclusion of GIN is insightful. In this study, our focus is primarily on evaluating the ability of message-passing models to identify and count structural features like common neighbors and triangles. While GAT employs a specialized form of message-passing through its attention mechanism, this specific feature is not the central focus of our research.

Both GAT and GIN, as instances of MPNNs, are limited by the expressiveness of the 1-WL test [1], making them less suitable for estimating complex structures like triangles. This limitation is not due to the specific message-passing mechanism but rather to the inherent constraints of the 1-WL framework. Consequently, we chose GCN and SAGE as representatives of 1-WL GNNs to investigate their capacity to count triangles. This ability arises not from the specific type of message-passing employed but rather from the use of quasi-orthogonal vectors as inputs and the inner product as the decoder.

In light of these considerations, we did not include GAT or GIN in our experiments, as their inclusion would not have significantly contributed to the specific aims of our study.

Reviewer comment (Experiments):
The node-label estimation of CN and DE is promising. However, it seems limited in scope since it approximates just CN or DE and does not extend further to DRNL or DE+. This may be due to concerns of tractable computation but given SEAL's explicit testing of both DRNL and DE++ as labelling tricks, this seems like an important inclusion to evaluate MPLP fully.

Author response:
We value your suggestion about integrating DRNL into MPLP. Indeed, we have explored this integration and faced certain challenges. DRNL utilizes a specific 'masking trick' to compute the distance between two nodes, as detailed in Appendix D of [2]. This approach necessitates that the DRNL calculation be conditioned on each specific edge, leading to a computational overhead akin to that of SEAL.

Considering that DE alone often sufficiently represents graph structure, we decided against incorporating DRNL into our model. The additional computational burden and the marginal benefits in most use cases did not justify the integration of DRNL into the MPLP framework.

[1] Xu, Keyulu, et al. "How powerful are graph neural networks?." (2018).

[2] Zhang, Muhan, et al. "Labeling trick: A theory of using graph neural networks for multi-node representation learning." Advances in Neural Information Processing Systems 34 (2021).

Reviewer comment (Experiments):
What was the reasoning behind not including the remaining OGBL datasets? It seems that MPLP's scalability is an important component based on the inference, dimensions, and ablation studies but why not bring the scalability to the forefront of the paper instead of placing the ablation studies in the appendix.

Author response:
We appreciate your inquiry regarding our choice to omit some OGBL datasets. This decision was primarily influenced by MPLP’s performance characteristics in relation to graph density, as detailed in Appendix Section F. Theorem 2 in our paper reveals that MPLP's estimation error is closely tied to graph density. While MPLP excels in sparse graph environments, its performance tends to decline in denser graphs or those with a high concentration of hubs. The estimation error, in these cases, can escalate quadratically.

To illustrate this, we analyzed the average degree of other OGBL datasets like DDI and PPA. Our findings showed that these datasets have significantly higher average degrees compared to the Collab dataset, leading to a marked increase in the estimation error for structural features. Additionally, considering 2-hop structural features in datasets like Citation2 exacerbates this issue due to the extensive size of the 2-hop neighborhoods.

Given these factors, we observed that MPLP’s performance becomes less stable on these datasets, primarily due to increased variance. Currently, we are actively working on extending MPLP's capabilities to more densely connected graphs and consider this an important area for future research.


|            | DDI    | PPA   | Collab | Citataion2 |
|------------|--------|-------|--------|------------|
| Avg Degree | 312.84 | 52.62 | 5.45   | 10.44      |
|            |        |       |        |            |
| Histogram  | of     | #     | 2-hop  | neighbors  |
| [0,1)      | 0.0    | 0.0   | 42.5   | 0.0        |
| [1,10)     | 0.0    | 0.4   | 7.5    | 3.1        |
| [10,50)    | 0.2    | 1.4   | 24.3   | 9.9        |
| [50,100)   | 0.0    | 1.5   | 11.7   | 8.3        |
| [100,500)  | 0.6    | 9.4   | 12.7   | 31.0       |
| [500,1000) | 3.3    | 11.4  | 0.3    | 15.3       |
| [1000,inf) | 95.8   | 75.8  | 0.3    | 32.4       |

Reviewer comment (Theory):
In order to estimate pairwise features, the pure message passing adopted in MPLP loses the non-linearity of MPNN, which comprises the expressiveness of the framework.

Author response:
Thank you for raising a critical point about the non-linearity of MPNNs. MPLP indeed incorporates vanilla GNNs, as represented by $ GNN(\cdot) $ in Equations (4) and (8). This integration ensures that MPLP retains at least the same level of expressiveness as any non-linear MPNN. Moreover, it's important to emphasize that MPLP primarily employs linear message passing for estimating structural features. This approach not only maintains the framework’s expressiveness but potentially enhances it by leveraging the strengths of both linear and non-linear methodologies.

Reviewer comment (Novelty):
The adopted QO vector from DotHash enables more types of pairwise structural features (AA, RA), but it is still heuristic-based. Those features are empirically helpful for link prediction but also lose flexibility and capacity since they are fixed and predefined.

Author response:
We acknowledge the heuristic-based characteristics of MPLP. However, MPLP is designed to learn flexible pairwise features autonomously. As indicated in Equation (4), MPLP derives these features from the node representations generated by any selected $ GNN(\cdot) $. This capability allows MPLP to adapt and fit to diverse data sets, mitigating concerns about the rigidity and predefined nature of its features.

Reviewer comment (Theory):
How would the sampling of neighborhoods in SAGE affect Thereom 1 and the estimator in Eq. (3)?

Author response:
In our manuscript, "SAGE" specifically denotes the encoder component of the GraphSAGE model [6], excluding neighborhood sampling. We adhere to using the entire neighborhood for message passing, a method consistent with several other studies utilizing SAGE for link prediction [1,3,4,5]. Our focus remains on the encoder's performance without altering the graph structure, as such modifications lean more towards data augmentation rather than enhancing the link prediction task itself.

Reviewer comment (Experiments):
What is the cost of generating QO vectors? Is there a principal way to pick its dimension? Will it scale to graphs larger than collab? How would the density of the graph affect its performance of runtime and estimation error?

Author response:
- **Cost Effectiveness**: Generating QO vectors incurs minimal computational cost. Its time complexity, $ O(|V|F) $, similar to the process of loading node features into memory, thereby makes it a highly efficient step in most scenarios.

- **Dimension Selection Strategy**: The selection of dimensions for QO vectors can be approached in a pragmatic manner. Ideally, dimensions should be increased until a point where further increase does not significantly reduce estimation variance. In practice, one may balance the trade-off between the variance and the computational cost. 

- **Scalability to Larger Graphs**: MPLP demonstrates robust scalability to graphs larger than the Collab dataset. However, it is important to note that graph density plays a pivotal role in influencing both runtime performance and estimation accuracy. The runtime of MPLP, similar to other message-passing GNNs, scales linearly with the number of edges. Additionally, as indicated in Theorem 2 of our study, denser graphs can lead to heightened estimation errors. This aspect is crucial in understanding the applicability of MPLP across varying graph densities.

Reviewer comment (Experiments):
Sec 4.3 mentioned that Eq. (7) can be used for estimating triangles. Can MPLP be used for the triangle estimation task in Sec 6.1 of [2]?

Author response:
Thank you for highlighting the potential of MPLP in the context of triangle estimation as discussed in Section 6.1 of [2]. Indeed, MPLP is adept at estimating triangles through its pure message-passing mechanism. This capability places MPLP on par with the more complex GNNs referenced in Section 6.1 of [2], particularly in terms of accurately counting triangles. Furthermore, MPLP achieves this with an efficiency level comparable to 1-WL GNNs, such as GCN, SAGE, and GIN. 

This aspect of MPLP not only demonstrates its versatility but also suggests its potential for broader applications. Specifically, MPLP's ability to estimate triangles is indicative of its potential to effectively estimate other subgraph structures. We are excited about the prospects of extending MPLP's capabilities in future research, exploring its application in estimating a wider range of substructures.

Reviewer comment (Writing):
Is the $\text{GNN}(\cdot)$ in Eq. (4) and Eq. (8) referring to valina GNNs or the linearized version proposed in Eq. (3)?

Author response:
In our manuscript, the GNNs referenced in Equations (4) and (8) are indeed vanilla GNNs. These can be any generic GNN models capable of encoding node representations, such as GCN, SAGE, or GAT. This choice allows for a broad applicability of our approach, as it is not restricted to any specific type of GNN architecture.

Reviewer comment (Experiments):
It would be great to provide more details regarding the baselines, especially differentiating ELPH and BUDDY [1], and NCN/NCNC-k [3] for interpreting the results of Table 2 and Fig 4.

Author response:
Due to constraints on the length of our paper, we focused on including the most impactful variants of the baseline models in our study. Consequently, we chose to highlight ELPH over BUDDY as mentioned in [1], and NCNC over NCN as discussed in [3]. This decision was guided by the aim to present the most powerful and relevant comparisons. We have expanded upon these choices and provided additional details in the appendix of our revised paper, marked in blue, for further clarity.