Paper: Bag of Features: New Baselines for GNNs for Link Prediction (ICLR-2024)

Reviewer comment (Evaluation):
Rectify the discrepancies between baseline performances in Table 5 and those on the OGB website (refer to Comment 1 for details).

Author response:
$\textbf{Q1. Table Discrepancy.}$

>Thanks for sharing this point. For OGB-Collab, we used the results given in [1],  a review paper that compares the performances of recent Graph Neural Networks (GNNs) in the link prediction task. All the reported performances are extracted from this source. On the OGB-Leaderboard, we observed that GraphSage's performance was suboptimal, so we opted for another configuration with improved hyperparameters.

>In response to Reviewer Dape's request, we included additional GNN baselines in Table 5. These supplementary results are sourced from [1], specifically Table 2 in the reference.

>[1] Li, J., Shomer, H., Mao, H., Zeng, S., Ma, Y., Shah, N., Tang, J. and Yin, D., 2023. Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking. arXiv preprint arXiv:2306.10453.

Reviewer comment (Experiments):
Broaden the evaluation datasets to encompass a variety of network structures and domains (see Comments 2 and 3 for clarification).

Author response:
$\textbf{Q2. Extra Datasets:}$ Tables 3 and 4 follow the methods of (Zhao et al., 2022) and (Guo et al., 2022), respectively. To enhance comprehensiveness, the authors should consider adding Facebook and OGB-DDI to Table 3, and Cora, Citeseer, and DBLP to Table 4.

>Thank you very much for this suggestion. These two papers use different splits (70/10/20 and 85/5/10) so we did not add our CORA/CITESEER results from Table 3 to Table 4 not to cause any confusion. We add these results to the appendix with different splits (Table 9). We will try to complete the experiments for the suggested datasets (FACEBOOK, DBLP, and OGB dataset) before the rebuttal deadline, and let you know the results.

Reviewer comment (Experiments):
Perform ablation studies to gain a deeper understanding of the contributions made by the Bag of Features model components (refer to Comment 4 for more information).

Author response:
$\textbf{Q4. Ablation Study for Individual Features.}$ 

>Thank you very much for this suggestion. We added the ablation study table for feature importance in the appendix (Table 7)

Reviewer comment (Evaluation):
A discrepancy exists between the performance reported in Table 5 and the OGB website. For instance, GraphSAGE shows a 48.10 on the OGB website, but the paper reports 56.88. The performance in Table 5 closely aligns with the validation performance on the OGB website. The authors should verify if Table 5 reflects the validation dataset performance and determine the cause of this inconsistency.

Author response:
$\textbf{Q1. Table Discrepancy.}$

>Thanks for sharing this point. For OGB-Collab, we used the results given in [1],  a review paper that compares the performances of recent Graph Neural Networks (GNNs) in the link prediction task. All the reported performances are extracted from this source. On the OGB-Leaderboard, we observed that GraphSage's performance was suboptimal, so we opted for another configuration with improved hyperparameters.

>In response to Reviewer Dape's request, we included additional GNN baselines in Table 5. These supplementary results are sourced from [1], specifically Table 2 in the reference.

>[1] Li, J., Shomer, H., Mao, H., Zeng, S., Ma, Y., Shah, N., Tang, J. and Yin, D., 2023. Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking. arXiv preprint arXiv:2306.10453.

Reviewer comment (Experiments):
Tables 3 and 4 follow the methods of (Zhao et al., 2022) and (Guo et al., 2022), respectively. To enhance comprehensiveness, the authors should consider adding Facebook and OGB-DDI to Table 3, and Cora, Citeseer, and DBLP to Table 4.

Author response:
$\textbf{Q2. Extra Datasets:}$ Tables 3 and 4 follow the methods of (Zhao et al., 2022) and (Guo et al., 2022), respectively. To enhance comprehensiveness, the authors should consider adding Facebook and OGB-DDI to Table 3, and Cora, Citeseer, and DBLP to Table 4.

>Thank you very much for this suggestion. These two papers use different splits (70/10/20 and 85/5/10) so we did not add our CORA/CITESEER results from Table 3 to Table 4 not to cause any confusion. We add these results to the appendix with different splits (Table 9). We will try to complete the experiments for the suggested datasets (FACEBOOK, DBLP, and OGB dataset) before the rebuttal deadline, and let you know the results.

Reviewer comment (Experiments):
The paper focuses on homophilous graph datasets, but the proposed method could also be applied to heterophyllous graph datasets. Including these datasets would increase the paper's impact.

Author response:
$\textbf{Q3. Heterophilic Datasets.}$

>Thank you very much for this great suggestion. We will add experiments for some heterophilic datasets before the rebuttal deadline, and let you know the results.

Reviewer comment (Experiments):
The ablation studies only examine the removal of all structural or domain features. A more detailed table illustrating the impact of each individual feature on the datasets would make the results more appealing.

Author response:
$\textbf{Q4. Ablation Study for Individual Features.}$ 

>Thank you very much for this suggestion. We added the ablation study table for feature importance in the appendix (Table 7)

Reviewer comment (Experiments):
Experiments are not conducted on OGB datasets other than collab.

Author response:
We are working on additional OGB experiments. We hope to provide the results before the rebuttal deadline.

Reviewer comment (Experiments):
Please add experiments on other OGB datasets.

Author response:
Unfortunately, due to unforeseen issues, we could not finish OGBL-PPA experiments yet. However, we will add the results to our paper when finished.

Reviewer comment (Novelty):
The paper's novelty appears constrained. The significance of graph structural features in the realm of link prediction has been previously underscored by several studies, such as [1][2][3].

Author response:
>Thank you very much for bringing these references to our attention. We added these references to our paper. We wish to emphasize that the novelty of our work lies in substantiating that traditional feature engineering methods can stand shoulder-to-shoulder with cutting-edge GNN models. We argue this based on consistently outperforming powerful GNN methods on various datasets, which are published in top conferences. This supports our claim that our Bag of Features model is a practical and effective alternative to these computationally intensive models.

Reviewer comment (Experiments):
The evaluation omits several critical baselines that similarly exploit graph structural features. Notably absent are BUDDY, Neo-GNN, NCN, NCNC, and NBFNet.

Author response:
>By using reference [1], we give the performance comparison of these GNN models with our model. For OGBL-COLLAB, we report Hits@50 results below, and add them into our accuracy table (Table 5). However, for CORA, CITESEER, and PUMED datasets, the reported performances use different splits. Considering our model uses less training data, it produces highly competitive results with these cutting edge GNN models. Since the experimental setups are different, we opt out these performances  in our accuracy table in the main text, however we added this table to appendix (Table 7). 

>OGBL-COLLAB
| Model   | Hits@50      |
|---------|--------------|
| SEAL    | 63.37 ± 0.69 |
| BUDDY   | 64.59 ± 0.46 |
| Neo-GNN | 66.13 ± 0.61 |
| NCN     | 63.86 ± 0.51 |
| NCNC    | 65.97 ± 1.03 |
| NBFNet  | OOM          |
| Ours    | 76.50 ± 0.27 |

>| Model    | Split      | CORA         | CITESEER     | PUBMED       |
|----------|------------|--------------|--------------|--------------|
| BUDDY    | 85:05:10    | 95.06±0.36   | 96.72±0.26   | 98.20±0.05   |
| Neo-GNN  | 85:05:10    | 93.73±0.36   | 94.89±0.60   | 98.71±0.05   |
| NCN      | 85:05:10    | 96.76±0.18   | 97.04±0.26   | 98.98±0.04   |
| NCNC     | 85:05:10    | 96.90±0.28   | 97.65±0.30   | 99.14±0.03   |
| NBFNet   | 85:05:10    | 92.85±0.17   | 91.06±0.15   | 98.34±0.02   |
| Ours     | 70:10:20    | 94.54±0.30   | 95.01±0.35   | 94.73±0.15   |

>[1] Li, J., Shomer, H., Mao, H., Zeng, S., Ma, Y., Shah, N., Tang, J. and Yin, D., 2023. Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking. arXiv preprint arXiv:2306.10453.

Reviewer comment (Experiments):
The experimental setup seems to miss out on key datasets such as ogbl-ddi, ogbl-ppa, and ogbl-citation2.

Author response:
>We are working on additional OGB experiments. We hope to provide the results before the rebuttal deadline. We will let you know when we have the results.

Reviewer comment (Experiments):
There are no ablation studies about the importance of each feature.

Author response:
>Thank you very much for this suggestion. We added the ablation study table reporting the impact of each individual feature in the appendix (Table 7), which you can see in our revision.

Reviewer comment (Experiments):
This paper overclaims its contribution. The method and experiments of this paper are designed for link prediction, but the contribution of this paper lies in discussing that GNN is not good at graph learning. As far as I know, heuristics perform better than GNN only in link prediction, and there seems to be no similar phenomenon in other domains such as graph classification, node classification, and graph regression.

Author response:
>Thank you for raising this issue. It's accurate to observe that our focus in this manuscript centers specifically on the challenge of link prediction. The decision to narrow our attention to this particular task within graph representation learning was primarily influenced by space limitations. In a distinct investigation, we employed similar techniques to compare Graph Neural Networks (GNNs) with traditional feature extraction in the context of a node classification task. Across six benchmark datasets in that study, we outperformed all state-of-the-art GNNs. While our findings with the OGBGN dataset did not outperform the OGB leaders, but gave highly competitive results. To preserve anonymity, we refrain from providing additional details.

>Given the inherent differences between node classification and link prediction tasks, the employed feature extraction methods also diverge significantly. Consequently, we opted to present these two studies in separate papers. Nevertheless, the outcomes from both studies imply that existing GNNs do not exhibit a substantial superiority over traditional feature engineering models. It is crucial to clarify that our intention in this paper is not to advocate for the adoption of feature engineering methods moving forward. Instead, we aim to document that current GNN models do not demonstrate a significant advantage over them. This revelation should illuminate the current state of affairs with GNNs and motivate researchers to advance and refine these models further.

Reviewer comment (Experiments):
The experimental part of this article does not compare with state-of-the-art GNN methods, such as NBFNet[1] and Seal[2].

Author response:
>Thanks for pointing out these baseline references. It's important to mention that SEAL [2] has already been given in Table 3 - row 4. For NBFNet, we report the result for a different split of 85:5:10 below by using reference [3]. However, since we use a different split for Table 3 (70:10:20), we opted not to include NBFNet in Table 3. On the other hand, with the suggestion of Reviewer Dape, we added several recent GNN baselines to our Table 5, and Table 8. We also acknowledged this reference in our related work section.

>The specifics of AUC performances are outlined in the table provided below.
|      | Split      | CORA  | CITESEER | PUBMED |
|------|------------|-------|----------|--------|
| SEAL | 70:10:20    | 92.55 | 85.82    | 96.36  |
| NBFNet | 85:05:10  | 95.60  | 92.30     | 98.30   |
| Ours | 70:10:20    | 94.54 | 95.01    | 94.73  |

>[3] Li, J., Shomer, H., Mao, H., Zeng, S., Ma, Y., Shah, N., Tang, J. and Yin, D., 2023. Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking. arXiv preprint arXiv:2306.10453.