Paper: Revisiting Link Prediction: a data perspective (ICLR-2024)

Reviewer comment (Presentation):
Readability. As it is stated in the submission instructions "reviewers are not required to read the appendix". In this case, I did not read the appendix. For this reason, I could understand specific details from the paper. While I understand the main idea, there are details, such as the parameter r that I could not understand. Unfortunately, the paper uses the appendix to explain very important parts of the paper; for example: "Limitation, broader impact, and future works are in Appendix G, H, and I". So, if you can not reduce and contain the paper to 9 pages, I suggest you submit the paper to a journal, instead of a conference. Note, I usually read the appendices of the paper. They usually have important proof and complementary results, but in this case, it is being used as a main part of the paper.

Author response:
We thank the reviewer for the concerns on the readability. We are also glad to see that the reviewer appreciates "our paper showing a very significant result and contribution". 

For the readability issue, we worked hard to improve the writing to make the revision self-contained. **The reviewer can directly see our new updated revision that is better self-contained. If you still have any concerns, please let us know immediately, and we will make sure that the paper is self-contained with clear writing.** In the following part, we first summarize your questions, explain them in detail, and show our improvement in the revision. Besides, we carefully check all the appendix sections to avoid missing any important parts of the paper in the main content. If you still have any concerns about the readability, please let us know. We are eager to **make sure the paper is self-contained within the 9-page limitation to meet the standard for ICLR.** 

**Questions are summarized and retold as follows**: Questions are majorly three-fold: (1) The main paper lacks some important details about parameters $r$ and $\beta$. (2) Limitation, broader impact, and future works are in Appendix G, H, and I" (3) I understand the main idea of the theoretical model and its importance in the paper, but this is not used in the experiment section of the main paper. We then provide explanations and the corresponding revision to those three questions step by step. 

**Question1:** The main paper lacks some important details about the parameter $r$ and $\beta$.

**Response1:** Thanks for pointing out this issue. In the original paper, we put most detailed explanations of symbols and parameters in Section 3.2 when we first introduce the latent space model. However, we do not repeat the detailed meanings of those parameters or notify the reader where they can find the detailed explanations in each lemma and proposition. We have made the change in the modification accordingly to avoid potential confusion. 

**The detailed explanation of those parameters is as follows.**  We first clarify that all those parameters are only utilized for the theoretical analysis, corresponding to different intrinsic properties of datasets. They show no relevance in training a deep model process. Therefore, all those stuff in the theoretical analysis should be called parameters, not hyperparameters. 

**Detail explanation for parameter $r$ is as follows.**
> $r_i$ is a connecting threshold parameter corresponding to node $i$.With $\alpha = +\infty$, $\frac {1}{1+ e^ {\alpha (d_{ij}-\max\{r_{i},r_{j}\})}}=0$ if $d_{ij} > \max\{r_i, r_j\}$, otherwise it equals to $1$. Therefore, a large $r_i$ indicates node $i$ is more likely to form edges, leading to a potentially larger degree. Nodes in the graph can be associated with different $r$ values, allowing us to model graphs with various degree distributions. Such flexibility enables our theoretical model to be applicable to more real-world graphs.

The detailed discussion of parameter $r$ can be found at the beginning of Section 3.2 theoretical model. 

**Detail explanation for parameter $\beta$ is as follows.**
> Feature perspective provides complementary information, considering two nodes with high feature proximity but located distantly in the latent space should also be potentially connected. In line with this, we introduce the feature proximity parameter $\beta_{ij}$ in the latent space. A larger $\beta_{ij}$ indicates more likely for node $i$ and $j$ to be connected. 

The detailed discussion of parameter $\beta$ in Section 3.2 can be found on, the 4th line of page 5, starting with **(ii)**. Moreover, $\beta$ is a feature proximity parameter in the theoretical model which does not have a direct calculation formulation. It is just for the theoretical analysis. In the experiment part, we calculate the heuristic method, FH, corresponding to feature proximity $\beta$ for calculation. Details in the paper are shown in Section 3.1.  

**Our modification in the revision:** We carefully check whether there is a description each time we mention each parameter. We add a simple description **each time** we mention the parameter and let the reader see more details in the corresponding part in section 3.2 of the main paper. The newly added content is as follows
> $r_i$ is a connecting threshold parameter corresponding to node $i$ while a large $r_i$ indicates node $i$ is more likely to form edges. A more detailed description of $r$ can be found under Eq.(1). $\beta_{ij}$ is the feature proximity parameter between nodes $i$ and $j$ in the latent space while a large $\beta_{ij}$ indicates large feature proximity. A more detailed description of $\beta$ can be found in Section 3.2.

Reviewer comment (Reproducibility):
The details of the theoretical models are not explained in the main paper. There are several details without any type of information, for example, the parameter r. It seems then rather than a parameter, r is a hyperparameter, but no details about this are given in the main paper. I have similar issues with the \beta parameter/hyperparameter, which is described as a feature proximity parameter, but no details about its calculation or suggestion are given.

Author response:
We thank the reviewer for the concerns on the readability. We are also glad to see that the reviewer appreciates "our paper showing a very significant result and contribution". 

For the readability issue, we worked hard to improve the writing to make the revision self-contained. **The reviewer can directly see our new updated revision that is better self-contained. If you still have any concerns, please let us know immediately, and we will make sure that the paper is self-contained with clear writing.** In the following part, we first summarize your questions, explain them in detail, and show our improvement in the revision. Besides, we carefully check all the appendix sections to avoid missing any important parts of the paper in the main content. If you still have any concerns about the readability, please let us know. We are eager to **make sure the paper is self-contained within the 9-page limitation to meet the standard for ICLR.** 

**Questions are summarized and retold as follows**: Questions are majorly three-fold: (1) The main paper lacks some important details about parameters $r$ and $\beta$. (2) Limitation, broader impact, and future works are in Appendix G, H, and I" (3) I understand the main idea of the theoretical model and its importance in the paper, but this is not used in the experiment section of the main paper. We then provide explanations and the corresponding revision to those three questions step by step. 

**Question1:** The main paper lacks some important details about the parameter $r$ and $\beta$.

**Response1:** Thanks for pointing out this issue. In the original paper, we put most detailed explanations of symbols and parameters in Section 3.2 when we first introduce the latent space model. However, we do not repeat the detailed meanings of those parameters or notify the reader where they can find the detailed explanations in each lemma and proposition. We have made the change in the modification accordingly to avoid potential confusion. 

**The detailed explanation of those parameters is as follows.**  We first clarify that all those parameters are only utilized for the theoretical analysis, corresponding to different intrinsic properties of datasets. They show no relevance in training a deep model process. Therefore, all those stuff in the theoretical analysis should be called parameters, not hyperparameters. 

**Detail explanation for parameter $r$ is as follows.**
> $r_i$ is a connecting threshold parameter corresponding to node $i$.With $\alpha = +\infty$, $\frac {1}{1+ e^ {\alpha (d_{ij}-\max\{r_{i},r_{j}\})}}=0$ if $d_{ij} > \max\{r_i, r_j\}$, otherwise it equals to $1$. Therefore, a large $r_i$ indicates node $i$ is more likely to form edges, leading to a potentially larger degree. Nodes in the graph can be associated with different $r$ values, allowing us to model graphs with various degree distributions. Such flexibility enables our theoretical model to be applicable to more real-world graphs.

The detailed discussion of parameter $r$ can be found at the beginning of Section 3.2 theoretical model. 

**Detail explanation for parameter $\beta$ is as follows.**
> Feature perspective provides complementary information, considering two nodes with high feature proximity but located distantly in the latent space should also be potentially connected. In line with this, we introduce the feature proximity parameter $\beta_{ij}$ in the latent space. A larger $\beta_{ij}$ indicates more likely for node $i$ and $j$ to be connected. 

The detailed discussion of parameter $\beta$ in Section 3.2 can be found on, the 4th line of page 5, starting with **(ii)**. Moreover, $\beta$ is a feature proximity parameter in the theoretical model which does not have a direct calculation formulation. It is just for the theoretical analysis. In the experiment part, we calculate the heuristic method, FH, corresponding to feature proximity $\beta$ for calculation. Details in the paper are shown in Section 3.1.  

**Our modification in the revision:** We carefully check whether there is a description each time we mention each parameter. We add a simple description **each time** we mention the parameter and let the reader see more details in the corresponding part in section 3.2 of the main paper. The newly added content is as follows
> $r_i$ is a connecting threshold parameter corresponding to node $i$ while a large $r_i$ indicates node $i$ is more likely to form edges. A more detailed description of $r$ can be found under Eq.(1). $\beta_{ij}$ is the feature proximity parameter between nodes $i$ and $j$ in the latent space while a large $\beta_{ij}$ indicates large feature proximity. A more detailed description of $\beta$ can be found in Section 3.2.

Reviewer comment (Experiments):
I understand the main idea of the theoretical model and its importance in the paper, but this is not used in the experiment section of the main paper.

Author response:
**Question3:** I understand the main idea of the theoretical model and its importance in the paper, but this is not used in the experiment section of the main paper.

**Response3:** Thanks for your question on the connection between the theoretical model and the experiment in the paper. Our primary answers are (1) for all the theories, we have the corresponding experimental results to support our claims. (2) **We indeed utilize the qualitative insight we gain from the theoretical analysis to guide the model design and dataset selection.** We have made changes in the revision accordingly to make it clear and easy to understand. 

We first show our theoretical insights and how they relate to experiments. Our theoretical insights are majorly two-fold: (1) Global structural proximity (GSP) only shows effectiveness when local structural proximity (LSP) is deficient. (2) The incompatibility between LSP and feature proximity (FP). It is unlikely that both large LSP and FP co-exist for a single node pair. Those insights guide the experiment design in section 3.4 and 4. 

In Section 3.4, we study how the incompatibility between structural proximity and feature proximity affects the effectiveness of GNN4LP models. In this section, we are majorly inspired by insight (2): the incompatibility between structural and feature factors. Due to such incompatibility, it could be hard for a single model to benefit both node pairs with feature proximity factors with opposite information. Therefore, we observe that the GNN4LP with specific designed for structural proximity will perform better on the node pairs with structural proximity but not the feature ones

In Section 4.1, we study the guidance for the model design inspired by insight (2): the incompatibility between structural and feature factors. Since we observe when both structural and feature factors come into play simultaneously, there is a potential for them to provide conflicting supervision to the model. Then we propose the decouple the model design which learns the feature proximity factors and pairwise structural ones independently before integrating their outputs, to avoid the conflict from insight 3. 

In Section 4.2, we illuminate the complete dataset landscape inspired by insight (1): The LSP and GSP can not be effective simultaneously in the same dataset. To get the full landscape, we enumerate all potential combinations among different data factors. As there are three factors, the potential combination should be $C_3^3 + C_3^2 + C_3^1=7$. Nonetheless, given the constraint from insight (1), the potential combinations shrink with only four combinations: Category 1: both LSP and FP factors dominate. Category 2: Only the LSP factor dominates. Category 3: both GSP and FP factors dominate. Category 4: Only the GSP factor dominates. The theoretical insight helps us to clearly identify the different types of datasets in the link prediction properly, avoiding those inexistent categories. 

We believe that the **qualitative insights** from theoretical analysis indeed provide insights for the experiment part, especially for the model design and dataset selection. 


**Our modification in the revision:** We carefully added the key concepts of the writing when we connected the theoretical limitation, broader impact, and future works in the main content. **We add more intuition about the connection between the theory insight and experiment at the beginning of each subsection. Please see the revision.**


We are committed to further enhancing the accessibility and clarity of our manuscript and appreciate your constructive feedback in this regard. Thank you for your consideration and guidance. **Moreover, we carefully check all the appendix sections to avoid missing any important parts of the paper in the main content. Hope that the reviewer finds the revision meets the requirement to be self-contained in 9 pages. If you have any further questions, feel free to mention them to us, and we will make modifications immediately. **

Reviewer comment (Theory):
What is the role of heterophily when calculating Feature proximity? Moreover, could you give an example in homophilic graph that FP and LSP are conflictive or FP and GSP are conflictive. I am asking this because I feel the conflict comes from the different link forming mechanism where FP is just not appropriate.

Author response:
Thanks for your question about the role of heterophily when calculating feature proximity. We are not quite sure whether we exactly understand some details of your question. Therefore, we will first retell your question and answer it accordingly. If there exists any misunderstanding, please let us know, we are happy to solve all your valuable concerns and improve our paper accordingly. 

Our current understanding of your question is two-fold: (1) How to describe the heterophily with the feature proximity factor? and (2) Does the conflict only happen when low FP(heterophily) but high LSP? How can the conflict happen when high FP(homophily) but low LSP? Can you show an example?

**R1:** Heterophily is the opposite to the well-known homophily concept in social science [1] with the principle "similarity breeds connection". More specifically, homophily indicates that nodes with larger feature similarity (high feature proximity). In contrast, heterophily indicates smaller feature similarity (small feature proximity). The ogbl-ddi dataset is the typical example where most node pairs are feature heterophilic as indicated in Figure 2. 

**R2:** No, the conflict can also happen in the homophily case with high FP(homophily) and low LSP. 
- We first clarify that there exists the case where node pairs with high FH but low LSP. The theoretical evidence and the empirical verification can be found in Lemma 2 and Figure 4, respectively. In Figure 4, we can observe that there is a large proportion of test node pairs that can be predicted correctly by heuristic-derived feature proximity (high FP) but predicted wrongly by heuristic derived local structural proximity (low LSP). 
- We then explain more on why there exists the case feature-homophilic pairs that FP and LSP are conflictive. **The key concept is, that link prediction is to predict the node pairs that aren't connected yet, excluding those node pairs that are already connected. When node pairs have both high FP and LSP, they are more likely to be with a link that exists already, rather than still unconnected for prediction.** We are then to predict those unconnected node pairs without both high FP and LSP. In such cases, high FP or high LSP individually is strong evidence of a potential link connected. Therefore, in a homophilic graph, it is likely that the unconnected node pairs with high FP and low LSP(conflictive) have a high probability of being connected.
- We want to clarify that in the link prediction task, feature proximity and structural proximity are all important, rather than the feature proximity dominating this problem. Evidence can be found in proposition 1 and 3. The feature and structure are two perspectives that may not necessarily be aligned with each other. Similar thoughts can also be found in social network analysis[2].


[1] McPherson, M., Smith-Lovin, L., & Cook, J. M. (2001). Birds of a feather: Homophily in social networks. Annual Review of Sociology.
[2] Abebe, Rediet, et al. "On the effect of triadic closure on network segregation." Proceedings of the 23rd ACM Conference on Economics and Computation. 2022.

Reviewer comment (Novelty):
The paper introduces a latent space model for link prediction and provides theoretical guarantees for the identified data factors. Nonetheless, why to use such model is still so clear and motivated.

Author response:
**Response:** Thanks for your great question about the motivation for the latent space model. The primary answer is that the latent space model is particularly suited for link prediction as it connects the likelihood of a link forming between two nodes to their proximity in a latent metric space. It is developed based on the most central principle in link prediction: Similarity breeds connection [6]. That is, there is a higher probability of forming a link if two nodes have similar characteristics. 

The latent space model is a widely adopted **theoretical model** that helps design many real-world problems, especially for network analysis [1,2,3,4] and physics. Particularly, the latent space model in network analysis aims to locate network information in some latent space that can describe both local and global structures. The selected latent space can be Euclidean space [1], unit hypersphere space [5], and so on to describe different properties.

We then utilize the latent space model for link prediction, based on the important data factors, considering both feature and structure perspectives comprehensively. Each node is associated with a location in a D-dimensional latent space. The latent space for link prediction is typically utilized to describe node proximity mathematically, where nodes close in the latent space are likely to share particular characteristics. We can easily connect the distance in the latent space with the probability of forming a link. And the distance in the latent space can be approximated with different data factors. The theoretical model is utilized to mathematically formulate the graph data and link prediction task. Such mathematical description can help us conduct theoretical analysis and bring insights into the effectiveness of these data factors (Proposition 1,2,3) and the relationship between different factors (Lemma 1,2). 

[1] Peter D Hoff, Adrian E Raftery, and Mark S Handcock. Latent space approaches to social network analysis. Journal of the american Statistical association, 97(460):1090–1098, 2002.
[2] Sewell, Daniel K., and Yuguo Chen. "Latent space models for dynamic networks." Journal of the american statistical association 110.512 (2015): 1646-1657.
[3] Smith, Anna L., Dena M. Asta, and Catherine A. Calder. "The geometry of continuous latent space models for network data." Statistical science: a review journal of the Institute of Mathematical Statistics 34.3 (2019): 428.
[4] Gormley, Isobel Claire, and Thomas Brendan Murphy. "A latent space model for rank data." ICML Workshop on Statistical Network Analysis. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006.
[5] Jiang, Diqiong, et al. "Sphere Face Model: A 3D morphable model with hypersphere manifold latent space using joint 2D/3D training." Computational Visual Media 9.2 (2023): 279-296.
[6] McPherson, M., Smith-Lovin, L., & Cook, J. M. (2001). Birds of a feather: Homophily in social networks. Annual Review of Sociology.

Reviewer comment (Evaluation):
Instead of hit metric, mrr is also very important for link prediction. It could be better if there is additional analysis on MRR.

Author response:
**Response**: We agree that MRR is also a very important metric. As such, we also add results with MRR. They are added to the revision in Appendix E: "ADDITIONAL RESULTS IN MAIN ANALYSIS". We observe that our existing conclusions are unchanged via the inclusion of MRR and in fact, are further verified.

Reviewer comment (Evaluation):
The work somehow shows correlation with the network evolution. Can you add some detailed discussion with the network evolution especially the dynamic case?

Author response:
**Response:** Thanks for your great question for the discussion about the insights in network evolution and link prediction. We provide the definition of network evolution and link prediction and make a comparison between them. We take the most well-adopted definition as follows. Link prediction is concerned with determining the likelihood of a link forming between two nodes in a network, given its current state. Network evolution is about understanding and modeling how networks change over time. This includes the formation of new links as well as the deletion of existing ones. Link prediction considers the network at a fixed point in time and doesn't account for how the network might evolve and grow temporally. Moreover, link prediction generally only considers a link addition while ignoring the link deletion.
 

Our paper focuses on a static link prediction setting, it is different from the network evolution considering temporal information. We leave extending our understanding of link prediction to network evolution as the future work. The core future steps for extension are:
- **link deletion:** The current latent space model considers node pairs with high proximity to be more likely to be connected. Future work can extend the model verifying that node pairs with low proximity are more likely to be deleted.
- **Temporal feature:** Network evolution introduces a time component, that further how time impacts the probability of a link occurring. Further work can consider temporal proximity as an additional data factor besides the existing three data factors we examined. 

The above discussion on the future is added in Appendix I: Future Work.