Paper: NetInfoF Framework: Measuring and Exploiting Network Usable Information (ICLR-2024)

Reviewer comment (Evaluation):
"How to connect the graph information with the performance metric on the task?" Here are some missed related works [1,2,3,4,5].

Author response:
Thank you for your supplement on related works. 
We have added them to Section 2 (Page 2) in our revised paper.
Noting that our focus has two major differences from [1, 2, 3, 4, 5]:
1. These works focus on node classification, while our work majorly focuses on **link prediction**.
2. These works focus on graphs that are heterophily defined by the node labels, while our work focuses on cases such as heterophily w.r.t. node features ("talkative people have silent friends"), as well as heterophily w.r.t. graph structure (forming bipartite cores). We did not consider node labels in link prediction.

Reviewer comment (Writing):
What is $f$ in C4? I suggest not using $f$, because you already use it as the number of features.

Author response:
As mentioned in the last sentence in C4 (Section 3.2, Page 3), $f$ is the column-wise L2 normalization. 
We replace it with $l$ in our revised paper for increased clarity.

Reviewer comment (Theory):
In theorem 1, "Given a discrete random variable Y , we have..." What is "accuracy(Y)"? Accuracy of a a discrete random variable? This looks weird and I suggest giving a more strict and accurate statement. What is $p_y$?

Author response:
As shown in Equation 1 (Page 4), given a discrete random variable, its accuracy is the probability of its most probable outcome. 
This is achieved by majority voting, which is the best we can do without any other information.
It is also described in Appendix A.1 (Page 13).

Reviewer comment (Theory):
"Predicting links by GNNs relies on measuring node similarity, which is incorrect if the neighbors have heterophily embeddings." The embedding will be similar even with heterophily connections if the nodes share similar neighborhood patterns [6].

Author response:
The main point is that we use **structure** embeddings (C1, Section 3.2, Page 3).

We have clarified our wording as follows (Page 5, 4 lines from the top):
*“Predicting links by GNNs relies on measuring node similarity, which is incorrect if the neighbors are expected to have dissimilar embeddings; for example, in a bipartite graph, …”* in our revised paper.

We agree with the reviewer that in the (label-)heterophily graphs, the **propagated** embeddings (C4 and C5) of connected nodes will be proximity embeddings, and are very likely to be similar.
However, for a (structure-)heterophily graph (e.g., consisting of bipartite cores), our proposed **structure** embeddings (C1), will not be similar even if they share similar neighborhood patterns.

Reviewer comment (Theory):
"It results in low value when node i and node j have heterophily embeddings, even if they are connected by an edge." This is not correct, see [2,3,6].

Author response:
The statement in the paper is correct, when we use **structure** embeddings (C1, Section 3.2, Page 3) in a (structure-)heterophily graph.

In our revised paper, we clarified it with the sentence (Page 5, Section 4.1): *“However, even if node $i$ and node $j$ are connected by an edge, it may result in low value if they are expected to have dissimilar embeddings (e.g. structure embeddings in a bipartite graph).”*

In more detail, in the cited papers [2, 3, 6], these are node features propagated through graph structure, which are similar to our derived **propagated** embeddings (C4 and C5).
As discussed in W4, the **propagated** embeddings of connected nodes can be similar in a (label-)heterophily graph, while the **structure** embeddings will not be similar in a (structure-)heterophily graph.

Reviewer comment (Experiments):
How does the NETINFOF framework handle noisy or incomplete graph data? Can it effectively measure and exploit network usable information in such scenarios?

Author response:
Yes, in such cases, NetInfoF will give low NetInfoF_Score to the corresponding noisy components in Section 3.2 (Page 3), and down-weigh them for the upcoming predictions.

Different graph scenarios are studied in Table 6 in Appendix C.1 (Page 16): 
For example, if the given graph has useful structure but very noisy features, NetInfoF gives high scores to the structural embeddings (C1 and C2), and a low score to the feature embedding (C3).

Reviewer comment (Experiments):
Are there specific types of graph structures or node features for which NETINFOF may not perform well? How robust is the framework in diverse graph settings?

Author response:
NetInfoF makes no assumption for either the graph structure or the node feature. 
In fact, as shown in Table 2 (Page 8) and Table 9 (Page 17), NetInfoF successfully tackles multiple graph scenarios on link prediction and node classification, respectively.

Reviewer comment (Experiments):
I have concerns regarding the sensitivity of NETINFOF to the estimated compatibility matrix (H) used in the framework. It would be beneficial if the authors could provide additional empirical results that examine the performance of NETINFOF under different label rates, as label rates can significantly impact the correctness of H.

Author response:
Thank you for your valuable suggestion. 
For link prediction, even a small fraction of existing links (=”labels”) is enough for estimating the compatibility matrix, as we show in the following table.
The table shows two medium size datasets “Computers” and “Actor”, where we vary the ratio of positive edges used for estimating the compatibility matrix.
NetInfoF still works well even with 10% of the edges for the compatibility matrix estimation, and it also performs better than without using the compatibility matrix.

For large graphs, we do not expect NetInfoF to be sensitive to the ratio of edge labels, since it requires only a few edge labels to well estimate the compatibility matrix. For example, in “ogbn-Products”, there are 43M (= 62M * 70%) edges in the training set; as we mentioned in T3 in Section 4.1, we sample only 200K for estimating the compatibility matrix, which is 0.5% (= 200K / 43M) edges in the training set.

| Edge Label Ratio | w/o CM | 10% | 30% | 50% | 70% |
| -------- | -------- | -------- | -------- | -------- | -------- |
| Comp.    | $27.9\pm0.2$ | $29.4\pm2.7$ | $33.0\pm1.8$ | $31.7\pm3.4$ | $31.1\pm1.9$ |
| Actor    | $29.5\pm1.8$ | $33.3\pm1.1$ | $33.9\pm1.1$ | $34.3\pm0.6$ | $36.2\pm1.2$ |

Reviewer comment (Theory):
The proof of Theorem 2 is problematic due to the incorrect use of conditional entropy. The proof incorrectly states the conditional entropy in Appendix A.2 as:
$$
H(Y|X) = \sum_{i=1}^{m} p_i \left(-\sum_{j=1}^{n} p_{ij} \log_2 p_{ij}\right),
$$
which is incorrect because it uses the joint probabilities $p_{ij}$ instead of the conditional probabilities. The correct expression for conditional entropy, which is based on the conditional probabilities, is:
$$
H(Y|X) = -\sum_{i=1}^{m} \sum_{j=1}^{n} p_{ij} \log_2 P(Y=y_j|X=x_i),
$$
where $P(Y=y_j|X=x_i) = \frac{p_{ij}}{p_i}$ is the conditional probability of $Y$ given $X=x_i$. And $p_i = \sum_{j=1}^{n} p_{ij}$ represents the marginal probability of $X$ taking the value $x_i$. This definition adheres to the fundamental property that the conditional probabilities for a given $x_i$ should sum to 1, i.e., $\sum_{j=1}^{n} P(Y=y_j|X=x_i) = 1$.

The misuse of conditional entropy in the proof leads to an erroneous application of Jensen's Inequality and subsequently invalidates the derived bound on the NetInfoF score.

Author response:
> W1 & Q1. Can the authors provide a more detailed justification for Theorem 2, particularly in light of the concerns highlighted in the first weakness?

Thank you for pointing out the problem, which is caused by the mistyping of the notations.
We update the proof of Theorem 2 in Appendix A.2 (Page 13), where our conclusion remains the same.

Reviewer comment (Experiments):
(Minor) Given the paper is mostly dealing with link prediction tasks. It will be beneficial to include some subgraph-based methods as baselines, such as SEAL [1]. I understand that NetInfoF is intended as an advancement beyond traditional GNN approaches, but including subgraph-based methods will give the community a more comprehensive evaluation of NetInfoF's capabilities. It's evident NetInfoF is way more scalable than subgraph-based methods, but subgraph-based methods are still SOTA on some datasets, e.g. Ogbl-collab.

Author response:
> W2. It will be beneficial to include some subgraph-based methods as baselines, such as SEAL.

We add the citations in Section 2 and run the experiments of SEAL with the default settings provided by OGB and set the hidden size to 128 as the same as other baselines, please find additional results in the following table.

The experiments are run by an AWS EC2 g4dn.metal instance, with 384GB RAM and Tesla T4 GPUs.
Noting that SEAL runs out of memory (O.O.M.) in the preprocessing step for several graphs, and exceeds time limit (T.L.E.) on “Photo”, taking more than 11 hours to run one (out of five) split.
Furthermore, as shown by Table 1 (Page 2), subgraph GNNs miss several desired properties focused by this study, i.e., SEAL is not scalable and can not be used on node classification, and NetInfoF is the only one that matches all properties.
Due to the time constraints, we are not able to conduct hyperparameter search and will conduct a more rigorous experiment in the next version. 

| Dataset | Cora | CiteSeer | PubMed | Comp. | Photo | ArXiv | Products | Cham. | Squirrel | Actor | Twitch | Pokec |
| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| SEAL     | $65.2\pm0.5$ | $60.0\pm1.1$ | $\mathbf{60.9\pm2.1}$ | $\color{red}{O.O.M.}$ | $\color{red}{T.L.E.}$ | $\color{red}{O.O.M.}$ | $\color{red}{O.O.M.}$ | $\mathbf{87.6\pm0.9}$ | $\color{red}{O.O.M.}$ | $\mathbf{43.8\pm0.8}$ | $\color{red}{O.O.M.}$ | $\color{red}{O.O.M.}$ |
| NetInfoF | $\mathbf{81.3\pm0.6}$ | $\mathbf{87.3\pm1.3}$ | $59.7\pm1.1$ | $\mathbf{31.1\pm1.9}$ | $\mathbf{46.8\pm2.2}$ | $\mathbf{39.2\pm1.8}$ | $\mathbf{35.2\pm1.1}$ | $86.9\pm2.3$ | $\mathbf{24.2\pm2.0}$ | $36.2\pm1.2$ | $\mathbf{19.6\pm0.7}$ | $\mathbf{31.3\pm0.5}$ |

Reviewer comment (Theory):
Can the authors provide a more detailed justification for Theorem 2, particularly in light of the concerns highlighted in the first weakness? I will raise my score if the authors can give a reasonable update.

Author response:
> W1 & Q1. Can the authors provide a more detailed justification for Theorem 2, particularly in light of the concerns highlighted in the first weakness?

Thank you for pointing out the problem, which is caused by the mistyping of the notations.
We update the proof of Theorem 2 in Appendix A.2 (Page 13), where our conclusion remains the same.

Reviewer comment (Theory):
How are the "Our bound" lines in Figures 3 and 4 derived? Additional details on this calculation would aid in understanding these figures.

Author response:
> Q2. How are the "Our bound" lines in Figures 3 and 4 derived?

We update the annotations in Figure 3 (Page 4) to make it more clear.
In more detail, according to Theorem 2, the training accuracy can not be lower than NetInfoF_Score, and thus our bound is the dash $x=y$ line in Figure 3. 
We show that even for validation accuracy, the results still obey Theorem 2.
In Figure 4, the fitting lines have high $R^2$ values, indicating that NetInfoF_Score is highly correlated with the test performance. 
Since it is difficult to connect a metric to the test performance, the ideal metric is expected to be highly correlated with the test performance.

Reviewer comment (Writing):
How NetInfoF_Probe estimates and calculates the discretizer in section 4.2? Maybe some examples or explanations will help reader understand this.

Author response:
> Q3. How NetInfoF_Probe estimates and calculates the discretizer in section 4.2?

In Section 4.2, for link prediction, we did the quantile bucketization with $k=8$ bins on the adjusted node similarity.
Due to space limitation, the detailed algorithm was provided in Algorithm 2 in the appendix (Page 15). 

Moreover, we conduct a sensitivity analysis on a medium size dataset “Computers" (see table below) and show that the result is insensitive to the exact value of $k$, forming a plateau when $k$ increases.
We put these results in the appendix (Table 8 and Figure 7, Page 16) in our revised paper. 

| Number of Bins | k=4 | k=8 | k=16 | k=32 | k=64 |
| -------- | -------- | -------- | -------- | -------- | -------- |
| C1: U    | $74.2\pm0.3$ | $79.7\pm0.2$ | $80.4\pm0.3$ | $80.8\pm0.3$ | $81.0\pm0.3$ |
| C2: R    | $76.5\pm0.2$ | $80.4\pm0.0$ | $81.2\pm0.3$ | $81.4\pm0.3$ | $81.5\pm0.3$ |
| C3: F    | $63.8\pm0.1$ | $64.9\pm0.2$ | $65.0\pm0.1$ | $65.0\pm0.1$ | $65.1\pm0.1$ |
| C4: P    | $77.3\pm0.2$ | $82.1\pm0.1$ | $83.0\pm0.3$ | $83.3\pm0.3$ | $83.4\pm0.3$ |
| C5: S    | $77.3\pm0.1$ | $82.5\pm0.1$ | $83.3\pm0.1$ | $83.6\pm0.2$ | $83.7\pm0.2$ |

Reviewer comment (Experiments):
An ablation study detailing the individual contributions of the five different node embedding components mentioned in Section 3.2 would be beneficial. Are these components equally influential in terms of accuracy, or do some weigh more significantly than others?

Author response:
> Q4. An ablation study detailing the individual contributions of the five different node embedding components mentioned in Section 3.2 would be beneficial. Are these components equally influential in terms of accuracy, or do some weigh more significantly than others?

In short, different components have variable impact depending on the input graph. 
It is an advantage of NetInfoF that it can pinpoint which component is useful.

In more detail, we studied the information of individual components in Appendix C.1 (Page 16) on the synthetic datasets.
In addition, we further conduct this experiment on real-world datasets, please find additional results in the following table (major component: in bold). 

This conclusion is similar to the one we have in Appendix C.1 (Page 16), where different input graphs may have different preferences on the components depending on the underlying scenarios.
Therefore, by combining all five components of node embeddings, NetInfoF is effective as well as robust.
We put this table in the appendix (Table 7, Page 16) in our revised paper.

| Dataset | Cora | CiteSeer | PubMed | Comp. | Photo | ArXiv | Products | Cham. | Squirrel | Actor | Twitch | Pokec |
| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| C1: U    | $48.3\pm0.8$ | $36.9\pm2.5$ | $43.7\pm1.8$ | $24.0\pm1.8$ | $38.5\pm2.6$ | $18.1\pm0.6$ | $13.2\pm0.3$ | $75.0\pm5.2$ | $12.3\pm1.8$ | $23.2\pm2.2$ | $\mathbf{15.8\pm0.2}$ | $\mathbf{16.2\pm0.6}$ |
| C2: R    | $61.5\pm1.2$ | $50.0\pm1.9$ | $47.9\pm0.8$ | $19.4\pm0.8$ | $36.8\pm3.3$ | $12.3\pm1.0$ | $09.4\pm0.7$ | $64.6\pm3.2$ | $08.2\pm1.4$ | $\mathbf{34.2\pm2.3}$ | $01.5\pm0.2$ | $05.2\pm0.2$ |
| C3: F    | $58.3\pm3.2$ | $71.5\pm2.4$ | $41.6\pm0.4$ | $07.1\pm0.4$ | $15.6\pm1.2$ | $04.9\pm0.2$ | $00.4\pm0.1$ | $10.7\pm1.0$ | $00.6\pm0.1$ | $02.6\pm0.5$ | $01.7\pm0.6$ | $00.5\pm0.2$ |
| C4: P    | $67.3\pm1.8$ | $60.9\pm2.5$ | $48.1\pm1.8$ | $\mathbf{28.6\pm2.4}$ | $\mathbf{42.4\pm1.4}$ | $\mathbf{34.2\pm1.0}$ | $27.6\pm0.5$ | $\mathbf{84.3\pm1.8}$ | $\mathbf{20.9\pm2.3}$ | $13.1\pm1.1$ | $07.3\pm1.0$ | $12.4\pm0.8$ |
| C5: S    | $\mathbf{82.4\pm1.0}$ | $\mathbf{88.7\pm1.5}$ | $\mathbf{63.3\pm1.1}$ | $29.0\pm2.3$ | $40.3\pm1.1$ | $33.6\pm1.1$ | $\mathbf{28.1\pm0.7}$ | $80.5\pm2.7$ | $19.2\pm1.2$ | $22.3\pm0.9$ | $02.7\pm1.8$ | $\mathbf{16.2\pm0.6}$ |