Paper: Noise Robust Graph Learning under Feature-Dependent Graph-Noise (ICLR-2024)

Reviewer comment (Writing):
Minor issues about the typo: “the graph structure OF node labels” in line 4 of the abstract should be “the graph structure OR node labels” if I am not misunderstanding.

Author response:
We sincerely appreciate your valuable feedback on our work and for recognizing the contributions made by our research! We have diligently addressed the typos in the manuscript to enhance its quality. The revised content is distinctly highlighted in orange.

Reviewer comment (Writing):
In line 5 of page 5, “introduces” should be “introduce”.

Author response:
We sincerely appreciate your valuable feedback on our work and for recognizing the contributions made by our research! We have diligently addressed the typos in the manuscript to enhance its quality. The revised content is distinctly highlighted in orange.

Reviewer comment (Novelty):
The proposed setting is a combination of popular GNN with label noise and [1]. It is better to clarify more application examples in real-world.

Author response:
**(W1-1) FDGN is a combination of structure noise and label noise.**

We would like to emphasize that our proposed FDGN is not a naive combination of independent node label noise and graph structure noise (which is referred to [1] by the reviewer). In this work, we introduce a more realistic noise scenario, where **noisy node features may entail both structure and label noise**, which we call feature-dependent graph-noise (FDGN). In other words, we assume that there exist "causal relationships" among node feature noise, graph structure noise, and node label noise, rather than assuming that these noise occur independently, lacking any causal connections between them. We argue that the proposed FDGN is more comprehensive and applicable in various real-world scenarios compared to the naive combination, because such causal relationships are evident in a range of real-world applications (as will be described below).

**(W1-2) More application examples in the real-world of FDGN**

Upon the reviewer’s request for additional application examples of FDGN, we provide additional examples of real-world applications:

**Biological networks**

A cell-cell graph is widely used in computational biology [1,2,3]. In the cell-cell graph, a node denotes a cell, each node is associated with features that represent gene expression values, and each node is labeled with the cell type. As the graph structural information is missing by nature, a cell-cell graph is usually constructed based on the node feature similarity [1,2,3]. However, gene expression values (i.e., cell-gene count matrix) often contain noise due to dropout phenomenon and batch effect, and such noise may entail noisy cell-cell graph structures, which can be referred to as FDGN. Furthermore, since the cell type (i.e., node label) is annotated by using transcripted marker genes (i.e., important features), noisy node features may lead to noisy node labels, which can be referred to as FDGN.

[1] scGNN is a novel graph neural network framework for single-cell RNA-Seq analyses. Nature communications 2021

[2] scGCL: an imputation method for scRNA-seq data based on graph contrastive learning. Bioinformatics 2023

[3] Single-cell RNA-seq data imputation using Feature Propagation. ICML 2023 Workshop on Computational Biology

**Recommender systems in many domains**

In recommender systems of various domains (e.g., e-commerce, news, movie, etc), user-item interaction graphs are common, where node feature represents the information of users/items, and the user-item interaction represents the graph structures. Besides, users’ communities (interests) and items’ categories denote node labels. In this situation, if a user creates a fake or incomplete profile due to various reasons (e.g., privacy), node features would become noisy. Due to such noisy node features, items that are irrelevant to the user’s genuine interest can be exposed to users, which would make them interact with irrelevant items, resulting in noisy user-item interaction graph structures. This is an example of FDGN. Furthermore, such noisy information of users and noisy user-item interactions may also eventually change the users’ communities (i.e., noisy node labels), which can be referred to as FDGN.

We appreciate the feedback and have included these examples in Appendix H. We believe it effectively addresses the concern.

Reviewer comment (Novelty):
A lot of GNN with label noise works are missed [2].

Author response:
**(W2) A lot of GNN with label noise works are missed**

We agree with the reviewer’s comment about the missing baselines. Hence, we further compare PRINGLE with the widely used label noise baselines: Co-teaching+ [1], CP [2], D-GNN [3], and CGNN [4].

[1] How does disagreement help generalization against label corruption? ICML 2019

[2] Adversarial Label-Flipping Attack and Defense for Graph Neural Networks. ICDM 2020.

[3] Learning graph neural networks with noisy labels. ICLR LLD Workshop 2019

[4] Learning on Graphs under Label Noise. ICASSP 2023

From the below table, we clearly see that the proposed method, PRINGLE, outperforms all the baselines in terms of the node classification accuracy. We argue that these baseline methods are designed to tackle the noisy node labels while assuming that both node features and graph structures are noise-free. However, the proposed FDGN introduces a more realistic noise scenario, wherein node features, graph structures, and node labels simultaneously contain noise. As a consequence, the performance of these existing methods is considerably more constrained compared to PRINGLE.

|  Dataset |  Setting  | Coteaching+ |    CP    |   D-GNN  |   CGNN   |  PRINGLE |
|:--------:|:---------:|:-----------:|:--------:|:--------:|:--------:|:--------:|
|   Cora   |   Clean   |   84.7±0.9  | 84.3±0.4 | 83.7±0.5 | 85.2±0.7 | **86.2±0.7** |
|          | FDGN-10\% |   76.9±0.5  | 78.7±0.6 | 78.6±0.5 | 77.4±0.3 | **82.9±0.6** |
|          | FDGN-30\% |   66.0±0.8  | 68.2±0.5 | 68.5±0.4 | 69.2±0.8 | **78.2±0.3** |
|          | FDGN-50\% |   55.0±0.1  | 53.1±0.9 | 57.7±0.3 | 55.1±0.2 | **69.7±0.6** |
| Citeseer |   Clean   |   72.7±0.4  | 72.8±0.9 | 75.1±0.2 | 71.1±0.9 | **77.3±0.6** |
|          | FDGN-10\% |   67.7±0.6  | 68.4±0.8 | 69.0±0.5 | 65.6±0.4 | **74.3±0.9** |
|          | FDGN-30\% |   55.0±0.9  | 56.8±0.9 | 54.0±0.7 | 54.1±0.3 | **65.6±0.6** |
|          | FDGN-50\% |   47.4±0.8  | 46.5±1.0 | 44.7±0.1 | 46.9±1.4 | **59.0±1.8** |
|   Photo  |   Clean   |   93.1±0.0  | 93.3±0.5 | 93.1±0.1 | 92.7±0.5 | **94.8±0.3** |
|          | FDGN-10\% |   87.9±0.8  | 90.5±0.5 | 90.3±0.6 | 87.1±0.3 | **93.2±0.2** |
|          | FDGN-30\% |   83.1±0.2  | 85.1±1.0 | 85.9±0.3 | 85.1±0.2 | **90.5±0.4** |
|          | FDGN-50\% |   81.5±0.3  | 80.4±0.6 | 85.1±0.8 | 80.5±0.7 | **87.6±0.2** |
|   Comp  |   Clean   |   88.6±0.8  | 90.7±0.3 | 89.4±0.9 | 90.0±0.5 | **92.2±0.0** |
|          | FDGN-10\% |   85.6±0.6  | 87.1±0.8  | 86.8±0.4 | 83.0±0.4 | **89.8±0.2** |
|          | FDGN-30\% |   81.5±0.3  | 82.8±0.6 | 82.9±0.5 | 82.3±0.8 | **86.9±0.3** |
|          | FDGN-50\% |   72.8±0.9  | 74.3±1.0 | 74.5±0.6 | 75.3±0.1 | **82.2±0.4** |

We acknowledge and appreciate the feedback provided, and as a response, we have incorporated the corresponding results in Appendix I. We are confident that this inclusion effectively addresses the raised concerns.

Reviewer comment (Experiments):
Why the last three losses share the same weights in Eq. 4?

Author response:
**(W4) Why the last three losses share the same weights in Eq. 4?**

In our experiments, we observed that the last three loss terms, namely $L_{rec-feat}$, $L_{cls-dec}$, and $L_{p}$, have a relatively minor impact on the model's performance compared to the others. As a result, we have made a strategic decision to simplify the hyperparameter search process and improve the practicality of PRINGLE by sharing the weights $\lambda_3$ among these last three loss terms.
 
We appreciate the reviewer's feedback and have addressed this concern by including a detailed explanation in the implementation details section in Appendix D.5. The revised content is distinctly highlighted in orange.

Reviewer comment (Theory):
Why the generative methods can release the label noise?

Author response:
**(W5) Why the generative methods can release the label noise?**

We acknowledge that deep generative models typically do not address label noise as a primary focus. However, in the context of our proposed FDGN, the DGP accounts for label noise that is generated in a feature-dependent manner. Hence, directly modeling the DGP of FDGN is beneficial for the model to recognize the way that the label noise is generated. Consequently, our modeling approach effectively mitigates the impact of the label noise.

Reviewer comment (Novelty):
The proposed solution lacks technical novelty, using VAE to model the causal relationship and counter noise has already been proposed by [1]. This paper only incrementally adapts that solution on the graph.

Author response:
**(W1) This paper only incrementally adapts CausalNL [1] on the graph.**

We agree with the reviewer's opinion that our work may appear to lack technical novelty in comparison to [1]. However, it is important to note that we tackle additional, more challenging aspects unique to our specific problem that do not exist in [1]. Specifically, when additionally introducing $A$, we handle four more causal relationships: $\epsilon → A, X → A, A → Y, Z_A → A$, each of which is non-trivial to consider. We would like to specify the challenges in instantiations caused by their additional introduction.

**Inference of $\epsilon$**

In contrast to [1], which assumes that $\epsilon$ is a cause of $X$, our proposed FDGN consider that $\epsilon$ is a cause of both $X$ and $A$. In other words, the DGP of FDGN contains the causal relationships $\epsilon → X$ and $\epsilon → A$, as real-world applications often exhibit graph structure noise originating from arbitrary sources (i.e., $\epsilon$) in addition to the feature-dependent noise. Therefore, this scenario is a unique characteristic of our problem and not addressed in [1]. To deal with this, we decompose $q_{\phi_2}(\epsilon |X,A,Z_Y)$ into $q_{\phi_{21}}(\epsilon_X | X, Z_Y)$ and $q_{\phi_{22}}(\epsilon_A|X,A)$. While the instantiation of $q_{\phi_{21}}(\epsilon_X | X, Z_Y)$ is similar to [1], that of $q_{\phi_{22}}(\epsilon_A|X,A)$ is non-trivial and is absent in [1]. In our approach, we regard $\epsilon_A$ as a set of scores indicating the likelihood of each observed edge being noisy or not. Moreover, we leverage the concept of early-learning phenomenon to infer $\epsilon_A$. It is worth emphasizing once again that the instantiation of this scenario is novel and not a straightforward extension of [1].

**Loss term $kl(q_{\phi_1}(Z_A|X,A)||p(Z_A))$**

To compute this loss term, we encounter two primary challenges:
1. Designing an appropriate prior for the latent graph structure $p(Z_A)$
2. Addressing the complexity associated with calculating the KL divergence between the two matrices sampled from Bernoulli distributions.

In response to the first challenge, we employ the $\gamma$-hop subgraph similarity as a metric to identify assortative edges.
Regarding the second challenge, we introduce a predefined candidate graph, which includes the observed edge set along with a $k$-NN graph based on the $\gamma$-hop subgraph similarity. Both of these challenges represent non-trivial aspects of our approach and are absent in [1].

**Loss term $kl(q_{\phi_3}(Z_Y|X,A)||p(Z_Y))$**

To compute this loss term, [1] employs a uniform distribution as the prior $p(Z_Y)$. In contrast, we introduce the concept of class homophily to effectively regularize the inference of latent clean node label $Z_Y$. Specifically, we encourage $Z_Y$ to align with our prior knowledge, i.e., $p(Z_Y)$, that the two end nodes on the accurately inferred latent graph structure $Z_A$ are expected to have identical latent labels. Therefore, using this prior helps alleviate the noisy node label issues.

It is essential to highlight that this instantiation effectively utilizes the property unique to the graph domain, which is not present in [1]. It is worth emphasizing once again that such an instantiation is novel and not a straightforward extension of [1].

*To be continued in the following post*

Reviewer comment (Theory):
The proposed solutions lack theoretical support, the derivation on ELBO are well-known results, and the authors are only re-stating them here.

Author response:
**(W2) The derivation on ELBO are well-known results, and the authors are only re-stating them here.**


We acknowledge that the derivation of the Evidence Lower Bound (ELBO) is based on well-known results. However, we would like to emphasize that our primary contribution lies in proposing the DGP of FDGN, which is a more realistic scenario in graph learning, not in the derivation of ELBO. In other words, considering that the ELBO of our objective has been derived by following the well-known derivation procedure, our main focus was on how to implement each term in the derived ELBO, which is non-trivial. Please refer to our response to [W1](https://openreview.net/forum?id=kNGxg8shA1&noteId=bVLGnXwfMA) for more detail on how each term is implemented. We fully agree with the reviewer’s opinion that the re-stating of the ELBO derivation may seem redundant as they are well-known, and we are willing to remove them from the paper.

Reviewer comment (Evaluation):
The proposed solution seems to have very high complexity (there are three encoder-decoder pairs, and three objectives to compute), therefore an efficiency analysis is needed.

Author response:
**(W3) An efficiency analysis is needed.**

We thank the reviewer for pointing out the potential high computational complexity of our proposed method.

Hence, we compare the training time of PRINGLE with the baselines to analyze the computational complexity of PRINGLE. In the below table, we report the total training time and training time per epoch on Cora with FDGN 50% for all models. Note that since STABLE is a 2-stage method, we did not report the training time per epoch. The results show that PRINGLE requires significantly less total training time and training time per epoch compared with WSGNN, ProGNN, RSGNN, STABLE, NRGNN, and RTGNN. This suggests that PRINGLE can be efficiently trained, while still achieving substantial performance improvements.

Although AirGNN and EvenNet require much less training time than PRINGLE, their node classification accuracy is notably worse than other methods, including PRINGLE. This indicates that, despite their fast training time, they may not be suitable for real-world scenarios. In summary, PRINGLE demonstrates superior performance compared to the baselines while maintaining acceptable training time.

|                             | WSGNN  | AirGNN | ProGNN | RSGNN  | STABLE | EvenNet | NRGNN  | RTGNN  | PRINGLE |
|-----------------------------|--------|--------|--------|--------|--------|---------|--------|--------|---------|
| Total training time (sec)   | 93.90  | 20.9   | 702.14 | 159.87 | 53.33  | 0.81    | 100.33 | 118.76 | 46.27   |
| Training time / epoch (sec) | 0.19   | 0.04   | 1.77   | 0.16   | -      | 0.004   | 0.20   | 0.18   | 0.09    |


We appreciate the feedback and have incorporated these experimental results into Appendix J. We believe it effectively addresses the concerns related to computational complexity.