Paper: Robust Graph Neural Networks via Unbiased Aggregation (ICLR-2024)

Reviewer comment (Novelty):
The contributions are weak as the formulation is essentially based on TWIRLS with the MCP penalty by Zhang, 2010. Robustness issues in Lasso regression have also been well-studied.

Author response:
We respectfully disagree with this comment. Our contributions are significant from many perspectives.

(1) First of all, the graph signal denoising perspective [1] is generally discovered as a powerful and principled paradigm for encoding prior knowledge about data. TWIRLS is just one such example. Our RUNG method differs from TWILRS from two key perspectives: different formulation and optimization solver. Therefore, our graph convolution layer in Eq. (7) derived from the proposed Quasi-Newton IRLS algorithm is totally different from the TWIRLS. Moreover, the robustness of our RUNG model is significantly stronger than TWIRLS in our experiments, especially under large attacking budgets.

(2) While the robustness issue in Lasso regression is well-studied, there is no work revealing the negative impact of these issues on the robustness of GNNs. In this paper, we provide the first study and unified view to discover that the robustness degradation of a class of state-of-the-art $\ell_1$-based GNNs including SoftMedian, ElasticGNN, and TWIRLS is caused by the accumulation of estimation bias. Note that such accumulation is not observed in Lasso regression. From this novel discovery, we design a novel RUNG model to significantly avoid such strong robustness degradation, which clearly advances the state-of-the-art in GNN defense.

(3) We would also like to advocate that it is imperative to design deep learning architectures with well-understood statistical principles (such as the MCP penalty we study in this paper). The recent work [2] already revealed a bitter lesson: many heuristically designed robust GNNs are not truly robust, majorly because of the lack of principled design and understanding of their robustness. On the contrary, the principally designed $\ell_1$-based GNNs exhibit the best robustness, and this work further pushes this research boundary and makes a solid step forward. 

Due to the above reasons, we believe the contributions and impact of our work are significant.

[1] Ma et al., "A Unified View on Graph Neural Networks as Graph Signal Denoising", CIKM'21

[2] Mujkanovic et al., "Are Defenses for Graph Neural Networks Robust?", NIPS 2022

Reviewer comment (Novelty):
Most of the cited works are from 2021 or before. The authors are missing recent GNN robustness works like “On the robustness of graph neural diffusion to topology perturbations", NeurIPS 2022 and “Graph-coupled oscillator networks", ICLR 2022.

Author response:
Thanks for your suggestions. We have evaluated the robustness of more than 8 robust GNN baselines such as SoftMedian [1] and GNNGuard [3], which are widely recognized as state-of-the-art defenses [2] and are still competitive with or superior to more recent models. Nevertheless, we have also included two more baselines published recently, EvenNet and GARNET as shown below.


*Global adaptive evasion attack on Cora. **Best** and <u>second</u> highlighted. Prediction accuracy in percentage:*
| Budget                 | Clean          | 5\%                        | 10\%                       | 20\%                       | 30\%                       | 40\%                     |
| ---------------------- | -------------- | -------------------------- | -------------------------- | -------------------------- | -------------------------- | ------------------------ |
| GCN                    | $85.0 \pm 0.4$ | $75.3 \pm 0.5$             | $69.6 \pm 0.5$             | $60.9 \pm 0.7$             | $54.2 \pm 0.6$             | $48.4\pm0.5$             |
| EvenNet                | $84.8 \pm 0.9$ | $75.8 \pm 0.9$             | $70.7 \pm 0.6$             | $63.9 \pm 0.5$             | $58.7 \pm 0.7$             | $54.7 \pm 0.8$           |
| GARNET                 | $84.0 \pm 0.5$ | $76.5 \pm 0.4$             | $72.1 \pm 0.1$             | $66.4 \pm 0.7$             | $62.1 \pm 1.3$             | $58.7 \pm 1.5$           |
| SoftMedian             | $85.0 \pm 0.7$ | $\underline{78.6 \pm 0.3}$ | $\underline{75.5 \pm 0.9}$ | $\underline{69.5 \pm 0.5}$ | $62.8 \pm 0.8$             | $58.1\pm0.7$             |
| RUNG-$\ell_{1}$ (ours) | $85.8 \pm 0.5$ | $78.4 \pm 0.4$             | $74.3 \pm 0.3$             | $68.1 \pm 0.6$             | $\underline{63.5 \pm 0.7}$ | $\underline{59.8\pm0.8}$ |
| RUNG (ours)            | $84.6 \pm 0.5$ | $\mathbf{78.9 \pm 0.4}$    | $\mathbf{75.7 \pm 0.2}$    | $\mathbf{71.8 \pm 0.4}$    | $\mathbf{67.8 \pm 1.3}$    | $\mathbf{65.1 \pm 1.2}$  |

*Global adaptive evasion attack on Citeseer. **Best** and <u>second</u> highlighted. Prediction accuracy in percentage:*
| Budget                 | Clean          | 5\%                        | 10\%                       | 20\%                       | 30\%                       | 40\%                     |
| ---------------------- | -------------- | -------------------------- | -------------------------- | -------------------------- | -------------------------- | ------------------------ |
| GCN                    | $74.8 \pm 1.2$ | $66.1 \pm 1.0$             | $60.9 \pm 0.8$             | $53.0 \pm 1.0$             | $47.0 \pm 0.8$             | $41.2\pm1.1$             |
| EvenNet                | $74.6 \pm 0.5$ | $66.8 \pm 0.5$             | $62.0 \pm 0.6$             | $55.9 \pm 0.4$             | $51.1 \pm 0.4$             | $47.4 \pm 0.8$           |
| GARNET                 | $74.8 \pm 1.3$ | $68.0 \pm 0.9$             | $64.0 \pm 1.1$             | $58.2 \pm 0.7$             | $53.9 \pm 0.8$             | $51.0 \pm 0.9$           |
| SoftMedian             | $74.6 \pm 0.7$ | $68.0 \pm 0.7$             | $64.4 \pm 0.9$             | $59.3 \pm 1.1$             | $55.2 \pm 2.0$             | $51.9\pm2.1$             |
| RUNG-$\ell_{1}$ (ours) | $75.5 \pm 1.1$ | $\underline{69.3 \pm 1.2}$ | $\underline{65.9 \pm 1.2}$ | $\underline{61.1 \pm 1.1}$ | $\underline{57.2 \pm 1.4}$ | $\underline{53.9\pm1.3}$ |
| RUNG (ours)            | $74.3 \pm 0.7$ | $\mathbf{71.4 \pm 1.0}$    | $\mathbf{69.8 \pm 1.3}$    | $\mathbf{67.6 \pm 1.2}$    | $\mathbf{66.5 \pm 1.3}$    | $\mathbf{65.3 \pm 1.5}$  |

From these results, we found that the more recent baselines are still less robust than $\ell_1$ baselines. 

For the two models you mentioned: 
(1) The proposed method in [6] is reported to underperform GNNGuard which has been included in the baselines of our submission; (2) The Graph Coupled Oscillator in [7] is not a GNN defense method and is certainly not robust.

Therefore, we believe our evaluation has demonstrated our state-of-the-art robustness performance.



[1] Geisler, Simon, et al. "Robustness of Graph Neural Networks at Scale.", NIPS'21

[2] Mujkanovic et al., "Are Defenses for Graph Neural Networks Robust?", NIPS 2022

[3] Zhang et al., "GNNGuard: Defending Graph Neural Networks against Adversarial Attacks", NIPS'20

[4] Lei et al., "EvenNet: Ignoring Odd-Hop Neighbors Improves Robustness of Graph Neural Networks", NIPS'22

[5] Song et al., "On the robustness of graph neural diffusion to topology perturbations", NIPS'22

[6] Rusch et al., "Graph-coupled oscillator networks", ICML'22

Reviewer comment (Experiments):
The experiments are limited with tests only on Cora and CiteSeer. Attacks like TDGIA and MetaGIA are not considered.

Author response:
We did experiments on the widely used Cora and Citeseer datasets since the strong adaptive attack on these datasets is well-studied. We believe they serve as a credible verification of the robustness of RUNG which avoids the false sense of robustness when evaluating our model. Nevertheless, we have done new experiments that provide stronger evidence of RUNG's superior robustness.

**ogbn-arxiv**

To demonstrate the scalability of RUNG, we have also conducted experiments on ogbn-arxiv.

*Global evasion attack on ogbn-arxiv. Prediction accuracy in percentage:*
| Budget               | Clean  | 1\%                   | 5\%                   | 10\%                  |
| -------------------- | ------ | --------------------- | --------------------- | --------------------- |
| GCN                  | $71.9$ | $63.1\pm0.4$          | $48.9\pm 3.2$         | $41.8\pm 0.5$         |
| RUNG-$\ell_1$ (ours) | $71.6$ | $\mathbf{65.5\pm0.4}$ | $55.0\pm 1.1$         | $49.6\pm1.1$          |
| RUNG (ours)          | $70.2$ | $65.2\pm0.2$          | $\mathbf{64.0\pm0.8}$ | $\mathbf{61.2\pm0.7}$ |

The results verified the superior robustness of RUNG model especially under larger attack budgets. 

RUNG's robustness outperforms its $\ell_1$ variant which delivers similar performance as SoftMedian and Elastic GNN due to their shared $\ell_1$ graph smoothing, which is also the strongest baseline as shown in our experiments. We will add other baselines (if scalable) in the final version of our paper. 


**TDGIA**

In order to more comprehensively evaluate the robustness of RUNG, we also conducted injection attack following your suggestion of considering TDGIA [1]. In the experiment, we set the budget on the number of injected nodes as 100 and the budget on degree as 200.

*Prediction accuracy in percentage, Targeted Injection Attack on Citeseer:*
| Budget        | Clean   | Attacked |
| ------------- | ------- | -------- |
| GCN           | $75.4$  | $28.14$  |
| RUNG-$\ell_1$ | $75.01$ | $39.22 $ |
| RUNG (ours)   | $75.65$ | $51.13$  |

The results show that our RUNG significantly outperforms the baseline models. Its improvement over RUNG-$\ell_1$ again demonstrates RUNG's capability of addressing the bias accumulation problem of $\ell_1$ models.

Reviewer comment (Novelty):
A quasi-Newton optimization approach is proposed. Is this related to standard quasi-Newton optimization procedures like BFGS, Broyden, etc.?

Author response:
Thanks for your question. Our Quasi-Newton IRLS in RUNG model is very different from standard quasi-Newtom methods such as BFGS and Broyden. These two methods are not compatible with the efficiency and simplicity requirements of GNNs. Therefore, we design a novel quasi-Newton method that is customized for our problem. The approximate Hessian of our method is based on the upper bound of the optimization objective and is diagonal, which delivers great computation efficiency, preserving RUNG's scalability and meanwhile offering a convergence guarantee for the RUGE objective as we proved in Appendix B.

Reviewer comment (Theory):
RUGE is claimed to be unbiased. Is this true and is it proven? An analytical robustness measure is also not provided.

Author response:
Thanks for your comments. Actually, the unbiasedness of RUGE has both the theoretical guarantee and the empirical evidence. In our paper, we provide multiple evidence from various perspectives to clarify this:

(1) From the theoretical perspective, different from the $\ell_1$ regularization which induces an estimation bias, the MCP-based problem has a nearly unbiased solution as proved in [1,2]. Roughly speaking, as shown in Figure 3, if the edge difference is larger than some threshold, the bias is totally removed; if the difference is small, the bias is
also limited. 

(2) From the algorithm perspective, the firm thresholding operator $F_{\lambda,\gamma}(\theta)$ induced by the MCP-based problem is $\frac{\gamma}{\gamma-1}S_\lambda( \theta)$ when $ |\theta|<\gamma\lambda$ and keep the identity $\theta$ when $|\theta|>\gamma\lambda$. When $\theta$ is larger than the threshold $\gamma\lambda$, the estimator keeps unbiased instead of being biased towards 0 as soft-thresholding.

(4) From the performance perspective, our RUNG outperforms the $\ell_1$-based GNNs such as SoftMedian, TWIRLS, and RUNG-$\ell_1$ (the $\ell_1$ variant of our model). It can tolerate severe feature differences when the attack budget increases, avoiding suffering from significant performance degradation. 

(5) From our ablation study in Figure 6, we quantify the estimation bias of the aggregated feature $f_i^\star$ on the attacked graph from the feature $f_i$ on the clean graph: $\sum_{i\in\mathcal{V}} \|f_i-f_i^*\|_2^2$. The results demonstrate that our proposed RUNG method exhibits a nearly zero estimation bias under adversarial attacks.

We believe all of this evidence can convincingly support our claim that our RUNG estimator is nearly unbiased, which validates the motivation of our robust GNN model. We will incorporate these discussions in our revision to further clarify the unbiasedness of RUNG.


[1] Varma, Rohan, et al. “Vector-Valued Graph Trend Filtering with Non-Convex Penalties.” IEEE Transactions on Signal and Information Processing over Networks, vol. 6, 2020, pp. 48–62.

[2] Zhang, Cun-Hui. “Nearly Unbiased Variable Selection under Minimax Concave Penalty.” The Annals of Statistics, vol. 38, no. 2, Apr. 2010.

Reviewer comment (Experiments):
Authors leverage QN-IRLS to approximate the inverse Hessian matrix to address the scalability issue. However, the experiments are only conducted on two small graphs, making it unclear whether the proposed approach would still work on large graphs.

Author response:
We understand your concern. We have added an analysis of our space and time complexity in Appendix C. Let us address your concern about the scalability of our RUNG here:

(1)  RUNG is scalable with time complexity of $O(k(m + n)d)$ and space complexity $O(m+nd)$, where $m$ is the number of edges, $d$ is the number of features, $n$ is the number of nodes, and $k$ is the number of GNN layers. Therefore, the complexity of our RUNG is comparable to normal GCN (with a constant difference) and it is feasible to implement.

(2)  To further mitigate your concern with empirical evidence, we also provide a comparison between the runtime of GCN and RUNG with the same layer number (3 layers) on ogbn-arxiv. The result verifies the efficiency of RUNG.

*Training speed on ogbn-arxiv:*
| Model          | GCN         | RUNG        |
| -------------- | ----------- | ----------- |
| Training speed | $23.3$ it/s | $10.9$ it/s |


To verify the robustness of RUNG when scaling up, we conducted experiments on the larger dataset of ogbn-arxiv.

*Global evasion attack on ogbn-arxiv. Prediction accuracy in percentage:*
| Budget        | Clean           | 1%                    | 5%                    | 10%                   |
| ------------- | --------------- | --------------------- | --------------------- | --------------------- |
| GCN           | $\mathbf{71.9}$ | $63.1\pm0.4$          | $48.9\pm 3.2$         | $41.8\pm 0.5$         |
| RUNG-$\ell_1$ | $71.6$          | $\mathbf{65.5\pm0.4}$ | $55.0\pm 1.1$         | $49.6\pm1.1$          |
| RUNG (ours)   | $70.2$          | $65.2\pm0.2$          | $\mathbf{64.0\pm0.8}$ | $\mathbf{61.2\pm0.7}$ |

The results verified the superior robustness of the RUNG model especially under larger attack budgets. 

RUNG's robustness outperforms its $\ell_1$ variant (RUNG-$\ell_1$) which delivers similar performance as SoftMedian and Elastic GNN due to their shared $\ell_1$ graph smoothing, which is also the strongest baseline as shown in our experiments. We will add other baselines (if scalable) in the final version of our paper.

Reviewer comment (Experiments):
The accuracy improvement under small attack budgets is less convincing. For instance, the accuracy gap between RUNG and the runner-up model is often smaller than the standard deviation, as observed under local attacks with a 20% budget on CoraML. My concern is that RUNG might underperform the baselines with a new random seed, considering that the authors obtained averaged accuracy using only 5 different random splits.

Author response:
We understand your concern and let's address it as follows. 

(1) In most cases where the performance gap is close, the runner-up model is an $\ell_1$ model (SoftMedian or RUNG-$\ell_1$). However, the $\ell_1$ model is actually a special case of RUNG, with $\gamma\rightarrow\infty$. Therefore, RUNG can always achieve at least the performance of the $\ell_1$ model.

(2) When the attack budget is small, $\ell_1$ models do not suffer from a strong accumulation of bias so they are comparable to RUNG. However, when the attack budget escalates, the bias effect will become dominant and RUNG will demonstrate a significant improvement. In fact, the large-budget performance is our emphasis when designing RUNG. An example shown below, in addition to the experiments in the paper, supports our claim.

| Budget        | 50\%           | 80\%           | 100\%          |
| ------------- | -------------- | -------------- | -------------- |
| MLP           | $65.0 \pm 1.0$ | $65.0 \pm 1.0$ | $65.0 \pm 1.0$ |
| GCN           | $43.2 \pm 1.2$ | $31.7 \pm 1.6$ | $25.7 \pm 2.8$ |
| RUNG-$\ell_1$ | $52.8 \pm 1.4$ | $46.5 \pm 1.8$ | $44.5 \pm 2.5$ |
| RUNG (ours)   | $68.4 \pm 1.2$ | $66.6 \pm 1.4$ | $65.4 \pm 1.7$ |

Reviewer comment (Experiments):
Authors have not performed sensitivity analyses on critical hyperparameters such as $\lambda$ and $\gamma$. It's unclear how these hyperparameters affect the performance of RUNG.

Author response:
Thank you for the comments. We followed your suggestions to conduct a study on the accuracy of RUNG with different combinations of $\gamma$ and $\lambda$ under different attack intensities in Appendix G.1. 

The tuning of $\lambda$ is similar to other graph-smoothing based models [1-3] and has been well studied. Moreover, its optimal value is stable across budgets, so $\lambda$ is omitted from the result below. Now, we show the accuracy of RUNG under different attack budgets.

*Global adaptive evasion attack on Cora. Prediction accuracy in percentage:*
| $\downarrow\gamma$ \ Budget $\rightarrow$ | $5\%$            | $10\%$           | $20\%$           | $40\%$           |
| ----------------------------------------- | ---------------- | ---------------- | ---------------- | ---------------- |
| $0.5$                                     | $ 73.1 \pm  0.8$ | $ 72.8 \pm  0.7$ | $ 72.2 \pm  0.6$ | $ 71.7 \pm  0.6$ |
| $1$                                       | $ 74.8 \pm  0.5$ | $ 73.5 \pm  0.4$ | $ 71.6 \pm  0.5$ | $ 69.4 \pm  0.8$ |
| $3$                                       | $ 79.1 \pm  0.4$ | $ 75.7 \pm  0.3$ | $ 71.4 \pm  0.5$ | $ 65.4 \pm  1.6$ |
| $5$                                       | $ 79.4 \pm  0.3$ | $ 75.9 \pm  0.2$ | $ 71.0 \pm  0.8$ | $ 64.0 \pm  2.3$ |


The result above verifies that RUNG's performance is not very sensitive to the value of $\gamma$ and it varies in a simple way: under small budgets, a larger $\gamma$ is better, while when the budget is larger, a smaller $\gamma$ is more favorable. $\gamma$ can thus be easily tuned. 

We would like to emphasize that in our experiments, we keep $\gamma$ and $\lambda$ the same under different budgets for the sake of fairness. However, if the attack budget is known in advance, RUNG can be easily tuned to achieve even better performance than reported, which is also a significant advantage of our model.

Reviewer comment (Experiments):
Some recent defense models (e.g., [1, 2, 3]) are not compared in this work.

Author response:
Thanks for your comments. We would like to point out that STABLE and GARNET are the purification methods as ProGNN which is already included in our submission and has been verified to be weak in robustness under the adaptive attack. 

Following your suggestions, we added these baselines to our experiments to make a comparison, except STABLE [1] on which how to conduct adaptive attack remains unknown. Nevertheless, the experiment results in [4] show that the robustness of STABLE is worse than RGCN (included in our experiments) across several attacks (e.g., Nettack, TDGIA, G-NIA, etc.).

The experimental results with the new baselines are shown below. 

*Global adaptive evasion attack on Cora. **Best** and <u>second</u> highlighted. Prediction accuracy in percentage:*
| Budget                 | Clean          | 5\%                        | 10\%                       | 20\%                       | 30\%                       | 40\%                     |
| ---------------------- | -------------- | -------------------------- | -------------------------- | -------------------------- | -------------------------- | ------------------------ |
| GCN                    | $85.0 \pm 0.4$ | $75.3 \pm 0.5$             | $69.6 \pm 0.5$             | $60.9 \pm 0.7$             | $54.2 \pm 0.6$             | $48.4\pm0.5$             |
| EvenNet                | $84.8 \pm 0.9$ | $75.8 \pm 0.9$             | $70.7 \pm 0.6$             | $63.9 \pm 0.5$             | $58.7 \pm 0.7$             | $54.7 \pm 0.8$           |
| GARNET                 | $84.0 \pm 0.5$ | $76.5 \pm 0.4$             | $72.1 \pm 0.1$             | $66.4 \pm 0.7$             | $62.1 \pm 1.3$             | $58.7 \pm 1.5$           |
| SoftMedian             | $85.0 \pm 0.7$ | $\underline{78.6 \pm 0.3}$ | $\underline{75.5 \pm 0.9}$ | $\underline{69.5 \pm 0.5}$ | $62.8 \pm 0.8$             | $58.1\pm0.7$             |
| RUNG-$\ell_{1}$ (ours) | $85.8 \pm 0.5$ | $78.4 \pm 0.4$             | $74.3 \pm 0.3$             | $68.1 \pm 0.6$             | $\underline{63.5 \pm 0.7}$ | $\underline{59.8\pm0.8}$ |
| RUNG (ours)            | $84.6 \pm 0.5$ | $\mathbf{78.9 \pm 0.4}$    | $\mathbf{75.7 \pm 0.2}$    | $\mathbf{71.8 \pm 0.4}$    | $\mathbf{67.8 \pm 1.3}$    | $\mathbf{65.1 \pm 1.2}$  |

*Global adaptive evasion attack on Citeseer. **Best** and <u>second</u> highlighted. Prediction accuracy in percentage:*
| Budget                 | Clean          | 5\%                        | 10\%                       | 20\%                       | 30\%                       | 40\%                     |
| ---------------------- | -------------- | -------------------------- | -------------------------- | -------------------------- | -------------------------- | ------------------------ |
| GCN                    | $74.8 \pm 1.2$ | $66.1 \pm 1.0$             | $60.9 \pm 0.8$             | $53.0 \pm 1.0$             | $47.0 \pm 0.8$             | $41.2\pm1.1$             |
| EvenNet                | $74.6 \pm 0.5$ | $66.8 \pm 0.5$             | $62.0 \pm 0.6$             | $55.9 \pm 0.4$             | $51.1 \pm 0.4$             | $47.4 \pm 0.8$           |
| GARNET                 | $74.8 \pm 1.3$ | $68.0 \pm 0.9$             | $64.0 \pm 1.1$             | $58.2 \pm 0.7$             | $53.9 \pm 0.8$             | $51.0 \pm 0.9$           |
| SoftMedian             | $74.6 \pm 0.7$ | $68.0 \pm 0.7$             | $64.4 \pm 0.9$             | $59.3 \pm 1.1$             | $55.2 \pm 2.0$             | $51.9\pm2.1$             |
| RUNG-$\ell_{1}$ (ours) | $75.5 \pm 1.1$ | $\underline{69.3 \pm 1.2}$ | $\underline{65.9 \pm 1.2}$ | $\underline{61.1 \pm 1.1}$ | $\underline{57.2 \pm 1.4}$ | $\underline{53.9\pm1.3}$ |
| RUNG (ours)            | $74.3 \pm 0.7$ | $\mathbf{71.4 \pm 1.0}$    | $\mathbf{69.8 \pm 1.3}$    | $\mathbf{67.6 \pm 1.2}$    | $\mathbf{66.5 \pm 1.3}$    | $\mathbf{65.3 \pm 1.5}$  |

As can be observed, these new baselines are close to or weaker than $\ell_1$ models, which validates our claims on $\ell_1$ models as well as the superior performance of RUNG.


[1] Li et al., “Reliable Representations Make A Stronger Defender: Unsupervised Structure Refinement for Robust GNN.”, KDD'22
[2] Deng, Chenhui, et al. "GARNET: Reduced-Rank Topology Learning for Robust and Scalable Graph Neural Networks.", LoG'22
[3] Lei et al., "EvenNet: Ignoring Odd-Hop Neighbors Improves Robustness of Graph Neural Networks", NIPS'22.
[4] Tao et al., "IDEA: Invariant Causal Defense for Graph Adversarial Robustness", arXiv:2305.15792

Reviewer comment (Presentation):
Typo: the runner-up model is SoftMedian rather than RUNG-$l_1$ under the 5% attack budget in Table 2.

Author response:
Thank you for pointing out the typo. We have amended that.

Reviewer comment (Writing):
What is $\beta$ in Section 2.3? I would suggest authors to use another notation, since $\beta$ has been used for SoftMedian.

Author response:
Thank you for the suggestion. $\beta$ in section 2.3 denotes the parameters in the regression model. We have changed it to another commonly used notation $\theta$.

Reviewer comment (Reproducibility):
How do authors perform adaptive PGD attack on Jaccard-GCN, which preprocesses the adjacency matrix?

Author response:
Jaccard-GCN drops edges based on the node features in a preprocessing manner. By doing the Hadamard product between the original adjacency matrix and the edge-dropping mask matrix, there will be no problem in the attack gradient evaluation. As our results showed, the adaptive attack setting successfully broke the defense of Jaccard-GCN and is thus effective. This adaptive attack setting follows that in [1].

[1] Mujkanovic et al., "Are Defenses for Graph Neural Networks Robust?", NIPS'22

Reviewer comment (Experiments):
Authors mention SVD-GCN as one of the baselines. Where are the results of SVD-GCN?

Author response:
We apologize for our oversight. We missed SVD-GCN by mistake in our table and will include it in the revision. It does not show notable robustness improvement over base models under adaptive attack, which is consistent with the results reported in [1].

*Global adaptive global evasion on Cora. Prediction accuracy in percentage:*
| Budget      | 5\%                     | 10\%                    | 20\%                    | 30\%                    | 40\%                    |
| ----------- | ----------------------- | ----------------------- | ----------------------- | ----------------------- | ----------------------- |
| GCN         | $75.3 \pm 0.5$          | $69.6 \pm 0.5$          | $60.9 \pm 0.7$          | $54.2 \pm 0.6$          | $48.4\pm0.5$            |
| SVD-GCN     | $68.7 \pm 0.6$          | $62.2 \pm 0.9$          | $55.5 \pm 1.4$          | $50.4 \pm 2.3$          | $46.6 \pm 1.5$          |
| RUNG (ours) | $\mathbf{78.9 \pm 0.4}$ | $\mathbf{75.7 \pm 0.2}$ | $\mathbf{71.8 \pm 0.4}$ | $\mathbf{67.8 \pm 1.3}$ | $\mathbf{65.1 \pm 1.2}$ |

*Global adaptive global evasion on Citeseer. Prediction accuracy in percentage:*
| Budget      | 5\%                     | 10\%                    | 20\%                    | 30\%                    | 40\%                    |
| ----------- | ----------------------- | ----------------------- | ----------------------- | ----------------------- | ----------------------- |
| GCN         | $66.1 \pm 1.0$          | $60.9 \pm 0.8$          | $53.0 \pm 1.0$          | $47.0 \pm 0.8$          | $41.2\pm1.1$            |
| SVD-GCN     | $60.4 \pm 0.8$          | $54.7 \pm 1.6$          | $41.2 \pm 13.2$         | $43.9 \pm 1.3$          | $40.8 \pm 3.9$          |
| RUNG (ours) | $\mathbf{71.4 \pm 1.0}$ | $\mathbf{69.8 \pm 1.3}$ | $\mathbf{67.6 \pm 1.2}$ | $\mathbf{66.5 \pm 1.3}$ | $\mathbf{65.3 \pm 1.5}$ |



[1] Mujkanovic et al., "Are Defenses for Graph Neural Networks Robust?", NIPS'22

Reviewer comment (Experiments):
It's unclear to me whether the defense under large attack budgets is practical. Is there any realistic application/scenario where an attacker can largely perturb the graph structure?

Author response:
Yes, there are many real-world applications where the attacker can largely perturb the graph structure. For instance, Web spam on the internet [1][2] and fake accounts in social media [3] can be easily generated at scale, which could lead to a large number of adversarial edges. In financial systems like blockchain, the accounts are anonymous so it is easy to generate lots of fake accounts and transactions [4][5]. In addition, the robustness under large budgets can be seen as the worst-case scenario, which prevents stronger attacks that may not happen in the past but could happen in the future.


[1] Gyongyi et al., "Combating Web Spam with TrustRank", VLDB'04

[2] Spirin et al., "Survey on web spam detection: principles and algorithms", KDD'12

[3] Khaled et al., "Detecting Fake Accounts on Social Media", IEEE BigData'18 

[4] Xia et al., "Characterizing cryptocurrency exchange scams", Computers & Security, vol. 98, 2020

[5] Chen et al., "Do cryptocurrency exchanges fake trading volumes? An empirical analysis of wash trading based on data mining", Physica A: Statistical Mechanics and its Applications, vol 586, 2022

Reviewer comment (Writing):
The definition of $l_1$-based graph smoothing is not clear. The MCP function $\rho_\gamma$ used in Equation 4 and Equation 7 is not defined.

Author response:
We have defined the MCP in Equation 4 and showed the difference between MCP and other penalty functions in Figure 3. For the $\ell_1$-based graph smoothing models, we show three examples in Section 2.2 and summarize them in the last sentence in "A Unified View of Robust Estimation". Specifically, $\ell_1$-based GNNs share a similar graph smoothing objective with edge difference penalties $\lVert f_i-f_j\rVert_1$ or $\lVert f_i-f_j\rVert_2$., following the definitions in ElasticGNN [1].  We will make this clear in our revision.

[1] Liu et al., "Elastic Graph Neural Networks.", ICML'21

Reviewer comment (Experiments):
The paper does not provide an analysis of the computational complexity of RUNG, leaving its efficiency and applicability to larger datasets, such as Ogbn[1], unclear. Additionally, numerical experiments are limited, being only applied to Cora-ML and Citeseer datasets. It would be beneficial to see how RUNG performs on larger-scale datasets.

Author response:
We understand your concern. We have added an analysis of our space and time complexity in Appendix C. Let us address your concern about the scalability of our RUNG here:

(1)  RUNG is scalable with time complexity of $O(k(m + n)d)$ and space complexity $O(m+nd)$, where $m$ is the number of edges, $d$ is the number of features, $n$ is the number of nodes, and $k$ is the number of GNN layers. Therefore, the complexity of our RUNG is comparable to normal GCN (with a constant difference) and it is feasible to implement.

(2)  To further mitigate your concern with empirical evidence, we also provide a comparison between the runtime of GCN and RUNG with the same layer number (3 layers) on ogbn-arxiv. The result verifies the efficiency of RUNG.

*Training speed on ogbn-arxiv:*
| Model          | GCN         | RUNG        |
| -------------- | ----------- | ----------- |
| Training speed | $23.3$ it/s | $10.9$ it/s |


To verify the robustness of RUNG when scaling up, we conducted experiments on the larger dataset of ogbn-arxiv.

*Global evasion attack on ogbn-arxiv. Prediction accuracy in percentage:*
| Budget        | Clean           | 1%                    | 5%                    | 10%                   |
| ------------- | --------------- | --------------------- | --------------------- | --------------------- |
| GCN           | $\mathbf{71.9}$ | $63.1\pm0.4$          | $48.9\pm 3.2$         | $41.8\pm 0.5$         |
| RUNG-$\ell_1$ | $71.6$          | $\mathbf{65.5\pm0.4}$ | $55.0\pm 1.1$         | $49.6\pm1.1$          |
| RUNG (ours)   | $70.2$          | $65.2\pm0.2$          | $\mathbf{64.0\pm0.8}$ | $\mathbf{61.2\pm0.7}$ |

The results verified the superior robustness of RUNG model especially under larger attack budgets. 

RUNG's robustness outperforms its $\ell_1$ variant (RUNG-$\ell_1$) which delivers similar performance as SoftMedian and Elastic GNN due to their shared $\ell_1$ graph smoothing, which is also the strongest baseline as shown in our experiments. We will add other baselines (if scalable) in the final version of our paper.

Reviewer comment (Experiments):
The description of the attack setting in Section 4.1 lacks clarity. Could you provide more details on how the adaptive evasion attack is designed based [2] you referenced? Specifically, what type of perturbations are performed during the attack - are they feature perturbations, graph topology perturbations, or both?

Author response:
The adaptive evasion attack we employ uses the graph topology PGD attack that perturbs the adjacency matrix, which has been explicitly mentioned in the revision. Additionally, we are happy to clarify attack settings we adopted here. Evasion attack means the data fed into a trained GNN model is perturbed rather than the data used to train a GNN, while adaptive means the attack is tailored for different GNN model respectively instead of using the same perturbation across different models. Specifically, the adaptive attack is realized through PGD, which leverages gradient information of the attacked GNN to precisely identify the adversarial edges, so as to modify the graph topology and mislead the GNN predictions. We didn't describe the details of PGD because it is a well-known attack [1] and it's not our contribution, but for completeness, we added some details of it in Appendix E. PGD evasion attack is claimed to be the strongest adaptive attack so far in [2], which can better reflect the true robustness of the GNNs.
 

[1] Xu, Kaidi, et al., "Topology Attack and Defense for Graph Neural Networks: An Optimization Perspective.", IJCAI'19

[2] Mujkanovic, Felix, et al., "Are Defenses for Graph Neural Networks Robust?", NIPS'22

Reviewer comment (Evaluation):
The robustness validation of RUNG in the paper is notably insufficient, solely relying on the PGD attack. A comprehensive assessment using various attacks, as detailed in references [3][4][5][6], is necessary for a credible demonstration of RUNG's robustness.

Author response:
Thank you for your advice. We would like to address your concerns as follows.


(1) First of all, we would like to point out that one notable strength of our work is to avoid the false sense of security as much as we can since the recent work [1] reveals that most existing GNN defenses are not robust although they have been shown to be robust under various settings such as black-box attack and poison attack. Therefore, to obtain a reliable robustness estimation, we mainly use the strongest attack setting we have found so far: white-box adaptive attack in the evasion setting, following the work [1].

(2) Since PGD generates attacks on all edges simultaneously with the relaxation of edge width instead of flipping edges greedily one by one, it can generate a stronger attack. Specifically, PGD attack is reported to be the strongest in the recent work [1], almost always outperforming Nettack [6]. Besides, the PGD-based poisoning attack named Meta-PGD [1] is also reported to outperform Metattack in most cases. Therefore, our choice of PGD attack should reflect the true robustness of our RUNG.

(3) The effect of injection [4] and modification attacks are similar in terms of producing estimation bias by introducing outliers. The same applies to the poisoning [5] attack and the evasion topology attack. Therefore, a comprehensive evaluation of the modification evasion setting should be able to demonstrate the robustness of our RUNG.

(4) To further test the robustness of RUNG in different scenarios, we still provide new experimental evidence including poisoning attacks and injection attacks following your suggestions. The results are shown below:

*Prediction accuracy in percentage, global adaptive poisoning on Cora:*
| Budget        | 5\%                     | 10\%                    | 20\%                    | 30\%                    | 40\%                    |
| ------------- | ----------------------- | ----------------------- | ----------------------- | ----------------------- | ----------------------- |
| GCN           | $74.9 \pm 0.4$          | $69.7 \pm 0.7$          | $60.7 \pm 0.7$          | $54.0 \pm 1.0$          | $48.7 \pm 1.0$          |
| APPNP         | $76.3 \pm 0.9$          | $71.1 \pm 1.2$          | $63.0 \pm 1.3$          | $57.1 \pm 0.6$          | $53.2 \pm 1.1$          |
| SoftMedian    | $79.2 \pm 0.7$          | $75.6 \pm 0.3$          | $67.8 \pm 0.6$          | $62.9 \pm 1.0$          | $58.6 \pm 0.7$          |
| RUNG-$\ell_1$ | $\mathbf{79.7 \pm 0.6}$ | $\mathbf{76.4 \pm 0.6}$ | $68.1 \pm 0.6$          | $63.8 \pm 0.5$          | $60.1 \pm 0.9$          |
| RUNG (ours)   | $78.5 \pm 0.5$          | $75.5 \pm 0.3$          | $\mathbf{71.5 \pm 0.4}$ | $\mathbf{67.1 \pm 1.6}$ | $\mathbf{64.6 \pm 1.3}$ |

*Prediction accuracy in percentage, global adaptive poisoning on Citeseer:*
| Budget        | 5\%                     | 10\%                    | 20\%                    | 30\%                    | 40\%                    |
| ------------- | ----------------------- | ----------------------- | ----------------------- | ----------------------- | ----------------------- |
| GCN           | $65.5 \pm 1.1$          | $59.8 \pm 1.0$          | $51.0 \pm 1.0$          | $44.0 \pm 1.2$          | $37.9 \pm 1.0$          |
| APPNP         | $64.2 \pm 1.8$          | $58.1 \pm 2.6$          | $49.8 \pm 2.5$          | $43.4 \pm 2.3$          | $40.6 \pm 2.7$          |
| SoftMedian    | $67.1 \pm 1.0$          | $63.8 \pm 1.0$          | $58.5 \pm 1.1$          | $54.3 \pm 1.9$          | $51.2 \pm 2.4$          |
| RUNG-$\ell_1$ | $68.9 \pm 1.0$          | $65.9 \pm 1.1$          | $61.0 \pm 1.0$          | $57.2 \pm 1.1$          | $53.9 \pm 1.3$          |
| RUNG (ours)   | $\mathbf{72.4 \pm 0.9}$ | $\mathbf{72.1 \pm 1.2}$ | $\mathbf{71.3 \pm 1.4}$ | $\mathbf{70.8 \pm 1.3}$ | $\mathbf{69.7 \pm 1.4}$ |

The poisoning attack follows the setting in [1]. As can be seen from the poisoning results, the bias accumulation effect of $\ell_1$ models (SoftMedian and RUNG-$\ell_1$) also emerges. Our RUNG, however, alleviates this problem and shows a substantial improvement over the baselines.

*Prediction accuracy in percentage, Targeted Injection Attack on Citeseer:*
| Budget        | Clean   | Attacked |
| ------------- | ------- | -------- |
| GCN           | $75.4$  | $28.14$  |
| RUNG-$\ell_1$ | $75.01$ | $39.22 $ |
| RUNG (ours)   | $75.65$ | $51.13$  |

The injection attack was conducted following the settings in [2] to evaluate the robustness of different methods. We set up the budget on the number of injected nodes as 100 and the budget on degree as 200. The results show that our RUNG significantly outperforms the baseline models.

We will incorporate these experiments in the final version of the paper.


[1] Mujkanovic et al., "Are Defenses for Graph Neural Networks Robust?", NIPS'22

[2] Zou et al. “TDGIA:Effective Injection Attacks on Graph Neural Networks.”, KDD'21

[3] Zheng et al. "Graph robustness benchmark: Benchmarking the adversarial robustness of graph machine learning", NIPS'21

[4] Chen et al., "Understanding and improving graph injection attack by promoting unnoticeability", ICLR'22

[5] Zügner et al., “Adversarial attacks on graph neural networks via meta learning,”, ICLR'19.

[6] Zügner et al., “Adversarial attacks on neural networks for graph data,”, KDD'18

Reviewer comment (Reproducibility):
Implementation code was not provided.

Author response:
We would like to kindly point out that the code is not mandatory at the submission stage. However, we will be happy to make our code available upon the acceptance of our work.

Reviewer comment (Theory):
How to calculate the derivative in the $W_{ij}^{(k)}$ of Equation (7)?

Author response:
Thank you for asking. We have revised Equation 8 (numbered Equation 7 in the initial submission) for enhanced clarity. For the detailed proof of Equation 8, we put it in Appendix B due to the space limit. We are happy clarify the explicit formulation of $W_{ij}^{(0)}$ and also a sketch of its derivation here:

(1) We acquired the graph smoothing objective $\sum_{(i,j)\in\mathcal{E}}\rho(\lVert f_i-f_j\rVert_2)+\lambda \sum_k \lVert f_i-f_k^{(0)}\rVert_2$, and we proposed QN-IRLS to construct in each iteration a new quadratic upper bound of the original objective. 

(2) $W^{(k)}$ is the coefficient of the quadratic bound in the $k$-th iteration and it is proved to be $W_{ij}^{(k)} = \frac{\partial \rho(y)}{\partial y^2}|_{y=\lVert f_i^{(k)}-f_j^{(k)}\rVert_2}$ (note that we are taking the first order derivative w.r.t. $y^2$ instead of the second order derivative).

(3) In RUNG, $\rho(y)$ takes the MCP penalty $\rho_\gamma(y)$. Therefore, $\rho(y)=\rho_\gamma(y) = 1_{y\le \gamma}(y - \frac{y^2}{2\gamma} )+ 1_{y > \gamma} \frac{\gamma}{2}$ and thus $W_{ij}^{(k)} = \max(0, \frac{1}{2\lVert f_i^{(k)} - f_j^{(k)}\rVert_2} - \frac{1}{2\gamma})$. 


(We omitted the degree normalization of $f_i$ and $f_j$ here for simplicity.)

Reviewer comment (Experiments):
The hyperparameter of $\lambda$ and $\gamma$ appears crucial in the model. Could you provide an ablation study to demonstrate the impact of varying these values? It would be insightful to understand how sensitive the performance of RUNG is to the selection of $\lambda$ and $\gamma$.

Author response:
Thank you for the question. We followed your suggestions to conduct a study on the accuracy of RUNG with different combinations of $\gamma$ and $\lambda$ under different attack intensities in Appendix G.1. 

In summary, $\lambda$ is the regularization intensity in front of $\sum_{i\in\mathcal{V}}\lVert f_i-f_i^{(0)}\rVert$ in graph smoothing and its optimal value is stable across different attack budgets, while $\gamma$ controls the tradeoff between clean accuracy and robustness of RUNG. 

The tuning of $\lambda$ is similar to other graph-smoothing based models [1][2][3]. Moreover, its optimal value is stable across budgets, and we can simply tune it according to the clean scenario and fix it as the optimal value $0.11$. Therefore, $\lambda$ is omitted from the result below. Now, we show the accuracy of RUNG under different attack budgets.

*Global adaptive evasion attack on Cora. Prediction accuracy in percentage:*
| $\downarrow\gamma$ \ Budget $\rightarrow$ | $5\%$            | $10\%$           | $20\%$           | $40\%$           |
| ----------------------------------------- | ---------------- | ---------------- | ---------------- | ---------------- |
| $0.5$                                     | $ 73.1 \pm  0.8$ | $ 72.8 \pm  0.7$ | $ 72.2 \pm  0.6$ | $ 71.7 \pm  0.6$ |
| $1$                                       | $ 74.8 \pm  0.5$ | $ 73.5 \pm  0.4$ | $ 71.6 \pm  0.5$ | $ 69.4 \pm  0.8$ |
| $3$                                       | $ 79.1 \pm  0.4$ | $ 75.7 \pm  0.3$ | $ 71.4 \pm  0.5$ | $ 65.4 \pm  1.6$ |
| $5$                                       | $ 79.4 \pm  0.3$ | $ 75.9 \pm  0.2$ | $ 71.0 \pm  0.8$ | $ 64.0 \pm  2.3$ |


The result above verifies that RUNG's performance is not very sensitive to the value of $\gamma$ and it varies in a simple way: under small budgets, a larger $\gamma$ is better, while when the budget is larger, a smaller $\gamma$ is more favorable. $\gamma$ can thus be easily tuned. 

We would like to emphasize that in our experiments, we keep $\gamma$ and $\lambda$ the same under different budgets for the sake of fairness. However, if the attack budget is known in advance, RUNG can be easily tuned to achieve even better performance than reported, which is also a significant advantage of our model. 


[1] Gasteiger et al., "Predict then propagate: Graph neural networks meet personalized pagerank." ICLR'18
[2] Liu et al., "Elastic graph neural networks.", ICML'21.
[3] Yang, Yongyi, et al., "Graph Neural Networks Inspired by Classical Iterative Algorithms." , ICML'21

Reviewer comment (Experiments):
Empirical evaluation only using Cora ML and Citeseer is insufficient. Consider larger graphs like ogbn-arxiv as well.

Author response:
Thank you for the suggestion. Following your suggestion, we conducted new experiments on ogbn-arxiv.

*Global evasion attack on ogbn-arxiv:*
| Budget        | Clean           | 1\%                   | 5\%                   | 10\%                  |
| ------------- | --------------- | --------------------- | --------------------- | --------------------- |
| GCN           | $\mathbf{71.9}$ | $63.1\pm0.4$          | $48.9\pm 3.2$         | $41.8\pm 0.5$         |
| RUNG-$\ell_1$ | $71.6$          | $\mathbf{65.5\pm0.4}$ | $55.0\pm 1.1$         | $49.6\pm1.1$          |
| RUNG (ours)   | $70.2$          | $65.2\pm0.2$          | $\mathbf{64.0\pm0.8}$ | $\mathbf{61.2\pm0.7}$ |

The results verified the superior robustness of RUNG model especially under larger attack budgets. 

RUNG's robustness outperforms its $\ell_1$ variant which delivers similar performance as SoftMedian and Elastic GNN due to their shared $\ell_1$ graph smoothing, which is also the strongest baseline as shown in our experiments. We will add other baselines (if scalable) in the final version of our paper.

Reviewer comment (Experiments):
The authors should show that their method breaks. For example, complement Figure 2 with a setting where the method fails.

Author response:
Thank you for the comments and suggestions. We believe there exist some misunderstandings which we would like to clarify in (1), and a discussion on the estimators' performance beyond the breakdown point follows in (2). The connection to the graph scenario follows in (3).

(1) Our outlier ratio is calculated as $\frac{m}{n}$ instead of $\frac{m}{m+n}$, where $m$ is the number of outliers and $n$ is the number of clean samples. For example, the number of outliers does not exceed that of the clean samples in our simulation (Figure 2) since "$80\%$ outliers" in our paper is actually $44\%$ of all samples. We have changed the notation in our paper to $\frac{m}{m+n}$ and added the definition for this ratio in our revision to avoid potential confusion.

(2) In our simulation of Section 2.3, the mean estimators do fail when the outlier ratio is above the optimal breakdown point of $50\%$. In our setting, when the number of outliers is larger than clean samples, the outliers become the dominating class and the mean estimator will find the center of this dominating outlier class as is shown in our new experiments. These results are provided in Appendix A.2. Therefore, We have revised our phrasing following your comment to be more precise. Nevertheless, our conclusions on the robustness of different estimators are not affected.

(3) We would like to emphasize that, under a budget larger than the breakdown point ($50\%$ modification ratio of edges), RUNG is still able to achieve reasonably good performance because of the extra feature similarity term $\lVert f_i-f_i^{0}\rVert_2$ in the graph estimator. This will be further elaborated in our response to your questions 1 and 2,

Reviewer comment (Theory):
The authors state "The simulation in Figure 2 verifies that our proposed estimator (η(x) := ργ(∥x∥2)) recovers the true mean regardless of the increasing outlier ratio." Which is somewhat misleading since the asymptotically optimal breakdown point of a location estimator is 50% (under certain assumptions, e.g., without constraining the value range of the estimation).

Author response:
Thank you for the comments and suggestions. We believe there exist some misunderstandings which we would like to clarify in (1), and a discussion on the estimators' performance beyond the breakdown point follows in (2). The connection to the graph scenario follows in (3).

(1) Our outlier ratio is calculated as $\frac{m}{n}$ instead of $\frac{m}{m+n}$, where $m$ is the number of outliers and $n$ is the number of clean samples. For example, the number of outliers does not exceed that of the clean samples in our simulation (Figure 2) since "$80\%$ outliers" in our paper is actually $44\%$ of all samples. We have changed the notation in our paper to $\frac{m}{m+n}$ and added the definition for this ratio in our revision to avoid potential confusion.

(2) In our simulation of Section 2.3, the mean estimators do fail when the outlier ratio is above the optimal breakdown point of $50\%$. In our setting, when the number of outliers is larger than clean samples, the outliers become the dominating class and the mean estimator will find the center of this dominating outlier class as is shown in our new experiments. These results are provided in Appendix A.2. Therefore, We have revised our phrasing following your comment to be more precise. Nevertheless, our conclusions on the robustness of different estimators are not affected.

(3) We would like to emphasize that, under a budget larger than the breakdown point ($50\%$ modification ratio of edges), RUNG is still able to achieve reasonably good performance because of the extra feature similarity term $\lVert f_i-f_i^{0}\rVert_2$ in the graph estimator. This will be further elaborated in our response to your questions 1 and 2,

Reviewer comment (Experiments):
It would be interesting to see how the perturbations of RUNG transfer to the other defenses as well.

Author response:
Thanks for your suggestion. We have added new experiments of transfer attack among different baselines across various budgets in Appendix F. Additionally, we observed that attacks generated by RUNG are stronger when applied to more robust models like SoftMedian, while not as strong against undefended or weakly defended models. The results indicate that transfer attack performances are similar when the surrogate and victim models share a similar architecture. This again justifies the importance of adaptive attacks in avoiding the false sense of robustness since the attacks on one model might not transfer well to another model.

Reviewer comment (Writing):
Section 2.2 could benefit from some notes on the composition of multiple GNN layers.

Author response:
Thanks for your suggestion. Section 2.2 focuses on providing a unified view and connection of several existing robust baselines, therefore we mainly describe the similar formulations of the objectives solved by the robust GNNs.  We will add some descriptions of the multi-layer architecture of the mentioned models in our revision following your suggestion.

Reviewer comment (Theory):
Can the authors elaborate more on how the model behaves if the perturbation strength is very high? I.e., does the model then effectively become an MLP?

Author response:
Thank you for your insightful questions. Based on your questions, we did additional experiments by increasing the budget of the global evasion attack on Cora to $50\%$, $80\%$, and $100\%$ and evaluated the robustness of RUNG.

*Prediction accuracy under global evasion attack on Cora:*
| Budget     | 50%           | 80%           | 100%          |
| ---------- | -------------- | -------------- | -------------- |
| MLP        | $65.0 \pm 1.0$ | $65.0 \pm 1.0$ | $65.0 \pm 1.0$ |
| GCN        | $43.2 \pm 1.2$ | $31.7 \pm 1.6$ | $25.7 \pm 2.8$ |
| RUNG       | $62.9 \pm 1.6$ | $56.3 \pm 2.7$ | $53.8 \pm 2.5$ |
| RUNG-tuned | $68.4 \pm 1.2$ | $66.6 \pm 1.4$ | $65.4 \pm 1.7$ |


Observations of the experiment and discussions are summarized as follows:

(1) The result in the table shows that our RUNG ("RUNG" in the table) can achieve a tremendous robustness improvement compared to GCN. However, it performs worse than MLP because in our experiments the hyperparameters are set the same for different attack budgets ("RUNG" in the table takes the hyperparameters suitable for smaller budgets in our initial experiments with a largest budget of $40\%$).

(2) Nevertheless, we can easily decrease the hyperparameter $\gamma$ to reach a better performance ("RUNG-tuned" in the table) if we focus on the large-budget robustness. According to the formulation of our model, we can either take $\lambda\rightarrow\infty$ or $\gamma\rightarrow 0$ to cover MLP, and in our experiments we found that under sufficient attack budgets ($> 100\%$ modification), the optimal RUNG ("RUNG-tuned") performs as if MLP because no graph information can be utilized then.

(3) It is worth noting that, however, even under very large attack budgets greater or equal to the optimal breakdown point of a mean estimator (e.g., $50\%$ and $80\%$ in the table), the optimal RUNG("RUNG-tuned" in table) can still outperform MLP. This also reflects RUNG's capability of learning on a very noisy graph in the worst case.

Similar observations can be made from the local attack setting in Table 1 and Table 3 of our paper, which, once again, verifies the soundness of our robustness evaluation. 

The behavior of RUNG under budgets larger than the breakdown point is surprising, especially when compared to the mean estimation problem. Therefore, we now elaborate on how RUNG defies the optimal breakdown point. The reason lies in an additional feature proximity term $ \sum_{i \in \mathcal{V}} \lVert f_i - f_i^{(0)}\rVert_2^2$ of our graph estimator design (RUGE in Section 3.1, Eq. (3)). This proximity (or regularization) term enforces the similarity between aggregated feature $f_i$ and the original node feature $f_i^0$, which does not exist in the mean estimator. Therefore, this regularization can provide the estimator $\min\sum_{(i,j)\in\mathcal{E}}\rho(\lVert f_i-f_j\rVert_2)$ with additional information, enabling it to correctly identify the adversarial edges and achieve superior performance even beyond the theoretical optimal breakdown point.

Reviewer comment (Theory):
How does RUNG defy the breakdown point?

Author response:
Thank you for your insightful questions. Based on your questions, we did additional experiments by increasing the budget of the global evasion attack on Cora to $50\%$, $80\%$, and $100\%$ and evaluated the robustness of RUNG.

*Prediction accuracy under global evasion attack on Cora:*
| Budget     | 50%           | 80%           | 100%          |
| ---------- | -------------- | -------------- | -------------- |
| MLP        | $65.0 \pm 1.0$ | $65.0 \pm 1.0$ | $65.0 \pm 1.0$ |
| GCN        | $43.2 \pm 1.2$ | $31.7 \pm 1.6$ | $25.7 \pm 2.8$ |
| RUNG       | $62.9 \pm 1.6$ | $56.3 \pm 2.7$ | $53.8 \pm 2.5$ |
| RUNG-tuned | $68.4 \pm 1.2$ | $66.6 \pm 1.4$ | $65.4 \pm 1.7$ |


Observations of the experiment and discussions are summarized as follows:

(1) The result in the table shows that our RUNG ("RUNG" in the table) can achieve a tremendous robustness improvement compared to GCN. However, it performs worse than MLP because in our experiments the hyperparameters are set the same for different attack budgets ("RUNG" in the table takes the hyperparameters suitable for smaller budgets in our initial experiments with a largest budget of $40\%$).

(2) Nevertheless, we can easily decrease the hyperparameter $\gamma$ to reach a better performance ("RUNG-tuned" in the table) if we focus on the large-budget robustness. According to the formulation of our model, we can either take $\lambda\rightarrow\infty$ or $\gamma\rightarrow 0$ to cover MLP, and in our experiments we found that under sufficient attack budgets ($> 100\%$ modification), the optimal RUNG ("RUNG-tuned") performs as if MLP because no graph information can be utilized then.

(3) It is worth noting that, however, even under very large attack budgets greater or equal to the optimal breakdown point of a mean estimator (e.g., $50\%$ and $80\%$ in the table), the optimal RUNG("RUNG-tuned" in table) can still outperform MLP. This also reflects RUNG's capability of learning on a very noisy graph in the worst case.

Similar observations can be made from the local attack setting in Table 1 and Table 3 of our paper, which, once again, verifies the soundness of our robustness evaluation. 

The behavior of RUNG under budgets larger than the breakdown point is surprising, especially when compared to the mean estimation problem. Therefore, we now elaborate on how RUNG defies the optimal breakdown point. The reason lies in an additional feature proximity term $ \sum_{i \in \mathcal{V}} \lVert f_i - f_i^{(0)}\rVert_2^2$ of our graph estimator design (RUGE in Section 3.1, Eq. (3)). This proximity (or regularization) term enforces the similarity between aggregated feature $f_i$ and the original node feature $f_i^0$, which does not exist in the mean estimator. Therefore, this regularization can provide the estimator $\min\sum_{(i,j)\in\mathcal{E}}\rho(\lVert f_i-f_j\rVert_2)$ with additional information, enabling it to correctly identify the adversarial edges and achieve superior performance even beyond the theoretical optimal breakdown point.

Reviewer comment (Experiments):
In the numerical simulations, "clean samples" and "outlier samples" are generated using a Gaussian distribution. The designation of red dots as outliers is perplexing, as there's no discernible reason to not categorize the red dots as clean samples. Accepting this alternate categorization would render the entire example questionable. Furthermore, when the third plot indicates that 80\% of the data are outliers, the persistence of the estimation at the center of the blue dots raises concerns. This behavior might be influenced by factors not elaborated upon in the paper, making it premature to commend this as a positive outcome of the proposed method.

Author response:
Thank you for this thoughtful comment. We believe there exist some misunderstandings. Let's clarify it as follows.

(1) We would like to point out that the outlier ratio in Figure 2 of our initial submission was calculated as $\frac{m}{n}$ instead of $\frac{m}{m+n}$, where $m$ is the number of outliers and $n$ is the number of clean samples. Therefore, the number of outliers does not exceed that of the clean samples in our simulation (Figure 2) since "$80$% outliers" in our paper is actually about $44$% of all samples. We have changed the notation to $\frac{m}{m+n}$ and added the definition for this ratio in our revision to avoid potential confusion.

(2) In our simulation (Figure 2), the mean estimators are computed from the mixture of clean and outlier samples: 
$\min_z\sum_i^{n+m}\rho(\|z-x_i\|)$ where $\rho(y)=1_{y\le \gamma} (y - \frac{y^2}{2\gamma})+1_{y>\gamma}\frac{\gamma}{2}$ and $\gamma$ is the hyperparameter. When the number of outliers is larger than clean samples, we verified that the outliers become the dominating class and the mean estimator will be closer to this dominating class. This aligns well with your intuition. The results of this new experiment can be found in Appendix A.2.

(3) We would like to emphasize that in our final graph estimator design (Section 3.1, Eq. (3)), there is an additional feature proximity term $\sum_{i\in \mathcal{V}} \|f_i - f_i^{(0)}\|_2^2$ that enforces the similarity between aggregated feature $f_i$ and the original node feature $f_i^0$, in contrast to the mean estimator. Therefore, the regularization can reduce the impact of outliers even further, which explains why our RUNG method still performs reasonably well when the attack budget is more than $100\%$ in Section 4 (Table 1, local attack).

Reviewer comment (Theory):
The proposed method offers only a rudimentary approximation of one step in GNN aggregation (7). Even if the example in the numerical simulations were valid, it doesn't convincingly demonstrate robustness in GNNs. Given that $W^k$ fundamentally varies for each $k$, there's no assurance that the single-step gradient descent aligns with the theorem.

Author response:
We believe there are some misunderstandings of numerical simulation in Section 2.3 and the final GNN model in Section 3. We are happy to clarify as follows.

(1) The numerical simulation of mean estimators in Figure 2 (Section 2.3) aims to provide a more intuitive understanding of the bias accumulation effect of $\ell_1$ estimator. As mentioned in Section 2.3, these estimators are computed by the classic iterative Weiszfeld method for many steps until convergence since we do not worry about efficiency in such simulations.  

(2) In our final GNN design (Section 3), we need to develop an efficient algorithm (Quasi-Newton IRLS) to build up deep learning layers that are not only efficient but also compatible with back-propagation training. Our RUNG model stacks multiple aggregation layers using Eq. (7) rather than only one-step aggregation. Theorem 2 guarantees that each step of the iteration in Eq. (7) will decrease the objective we design (with a proof in Appendix B).

Reviewer comment (Theory):
The statement "which not only provides clear interpretability but also covers many classic GNNs as special cases" is misleading. The term "covers" is inappropriate. The paper doesn't truly encompass classic GNNs as special cases but rather offers an approximate perspective on them through the lens of Graph Signal processing.

Author response:
We precisely recover the aggregation of several classic GNNs in Section 3.3 (please refer to the paragraph "Relations with Existing GNNs"). Specifically, when $\rho(y)=y^2$, the graph aggregation layer in classic GNNs such as APPNP ($\lambda>0$) and GCN ($\lambda=0$) can be precisely recovered by Eq. (7) of our RUNG model. When $\rho(y)=y$, Eq. (7) of our RUNG model intrinsically solves the same objective as ElasticGNN, SoftMedian, and TWIRLS since they are inherently $\ell_1$-based GNNs. In this case, our solver is better than but different from their solvers. We will make this statement clear in our revision.

Reviewer comment (Evaluation):
The paper's limitations are not adequately addressed. Like other defense papers aiming to prune the influence of heterophilic edges, this method likely has restricted applicability, perhaps being best suited for datasets that inherently assume homophily.

Author response:
It is true that our defense is best suited for homophilic graphs, and we will make this point clear in our revision. We would also like to point out that the majority of GNN defenses[1-10] rely on homophily assumption. The defense of GNNs on heterophilic graphs is largely unexplored to the best of our knowledge. The major challenge is the ambiguity between adversarial attacks and heterophic edges. The adversarial robustness in heterophilic graphs will be an interesting future direction.


[1] Zhu, Dingyuan, et al. “Robust Graph Convolutional Networks Against Adversarial Attacks.”, KDD'2019

[2] Wu, Huijun, et al. "Adversarial Examples on Graph Data: Deep Insights into Attack and Defense.", IJCAI'19

[3] Jin, Wei, et al., "Graph Structure Learning for Robust Graph Neural Networks.", KDD'20

[4] Entezari, Negin, et al. “All You Need Is Low (Rank): Defending Against Adversarial Attacks on Graphs.”, WSDM'20

[5] Liu, Xiaorui, et al. "Elastic Graph Neural Networks.", ICML'21

[6] Yang, Yongyi, et al. "Graph Neural Networks Inspired by Classical Iterative Algorithms.", ICML'21

[7] Zhang, Xiang, and Marinka Zitnik. "GNNGuard: Defending Graph Neural Networks against Adversarial Attacks.", NIPS'20

[8] Geisler, Simon, et al. "Robustness of Graph Neural Networks at Scale.", NIPS'21

[9] Li, Kuan, et al. “Reliable Representations Make A Stronger Defender: Unsupervised Structure Refinement for Robust GNN.”, KDD'22

[10] Song, Yang, et al. "On the Robustness of Graph Neural Diffusion to Topology Perturbations.", NIPS'22

Reviewer comment (Evaluation):
A glaring omission is the lack of reported computational costs. This oversight leaves readers questioning the practical feasibility of the proposed approach.

Author response:
Thank you for your comment. We have added an analysis of our space and time complexity in Appendix C. Let us address your concern about the practical feasibility of our RUNG here:

(1)  RUNG is scalable with time complexity of $O(k(m + n)d)$ and space complexity $O(m+nd)$, where $m$ is the number of edges, $d$ is the number of features, $n$ is the number of nodes, and $k$ is the number of GNN layers. Therefore, the complexity of our RUNG is comparable to normal GCN (with a constant difference) and it is feasible to implement.

(2)  To further mitigate your concern in practice, we also provide a comparison between the runtime of GCN and RUNG with the same layer number (3 layers) on ogbn-arxiv with an RTX-A6000 graphics card. The result verifies the efficiency of RUNG.

*Training speed on ogbn-arxiv:*
| Model          | GCN         | RUNG        |
| -------------- | ----------- | ----------- |
| Training speed | $23.3$ it/s | $10.9$ it/s |

Reviewer comment (Experiments):
The experimental section is inadequate. The range of attacks considered is severely limited. It is imperative to incorporate evaluations against poison and evasion attacks, as well as both white-box and black-box scenarios, and to consider both injection and modification types.

Author response:
Thank you for your advice. We would like to address your concerns as follows.

(1) First of all, we would like to point out that one notable strength of our work is to avoid the false sense of security as much as we can since the recent work [1] reveals that most existing GNN defenses are not robust although they have been shown to be robust under various settings such as black-box attack and poison attack. Therefore, to obtain a reliable robustness estimation, we mainly use the strongest attack setting we have found so far: white-box adaptive attack in the evasion setting, following the work [1].

(2) In our initial submission, we evaluated the black-box attack (transfer attack) and white-box attack (adaptive attack). Our experiments in Figure 8 show that the black-box (transfer attack) is much weaker than the white-box attack and it induces a severe false sense of robustness. 

(3) The effect of injection and modification attacks are similar in terms of producing estimation bias by introducing outliers. The same applies to the poisoning attack and the evasion topology attack. Therefore, a comprehensive evaluation of the modification evasion setting should be able to demonstrate the robustness of RUNG.

(4) To further test the robustness of RUNG in different scenarios, we still provide new experimental evidence of surrogate evasion attacks poisoning attacks, and injection attacks, following your suggestions. The results are shown below: