Paper: EGALA: Efficient Gradient Approximation for Large-scale Graph Adversarial Attack (ICLR-2024)

Reviewer comment (Experiments):
Have authors adopted mini-batch training on large graphs? How does the mini-batch training affect the gradient approximation in EGALA?

Author response:
**[Q1] Mini-batch training on large graphs:**
We adopt full-batch training on large graphs, which is the same as the baselines. Specifically, they chunk some operations (e.g. matrix multiplication) within the message passing step to successfully scale to larger graphs.

Reviewer comment (Reproducibility):
How do authors perform hyperparameter tuning on all GNN models in the experiments?

Author response:
**[Q2] Hyperparameter tuning:**
To further explore parameter sensitivity and complexity, we have conducted additional ablation studies, presented in Appendix C, to analyze the influence of clustering parameters on our model's performance, both the accuracy and computational cost. For GNN models, we follow the training settings of the original paper of PRBCD and GRBCD.

Reviewer comment (Experiments):
The proposed method uses SGC as the surrogate model and cannot be extended to other surrogate models. Additionally, in the experiments, the surrogate model used in the baseline is SGC, which may reduce the baseline's attack capabilities.

Author response:
**[W1] Regarding the extension to other surrogate models and the surrogate model of baselines:**
Firstly, although our proposed method cannot be extended to other surrogate models, we demonstrate the importance of using SGC as our surrogate model for its simplicity, scalability and effectiveness.
SGC is commonly used as surrogate model in graph adversarial attacks [1][2]. Also, the simplicity of SGC makes it even more applicable to large graphs.
We have expanded our empirical evaluation, incorporating additional experiments to benchmark our EGALA model against adaptive PRBCD and GRBCD attacks. These new comparisons are detailed in the revised Appendix B.1. Using more advanced GNNs as surrogate models for attack does not necessarily improve the attack performance.
Additionally, we examined the computational resources required conducting attacks on more advanced GNNs. Both time and memory costs rise significantly with the complexity of GNN models, constraining the scalability of adaptive attacks.

Secondly, we carried out additional experiments to consider PRBCD and GRBCD utilizing their originally suggested surrogate model, Vanilla GCN. We present the results of this analysis in Appendix B.2.

Reviewer comment (Experiments):
The experiments in the paper are not comprehensive enough. Providing more ablation experiments would be beneficial—for example, the impact of Δ_t in the algorithm. I also want to know the performance comparison of the PDG topology attack [1] and EGALA on small datasets.

Author response:
**[W2] Regarding the distribution of budgets:**
In our experiemnt, the $\Delta_t$ we use is the same as that of GRBCD, where the budget is evenly distributed. This was done to ensure an equitable comparison of results. Furthermore, we acknowledge the potential for future work to explore more sophisticated budget distributions, as also suggested in the GRBCD original paper. Our current choice of $\Delta_t$ serves as a starting point, and we are open to investigating more complex allocation strategies as part of our continued research efforts.

**[W2] Regarding the comparison with PDG:**
We would like to clarify the rationale behind our baseline selection process. The PDG model has already been comprehensively compared with both PRBCD and GRBCD in their respective original publications. Consequently, we focused our comparative evaluation on the two more recent and advanced models: PRBCD and GRBCD. Nonetheless, we appreciate the importance of thorough baseline validation and are prepared to include PDG in our future evaluations to corroborate and extend the findings of the original papers.

Reviewer comment (Experiments):
EGALA is limited to attacking the surrogate model of two-layer linear GCN (essentially 2-layer SGC), and it can be only applied in the transfer attack setting when the victim model is not the same as the surrogate model. However, it has been shown in [1] that the transfer attack is much weaker than the adaptive attack, and the robustness evaluated under transfer attacks exhibits a strong false sense of security. This concern significantly weakens the contribution of this work.

Author response:
**[W1 \& W2] Regarding the adaptive attack:**
We expanded our empirical evaluation, incorporating additional experiments to benchmark our EGALA model against adaptive PRBCD and GRBCD attacks. These new comparisons are detailed in the revised Appendix B.1. Our results indicate that while adaptive attacks on the Vanilla GCN indeed demonstrate effectiveness, adaptive GRBCD does not consistently outperform its transfer attack, particularly within more complex GNN models like Vanilla GDC. The adaptive GRBCD fails to surpass its transfer counterpart in these contexts, suggesting that the advantage of adaptivity in greedy methods is non-universal and may not be decisive in all scenarios. Additionally, we examined the computational resources required for these adaptive attacks. We found that both time and memory costs rise significantly with the complexity of GNN models, constraining the scalability of adaptive attacks. This finding is important as it underscores the trade-offs between the intensity of attack methodologies and their practical deployability, especially with more elaborate GNN architectures.

Reviewer comment (Experiments):
The major baseline PRBCD is a randomized block coordinate method. PRBCD is efficient by selecting a small block size. More importantly, it is a general attack algorithm that can be applied to potentially any GNN model, without being limited to two-layer linear GCNs. Overall, the advantages of EGALA over PRBCD and GRBCD are not convincing enough. First, the attack performances of EGALA, PRBCD, and GRBCD are comparable in the transfer attack, while it is expected that PRBCD and GRBCD will provide much stronger adaptive attacks, especially when the evaluated model is robust GNNs (although no such study is presented). Second, the time complexity depends on many hyperparameters such as block size and number of clusters. However, there is no discussion and ablation study on the hyperparameter setting of baselines such as PRBCD and GRBCD. Therefore, the reported time cost comparison is not convincing enough.

Author response:
(1) I would like to clarify our evaluation process related to robust GNNs. In our paper, we indeed conducted experiments that evaluate the transfer attacks on robust GNNs. As you correctly pointed out, the distinction in architecture between vanilla and robust GNNs can significantly influence the success rate of transferability. To address this concern, we evaluate transfer attacks on robust GNN models. Specifically, we examined our proposed EGALA against models like SoftMedian GDC and SoftMedian PPRGo, which are designed to withstand adversarial attacks. The results of these evaluations are presented in Table 2 of our paper. The results indicate the efficacy of our proposed method, providing insights into its impact on models with inherent robustness features.

(2) As indicated in our experimental results, we observe that adaptive GRBCD do not consistently outperform transfer attacks on the datasets examined, such as Cora and CiteSeer, where the corresponding results are detailed in Tables 2 and 4 of our paper. Our findings are consistent with those reported in the original paper[1], where the corresponding results are detailed in Tables F.4 and F.5. To address your point concerning the selection of block size, we carefully selected a block size of $10^6$ for our experiments on both Cora and CiteSeer, following the recommendations from the original paper[1]. We ensure the use of a block size that does not compromise the model performance, and our experimental design avoids the introduction of additional variables that could skew the comparison between methods. We believe that the empirical evidence presented is intended to serve as a reference point for the performance of EGALA relative to existing methods. Nevertheless, we recognize the value of such discussions and would be open to exploring these comparisons in greater depth in future work.

Reviewer comment (Experiments):
There is a lack of time complexity comparison of EGALA and EGALA-N. It will be better to provide detailed analysis as well as corresponding ablation experimental results. In Table 3, the paper shows that EGALA and EGALA-N share the same time and memory cost, which raises concerns about the advantage of the proposed clustering-based ANNS. Additionally, it is unclear why the cost of EGALA on PubMed is higher than the other baselines.

Author response:
**[W2 \& W3] Regarding the computational cost:**
We would like to apologize for any confusion caused by our initial presentation of the computational cost comparison. We have revised Table 3 in Section 4.2 to seperate the computational analysis of both EGALA and EGALA-N. The the increment of computational cost of EGALA on small datasets can be attributed to the ANNS methodology employed in EGALA. Exploring more efficient ANNS implementations holds the potential to notably diminish the overall computation cost. More importantly, for smaller datasets like Cora and Citeseer, the direct computation of matrix multiplication proves to be both rapid and incurs minimal memory cost. Conversely, for larger datasets like Arxiv and Products, the incorporation of the ANNS mechanism significantly mitigates computational costs, justifies its application.

When comparing the computational cost of PRBCD and GRBCD, we used the recommended hyperparameter settings in the original papers to ensure fidelity in our comparative analysis. We also studied the influence of block size on the performance of the baseline. The results are shown in the following table.

| Block size | $10^3$ | $10^4$ | $10^5$ | $10^6$ | $10^7$ |
| --- | ---: | ---: | ---: | ---: | ---: |
| Accuracy | 0.645 | 0.630 | 0.622 | 0.616 | 0.619 |
| Time per step (s) | 0.014 | 0.014 | 0.015 | 0.051 | 0.195 |
|Memory (MB) | 649 | 735 | 797 | 1101 | 2097 |

Reviewer comment (Experiments):
Lack of comprehensive ablation studies on several components, e.g., clustering method, number of clusters and number of closest vector pairs, period of cluster update. These components or hyperparameters can influence the accuracy and computation cost. Ablation studies should be included to show the impact of each technical component.

Author response:
**[W4] Ablation studies:**
We conducted ablation studies, presented in Appendix C, to analyze the influence of clustering parameters on our model's performance, comparing both the accuracy and computational cost simultaneously.

Your feedback has been instrumental in propelling our research forward. We appreciate the opportunity to address these critical aspects of our research and believe that the revisions have significantly strengthened the contribution of our work.

Reviewer comment (Experiments):
The authors only consider a grey box setting where the perturbations are transferred between models. This setting certainly has its merits. However, as pointed out previously, it is vital to assess neural networks with adaptive attacks [I, II] to get a proper estimate of the model's robustness. The authors should have prominently placed disclaimers and a comprehensive discussion on for what purpose the attack could be used.

Author response:
We have expanded our empirical evaluation, incorporating additional experiments to benchmark our EGALA model against adaptive PRBCD and GRBCD attacks. These new comparisons are detailed in the revised Appendix B.1. Our results indicate that while adaptive attacks on the Vanilla GCN indeed demonstrate effectiveness, adaptive GRBCD does not consistently outperform its transfer attack, particularly within more complex GNN models like Vanilla GDC. The adaptive GRBCD fails to surpass its transfer counterpart in these contexts, suggesting that the advantage of adaptivity in greedy methods is non-universal and may not be decisive in all scenarios. Additionally, we examined the computational resources required for these adaptive attacks. We found that both time and memory costs rise significantly with the complexity of GNN models, constraining the scalability of adaptive attacks. This finding is important as it underscores the trade-offs between the intensity of attack methodologies and their practical deployability, especially with more elaborate GNN architectures.

Reviewer comment (Experiments):
The attack is model specific and thus, it is not straightforward to make it "adaptive" for other GNNs than SGC.

Author response:
We have expanded our empirical evaluation, incorporating additional experiments to benchmark our EGALA model against adaptive PRBCD and GRBCD attacks. These new comparisons are detailed in the revised Appendix B.1. Our results indicate that while adaptive attacks on the Vanilla GCN indeed demonstrate effectiveness, adaptive GRBCD does not consistently outperform its transfer attack, particularly within more complex GNN models like Vanilla GDC. The adaptive GRBCD fails to surpass its transfer counterpart in these contexts, suggesting that the advantage of adaptivity in greedy methods is non-universal and may not be decisive in all scenarios. Additionally, we examined the computational resources required for these adaptive attacks. We found that both time and memory costs rise significantly with the complexity of GNN models, constraining the scalability of adaptive attacks. This finding is important as it underscores the trade-offs between the intensity of attack methodologies and their practical deployability, especially with more elaborate GNN architectures.

Reviewer comment (Experiments):
In connection to 2 & 3, the authors should craft experiments where they compare their transfer EGALA with an adaptive PRBCD and GRBCD. For example, the authors could attack defenses like Jaccard GCN, or SVG GCN (see [I]).

Author response:
We have expanded our empirical evaluation, incorporating additional experiments to benchmark our EGALA model against adaptive PRBCD and GRBCD attacks. These new comparisons are detailed in the revised Appendix B.1. Our results indicate that while adaptive attacks on the Vanilla GCN indeed demonstrate effectiveness, adaptive GRBCD does not consistently outperform its transfer attack, particularly within more complex GNN models like Vanilla GDC. The adaptive GRBCD fails to surpass its transfer counterpart in these contexts, suggesting that the advantage of adaptivity in greedy methods is non-universal and may not be decisive in all scenarios. Additionally, we examined the computational resources required for these adaptive attacks. We found that both time and memory costs rise significantly with the complexity of GNN models, constraining the scalability of adaptive attacks. This finding is important as it underscores the trade-offs between the intensity of attack methodologies and their practical deployability, especially with more elaborate GNN architectures.

Reviewer comment (Experiments):
How is the computational cost affected by the hyperparameters?

Author response:
To further explore parameter sensitivity and complexity, we have conducted additional ablation studies, presented in Appendix C, to analyze the influence of clustering parameters on our model's performance, both the accuracy and computational cost. Also, it is noteworthy that the primary contributor to computational load in EGALA on small datasets is the Approximate Nearest Neighbor Search (ANNS). As such, pursuing more advanced ANNS methods is likely to reduce computational costs substantially.

Reviewer comment (Reproducibility):
Is it necessary to approximate the derivate d a_ij / d e_ij for scalability?

Author response:
The approximation of the derivative $\frac{\partial a_{ij}}{\partial e_{ij}}$ is necessary and indispensable. Empirically, the attack performance degrades if we ignore this part. Moreover, according to Equation (14) of our paper, $\frac{\partial a_{ij}}{\partial e_{ij}}$ is inversely proportional to the squared root of node degrees. By attacking edges connected to lower-degree nodes, the model incurs a higher misclassification rate, which is consistent with recent findings in [1]. The attack performance without the approximation of $\frac{\partial a_{ij}}{\partial e_{ij}}$ is shown in the following table.

| Model | SGC | GCN | GDC | PPRGo | S-GDC | S-PPRGo
|---|---:|---:|---:|---:|---:|---:|
| EGALA | 0.599 | 0.615 | 0.655 | 0.695 | 0.725 | 0.766 |
| No $\frac{\partial a_{ij}}{\partial e_{ij}}$ | 0.780 | 0.788 | 0.824 | 0.798 | 0.824 | 0.797 |