Paper: GNNCert: Deterministic Certification of Graph Neural Networks against Adversarial Perturbations (ICLR-2024)

Reviewer comment (Novelty):
The paper fails to cite prior work on derandomized smoothing which is strikingly similar to GraphGuard. For example, Levine & Feizi (2020) designed a certifiably robust classifier that also splits the input into sub-parts, classifies the sub-parts using a base model, and makes the final classification using majority vote. While their method is applied to image classification to certify against patch attacks, the fundamental design pattern (and analysis) is similar to GraphGuard. More recently, Hammoudeh & Lowd (2023) showed that the same design pattern can be used to achieve certified robustness against L0 perturbations (including patch attacks) at training-time and test-time. Given this context, GraphGuard could be viewed as an extension of derandomized smoothing to a new domain (graphs), which could impact the assessment of technical novelty. **Update 21/11: The authors have promised to address this issue. I have therefore raised my _Presentation_ score from 1 to 3.**

Author response:
**Comment-1: Prior work (Levine & Feizi (2020) and Hammoudeh & Lowd (2023)).**

Thanks for pointing them out. We agree with the reviewer that splitting&predictions&majority vote is a general randomized smoothing-based framework for certification. The key challenge is how to tailor this framework for different domains. For instance, (de)randomized smoothing (Levine & Feizi (2020)) divides an image into different blocks or bands, which are customized for images. Hammoudeh & Lowd (2023) extend this general framework to certify $\ell_0$-norm perturbation for image and tabular data (all inputs have the same size) as mentioned by the reviewer.  By contrast, we focus on the graph domain, where the number of nodes and edges varies in each graph, making existing methods inapplicable. We develop three different methods to divide a graph into different sub-graphs. Our designs of three division methods (e.g., structure, feature, structure-feature division) are tailored to the graph domain. We will cite and discuss the difference with Levine & Feizi (2020) and Hammoudeh & Lowd (2023) as suggested.

Reviewer comment (Experiments):
Although the experiments are comprehensive, I believe there is a crucial baseline missing: the base graph classifier (GIN) _without_ GraphGuard. Specifically, I would like to see a comparison of standard accuracy for GIN trained normally on full graphs, and GraphGuard with GIN trained on sub-graphs. This would allow for an assessment of GraphGuard’s impact on accuracy. This is important, as the standard accuracy of GraphGuard is fairly low for most datasets (around 70% based on Fig. 2)—it’s not clear whether this is due to GraphGuard or the inherent difficulty of the datasets. Incidentally, it may be interesting to report the accuracy of the base classifiers for all certified methods. Levine & Feizi (2020) found much higher base classifier accuracies for derandomized smoothing (which is similar to GraphGuard) than randomized ablation (which is similar to Zhang et al., 2021b). The same explanation may apply here.

Author response:
**Comment-2: Comparison of the accuracy of the normally trained GIN and GraphGuard with GIN trained on sub-graphs.**

Thanks for the suggestions. We add experiments to compare the accuracy as suggested. We have the following observations from the experimental results. On MUTAG, DD, COLLAB, DBLP, ENZYMES, PROTEINS datasets, the accuracy of GraphGuard is at most 3\% lower than that of normally trained GIN. On REDDIT and NCI1 datasets, the accuracy of GraphGuard is at most 15\% lower than that of normally trained GIN. The reason is that there is a tradeoff between accuracy without attacks and robustness (such tradeoff widely exists for certified defenses). Our GraphGuard (with GIN as the base classifier) reduces to GIN when $T_s=1$. To reduce the accuracy drop, we could set a smaller $T_s$ in practice. We will add the results to our paper and discuss this tradeoff. 

We also compare the accuracy of base classifiers for different certified methods. As mentioned by the reviewer, the accuracy of GraphGuard’s base classifier is indeed higher than those of other defenses, which explains why our GraphGuard outperforms them. For instance, on MUTAG dataset, the accuracy of GraphGuard’s base classifier is 91.85\%. By contrast, the accuracy of base classifiers for Bojchevski et al., Wang et al., Zhang et al. are 67.88\%, 88.89\%, and 82.59\%, respectively. We will add the explanation. Moreover, we will discuss our observation is consistent with that in the image domain (Levine & Feizi (2020)).

Reviewer comment (Evaluation):
While GraphGuard outperforms prior methods in terms of certified accuracy, the standard accuracy is rather low (at around 70% for 6 of the 8 datasets). This has a bearing on the significance of the paper in my view, as the sacrifice in terms of accuracy seems too high to be practical. It’s also worth noting that the variant of GraphGuard that protects against structure and feature perturbations simultaneously suffers a severe accuracy drop for some of the datasets (DBLP and ENZYMES). For these datasets, the classifiers are no better than random based on the standard accuracy in Figs. 9 and 10.

Author response:
**Comment-3: The accuracy drop for protecting against structure and feature perturbations simultaneously on DBLP and ENZYMES.**

We agree with the reviewer that the accuracy of our GraphGuard drops on some datasets such as DBLP and ENZYMES for structure-feature division. The reason is that we aim to certify structure and feature perturbations simultaneously. Additionally, we consider an attacker could arbitrarily manipulate the features of a number of nodes. In other words, we consider a very strong attacker. To reduce the accuracy drop, we could divide a graph into less number of sub-graphs. For instance, on DBLP dataset, our GraphGuard achieves a 76\% accuracy when dividing each graph into 9 sub-graphs ($T_s=3$, $T_f=3$).

Reviewer comment (Experiments):
Section 4.2: should alpha be 0.001?

Author response:
**Comment-5: $\alpha$ should be 0.001.**

We really appreciate the reviewer for pointing this out. We will fix the typo.

Reviewer comment (Writing):
On Page 4, at the end of line 4: Shouldn't the notation be $H(.)$%$T_s + 1$?

Author response:
**Comment-1: The notation at the end of the line 4 on Page 4.**

Thanks for pointing it out. It is indeed $H(\cdot)\\%T_s+1$. We will fix the typo.

Reviewer comment (Theory):
Based on my understnading $(S_{v_t} \bigoplus S_{v_j}) \neq (S_{v_j} \bigoplus S_{v_t})$. If that is correct, then it seems in structure division explained in 3.3.1 the same edge can have two different hash values depending on which end node is visited first. Can the authors clarify this?

Author response:
**Comment-2: The hash value depends on which node is visited first.**

Sorry for the confusion. We note that each node has a node ID. We always put the node with a smaller ID first. We also add an experiment to put the node with a larger ID first on MUTAG dataset. The certified accuracy is 92\% (smaller ID first) and 92\% (larger ID first) when an attacker arbitrarily adds/deletes one edge, respectively. The results demonstrate that our method is insensitive to the orders of nodes. We will clarify and add results to our paper.

Reviewer comment (Experiments):
Could the authors provide an intuitive explanation for the preference of hash-based subsampling over methods like $\tau$ fraction-based sampling (referenced in the benchmarks) or other similar techniques? What advantages does this approach offer and what are its potential limitations?

Author response:
**Comment-4: Preference of hash-based sub-sampling over $\tau$ fraction-based sampling.**

Our hash-based sub-sampling has the following advantages. First, our hash-based sub-sampling enables our defense to certify feature perturbation as well as both feature and structure perturbations for graph classification. By contrast, existing sub-sampling methods cannot. Second, hash-based sub-sampling divides a graph into sub-graphs deterministically while $\tau$ fraction-based sampling generates sub-graphs probabilistically. As a result, our method could provide deterministic robustness guarantees while existing methods cannot due to the randomness in their sub-sampling. Third, our hash-based division makes our defense very efficient (as shown in Table 1). The reason is that our hash-based sub-sampling divides a graph into a moderate number of (e.g., 30) sub-graphs. By contrast, existing  $\tau$ fraction-based sampling requires sampling hundreds of sub-sampled graphs to estimate the robustness guarantees (as they utilize Monte-Carlo to estimate them). Our experimental results in Figure 8 (in Appendix) show the certified accuracy of existing methods is much worse when reducing the number of sub-sampled graphs for them. 

In general, there is a tradeoff between utility (measured by the accuracy without attacks) and robustness. One limitation of our defense is that it also has such a tradeoff. For instance, when the number of groups is very large, the accuracy without attacks of our defense slightly drops. The experimental results show our defense achieves a better tradeoff than existing methods.

Reviewer comment (Experiments):
How do the authors determine the values for $T_s$ and $T_f$? It would be beneficial if this determination were associated with specific graph properties (e.g., the number of edges, nodes, diameter, path length, clustering coefficient, etc.), as this would guide potential users of this defense strategy in tailoring it to their datasets.

Author response:
**Comment-5: How to determine the values of $T_s$ and $T_f$.**

In our experiment, we use a fixed $T_s=30$ and $T_f=30$ for all 8 datasets to be consistent. Additionally, we show the impact of $T_s$ and $T_f$ in Figures 4 and 5. Our results show GraphGuard could achieve better robustness guarantees when $T_s$ and $T_f$ are larger. We could set $T_s$ and $T_f$ based on the number of edges and nodes, respectively. In particular, we could set a larger $T_s$ and $T_f$ when the number of edges and nodes in a graph is larger to ensure better robustness.

We really appreciate the reviewer for the comment. We will add the discussion on setting hyperparameters $T_s$ and $T_f$. Additionally, we will conduct more experiments to connect $T_s$ and $T_f$ with graph properties  (e.g., the number of edges and nodes as mentioned by the reviewer) to further optimize our results. We thank the reviewer again for the constructive feedback, which significantly improves the quality of our work.