Paper: When Witnesses Defend: A Witness Graph Topological Layer for Adversarial Graph Learning (ICLR-2024)

Reviewer comment (Presentation):
The threat model should be moved to the main body of the paper, instead of in the appendix.

Author response:
Thank you for this suggestion. Due to page limitations, we are unable to move Appendix B to the main body. Presently, we provide a brief description of different attacks considered in the experiments (Section 5) in the paragraph “Adversarial Attacks: Local and Global.” (page 7). However, if the reviewer thinks it to be critical, we will try our best to do it.

Reviewer comment (Experiments):
There is a lack of comparisons with previous non-topological works. While I agree that this work would be the first to utilize persistent homology, previous baseline methods also consider similar information/knowledge, such as neighboring structure or connectivity patterns using other concepts. Simply demonstrating improvements over the vanilla GCN or combining with a single specific baseline may not be entirely convincing.

Author response:
Thank you for pointing out this limitation. We have compared and demonstrated improvement over non-topological works, in particular those that use neighboring structure or connectivity patterns using other concepts such as GraphSAGE, GNNGuard, and H$_2$GCN. 

(a)   **Performance with GraphSAGE [1]:** Table 16 shows the performance of GraphSAGE and the proposed GraphSAGE+WGTL on the Cora-ML and Polblogs datasets under mettack. We observe that GraphSAGE+WGTL improves robustness with respect to the baseline GraphSAGE by 3.2% - 42.6% on Cora-ML and 1.5% - 40.8% on Polblogs. Table 17 shows the performance under nettack. We observe that GraphSAGE+WGTL improves robustness with respect to the baseline GraphSAGE by 3% - 4% on Cora-ML and 0.4% - 1.7% on Polblogs. 

[(URL to Tables 16-17)](https://drive.google.com/file/d/138WvQLB6ieS-fojA9D1brAKc3NnbSm1H/view?usp=sharing)

(b)    **Comparison with GNNGuard [2]:** In Appendix F.2, we have compared the proposed WGTL with GNNGuard (Tables 18-19). On Cora-ML under mettack (Table 18), our method GCN+GNNGuard+WGTL yields more than 2.25%, 1.12%, and 1.85% relative improvements with respect to GCN, GCN+GNNGuard (SOTA defence), and GCN+WGTL, respectively. 

[(URL to Tables 18-19)](https://drive.google.com/file/d/13942jxN8ZntEusr3SgL93LScd2G_PRJ7/view?usp=sharing)

(c) **Performance with  H${_2}$GCN [3]:** Following [4], we run experiments on snap-patents, a strongly heterophilic graph under mettack. We incorporated global topology encoding into H${_2}$GCN [3], a popular method with 580+ citations proposed to handle heterophilic graphs. H${_2}$GCN proposes a set of key design techniques to improve the performance of GNNs on heterophilic graphs: (1) separation of ego- and neighbor-embedding, (2) incorporation of higher-order neighborhoods, and (3) combination of intermediate representations using skip-connections. In Table 20 of Appendix F3, we have shown that H${_2}$GCN+WGTL improves the robustness of H2GCN by up to 4%. 
Note that the most adversarially robust method on this dataset, APPNP by [5], has been shown to have an accuracy of $27.76\%$ under 20\% perturbation (c.f. Table 3, [4]). Improving on that, we observe that H${_2}$GCN+WGTL achieves $28.21\%$ average accuracy under $20\%$ perturbation.
$$
\bf{Table 20 =>}
\\begin{array}{|c|c|c|c|c|c|c|}
\\hline
 \text{Perturbation rates=>} & 0\\% & 5\\% & 10\\%  & 15\\% & 20\\% & 25\\% \\\\
\hline
H_2GCN & 27.71\pm 0.86  	& 27.55 \pm 0.19  & 28.62 \pm 0.38 & 28.40 \pm 1.38 & 27.77 \pm 0.30 & 27.45 \pm0.89  \\\\
H_2GCN+WGTL & 27.72 \pm 0.85 & 28.66 \pm 0.1.68  & 28.79 \pm 1.0 & 28.45 \pm 0.61 & 28.71 \pm 0.66 & 27.90 \pm 0.84 \\\\
\\hline
\\end{array}
$$

Reviewer comment (Experiments):
The entire set of experiments is conducted using GCN. The significant improvement observed in GCN may be attributed to its naive mechanism. Given that the incorporation of a multi-scale receptive field for node representation is already well-explored in the field of graph neural networks, it is highly recommended to conduct experiments on more recent graph neural networks.

Author response:
In addition to GCN in the main paper, we have added two more backbones, GAT [6] (Tables 14-15) and GraphSAGE [1] (Tables 16-17), in Appendix F and shown the effectiveness of WGTL on Cora-ML and Polblogs under mettack and nettack. We observe that incorporating WGTL into all of the backbones improves their corresponding performances under a range of perturbation rates. The following table summarizes the %improvement in mean accuracy with respect to the corresponding backbone models:
 
$$
\\begin{array}{|c|c|c|c|}
\\hline
 & & \\% \text{improvement on Cora-ML} & \\% \text{improvement on Polblogs}  \\\\
\hline
GAT+WGTL & Mettack & 0.1-13.8 & 0.6-50.2 \\\\
GraphSAGE+WGTL & Mettack &       3.2-42.6   &   1.5-40.8 \\\\
GAT+WGTL    	& Nettack &  0.7-1.1     &   0.14-1.5 \\\\
GraphSAGE+WGTL & Nettack &    3-4   &    0.4-1.7 \\\\
\\hline
\\end{array}
$$
[(URL to Tables 14-15)](https://drive.google.com/file/d/1315msJ6yJGCoKncR2Fd6sWimv3tSVj5n/view?usp=sharing)

[(URL to Tables 16-17)](https://drive.google.com/file/d/138WvQLB6ieS-fojA9D1brAKc3NnbSm1H/view?usp=sharing)

Reviewer comment (Experiments):
Can this method also perform well on heterophilous graphs or molecular graphs? The proposed method has only been validated on homophilous graphs, particularly citation networks. Therefore, it remains uncertain whether this strategy can exhibit versatility when applied to other types of graphs.

Author response:
As a response to this question, we refer the reviewer to point **(c) Comparison with  H${_2}$GCN [3]** of the answer to question (1), and Appendix F.3 in the revised draft.

Reviewer comment (Theory):
Witness complexes, although sometimes having advantages in terms of fewer simplices, are known to suffer from numerous drawbacks: calculation is heavily dependent on the choice of "landmark" points, bringing instability.

Author response:
>  (2) Discussions on different aspects of Witness complex construction 

Here, we elaborate on four points regarding different aspects of Witness complex construction and how we address in this work the four drawbacks mentioned by the reviewer.

a.    The landmark selection heuristic employed is deterministic because we select the top 5% highest-degree nodes as landmarks. Since there is no randomness in such deterministic selection, there is no instability in the final performance.

b.    The accuracy is indeed dependent on the number of landmarks. Figure 3 (Appendix E) shows that with an increased number of landmarks, the accuracy increases. 

c.    Indeed, with an increased number of landmarks, the computational cost increases, as we have shown in Figure 3. However, Table 13 indicates that even after such an increase, the computation time is still substantially less than that of Vietoris-Rips. Furthermore, Figure 3 (Appendix E) shows that the accuracy increases, becoming closer to what is obtained by GCN+VRGTL.

d.    It is not clear what the reviewer meant by “different situations”; hence, we consider the following two scenarios:
  
 1)    The relation between persistence diagrams induced by different sets of landmarks of the same cardinality: [8] show that irrespective of the choice of landmarks, if the landmarks are $\epsilon$-net of the graph $G = (V, E)$, it produces a $(3log3 + 2\epsilon)$-approximations to Vietoris-Rips complex of $G$. Let us assume two sets of landmarks  $L$, $L’$ such that $|L| = |L’|$ and both are $\epsilon$-net of the graph $G = (V, E)$. Then, both choices give us the same approximation error w.r.t Vietoris-Rips. 
        
 2)    The relations between encoding induced by different sets of #landmarks of the same cardinality:  Proposition 2 shows that the encoding $Z_{WGTL}$ is stable under adversarial perturbation. The guarantee is dependent on $\epsilon$ but does not depend on the particular choices of landmarks. Let us assume two sets of landmarks  $L$, $ L’$ such that $|L| = |L’|$ and both are $\epsilon$-net of the graph $G = (V, E)$. Then, both choices of the landmark set give us the same stability guarantee in terms of encoding $Z_{WGTL}$.

Reviewer comment (Experiments):
Witness complexes are sensitive to parameters — their setup involves the choice of several hyperparameters, such as the number of landmarks or the epsilon in the epsilon-net, etc.

Author response:
>  (2) Discussions on different aspects of Witness complex construction 

Here, we elaborate on four points regarding different aspects of Witness complex construction and how we address in this work the four drawbacks mentioned by the reviewer.

a.    The landmark selection heuristic employed is deterministic because we select the top 5% highest-degree nodes as landmarks. Since there is no randomness in such deterministic selection, there is no instability in the final performance.

b.    The accuracy is indeed dependent on the number of landmarks. Figure 3 (Appendix E) shows that with an increased number of landmarks, the accuracy increases. 

c.    Indeed, with an increased number of landmarks, the computational cost increases, as we have shown in Figure 3. However, Table 13 indicates that even after such an increase, the computation time is still substantially less than that of Vietoris-Rips. Furthermore, Figure 3 (Appendix E) shows that the accuracy increases, becoming closer to what is obtained by GCN+VRGTL.

d.    It is not clear what the reviewer meant by “different situations”; hence, we consider the following two scenarios:
  
 1)    The relation between persistence diagrams induced by different sets of landmarks of the same cardinality: [8] show that irrespective of the choice of landmarks, if the landmarks are $\epsilon$-net of the graph $G = (V, E)$, it produces a $(3log3 + 2\epsilon)$-approximations to Vietoris-Rips complex of $G$. Let us assume two sets of landmarks  $L$, $L’$ such that $|L| = |L’|$ and both are $\epsilon$-net of the graph $G = (V, E)$. Then, both choices give us the same approximation error w.r.t Vietoris-Rips. 
        
 2)    The relations between encoding induced by different sets of #landmarks of the same cardinality:  Proposition 2 shows that the encoding $Z_{WGTL}$ is stable under adversarial perturbation. The guarantee is dependent on $\epsilon$ but does not depend on the particular choices of landmarks. Let us assume two sets of landmarks  $L$, $ L’$ such that $|L| = |L’|$ and both are $\epsilon$-net of the graph $G = (V, E)$. Then, both choices of the landmark set give us the same stability guarantee in terms of encoding $Z_{WGTL}$.

Reviewer comment (Theory):
Witness complexes lack functoriality — the relations between results of calculations in different situations are more difficult to establish.

Author response:
>  (2) Discussions on different aspects of Witness complex construction 

Here, we elaborate on four points regarding different aspects of Witness complex construction and how we address in this work the four drawbacks mentioned by the reviewer.

a.    The landmark selection heuristic employed is deterministic because we select the top 5% highest-degree nodes as landmarks. Since there is no randomness in such deterministic selection, there is no instability in the final performance.

b.    The accuracy is indeed dependent on the number of landmarks. Figure 3 (Appendix E) shows that with an increased number of landmarks, the accuracy increases. 

c.    Indeed, with an increased number of landmarks, the computational cost increases, as we have shown in Figure 3. However, Table 13 indicates that even after such an increase, the computation time is still substantially less than that of Vietoris-Rips. Furthermore, Figure 3 (Appendix E) shows that the accuracy increases, becoming closer to what is obtained by GCN+VRGTL.

d.    It is not clear what the reviewer meant by “different situations”; hence, we consider the following two scenarios:
  
 1)    The relation between persistence diagrams induced by different sets of landmarks of the same cardinality: [8] show that irrespective of the choice of landmarks, if the landmarks are $\epsilon$-net of the graph $G = (V, E)$, it produces a $(3log3 + 2\epsilon)$-approximations to Vietoris-Rips complex of $G$. Let us assume two sets of landmarks  $L$, $L’$ such that $|L| = |L’|$ and both are $\epsilon$-net of the graph $G = (V, E)$. Then, both choices give us the same approximation error w.r.t Vietoris-Rips. 
        
 2)    The relations between encoding induced by different sets of #landmarks of the same cardinality:  Proposition 2 shows that the encoding $Z_{WGTL}$ is stable under adversarial perturbation. The guarantee is dependent on $\epsilon$ but does not depend on the particular choices of landmarks. Let us assume two sets of landmarks  $L$, $ L’$ such that $|L| = |L’|$ and both are $\epsilon$-net of the graph $G = (V, E)$. Then, both choices of the landmark set give us the same stability guarantee in terms of encoding $Z_{WGTL}$.

Reviewer comment (Theory):
The paper mentions some of the above concerns with witness complexes but does not explain convincingly how to overcome them.

Author response:
>  (2) Discussions on different aspects of Witness complex construction 

Here, we elaborate on four points regarding different aspects of Witness complex construction and how we address in this work the four drawbacks mentioned by the reviewer.

a.    The landmark selection heuristic employed is deterministic because we select the top 5% highest-degree nodes as landmarks. Since there is no randomness in such deterministic selection, there is no instability in the final performance.

b.    The accuracy is indeed dependent on the number of landmarks. Figure 3 (Appendix E) shows that with an increased number of landmarks, the accuracy increases. 

c.    Indeed, with an increased number of landmarks, the computational cost increases, as we have shown in Figure 3. However, Table 13 indicates that even after such an increase, the computation time is still substantially less than that of Vietoris-Rips. Furthermore, Figure 3 (Appendix E) shows that the accuracy increases, becoming closer to what is obtained by GCN+VRGTL.

d.    It is not clear what the reviewer meant by “different situations”; hence, we consider the following two scenarios:
  
 1)    The relation between persistence diagrams induced by different sets of landmarks of the same cardinality: [8] show that irrespective of the choice of landmarks, if the landmarks are $\epsilon$-net of the graph $G = (V, E)$, it produces a $(3log3 + 2\epsilon)$-approximations to Vietoris-Rips complex of $G$. Let us assume two sets of landmarks  $L$, $L’$ such that $|L| = |L’|$ and both are $\epsilon$-net of the graph $G = (V, E)$. Then, both choices give us the same approximation error w.r.t Vietoris-Rips. 
        
 2)    The relations between encoding induced by different sets of #landmarks of the same cardinality:  Proposition 2 shows that the encoding $Z_{WGTL}$ is stable under adversarial perturbation. The guarantee is dependent on $\epsilon$ but does not depend on the particular choices of landmarks. Let us assume two sets of landmarks  $L$, $ L’$ such that $|L| = |L’|$ and both are $\epsilon$-net of the graph $G = (V, E)$. Then, both choices of the landmark set give us the same stability guarantee in terms of encoding $Z_{WGTL}$.

Reviewer comment (Experiments):
It is not explained how to make the crucial choice concerning the number of landmarks for the pipeline to work.

Author response:
>  (3) Explaining the choice concerning the number of the landmarks and its impact

The choice of the number of landmarks is indeed an important hyperparameter for the pipeline to work. We varied the number of landmarks and made an assessment of their impact on accuracy as well as the increased computation time. Figure 3 (Appendix E) shows that increasing the number of landmarks indeed slightly increases the accuracy, albeit with the expense of increased computation time. Due to this trade-off, the selection of an optimal number of landmarks is dependent on how much robustness is desired by a user with a given computation time budget. Appendix E.1  includes a more elaborate discussion on this point. 

[(URL to Figure 3)](https://drive.google.com/file/d/1307sEUDuvneaBrEOgzaj9dny_7gr83X1/view?usp=sharing)

Reviewer comment (Experiments):
It is not clear why the standard Vietoris-Rips complexes, via GPU acceleration, could not be used instead to solve the described defense tasks.

Author response:
> (4) Using the standard Vietoris-Rips complexes in the pipeline

Thank you for bringing this interesting point to our attention. The Vietoris-Rips complexes can be used with the proposed pipeline, which is indicative of the flexibility offered by the proposed method. 

In Appendix E.2, we have implemented GCN + VRGTL, where instead of encoding witness topological features, we have encoded the Vietoris-Rips features. Table 12 shows that the accuracy is comparable to that of GCN+WGTL while at the cost of a significant increase in computation time (Table 13). In our implementation, we adopted Ripser (SOTA for CPU-based computation) for computing both Vietoris-Rips and Witness topological features. Hence, if a GPU-accelerated implementation,e.g., Ripser++, were to be used, it would have accelerated Vietoris-Rips as well as Witness feature computations. 

[(URL to Tables 12-13)](https://drive.google.com/file/d/13IFvmsAeIA7J6X3ShUIT9oWxa-7jU0R0/view?usp=sharing)

Reviewer comment (Writing):
The formulations of the theoretical results are not very clearly stated.

Author response:
> (5) Clarification of theoretical results

We have clarified the theoretical results in the main paper. In addition, we have included in Appendix D a nomenclature of notations (Appendix D.1) and detailed derivations of the proofs to enhance clarity. Please consider the revised draft for the changes.

Reviewer comment (Reproducibility):
In the description of the pipelines, in experiment details, and in the statements or the proofs of theoretical results, the dimensions of the computed persistence diagrams are not specified.

Author response:
> (6) Specifying the dimensions used in the persistence diagrams

Thank you for your detailed observation. Our theoretical results hold true once we fix the dimension of the persistence $d \in Z_{\geq 0}$ to be computed and use it throughout the pipeline. In our experiments, we used 0-dimensional persistence. We have added it in Section 5 (paragraph 1) of our updated manuscript.

Reviewer comment (Theory):
The reported computational complexity of the pipeline is not accurate; for example, it does not include the complexity of the geodesic distance on the graph.

Author response:
> (7) Computational complexity of the Witness complex-based topological features

Thank you for pointing out this issue. 
We have corrected the computational complexity segment in Section 5 as per the following:

Landmark selection (top-$|\mathfrak{L}|$ degree nodes) has complexity $\mathcal{O}(N\log(N))$. To compute witness topological features, one needs to compute (1) landmarks-to-witness distances costing $\mathcal{O}(|\mathfrak{L}|(N+|\mathcal{E}|))$ due to BFS-traversal from landmarks, (2) landmark-to-landmark distances costing $\mathcal{O}(|\mathfrak{L}|^2)$, and finally (3) persistent homology via boundary matrix construction and reduction [7]. Matrix reduction algorithm costs $\mathcal{O}(\zeta^3)$, where $\zeta$ is the number of simplices in a filtration. Overall, the computational complexity of computing witness topological feature on the graph is $\mathcal{O}(|\mathfrak{L}|(N+|\mathcal{E}|)+|\mathfrak{L}|^2+\zeta^3)$.

Reviewer comment (Writing):
On page 2, "complementary information" is unclear — complementary to what?

Author response:
> (8) Specific remarks

Thank you for pointing out these mistakes/typos. We have addressed them in our updated manuscript. 
Here, we address two specific questions mentioned in this list.

> page 2 "complementary information" - complementary to what? it is not very clear
  
By complementary information here, we understand more traditional graph summaries that are not extracted using any tools of persistent homology.

Reviewer comment (Writing):
On page 9, "the homologically persistent graph skeleton" — what is meant by this?

Author response:
> (8) Specific remarks

Thank you for pointing out these mistakes/typos. We have addressed them in our updated manuscript. 
Here, we address two specific questions mentioned in this list.

> page 9 "the homologically persistent graph skeleton" - what is meant by this?

We have reformulated this sentence as follows: “Another interesting research direction is to investigate the linkage between the attacker's budget, number of landmarks, and topological attacks targeting the skeleton shape, that is, topological properties of the graph induced by the most important nodes (landmarks).”

Reviewer comment (Experiments):
Question — Why could the standard Vietoris-Rips complexes, with e.g. subsampling and GPU acceleration, not be used to solve the described defense tasks?

Author response:
> (4) Using the standard Vietoris-Rips complexes in the pipeline

Thank you for bringing this interesting point to our attention. The Vietoris-Rips complexes can be used with the proposed pipeline, which is indicative of the flexibility offered by the proposed method. 

In Appendix E.2, we have implemented GCN + VRGTL, where instead of encoding witness topological features, we have encoded the Vietoris-Rips features. Table 12 shows that the accuracy is comparable to that of GCN+WGTL while at the cost of a significant increase in computation time (Table 13). In our implementation, we adopted Ripser (SOTA for CPU-based computation) for computing both Vietoris-Rips and Witness topological features. Hence, if a GPU-accelerated implementation,e.g., Ripser++, were to be used, it would have accelerated Vietoris-Rips as well as Witness feature computations. 

[(URL to Tables 12-13)](https://drive.google.com/file/d/13IFvmsAeIA7J6X3ShUIT9oWxa-7jU0R0/view?usp=sharing)