Paper: Boosting Graph Anomaly Detection with Adaptive Message Passing (ICLR-2024)

Reviewer comment (Theory):
Although the effectiveness and efficiency of the proposed method have been empirically studied from different aspects, this is no theoretical guarantee of the proposed method.

Author response:
**W1.** Although the effectiveness and efficiency of the proposed method have been empirically studied from different aspects, this is no theoretical guarantee of the proposed method.

Inspired by CoLA[1], in the first stage of the model, we employs a "node vs. local subgraph" contrastive learning approach to extract local anomalies. In this process, the model aims to maximize the similarity between nodes and their adjacent subgraphs. Nodes that are challenging to optimize receive higher anomaly scores due to their high local inconsistency. The motivation for such a contrastive learning framework is intuitive and fits well with the goal of mining local inconsistency. Meanwhile, our contrast learning framework has demonstrated its effectiveness in local anomaly detection, as affirmed by a considerable body of work[1,2,3].

In the second stage of the model, we focus on mining another kind of abnormal signals from the perspective of "node vs. global normal context". Inspired by previous work[4], we **theoretically demonstrate** that by optimizing the loss loss function Eq.(13), our model can make the distribution of learned abnormal node features spread uniformly as much as possible in the feature space. Thus GADAM can better deal with the diverse abnormal distributions.

We begin with the formal problem statement. Given a graph $G$ containing $N$ nodes, let $h_i$ be the embedding of node $v_i$, and let $q$ be the embedding for global context. Let $y_i$ be the label of node $v_i$, where $y_i=0$ corresponds to normal nodes, and $y_i=1$ indicates abnormal nodes. Assume there are $n$ normal nodes and $m$ abnormal nodes, satisfying $n+m=N$. We consider a relatively simple case by optimizing the loss function defined by Eq (13), where $n$ normal nodes are pulled closer to $q$, and the opposite for $m$ abnormal nodes:

$
\mathcal{L} = -\mathbb{E}_{i,j}[\text{log}(q^Th_i) + \text{log}(1-q^Th_j)], where ~ y_i=0, y_j=1
$

For better clarity, we use $h^+$ to represent the embedding of a normal node, and $h^-$ to represent the embedding of an abnormal node. We perform mean pooling on all nodes to obtain $q$, thus $q = \frac{1}{N}(\sum_{i=1}^n h_i^+ + \sum_{j=1}^m h_j^-)$. Then we have:

**Theorem 1**: Let $X^+$ denote the random variable of normal node and $p^+$ denote its marginal distribution, thus $X^+ \sim p^+(x^+)$. Likewise, we denote the abnormal node by $X^-$ and its marginal distribution by $p^-$. Assume $p^+$ and $p^-$ are mutually independent. Then we have: minimizing the loss $\mathcal{L}$ forms the lower bound of 1) the cross-entropy of the two data distributions $p^+$ and $p^-$ plus 2) the entropy of $p^-$. Formally, $\text{min}~ \mathcal{L} \iff \text{max}~[H(p^-,p^+) + H(p^-)]$.

The **proof of Theorem 1** can be found in Appendix F in our revision. Below are some key conclusions drawn from Theorem 1:
+ The cross-entropy term $H(p^-,p^+)$ in Theorem1 links to the node-wise contrastive loss, which is proved to be the cross-entropy between the predicted labels and the ground truth labels[5]. This enables GADAM to distinguish between normal and abnormal nodes effectively.
+ Maximizing the entropy of abnormal nodes ($H(p^-)$) can be viewed as a form of data augmentation, as it enrich the information of minor classes, which helps to reduce the label imbalance issue. Also, it helps GADAM to better deal with the diverse abnormal distributions.
+ GADAM obtains global normal context through mean pooling on high-confidence normal set $V_n$. This approach is more representative of the distribution of normal nodes when compared to the global context $q$, as it involves less contamination. 

In summary,  as per Theorem 1, GADAM can effectively handle diverse distributions of anomalies and enhance its anomaly detection capabilities with a solid theoretical guarantee.

Reviewer comment (Theory):
The first step of MLP-based LIM aims to identify local anomalies only. However, if the given data contains more contextual anomalies, will the proposed method fail to capture them?

Author response:
**W2.** Thank you for your detailed review. We will address your conceens one by one:

>(1) The first step of MLP-based LIM aims to identify local anomalies only.

Actually, MLP-based LIM has the ability to detect some structural anomalies. It's worth noting that the neighbors of some structural anomalies may include normal nodes(e.g. node 7 in Figure 2 of our paper). Consequently, these structural anomalies can exhibit high local inconsistency, making them detected in the first stage. Hence, $S^{local}$ remains effective in detecting structural anomalies with this characteristic.

>(2) If the given data contains more contextual anomalies, will the proposed method fail to capture them? The influence of different types of anomalies on the proposed method is not discussed.

This is a good idea to do further testing of our model. We noticed that you asked a similar question in Q3, so we will answer it in detail in Q3.

Reviewer comment (Evaluation):
The influence of different types of anomalies on the proposed method is not discussed.

Author response:
**Q3.** Thank you for your insightful question! Below are the detailed responses to each question:

>(2) What the performance will change if different numbers of contextual or structural anomalies are injected ?

Thanks for your valuable suggestion! To comprehensively assess the performance of our model with different numbers of anomalies injected, we selected three representative benchmark datasets for detailed testing: **Cora** represents a small dataset that is regular and has a low average node degree, **ACM** represents a larger dataset that contains a high number of anomalies, and **BlogCatalog** represents a dataset with a large average node degree that may pose a greater challenge to structural anomaly detection. We kept the total number of anomalous nodes unchanged (consistent with our paper, see appendix A.2) and conducted tests under three scenarios: contextual anomalies only, a balanced number of anomalies, and structural anomalies only.  We measured GADAM's effectiveness using the AUC-ROC metric, with bolded results indicating the scenarios in which GADAM excels in detecting the two types of anomalies. Using the Cora dataset as an example, the bolded results show that GADAM is best at detecting contextual anomalies in the contextual only case and structural anomalies in the structural only case:

**Cora**
| contextual only |             | balanced anomalies |              | structural only |               |
| :-------------- | ----------- | ------------------ | ------------ | --------------- | ------------- |
| contextual=150  | structual=0 | contextual=75      | structual=75 | contextual=0    | structual=150 |
| **0.9425**      | \           | 0.9416             | 0.9929       | \               | **0.9962**    |

**ACM**

| contextual only |             | balanced anomalies |               | structural only |               |
| --------------- | ----------- | ------------------ | ------------- | --------------- | ------------- |
| contextual=600  | structual=0 | contextual=300     | structual=300 | contextual=0    | structual=600 |
| **0.9607**      | \           | 0.9586             | **0.9721**    | \               | 0.9678        |

**BlogCatalog**

| contextual only |             | balanced anomalies |               | structural only |               |
| --------------- | ----------- | ------------------ | ------------- | --------------- | ------------- |
| contextual=300  | structual=0 | contextual=150     | structual=150 | contextual=0    | structual=300 |
| **0.9333**      | \           | 0.9094             | **0.7763**    | \               | 0.7553        |

The experimental results reveal the following insights:

+ Regarding the detection of contextual anomalies, our model consistently demonstrates optimal performance in scenarios where only contextual anomalies are included. This effectively addresses the concern expressed in W2 regarding the potential impact of continuously adding contextual anomalies. It serves as validation for the excellent performance of our model in mining local anomaly signals using the conflict-free Local Inconsistency Mining (LIM) approach.
+ In detecting structural anomalies, our model achieves competitive results when only structural anomalies are included, as compared to the scenario with a balanced number of anomalies. Surprisingly, it even outperforms on the Cora dataset. This results show that GADAM is also a strong detector of structural anomalies.

In summary, our model is capable of handling extreme anomaly distributions (contextual/structural only), and achieves very good performance in the more common balanced case. We appreciate your insightful questions, and we will integrate them in the later revision, which will help us to further improve the paper!

Reviewer comment (Experiments):
If there is no prior information about the ratio of anomalous nodes in a given graph, how should one select k_{ano}? If the selected k leads to more anomalies than actual ones, will the proposed method be impacted negatively?

Author response:
**Q2.** If there is no prior information about the ratio of anomalous nodes in a given graph, how should one select k_{ano}? If the selected k leads to more anomalies than actual ones, will the proposed method be impacted negatively?

Thanks for your thoughtful consideration. We'll respond to each in turn:

>(1) How to select $k_{ano}$ if there is no prior information about the ratio of anomalous nodes?

$k_{ano}$ determines the size of the high-confidence abnormal set $V_a$. So the setting of $k_{ano}$ relies on two principles: (1) making $V_a$ contain as many true anomalies as possible, and (2) making $V_a$ contain as few normal nodes as possible. Ideally, the optimal value of $k_{ano}$ should be the percentage of anomalies in each dataset, i.e., it relies on the percentage of anomalies in the dataset as a priori. But we kindly emphasize that **we do not rely on the anomaly proportion of the dataset as a priori when setting $k_{ano}$**. Given that anomalies typically represent minor classes, we just empirically set $k_{ano}$ within a reasonable interval, typically no greater than 5. Ablation studies conducted in Sec 4.5.1 also demonstrate the robustness of GADAM to different $k_{ano}$ values in this interval. 

>(2) If the selected k leads to more anomalies than actual ones, will the proposed method be impacted negatively?

If $k_{ano}$ continues to increase, it will clearly have a negative impact on the model, as it will cause $V_a$ to be contaminated by more normal nodes, thus providing the wrong supervised signals for model training. Also, we did observe the phenomenon of negative impact on the Pubmed dataset from the ablation study of Sec 4.5.1, i.e., when $k_{ano}=4$ is larger than the actual percentage of anomalies in Pubmed (3.1%), there is a slight decrease in the AUC of the model. However, on other datasets, the $k_{ano}=5$ setting in our experiments leads to more anomalies than actual ones on many datasets according to Appendix A2, but we empirically find that the model still reaches state-of-the-art performance by a large advantage.

In summary,we set $k_{ano}$ in a reasonable interval (not greater than 5) . Also within this interval GADAM shows strong robustness as shown in ablation studies conducted in Sec 4.5.1.

Reviewer comment (Experiments):
Performance reported for competing methods is lower than in the papers where these methods were proposed. It raises the question of whether the models are sufficiently tuned.

Author response:
Thanks for your careful review and bringing this up! We will provide a more detailed explanation in our responses to Q1 and Q2 below. In summary, we adhered to the hyperparameters recommended in the original papers and employed a standardized dataset generated uniformly with PyGod[1] library to ensure fair comparisons.

Reviewer comment (Experiments):
No comparison to methods in the vast literature outside deep learning / neural networks. See for example https://arxiv.org/abs/1404.4679.

Author response:
Thanks for pointing this out! After a thorough examination of the relevant literature, we have identified the following key findings:

+ We have incorporated the definition of anomaly from the literature into our paper. Specifically, in Section 2.1.2, it is mentioned that community based methods identify anomalies by finding densely connected nodes, which is consistent with our definition of structural anomaly.  Additionally, in Section 2.2.2, some methods mark nodes with deviate attributes from other members in a community as anomalies, which is consistent with our definition of contextual anomaly.  Consequently, certain methods from the literature hold relevance as baselines.
+ Many methods discussed in the literature operate under a singular assumption about anomalies, tailored for a specific anomaly type. For instance, LOF[2] relies solely on node attributes to identify contextual anomalies by gauging the degree of deviation of nodes and their neighbors. On the other hand, SCAN[3] relies solely on structural information within the graph, detecting clusters through clustering for the identification of structural anomalies. In contrast, existing deep learning methods go beyond these limitations by adeptly utilizing both graph attributes and structural information, enabling a more comprehensive anomaly detection approach.

Indeed, some methods outside deep learning / neural networks continue to hold relevance in contemporary anomaly detection. We refer to the benchmark[1] and chose LOF and SCAN as representative methods. Additionally, we included the widely adopted isolation forest (IF)[4] for further evaluation. Experimental results on three benchmark datasets are as follows:

AUC-ROC：

|              | Cora       |            |            | ACM        |            |            | BlogCatalog |            |            |
| ------------ | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ----------- | ---------- | ---------- |
|              | contextual | structural | overall    | contextual | structural | overall    | contextual  | structural | overall    |
| LOF          | 0.862      | 0.5156     | 0.7012     | 0.6167     | 0.4543     | 0.5567     | 0.3738      | 0.4741     | 0.4146     |
| SCAN         | 0.4548     | 0.8121     | 0.6463     | 0.4534     | 0.8182     | 0.6678     | 0.4970      | 0.7579     | 0.6564     |
| IF           | 0.7798     | 0.5054     | 0.6019     | 0.5655     | 0.5443     | 0.5602     | 0.6481      | 0.5024     | 0.5917     |
| GADAM (ours) | **0.9416** | **0.9929** | **0.9556** | **0.9586** | **0.9721** | **0.9603** | **0.9094**  | **0.7763** | **0.8117** |

The experimental results indicate that LOF and SCAN struggle to balance performance in detecting both anomaly types, aligning with our own observations. Furthermore, IF doesn't exhibit effectiveness, likely due to its lack of specialization for graph-based data. In contrast, our proposed model showcases effective detection capabilities for both contextual and structural anomalies, leveraging the combined information from graph attributes and structure.

We sincerely appreciate your insightful review, and we will integrate the experimental results into a later revision with references to these literatures. Thank you again for enriching our paper with your valuable comment!

Reviewer comment (Experiments):
The AnomalyDAE paper reports an AUC of over 97 for the BlogCatalog dataset. Do you have an explanation for the discrepancy in your paper (AUC of 76.58)? Were the authors of that paper injecting anomalies in a totally different way?

Author response:
Thank you for bringing up this concern! We've thoroughly reviewed the original AnomalyDAE paper and its associated source code[9]. However, it appears that the anomaly injection method is not clearly outlined in the article, and the provided source code only includes the BlogCatalog dataset with anomaly labels. To ensure a fair and comprehensive comparison, we utilized PyGOD[1], an open library from the NIPS'22 benchmark for unsupervised graph anomaly detection. We applied this library uniformly across all datasets to generate anomalies, striving for consistency and fairness.

Simultaneously, we conducted experiments using the parameter configurations specified in the original AnomalyDAE paper. The obtained results align with those reported in the updated work[6]. Regrettably, due to the lack of clarity on how AnomalyDAE injects anomalies and the absence of support for other datasets in its source code, we couldn't replicate experiments on additional datasets using the provided code. We made diligent efforts to ensure the fairness of our experiments through PyGOD, and observed performance variations in AnomalyDAE may be dataset-dependent.

Reviewer comment (Experiments):
The AUC for Sub-CR, DOMINANT, and COLA is higher in the Sub-CR paper. Do you think hyperparameters for the baseline models are sufficiently tuned?

Author response:
Thank you for your thoughtful consideration. Regarding CoLA and DOMINANT, we employed PyGOD for their implementation and adhered to the parameters suggested in the original papers. We have carefully compared the results with those reported in past works[5, 6, 7] and believe that the results we obtained for these two models are in the normal range.

Concerning Sub-CR, as it lacks open-source availability, we initiated communication with the authors to obtain the source code. Unfortunately, the provided source code only includes prepared datasets without an interface for anomaly injection. Consequently, we injected anomalies and tested model perfromance through PyGOD's unified interface to ensure a fair comparison. Additionally, we maintained consistency by employing the same hyperparameter settings as specified in the original paper[8].

Additionally, it's important to note that the results of the experiments will also vary with the anomaly injection implementation, software library versions, and hardware environments. In our efforts to uphold fairness, we have standardized the processes of dataset generation, baseline implementation, and model evaluation through PyGOD. We appreciate your understanding and consideration in this regard.

Reviewer comment (Reproducibility):
If in Point 1, it is not just 1-hop neighbors but all k-hop neighbors, then the size of the subgraphs can be potentially very large in densely connected graphs. How does the model address that?

Author response:
**W2.** If in (1), it is not just 1-hop neighbors but all k-hop neighbors, then the size of the subgraphs can be potentially very large in densely connected graphs. How does the model address that?

As mentioned above, our model uses only 1-hop neighbors, so the size of the subgraphs will not be too large. However, there are works such as CoLA[1] and ANEMONE[2] that utilize random walks to sample multi-hop neighbors for nodes, aiming to enhance the model's receptive filed while reducing the size of the neighborhood subgraph.  Nevertheless, these approaches have significant drawbacks:

+ Neighbor information obtained through sampling introduces considerable randomness, as explicitly stated in the original paper[1]. To address this randomness, CoLA constructs multiple contrastive instances by sampling each node with $R$ rounds in the final computation of the anomaly score, as described in Sec 2.2 in our paper. In CoLA, $R$ is recommended to be set to 256, meaning each node needs to sample 256 neighborhood subgraphs for stable and competitive results, imposing a significant computational overhead.
+ As we mentioned in W1, utilizing multi-hop neighbors may contaminate a node's neighborhood information to a greater extent, a factor that has been overlooked in existing works.

What's more, our experimental results show that our model not only surpasses them significantly in terms of computational efficiency and GPU memory overhead but also outperforms them in terms of detection performance. We hope our explanations have addressed your queries regarding neighborhood subgraph construction. Thank you for your question, which has provided us with the opportunity for further clarification!

Reviewer comment (Theory):
The representations of a subgraph G in the local context are just an average of the representation of the node embeddings in G. Then, the anomaly scores for each node are measured by the difference between its node representation and the subgraph representation. It is possible that the node embedding of each neighbor is really different than the target node's embedding, but the average of the neighbor embeddings is similar to the target node. Doesn't it also introduce an over-smoothing problem? It could be worse than the standard message passing, where the aggregation is not just standard averaging but parameterized by a weight matrix. Here, the aggregation is just a plain average.

Author response:
**W4.** The simple mean pooling method for subgraph embedding may introduce an over-smoothing problem. It could be worse than the standard message passing, where the aggregation is not just standard averaging but parameterized by a weight matrix.

We acknowledge your concerns. However, we empirically found that using only 1-hop neighbors and employing a simple mean pooling method for subgraph embedding is effective. **Introducing more parameters to learn subgraph embedding may lead to potential overfitting issues**. To address your concern, we experimented with using GCN and GAT instead of mean pooling in the first stage for subgraph embedding, while keeping the rest of GADAM unchanged. The results under AUC-ROC metric are as follows:

|              | Cora       | Citeseer   | Pubmed     | ACM        | BlogCatalog | Books      | Reddit     |
| ------------ | ---------- | ---------- | ---------- | ---------- | ----------- | ---------- | ---------- |
| GCN          | 0.6322     | 0.7995     | 0.7561     | 0.5803     | 0.5175      | 0.5328     | 0.4242     |
| GAT          | 0.4727     | 0.6009     | 0.7867     | 0.4001     | 0.6657      | 0.534      | 0.387      |
| Mean pooling | **0.9556** | **0.9415** | **0.9581** | **0.9603** | **0.8117**  | **0.5983** | **0.5809** |

The experimental results demonstrate that using more advanced methods for subgraph embedding results in a significant performance drop. The primary reason for this is that the first stage of GADAM operates in an unsupervised manner, where the model enhances the consistency of all nodes and their neighbors by optimizing the loss function in Eq.(1). In this process, most nodes with high consistency are further optimized by the model, while nodes with lower consistency are identified as anomalies due to the optimization difficulty. Introducing more parameters increases the expressive power of the model, but it also leads to overfitting of the loss function, resulting in a loss of local inconsistency of the anomalous nodes.

Reviewer comment (Reproducibility):
The experiment results of the baselines presented in the paper are rather different from the ones reported in the original paper for the same datasets. For example, in the original AnomalyDAE paper, the AUC are 97.81 (BlogCatalog), 90.05 (ACM); whereas in this paper, they are 76.58 (BlogCatalog), 75/16 (ACM). Could the authors explain more about it?

Author response:
**W6.** Issues about baseline performance reported in this paper, particularly in comparison to the AnomalyDAE paper. The AUC are 97.81 (BlogCatalog), 90.05 (ACM) in the original paper; whereas in this paper, they are 76.58 (BlogCatalog), 75.16 (ACM).

Thank you for bringing up this concern! We've thoroughly reviewed the original AnomalyDAE paper and its associated source code[3]. However, it appears that the anomaly injection method is not clearly outlined in the article, and the provided source code only includes the BlogCatalog dataset with anomaly labels. To ensure a fair and comprehensive comparison, we utilized PyGOD[4], an open library from the NIPS'22 benchmark for unsupervised graph anomaly detection. We applied this library uniformly across all datasets to generate anomalies, striving for consistency and fairness.

Simultaneously, we conducted experiments using the parameter configurations specified in the original AnomalyDAE paper. The obtained results align with those reported in the updated work[5]. Regrettably, due to the lack of clarity on how AnomalyDAE injects anomalies and the absence of support for other datasets in its source code, we couldn't replicate experiments on additional datasets using the provided code. We made diligent efforts to ensure the fairness of our experiments through PyGOD, and observed performance variations in AnomalyDAE may be dataset-dependent.

Reviewer comment (Experiments):
How to decide topk% included in the second stage. How does the selection of this percentage affect the results?

Author response:
**W7.** How to decide topk% included in the second stage. How does the selection of this percentage affect the results?

Thanks for brining this up! For both $k_{nor}$ and $k_{ano}$, the common setting principle is to include as many correct supervisory signals as possible.

+ $k_{nor}$ determines the size of the high-confidence normal set $V_{n}$。Given that normal nodes constitute the majority of the dataset,  $V_{n}$ generally includes a very small proportion of true anomalies. Consequently,  $k_{nor}$ can be set as long as it ensures an adequate supply of supervisory signals,  and we set it to a larger value in our experiments. We also found in our implementation that the setting of $k_{nor}$ has minimal impact on the model's performance.
+ $k_{ano}$ determines the size of the high-confidence abnormal set $V_a$, it seeks to include as many true anomalies as possible and exclude normal nodes. So $k_{ano}$ is ideally set to be as close as possible to, but not substantially larger than, the proportion ($p$) of anomalies in the datasets. Intuitively, when $k_{ano}>p$, a continuous increase in $k_{ano}$ will cause more normal nodes to be marked as anomalous, thus provide more false pseudo-labels for the second stage, and lead to model's performance drop. Importantly, we kindly emphasize that **we do not rely on the true anomaly proportion of the dataset as a priori when setting $k_{ano}$.** We just empirically set $k_{ano}$ within a reasonable interval, typically no greater than 5. Ablation studies conducted in Sec 4.5.1 also demonstrate the robustness of GADAM to different $k_{ano}$ values.

Reviewer comment (Novelty):
The paper is missing some unsupervised graph anomaly detection papers:
- Ding, Kaize, Jundong Li, Nitin Agarwal, and Huan Liu. "Inductive anomaly detection on attributed networks." In Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pp. 1288-1294. 2021.
- Huang, Yihong, Liping Wang, Fan Zhang, and Xuemin Lin. "Are we really making much progress in unsupervised graph outlier detection? Revisiting the problem with new insight and superior method." arXiv preprint arXiv:2210.12941 (2022).
- Yang, Shujie, Binchi Zhang, Shangbin Feng, Zhanxuan Tan, Qinghua Zheng, Jun Zhou, and Minnan Luo. "Ahead: A triple attention based heterogeneous graph anomaly detection approach." In Chinese Intelligent Automation Conference, pp. 542-552. Singapore: Springer Nature Singapore, 2023.
- Fathony, Rizal, Jenn Ng, and Jia Chen. "Interaction-Focused Anomaly Detection on Bipartite Node-and-Edge-Attributed Graphs." In 2023 International Joint Conference on Neural Networks (IJCNN), pp. 1-10. IEEE, 2023.
- Wang, Qizhou, Guansong Pang, Mahsa Salehi, Wray Buntine, and Christopher Leckie. "Cross-domain graph anomaly detection via anomaly-aware contrastive alignment." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 4, pp. 4676-4684. 2023.

Author response:
**W8.** The paper is missing some unsupervised graph anomaly detection papers。

We appreciate your thorough review and a wealth of related works you've provided. We carefully examined these works you mentioned and will introduce each method systematically:

+ **AEGIS[6]:** AEGIS mainly focuses on inductive learning, including an anomaly-aware GNN layer and gives the model the ability to detect new anomalies by generative adversarial learning. Unfortunately, AEGIS is not open sourced and unavailable through PyGOD, a fair comparison may not be feasible.
+ **VGOD[7]:**  VGOD designs two modules for structural and contextual anomalies respectively, which belongs to the same domain as our work. In the following section, we will conduct a detailed comparison with VGOD.
+ **AHEAD[8]:** AHEAD is designed for anomaly detection in heterogeneous graphs. Therefore its research has little relevance to us, and its open source code is specifically implemented for IMDB dataset, so we cannot include AHEAD in our baseline.
+ **GraphBEAN[9]:** GraphBEAN is designed for bipartite node-and-edge-attributed graphs, and detects anomalous nodes and edges simultaneously. Taking the example of consumer-purchase-product graph, GraphBEAN is designed to detect edges representing anomalous transactions, and users with anomalous behavior. Thus its research area has little overlap with our work, and we won't consider it as a baseline in our study.
+ **ACT[10]:** ACT focuses on cross-domain graph anomaly detection (CD-GAD), which describes the problem of detecting anomalous nodes in an unlabeled target graph using auxiliary, related source graphs with label nodes. Recognizing the specific requirements for auxiliary source graphs,  we agree that it's not suitable for inclusion in our baseline.

Reviewer comment (Theory):
The problem formulation seems to be problematic. Authors have mentioned "Structural anomalies are densely connected nodes in contrast to sparsely connected regular nodes". What about the networks that are dense by nature? Moreover, if density is the only criteria for being a structural anomaly, why can't we use degree of a node as the metric to find the structural anomalies? Are we over-complicating the solution of a relatively simple problem?

Author response:
**W1.** Thanks for brining up this fundamental question! We will address your concerns one by one:
> (1) Definition of structural anomaly

The definition of structural anomaly in our paper aligns with the latest benchmark from NIPS'22[1]. Notably, this definition is widely used in the field of unsupervised graph anomaly detection and is evident in the majority of existing works[2,3,4,5]. Structural anomaly is prevalent in the real world and is characterized by **nodes which are densely connected with abnormal links**. Examples range from members of organized fraud gangs, people from different communities but closely connected to each other, and even papers that engage in malicious citations. In our work, we adopt the anomaly injection method used in previous papers[1,2,3,4,5] to generate densely and abnormally connected nodes as structural anomalies (see Appendix A.1). While we acknowledge that solely emphasizing on "densely connected" can lead to ambiguities in the understanding of structural anomalies, and we will add a new term for "abnormal links" in a later revision.

> (2) What if the networks are dense by nature?

As discussed earlier, **structural anomalies are essentially clusters of densely and abnormally linked nodes**. Therefore, there is no inherent conflict between the graph itself being dense and the definition of structural anomalies. In a dense graph, normal nodes are normally connected to each other, while structural anomalies have abnormal connections to each other. As we introduced in Appendix A.1, when generating structural anomalies, we randomly select $m$ (e.g. $m=15$) nodes and connect them to each other. The randomly selected nodes come from different communities, thus creating densely and abnormally linked structural anomalies, which is consistent with the definition of structural anomalies.

> (3) Can we use node degrees to find structural anomalies？

This is indeed an interesting idea. However, as previously discussed, employing node degree is not aligned with the essence of detecting structural anomalies, since the degree does not reflect the abnormal links of nodes. For instance, in networks like Twitter's following network or paper citation networks such as Cora, it is clear that we cannot simply mark accounts with high number of followers or highly cited papers as anomalies, even though they have significantly higher node degrees compared to other nodes. We will delve deeper into validating the effectiveness of utilizing node degree for structural anomaly detection in response to W6.

> (4) Are we over-complicating the solution of a relatively simple problem?

Following the discussion above, we don't consider the detection of structural anomalies to be overly complicated. Utilizing node degree only exploits the phenomenon of being densely connected and can easily fail in graphs with high average node degrees, as we will demonstrate in response to W6. In contrast, our model takes a principled approach to anomaly detection by capturing two types of anomaly signals from both local and global perspectives. The anomaly signals are propagated through dense links in the cluster of abnormal nodes by adaptive message passing in the second stage. This process enhances the anomaly scores of structural anomalies as a whole, and we believe this approach possesses greater generalization capabilities and is more in line with the essence of anomaly detection.

Reviewer comment (Writing):
What is the difference between H_local and H_global? They seem to be computed in the same way through Eq. 3. Why are we calling the second one as "global"?

Author response:
**W3**. What is the difference between H_local and H_global? Why are we calling the second one as "global"?

Indeed, both $H_{local}$ and $H_{global}$ are calculated in the same way through Eq.(3), but as we claimed in  Sec. 3.2.1, $H_{global}$ employs a new MLP as an encoder.

To elaborate, in the first stage, we utilize one MLP on the raw node features to obtain $H_{local}$, employing the local inconsistency of the nodes as anomaly signals and learning $H_{local}$ through Eq.(1). While in the second stage, we deploy another MLP on the raw features to get $H_{global}$, mining the anomaly signals from the consistency of the nodes and global normal context, and learning $H_{global}$ through Eq.(13). We refer to the second one as "global" from the perspective of anomaly signaling.

Reviewer comment (Theory):
Some of the claims in the paper should have been explained more. Can you please elaborate the reason of "as the training advances, the influence of pre-attention gradually diminishes, while the opposite for post-attention".

Author response:
**W4**.  Can you please elaborate the reason of "as the training advances, the influence of pre-attention gradually diminishes, while the opposite for post-attention"?

Thanks for brining this up. We will explain the reasons for this in terms of formula definitions and design intuitions, respectively:

+ As defined in Ep.(12), pre-attention and post-attention regulate the weights by a weight coefficient $\beta_t$ that varies with the training step. In this process the weight of pre-attn is gradually decreased while the weight of post-attn is increased through $\beta_t$.
+ This is a special design with the following considerations: (1) At the beginning of training in the second stage,  $H_{global}$ is not fully optimized, and pre-attn serves a reliable criterion to judge whether a node should align with its neighbors. However, we consider that the local anomaly scores in the first stage may contain inherent errors. For example, it fails to detect certain structural anomalies with high local consistency. Gradually reducing the weight of pre-attn aims to prevent inherent errors in local anomaly scores from continuously misdirecting attention for nodes. (2) As training progresses, nodes adaptively align with their neighbors through message passing involving hybrid attention. Therefore, we increase the weight of post-attn based on feature similarity as a complementary and corrective measure to pre-attn.
+ In the ablation experiments in Section 4.5.3, we also examined various attentional mechanisms. The results demonstrate that the dynamic sum approach exhibits a substantial advantage over other alternatives.

The explanations provided above are also articulated in our paper, accompanied by visualizations of the attention coefficients in Appendix E. These visualizations demonstrate that hybrid attention effectively reduces the message passing of contextual anomalies while enhancing them for normal nodes and structural anomalies with their surroundings. We hope that our explanations could help you further understand our design motivations!

Reviewer comment (Evaluation):
In Table 1, overall anomaly detection performance is reported. I am curious to see the performance on contextual and structural anomalies separately.

Author response:
**W5.** I am curious to see the performance on contextual and structural anomalies separately.

Thanks for your valuable comment! To assess the efficacy of our model in detecting two types of anomalies, we selected three benchmark datasets for experimentation. **Cora** serves as a small, regular dataset with a low average node degree. **ACM**, on the other hand, represents a larger dataset with a substantial number of anomalies. Finally, **BlogCatalog** is characterized by a large average node degree, presenting a greater challenge for the detection of structural anomalies. Additionally, we chose representative models from the baseline families—DOMINANT, CoLA, Sub-CR and ComGA—for comparative experiments. The experimental results under the AUC-ROC metric are as follows:

|          | Cora       |            | ACM        |            | BlogCatalog |            |
| -------- | ---------- | ---------- | ---------- | ---------- | ----------- | ---------- |
|          | contextual | structural | contextual | structural | contextual  | structural |
| DOMINANT | 0.7173     | 0.9312     | 0.7045     | 0.8182     | 0.7371      | 0.7664     |
| CoLA     | 0.8934     | 0.8213     | 0.9103     | 0.8325     | 0.8585      | 0.7325     |
| ComGA    | 0.8173     | 0.9052     | 0.7525     | 0.8989     | 0.8294      | 0.7679     |
| Sub-CR   | 0.9057     | 0.9293     | 0.7067     | 0.7913     | 0.8267      | 0.7573     |
| GADAM    | **0.9416** | **0.9929** | **0.9586** | **0.9721** | **0.9094**  | **0.7763** |

The results show that:

+ Most of the baselines struggles to balance the detection of both types of anomalies, while GADAM achieves the best.
+ On the two sparser datasets, Cora and ACM, GADAM significantly outperforms baselines in structural anomaly detection. Even on the denser BlogCatalog dataset, GADAM continues to outperform all baselines, providing further validation of the effectiveness for structural anomaly detection. This outcome aligns with our model designs described in W2.

We intend to incorporate these valuable results into our later revisions, thanks again for your valuable comment!

Reviewer comment (Experiments):
As mentioned before, degree centrality seems to be a good metric to capture the structural anomalies. Can you please include this heuristic as a baseline to compare with?

Author response:
**W6.** Can you please include degree as a baseline to compare with?

Certainly. Given that degree is only capable of detecting structural anomalies, we evaluated its performance **specifically on structural anomalies** using the benchmark datasets mentioned in our paper. We have also provided the average node degree (avg_d) for each dataset, and listed the datasets in increasing avg_d to offer more clarity. The results, measured under the AUC-ROC metric, are as follows:

|        | Citeseer(avg_d = 1.4) | Pubmed(avg_d = 2.3) | ACM(avg_d = 4.4) | ogbn-Products(avg_d = 25.3) | BlogCatalog(avg_d = 33.1) |
| ------ | ------------------------- | ----------------------- | -------------------- | ------------------------------- | ----------------------------- |
| Degree | 0.9919                    | 0.9421                  | 0.9264               | 0.6397                          | 0.64                          |
| GADAM  | **0.9956**                | **0.9718**              | **0.9721**           | **0.7963**                      | **0.7763**                    |

As evident from the experimental results:

+ Degree is indeed a good metric for detecting structural anomalies in certain datasets but exhibits poor performance in datasets characterized by high average degrees (e.g., ogbn-Products, BlogCatalog). This limitation underscores the metric's challenges in achieving robust generalization across diverse datasets.
+ GADAM consistently outperforms Degree across all datasets, with the disparity being particularly pronounced in datasets featuring higher average degrees. This robust performance stems from GADAM's design, rooted in the fundamental principles of anomaly detection, enabling stronger generalization across diverse datasets.

We acknowledge that the effectiveness of using degree as a metric for detecting structural anomalies may be contingent on the limitations of current anomaly generation methods. Designing more realistic and flexible synthetic outlier generation in graphs is also a critical challenge[1]. Our work, while based on a general anomaly injection approach, strives to align with the essence of anomaly detection. We sincerely hope that our underlying motivations resonate with your understanding!

Reviewer comment (Theory):
How do you capture the nodes which are both contextual and structural anomalies? Also, will your approach be able to capture "Combined Outliers" (or combined anomalies) as introduced in "Outlier Aware Network Embedding for Attributed Networks" (AAAI-2019)?

Author response:
**W7.** How do you capture the nodes which are both contextual and structural anomalies?

Thanks for brining this up! Intuitively, our framework is poised to effectively capture nodes that exhibit both contextual and structural anomalies (mixed anomalies for short). The first stage of GADAM captures well the contextual abnormal features exhibited by mixed anomalies. In the second stage, GADAM facilitates the absorption of anomalous signals from the neighboring structural anomalies through adaptive message passing, further enhancing the detection of mixed anomalies.

Moreover, after a thorough examination of the mentioned paper (AAAI-2019), we confirm that "combined anomalies" align with our concept of mixed anomalies. To further address your concern, we generated a certain number of mixed anomalies on the five benchmark datasets. Specifically, we randomly selected $q$ structural anomalies from the original datasets in our paper, and employed the contextual anomaly injection method outlined in Appendix A.1 to substitute features for these nodes, thereby creating $q$ mixed anomalies. Our evaluation utilized the representative baselines mentioned in W5, and the performance of mixed anomaly detection under the AUC-ROC metric is as follows:

|          | Cora (q=15) | Citeseer(q=15) | Pubmed(q=60) | ACM(q=60) | BlogCatalog(q=30) |
| -------- | -------------- | ------------------ | ---------------- | ------------- | --------------------- |
| DOMINANT | 0.8354         | 0.8498             | 0.7904           | 0.7448        | 0.7723              |
| CoLA     | 0.8976         | 0.8928             | 0.9618           | 0.7909        | 0.7965              |
| ComGA    | 0.8828         | 0.9335             | 0.9255           | 0.8433        | 0.8275                |
| Sub-CR   | 0.9236         | 0.9400             | 0.9810           | 0.7518        | 0.7963                |
| GADAM    | **0.9970**      | **0.9994**         | **0.9863**       | **0.9884**    | **0.9685**            |

The results indicate that GADAM exhibits a strong ability to detect mixed anomalies across all datasets, aligning with the motivation behind the model design. We will incorporate these findings into a subsequent revision. Thanks for your insightful review!