Paper: New recipes for graph anomaly detection: Forward diffusion dynamics and graph generation (ICLR-2024)

Reviewer comment (Experiments):
The authors show that this rapid egonet dissimilarity decrease holds for CORA dataset, but the anomalies in the data were synthetically injected by Liu et al., 2022. For organic anomalies, this assumption might also be true on average (Liu et al., 2022), but the fact that DiffAD does not account for structural anomalies will likely prevent it from detecting anomalies if their initial egonet dissimilarity is low.

Author response:
In this paper, we assessed DiffAD's performance on three real datasets, namely, YelpChi, Reddit, and Weibo, apart from the synthesized datasets published in [1], i.e., CORA, ACM, BlogCatalog. Furthermore, we have conducted additional experiments on another three real-world datasets: Tfinance, Tolokers, and Questions, as reported in our **General Response to All Reviewers - Additional Experiments**. Overall, we have validate DiffAD's efficacy on **six real datasets and three synthetic datasets**. From the experimental results, even though these datasets encompass different types of anomalies, including contextual and structural anomalies, DiffAD consistently achieves superior performance compared to the baselines (three new methods have been added). 

We also want to emphasize that in real-scenarios, some anomalies can be categorized as both contextual anomalies and structural anomalies (such as the anomalies in Weibo dataset [3]). In addition, to detect contextual anomalies, we have explored both the graph structure and attributes. Although our proposed methods do not explicitly measure the abnormal structural patterns, some structural anomalies can also be detected due to their irregular attributes. From the results on Reddit, in which `the outliers in the Reddit have similar average neighbor feature similarities for outliers and inliers` [3] and the initial egonet dissimilarity is low, we can see that DiffAD achieves the best results with regard to all metrics.

Reviewer comment (Theory):
It is not clear how to extend these principles to consider for structural anomalies. The paper dismisses the need to consider structural anomalies that "form more densely [connected] links with other nodes", but there could be anomalies that also keep the density somewhat unchanged.

Author response:
Structural anomaly detection is a significant are. In fact, the GCN classifer employed in DiffAD is capable of detecting structural anomalies - just as the conventional GCN-based anomaly detectors employed in many existing methods like [5] [6]. For anomalies that exhibit similar density, we can explore how their attributes are different from their neighbors for detecting anomalies, which is the focus of this work.

Reviewer comment (Evaluation):
Is Observation I an expected result? If so, why?

Author response:
Yes, Observation I aligns with our expectations. In our preliminary study, we set the attribute distribution of contextual anomalies to be substantially different from normal nodes, following the definition in [1], [2], [3]. According to the definition, contextual anomalies are generated by different mechanisms, and their attributes significantly differ from those of their neighbors. Our preliminary study setup adheres to the implementation in [4]. We provided the rigorous analysis of this phenomenon through the proof of Proposition 1 in Appendix B.

Reviewer comment (Evaluation):
A fundamental question not addressed by the paper is: what is the anomaly detection performance on anomalous nodes whose initial ego dissimilarity is low?

Author response:
Good question. In fact, we have also studied this in our experiment. We specifically include Reddit, where `the outliers in the Reddit have similar average neighbor feature similarities for outliers and inliers` [3] and the initial egonet dissimilarity is low, to validate these questioned scenarios. From the results reported in Tables 1 and 4, we can see that DiffAD achieves the best results with regard to all metrics.

Reviewer comment (Theory):
Is it possible to extend the diffusion model to a case where A is changed over time?

Author response:
We appreciate the reviewer's interests in extending diffusion models to gradually corrupt the graph structure. It is worth noting that in the context of anomaly detection on static attributed graphs, the adjacency matrix contains binary/discrete values 1 and 0 to indicate the presence of edges. Different from the continuous node features, simply adding Gaussian noise to the discrete graph structure cannot meet the demand for gradually corrupting the graph structure, and from the equivalent score-based diffusion perspective [1] [2], calculating the derivative of a discrete-valued variable in the diffusion process is non-trivial. In fact, applying denoising diffusion models for discrete data remains an open problem and we recognize this as a potential research direction to detect anomalies.

Reviewer comment (Writing):
Please clarify:  
- In Eq. (18), should one of the $Z_t$ in the concatenation of the first GCN layer be $\bar Z_t$?  
- In Eq. (4), specify that this is the accumulated energy ration at rank $l$  
- p.2: Confronting -> Adhering to the principles  
- p.5: Eventual -> Eventually

Author response:
R6.1, R6.2, R6.3: No, $\mathbf{\bar{Z}}_t$ is concatenated only in the last layer.  
Thanks to the constructive comment. Eq. (4) is to the measure the accumulated energy ratio at rank $l$. We have revised the description above Eq. (4) as:
  
- `We further delve into the overall ratios of low and high frequency signals with regard to different thresholds at each step t by measuring the accumulated energy ratio at rank l following Tang et al. (2022)`.
  
Point taken. We have corrected both typos in the revision.

Reviewer comment (Experiments):
Graph anomaly detection often includes two types: attribute anomalies and structural anomalies. The author disrupts the original distribution by continuously injecting noise, which may be useful for detecting attribute anomalies. However, is this also effective for detecting structural anomalies?

Author response:
Our GCN classifier employed in DiffAD is capable of detecting structural anomalies following the message-passing scheme - just as the conventional GCN-based anomaly detectors employed in many existing methods, such as DAGAD [4] and other unsupervised detectors DOMINANT [5]. The experimental results and cases studies in Section 7 demonstrate the efficacy of our proposed method.

Reviewer comment (Experiments):
Baselines are too few to demonstrate the superiority of your method. You should choose some famous GAD methods, such as COLA, SL-GAD, rather than GCN, GAT, etc.

Author response:
Thanks for the constructive comment. We have conducted additional experiments with three most recent semi-/supervised baselines and two contrastive detectors, i.e., CoLA and CONAD, and the results are reported in the **General Response to All Reviewers - Additional Experiments**. From the results, we can see that our method outperforms all baselines.

Reviewer comment (Writing):
The definition of 'ego-net dissimilarity` is actually a GNN embedding in literature, so 'dissimilarity` is an inaccurate word.

Author response:
Thanks for the reminder. In this paper, we employ 'egonet dissimilarity' to specifically denote how each node's attributes are different from its egonet neighbors. We have revised the footnote on Page 3 to clarify this and to remove confusion, we can change it to 'egonet proximity' for the same purpose in the final version.

- `To eliminate confusion, we use 'egonet dissimilarity' to specifically denote how each node's attributes differ from its egonet neighbors, distinct from the embedding method used in previous works.`

Reviewer comment (Theory):
The time complexity should be very high due to the nature of the transformer and DDPM.

Author response:
For the transformer model (TRC-Trans), its complexity is the same as other graph transformer models. In the meanwhile, we present a skip-step algorithm in Section 5.1 to reduce the computational cost and the impact of this skip-step algorithm is also validated in Appendix J.6. For the generative method (DiffAD), it is true that our sampling process also needs to reverse the diffusion step by step, the same as the DDPM models.

In fact, reducing the computational cost of the Transformer model and accelerating the DDPM sampling process are currently hot research areas. These topics remain under investigation, and since this work primarily focuses on anomaly detection, we defer further enhancements to these methods to our future research efforts. In addition, with regard to your concern on the complexity, we have presented the complexity analysis in Appendix I.

Reviewer comment (Theory):
Why does the forward process need a separate loss function that utilizes the label information?

Author response:
We propose three methods, namely FSC, TRC and DIFFAD, for graph anomaly detection. In FSC and TRC, we train classifiers to identify anomalies with regard to the dynamics of the egonet dissimilarity. Therefore, label information is utilized. However, in DiffAD, the forward and reverse process involves no label information for graph generation, as formulated in Eqs. (20) and (70).

Reviewer comment (Experiments):
The paper's focus on a limited type of anomaly (contextual anomaly).

Author response:
R3-C3: Thanks for the constructive comment. It is true this our work primarily focuses on anomalies that exhibit deviating attributes compared to their egonet neighbors. 

However, in addition to our **R2** to Reviewer `1PRg`, we want to emphasize that categorizing graph anomalies as structural or contextual is to describe how anomalies distinct to normal nodes with respect to the graph structure and attributes. An anomalous node can belong to both categories and some structural anomalies can also be detected due to their irregular attributes. Therefore, our methods are capable of detecting structural anomalies that exhibit these characters, such as those in the widely-used Weibo dataset [1]. Furthermore, the competitive results of BWGNN, which is designed only for contextual anomaly detection, align with our argument, while our best-performing method surpasses the overall performance of the baselines.

Lastly, we state that we target contextual anomaly detection to concisely formulate the problem - `we explore both the graph structure and attribute to identify anomalies that exhibit deviating attributes to their neighbors` (defined in Section 2). 

We sincerely hope that our response can address your concerns, and we will gladly answer any additional questions you may have.

[1] BOND: Benchmarking unsupervised outlier node detection on static attributed graphs (NeurIPS 2022).

[2] Rethinking graph neural networks for anomaly detection (ICML 2022).

Reviewer comment (Experiments):
While the analysis of the forward diffusion process is well-done, the proposed method (Approach I) lacks sufficient evidence to demonstrate its superiority.

Author response:
Thank you for the insightful comment. In fact, the example in Fig. 2 is to illustrate our motivation of Approach I. The node feature in this preliminary study is 1-dimensional, as claimed in Section 4.1 - `The node attributes are randomly drawn from two Gaussian distributions, N(1, 1) for the normal class and N(1, 5) for anomalies, following Tang et al. (2022)`.

Given the high dimensionality of data in real-scenarios, the egonet dissimilarity will be divergent in each dimension. Such multi-dimensional data and pattern pose significant challenges to detect anomalies based on egonet dissimilarity at the initial stage, and this is one of the pivotal reasons for employing deep learning techniques for high-dimensional data analysis and graph anomaly detection. To this end, we propose two methods in Approach I to explore such patterns. Investigating other simple, effective methods based on egonet dissimilarity for high-dimensional graph data learning is a promising direction worth exploration.

Reviewer comment (Experiments):
The primary focus of this paper is on contextual anomalies that exhibit marked differences from their neighbors. However, it raises concerns about the representativeness of such anomalies in real-world diverse datasets. This could limit the applicability of the proposed approach. To address this, expanding the experimental evaluation to real datasets [1] would enhance the paper's contributions.

Author response:
We have conducted additional experiments on three more real-world datasets in [1], and the results are reported in the **General Response to All Reviewers - Additional Experiments**. It is worth mentioning that we already reported the results on the YelpChi, Reddit and Weibo datasets (in Tables 1 and 4), which are all collected from real-world websites.

Reviewer comment (Experiments):
While it is intuitive that adding noise would lead to a rapid drop in egonet dissimilarity for anomalies, it raises questions about whether complex methods in Approach I are necessary for anomaly detection. At the initial stages, the egonet dissimilarity of anomalies significantly outweighs that of normal instances as illustrated in Fig. 2. Therefore, it's worth considering if exploiting this difference directly could lead to more efficient detection.

Author response:
Thank you for the insightful comment. In fact, the example in Fig. 2 is to illustrate our motivation of Approach I. The node feature in this preliminary study is 1-dimensional, as claimed in Section 4.1 - `The node attributes are randomly drawn from two Gaussian distributions, N(1, 1) for the normal class and N(1, 5) for anomalies, following Tang et al. (2022)`.

Given the high dimensionality of data in real-scenarios, the egonet dissimilarity will be divergent in each dimension. Such multi-dimensional data and pattern pose significant challenges to detect anomalies based on egonet dissimilarity at the initial stage, and this is one of the pivotal reasons for employing deep learning techniques for high-dimensional data analysis and graph anomaly detection. To this end, we propose two methods in Approach I to explore such patterns. Investigating other simple, effective methods based on egonet dissimilarity for high-dimensional graph data learning is a promising direction worth exploration.

Reviewer comment (Evaluation):
The experimental results in Table 1 highlight that Propositions 1-based methods (FSC, TRC) often fail to outperform baseline approaches on multiple datasets. This further emphasizes the need to scrutinize the necessity and validity of Approach I.

Author response:
From the results on Weibo dataset, we can see that FSC and TRC methods achieve better results than others and `we attribute this to the significantly lower feature similarities between anomalies and normal nodes (0.004 vs. 0.993), which makes anomalies’ trajectories easier to be distinguished from normal nodes` (in Section 7.2). Although their performance are not the best on other datasets, we propose FSC and TRC to explore novel clues for graph anomaly detection and our experiment demonstrate that when the egonet dissimilarity is significant, FSC and TRC are even more effective than the baselines.

Reviewer comment (Theory):
Concerning loss function (Eq. 22), it's vital to ensure that the nodes in the graph \(G_a\) generated using denoising techniques align with the class of the original graph \(G\). Any inconsistencies could lead to conflicting optimization. Specifically, for an anomaly \(v_i\) in graph \(G\), how to ensure the generated corresponding node \(v_i\) in \(G_a\) is also an anomaly?

Author response:
We appreciate the reviewer for raising an insightful question regarding the consistency between the generated graph $\mathcal{G}_a$ and $\mathcal{G}$. Indeed, this aspect is pivotal for generating additional samples to enhance detection performance. Within the DDPM framework, this consistency subjects to the denoising neural network's ability to learn the original data distribution.

In our work, we aim to ensure this consistency by adhering to the two principles detailed on Page 5, as elaborated through our preliminary studies in Section 4. Specifically, by Principle I, we aim to ensure `the prior distribution` $p(\mathbf{X}_a|\mathbf{A})$ `of the generated graph aligns with the original distribution` $p(\mathbf{X}|\mathbf{A})$, `enabling the classifier to learn a more effective decision boundary to distinguish anomalies` and by Principle II, we expect to `recover the low frequency energy` to discriminate nodes in different classes, i.e., anomalous and normal.

Our proposed DiffAD model achieves both by employing GCN to capture the original graph $\mathcal{G}$'s local semantics and recover low frequency signals. We empirically validate the effectiveness of the generated graphs through extensive experiments in Section 7.2. The experimental results not only demonstrate the performance of DiffAD but also the efficacy of the generated graphs on boosting existing detectors, detailed in Section 7.3.

Reviewer comment (Experiments):
Although unsupervised methods cannot make use of label information, many of them [2][3] also leverage local inconsistencies for anomaly detection. Providing a comparative analysis against these existing methods would help establish DIFFAD's superiority effectively.

Author response:
Thanks for the constructive comment. For the limited space, we have added a discussion on these methods exploring local inconsistencies for anomaly detection in the related works in Appendix A.1.

-`Notably, there are plenty of unsupervised anomaly detection techniques, such as CONAD [2], SL-GAD [3] and Sub-CR [4], that investigate the consistency between anomalies and their neighbors in different contrastive views to measure node irregularity. While these methods investigate the egonet dissimilarity for anomaly detection, they only focus on the original graph, and the dynamics of egonet dissimilarity in the forward diffusion process remains unexplored.`

Reviewer comment (Experiments):
Limited experimental evaluation. No discussion of ablation or hyperparameter turning.

Author response:
For space limitation, we reported the additional ablation study and hyperparameter sensitivity analysis in Appendix J. These case studies are also referred to in Section 7:

- `We report additional experiments on exploring the impact of the generated graphs, key parameters and the skip-step algorithm, as well as an ablation study in Appendix J.`

Reviewer comment (Experiments):
No comparison to recently-proposed methods for graph anomaly detection, such as CoLA or CONAD. The justification is that such methods are not classifiers and they just assign an "anomaly score" to nodes. However, one could try a simple baseline of thresholding this anomaly score.

Author response:
To address your concern about baselines, we have compared DiffAD with three up-to-date semi-/supervised methods and two contrastive detectors, i.e., CoLA and CONAD, on three additional datasets, as reported in the **General Response to All Reviewers - Additional Experiments**. From the results, we can see that our method outperforms all baselines on all datasets.

Reviewer comment (Evaluation):
No discussion of running time.

Author response:
Thanks for the constructive comment. We have added the complexity analysis in Appendix I.

Reviewer comment (Reproducibility):
No discussion of inference for the DDPM.

Author response:
We provided the discussion on the reverse process (inference) in Section 3.2 and Appendix A.2. Our inference algorithm (formulated in Eq. (21)) was explained in Section 6.2 and Appendix H.

Reviewer comment (Experiments):
Would it be possible to compare the proposed method to contrastive detectors by applying a threshold?

Author response:
To address your concern about baselines, we have compared DiffAD with three up-to-date semi-/supervised methods and two contrastive detectors, i.e., CoLA and CONAD, on three additional datasets, as reported in the **General Response to All Reviewers - Additional Experiments**. From the results, we can see that our method outperforms all baselines on all datasets.

Reviewer comment (Reproducibility):
How is the inference for the DDPM performed?

Author response:
We provided the discussion on the reverse process (inference) in Section 3.2 and Appendix A.2. Our inference algorithm (formulated in Eq. (21)) was explained in Section 6.2 and Appendix H.

Reviewer comment (Theory):
What is the computational complexity of the proposed method as a whole?

Author response:
Thanks for the constructive comment. We have added the complexity analysis in Appendix I.

Reviewer comment (Writing):
One of my biggest concerns about this paper is its clarity of presentation. The paper is hard to understand. One factor is that the authors did not describe enough about how the diffusion process works on graphs. Section 3.2 only explains the generic diffusion process on regular data (x). How it applies in the context of graphs, which contain structure and attributes, is not clearly explained. After explaining Section 3.2, the authors directly jump to the graph observations without explaining how the diffusion process on graphs works.

Author response:
We provided preliminaries on the forward graph diffusion process in Section 2 - `The forward graph diffusion is referred to as injecting` $T$ `scales of scheduled noise to node attributes with the fixed graph structure`, which is to `gradually add scheduled Gaussian noise to the original data` $\mathbf{x}_0$ `through a `$T$`-step Markov chain such that the eventual distribution at the last step` q($\mathbf{x}_T$ |$\mathbf{x}_0$) ∼ $\mathcal{N}(0, \mathbf{I})$, formulated in Eq. (1) in Section 3.2. Per your comment and for space limitation, we have added another paragraph at the end of A.2 in Appendix A to better explain the forward graph diffusion process.

- `In this work, we investigate the DDPM framework specifically for graph anomaly detection and following our definition in Section 2, our forward graph diffusion process is to inject scheduled noise to the node attributes,` $\mathbf{X}$,` and the graph structure is fixed at all diffusion steps.`

Reviewer comment (Theory):
Since the authors did not clearly explain the diffusion process on graphs, many questions may arise about this, such as:
- How does the structure of the graph (A) contribute to the next node feature representation (X_{t+1})?
- Any effect of the neighboring node on a particular node representation in the next step?
- The diffusion process itself tries to model the data distribution, i.e., for graph cases, it models the data distribution of G={A, X}. As the first approach utilizes the diffusion information, why do the authors claim that the approach does not need to learn P(A, X)?
- And more questions that depend on the answer to the questions above.

Author response:
**R2.1**: In the forward diffusion process (as explained in Section 2), we fix the graph structure $\mathbf{A}$ and it does not affect the corrupted node features $\mathbf{X}_{t+1}$ at the next diffusion step.

**R2.2**: In addition to **R2.1** and following the DDPM framework, for a specific node $v_i$, its neighbors do not affect the corrupted node features $\mathbf{x}_{t+1}$ at $t+1$ in the `forward diffusion process`. However, it is worth noting that when measuring the egonet dissimilarity and graph energy distribution in the forward process, we can observe divergent evolving patterns between anomalies and normal nodes (our observation I), as well as the graph energy shifts (observation II).

In the `reverse process`, to obtain $\mathbf{X_t}$ from $\mathbf{X_{t+1}}$ and ensure that the generated graph adheres to the original graph semantics, we investigate GCN as a low-pass filter to recover low-frequency signals, following our devised design principles I and II in Section 4. From the spatial view, GCN is aggregating neighboring nodes' information to the target node; thus, neighbors will affect the target node in the reverse process. This can be considered as to generate node attributes conditioned on their neighbors and graph structure, which meets our expectation on synthesizing graphs that adhere to the original graph semantics.

**R2.3**: We appreciate the reviewer's insightful comment. By claiming `we first explore a novel paradigm to detect anomalies on a static graph without the need to explicitly learn its data distribution`, we want to state that in distinction to the conventional GNN methods, which employ neural networks to explicitly model the data distribution $p(\mathbf{A}, \mathbf{X})$ for downstream tasks, our Approach I alternatively involves neural networks to learn `the egonet dissimilarity changes in the forward diffusion process`. Instead of classifying nodes by learning the distribution $p(\mathbf{A}, \mathbf{X})$, our Approach I is proposed to learn the evolving patterns from $\lbrace \boldsymbol{\Omega_t} \rbrace_{t=0}^T$. In other words, our proposed methods, namely forward sequence classification (Section 5.2) and trajectory representation learning (Section 5.3), learn the evolving patterns rather than $p(\mathbf{A}, \mathbf{X})$ to detect anomalies.

Reviewer comment (Novelty):
The principle derived from the first observation is not surprising, i.e., "The denoising neural network should be capable of capturing the local information in egonets". In fact, I would argue that this principle is the bedrock of nearly all graph anomaly detection models.

Author response:
The denoising neural network is not for distinguishing anomalies; rather, it is proposed to reverse the diffusion process for graph generation, as explained in Section 3.2 and Section 6.1 - `Our goal is to learn the original graph data distribution through the reverse process, which can be practically described as to learn a denoising network` and `we train a two-layered GCN classifier` with the generated graphs (detailed in Section 6.3) for detecting anomalies.