Paper: Probabilistic Sampling-Enhanced Temporal-Spatial GCN: A Scalable Framework for Transaction Anomaly Detection in Ethereum Networks (ICLR-2024)

Reviewer comment (Writing):
The presentation and writing of the paper need improvement. Overall, the paper is not well-written with a somewhat unclear presentation and lacks critical details. It is necessary to describe the main contributions of the study and clearly explain how the existing challenges are addressed. Furthermore, as a key technology involved in this research, TRW should be introduced with more details, such as what specific temporal features are used and how they interact with spatial features, if applicable. Additionally, it is not clear which 7 features are extracted and why they were selected. Were they chosen based on findings from other existing studies? An example to demonstrate a sample and how the spatial and temporal patterns are involved would be helpful.

Author response:
- The paper has been structured to clearly present the main contributions (page 2), particularly focusing on the Temporal Random Walk (TRW) and Graph Convolutional Networks (GCN) methodology for anomaly detection in the Ethereum network. The empirical analysis section and the use of figures aim to make the presentation more understandable (Pages 8-9).
- TRW is introduced on page 3 with a focus on its ability to encapsulate temporal sequences and correlations of transactions to detect anomalies (page 4). It also explains how TRW impacts GCN performance, particularly in the context of Ethereum. This includes mathematical justification and algorithmic representation of the TRW-GCN combined method (page 11-14).
- The paper details the extraction and selection of 7 features, emphasizing their relevance in anomaly detection. These features include incoming value count, activity rate, change in activity, and time since the last transaction. The selection is based on their ability to capture spacial and temporal dynamics, which is crucial for detecting anomalies (Page 6-7).

Reviewer comment (Experiments):
There is a lack of comparison with existing approaches. As mentioned in the Introduction section, there are numerous related studies that employ various sampling methods. Liu et al. (2023) is introduced as an approach that "combines historical information over time". However, none of these approaches is considered as a benchmark method. It would be better and necessary to provide more experimental results demonstrating that the proposed approach offers a superior solution.

Author response:
- We evaluate the TRW-GCN methodology using four distinct machine learning techniques: DBSCAN, SVM, Isolation Forest (IsoForest), and Local Outlier Factor (LOF) (Page 7). This comparative analysis, especially in the context of different block sizes in Ethereum, demonstrates the effectiveness of the TRW-GCN approach over traditional methods. This also underscores the importance of algorithm selection, indicating a higher false positive rate with clustering methods.
- We also discuss the convergence of GCNs and TRW, emphasizing our approach in refining anomaly detection methods. This includes comparing different clustering and scoring methods to decrease false positives in detecting anomalous activities.
- We used 7 features for anomaly detection including outgoing_tx_count, incoming_tx_count, outgoing_value_count, incoming_value_count, activity_rate, change_in_activity, time_since_last (the last three are time-based features), that we added to detect time-sensitive transaction anomalies, and we showed that in Figure 2 that our TRW-GCN method distinguishes and detects such time-sensitive anomalies.
- We further introduced a scoring function for displaying nodes with an anomaly score beyond a certain threshold, section 3.3, using a z-score threshold of 2.0 (95% confidence). This is part of the methodology to identify anomalous nodes in the Ethereum network dataset (Page 7-8).
- We made a comparison between 4 features (without time-sensitive features) and 7 features (with time-sensitive features) to assure that with TRW-GCN method, we could detect more anomalies.

Reviewer comment (Reproducibility):
How to determine the threshold in practice?

Author response:
- We further introduced a scoring function for displaying nodes with an anomaly score beyond a certain threshold, section 3.3, using a z-score threshold of 2.0 (95% confidence). This is part of the methodology to identify anomalous nodes in the Ethereum network dataset (Page 7-8).

Reviewer comment (Evaluation):
Is the experimental result validated against any ground truth, and if there is ground truth, how is it obtained?

Author response:
- There is a lack of readily available ground truth for anomalies in Ethereum transactions; as we started on page 4, "In our work, the term 'anomaly' refers to patterns that are statistically uncommon or divergent from the norm based on the features learned by our model. These uncommon patterns, while not definitively erroneous, are of interest because they deviate from typical behavior. In the context of Ethereum transactions, such deviations could potentially indicate suspicious activities, novel transaction patterns, or transaction bursts."
- Ethereum, being a blockchain platform, records all transactions in a public ledger. These transactions include not just simple transfers of Ether. The variety of transaction types and the complexity of smart contract interactions make it difficult to establish a clear baseline of "normal" behavior.
- What constitutes an anomaly in Ethereum transactions can be subjective and context-dependent. For instance, a transaction pattern that seems unusual in one context might be perfectly normal in another.
- The Ethereum blockchain is continuously evolving, with new smart contracts and transaction types emerging regularly. This dynamic nature means that what is considered normal can change over time, further complicating the establishment of a static ground truth.

Reviewer comment (Experiments):
Are there any existing studies that could be compared as benchmarks? If there are no proper related studies, please explain why.

Author response:
- We evaluate the TRW-GCN methodology using four distinct machine learning techniques: DBSCAN, SVM, Isolation Forest (IsoForest), and Local Outlier Factor (LOF) (Page 7). This comparative analysis, especially in the context of different block sizes in Ethereum, demonstrates the effectiveness of the TRW-GCN approach over traditional methods. This also underscores the importance of algorithm selection, indicating a higher false positive rate with clustering methods.
- We also discuss the convergence of GCNs and TRW, emphasizing our approach in refining anomaly detection methods. This includes comparing different clustering and scoring methods to decrease false positives in detecting anomalous activities.

Reviewer comment (Novelty):
My main concern is that the contribution may be not enough for this conference. Although the application is novel, the idea of using temporal information to build graphs (and weighted adjacency matrix) is not new in pattern recognition and machine learning areas. In addition, the proposed method seems straightforward.

Author response:
We thank the reviewer for the in-depth and insightful comments, at the same time, we acknowledge that while the use of temporal information in graph construction is not new, its application in the context of Ethereum transaction analysis is novel. The integration of Temporal Random Walk (TRW) with Graph Convolutional Networks (GCNs) is presented as a significant contribution, particularly in detecting anomalies in blockchain transactions. This approach is tailored to the unique characteristics of Ethereum transactions, emphasizing the temporal aspects that are crucial for understanding and detecting anomalies (Page 4-5).

1. Paper Theory Contributions
- We aimed to prove three theorems in this paper (Section 2.3):

     - "Enhanced Anomaly Detection in GCNs through Temporal Random Walk Integration" (see page 4, 5).

     - "TRW sampling maintains higher temporal consistency than traditional random walk sampling" – added to section 2.3.

     - "Improvement of GCN performance with probabilistic sampling in the context of random walk sampling" (see appendix).

Reviewer comment (Experiments):
Lack of ablation experiment: While the paper presents the theoretical benefits of the TRW-GCN framework, there is a need for empirical evaluation to demonstrate its tangible benefits. Comparing the performance of GCN with and without TRW on a temporal dataset would provide more evidence of the model's effectiveness.

Author response:
3. Comparison and Validation
- We evaluate the TRW-GCN methodology using four distinct machine learning techniques: DBSCAN, SVM, Isolation Forest (IsoForest), and Local Outlier Factor (LOF) (Page 7). This comparative analysis, especially in the context of different block sizes in Ethereum, demonstrates the effectiveness of the TRW-GCN approach over traditional methods. This also underscores the importance of algorithm selection, indicating a higher false positive rate with clustering methods.
- We also discuss the convergence of GCNs and TRW, emphasizing our approach in refining anomaly detection methods. This includes comparing different clustering and scoring methods to decrease false positives in detecting anomalous activities.
- We used 7 features for anomaly detection including outgoing_tx_count, incoming_tx_count, outgoing_value_count, incoming_value_count, activity_rate, change_in_activity, time_since_last (the last three are time-based features), that we added to detect time-sensitive transaction anomalies, and we showed that in Figure 2 that our TRW-GCN method distinguishes and detects such time-sensitive anomalies.
- We further introduced a scoring function for displaying nodes with an anomaly score beyond a certain threshold, section 3.3, using a z-score threshold of 2.0 (95% confidence). This is part of the methodology to identify anomalous nodes in the Ethereum network dataset (Page 7-8). 
- We made a comparison between 4 features (without time-sensitive features) and 7 features (with time-sensitive features) to assure that with TRW-GCN method, we could detect more anomalies.

Reviewer comment (Presentation):
Lack of intuitive visualization of experimental results: The paper presents experimental results to demonstrate the superiority of the TRW-GCN framework, but the visualizations and figures provided are not clear and do not effectively convey the findings. The authors should consider improving the clarity and quality of the figures to make the experimental results more intuitive and easier to interpret.

Author response:
4. Clarity of Visualizations
- The paper includes figures to demonstrate the effectiveness of the TRW-GCN framework in anomaly detection. However, the reviewer's concern about the clarity and interpretability of these visualizations is noted. The paper could benefit from improving the quality and clarity of these figures to make the experimental results more intuitive and easier to understand. 
- For instance, we made Figure 1 and 2 bigger and more clear. 
- We numbered all the equations. 
- Figure 3 compares anomaly detection in different block sizes with varying features, in Figure 2, we explained the importance of time-sensitive feature selection with different colours. In Figure 4, we illustrated the improvement on GCN performance using random walk sampling.

Reviewer comment (Novelty):
Gaps in literature should be mentioned.

Author response:
We have stated the literature gaps on page 2.
See "Potential Contributions" on page 2, where we introduced the effectiveness of TRW in detecting anomalies, and suggested the importance of temporal patterns in identifying irregularities.

Reviewer comment (Presentation):
Equations should be numbered.

Author response:
All equations were numbered, and the font size of Figure 1 and 2 were increased.

Reviewer comment (Presentation):
Figure 1 and 2 text is too small, impossible to read.

Author response:
All equations were numbered, and the font size of Figure 1 and 2 were increased.

Reviewer comment (Theory):
The proofs are too short and it was tough to gauge if they were sound or not as presented.

Author response:
The proofs provided in the paper are concise, focusing on the fundamental aspects of the methodologies used. We added proof of theorem 2 in section 2.3. The paper also includes proof of theorem 3 in the appendix, demonstrating analytically how probabilistic sampling can improve GCN performance. This brevity is due to the focus on empirical analysis and practical application.
We aimed to prove three theorems in this paper (Section 2.3):

        - "Enhanced Anomaly Detection in GCNs through Temporal Random Walk Integration" (see page 4, 5).

        - "TRW sampling maintains higher temporal consistency than traditional random walk sampling" - added to section 2.3.

         - "Improvement of GCN performance with probabilistic sampling in the context of random walk sampling" (see appendix page 12-14).
Section 3.2 mainly concerns the empirical analysis for the 1st theorem.
Section 3.4 mainly refers to the empirical analysis done for the 3rd theorem.

Reviewer comment (Presentation):
No tabular data was presented.

Author response:
We evaluate the TRW-GCN methodology using four distinct machine learning techniques: DBSCAN, SVM, Isolation Forest (IsoForest), and Local Outlier Factor (LOF) (Page 6-7). This underscores the importance of algorithm selection, indicating a higher false positive rate with clustering methods.
We also discuss the convergence of GCNs and TRW, emphasizing our approach in refining anomaly detection methods. This includes comparing different clustering and scoring methods to decrease false positives in detecting anomalous activities.
We used 7 features for anomaly detection including outgoing_tx_count, incoming_tx_count, outgoing_value_count, incoming_value_count, activity_rate, change_in_activity, time_since_last (the last three are time-based features), that we added to detect time-sensitive transaction anomalies, and we showed that in Figure 2 that our TRW-GCN method distinguishes and detects such time-sensitive anomalies.
We further introduced a scoring function for displaying nodes with an anomaly score beyond a certain threshold, section 3.3, using a z-score threshold of 2.0 (95%). This is part of the methodology to identify anomalous nodes in the Ethereum network dataset (Page 8). We made a comparison between 4 features (without time-sensitive features) and 7 features (with time-sensitive features) to assure that with TRW-GCN method, we could detect more anomalies.
We also compared the accuracy of three GCN models on full graphs and subgraphs, highlighting the effectiveness of traditional random walk and temporal random walk in capturing essential graph properties (Page 9).
Although the paper extensively uses Figures for data representations. In Figure 3, we showed the list of addresses for detected anomalies, however lack of further tabular data is due to the nature of the data and the methods used, which would be more effectively represented visually. Including tables could enhance the clarity of results and comparisons, making it easier for readers to understand. We are still bound by the 9 pages though.

Reviewer comment (Novelty):
What is the overall novelty compared to current state of the art?

Author response:
The novelty of this research lies in its integration of TRW with GCN for anomaly detection in the Ethereum network. This approach is particularly innovative in its focus on the temporal sequences and correlations of transactions, a gap not extensively explored in existing literature. We suggest that this methodology offers a more refined lens for anomaly detection, capturing both spatial and temporal dynamics in Ethereum transaction data.
On page 2, we stated the literature gap and our potential contributions. This represents a significant advancement in the field.

Reviewer comment (Presentation):
The paper feels incomplete and not ready for review. There are multiple sections of the paper that are incomplete, for example: "Proof" in Section 2.3. What theorem to prove, and where is the proof? "caption." in Section 3.

Author response:
\section*{1. Incomplete Sections and Hypothetical Theorems:}
\begin{itemize}
    \item We aimed to prove three theorems in this paper (Section 2.3):
    \begin{enumerate}
        \item ``Enhanced Anomaly Detection in GCNs through Temporal Random Walk Integration'' (see page 4, 5).
        \item ``TRW sampling maintains higher temporal consistency than traditional random walk sampling'' - added to Section 2.3.
        \item ``Improvement of GCN performance with probabilistic sampling in the context of random walk sampling'' (see appendix page 11-14).
    \end{enumerate}
    \item Section 3.4 referred to the empirical analysis done for our 3rd theorem for which the proof was presented in the appendix (page 11-14). All the subsections related to "Theorem (Hypotehetical)" in Section 3.4 have been removed.
    \item by ``Potential Conclusions'' we meant ``Potential Contributions''in the context of Ethereum network and effectiveness of TRW in detecting anomalies (Page 2).
\end{itemize}

Reviewer comment (Theory):
The whole paper feels like a conjecture rather than proven or discovered findings. For example, the use of "Theorem (Hypotehetical)" in Section 3.4. What does that mean? Are they proven theorems or just hypotheses? Another example is the use of "Potential Conclusions" in Section 1.

Author response:
- In regard to your statement "the authors use the term 'hypothetical' in the theorem, which does not indicate a rigorous theorem. The current revised version expands the proof a bit and removes the word "hypothetical". However, the proofs are still not mathematically rigorous.", indeed there were some theorems written hypothetical in section 3.4, in the revised manuscript, we did not just remove the word hypothetical, but removed the theorems completely from section 3.4, instead we added a new theorem "TRW sampling maintains higher temporal consistency than traditional random walk sampling." in section 3.2 in order to make our mathematical section more rigorous. Due to lack of space, we pushed most of our proofs to the appendix.

- Rigor of Theorems: We understand your concern about the mathematical rigor of our theorems. The statement you referred to, "If the anomaly is temporally influenced, then h′_n should be significantly different from h_n due to the weights introduced by T_nj".  This statement is based on the underlying assumption that temporal influences in the network have a non-negligible effect on the node representations. The weights in T_nj are designed to capture these temporal dynamics, and our empirical observations support this assumption. We acknowledge that a more formal mathematical treatment could strengthen our argument, and we indeed added appendix A where We explore the nuances of how such weight cancellations can diminish the efficacy of anomaly detection and propose strategies to mitigate these issues.

Reviewer comment (Experiments):
The authors do not use proper baselines in the experiments. There are multiple GNN-based anomaly detection algorithms that have been proposed in the literature. None of them appeared in the experiment.

Author response:
\section*{2. Lack of Proper Baselines in Experiments:}
\begin{itemize}
    \item There is a lack of readily available ground truth for anomalies in Ethereum transactions; as we started on page 4, ``In our work, the term `anomaly' refers to patterns that are statistically uncommon or divergent from the norm based on the features learned by our model. These uncommon patterns, while not definitively erroneous, are of interest because they deviate from typical behavior. In the context of Ethereum transactions, such deviations could potentially indicate suspicious activities, novel transaction patterns, or transaction bursts.''
    \item Ethereum, being a blockchain platform, records all transactions in a public ledger. These transactions include not just simple transfers of Ether. The variety of transaction types and the complexity of smart contract interactions make it difficult to establish a clear baseline of ``normal'' behavior.
    \item What constitutes an anomaly in Ethereum transactions can be subjective and context-dependent. For instance, a transaction pattern that seems unusual in one context might be perfectly normal in another.
    \item The Ethereum blockchain is continuously evolving, with new smart contracts and transaction types emerging regularly. This dynamic nature means that what is considered normal can change over time, further complicating the establishment of a static ground truth.
\end{itemize}

- Baseline Selection: We respectfully disagree with the assertion that our baseline selection is insufficient. Our study includes a range of machine learning techniques, such as DBSCAN, SVM, Isolation Forest, and Local Outlier Factor, to provide a comprehensive evaluation of our model. While we acknowledge the existence of GNN-based models that incorporate temporal information, our focus was on demonstrating the efficacy of our Temporal Random Walk (TRW) integrated with Graph Convolutional Networks (GCNs) in the specific context of Ethereum transactions. Our selection of baselines was guided by the objective to compare our approach with a diverse set of well-established methods in anomaly detection.