Paper: EX-Graph: A Pioneering Dataset Bridging Ethereum and X (ICLR-2024)

Reviewer comment (Reproducibility):
The paper is not self-contained and a lot of things need to be assessed through the external links to the dataset or through the appendix. I am not very familiar with dataset papers, so maybe this is generally so.

Author response:
Thank you for your feedback regarding the self-contained nature of our paper. We understand your concern about the necessity to refer to external links and appendices for complete details. It is indeed common in dataset papers, especially those involving complex and extensive datasets like ours, to supplement the main text with additional resources. For example, [Huang et al., 2023] propose a Temporal Graph Benchmark for Machine Learning on Temporal Graphs, moving all experiment settings to the appendix and use an external website to contain many details. This approach helps maintain the clarity and conciseness of the paper while ensuring that detailed and technical aspects are readily accessible to those who seek more in-depth information. 
   
   Also, we acknowledge your concerns and have refined our paper by incorporating some key technical details, previously in the appendix into the main text. Notably, we have added an explanation for choosing BERT, now highlighted in blue in Section 3.3 of the revised paper.

Reviewer comment (Evaluation):
Q2 in the evaluation seems to make no sense to me: since there are only 3 matched addresses in the dataset, why should we anticipate improved performance with this dataset? The results are, thus, questionable to me regarding Q2.

Author response:
Thank you for your observations regarding Q2 in our evaluation. We understand your concern about the seemingly limited number of matched addresses in the dataset and its impact on performance improvement. To clarify, our experiments extended beyond merely analyzing matched Ethereum addresses. In response to the reviewer AoG7's Point 3, we discovered that incorporating Twitter data enhances the link prediction for non-matching Ethereum addresses as well. This improvement is largely attributed to the message-passing mechanism inherent in Graph Neural Networks (GNN). Through this mechanism, semantic information from the few matched addresses can disseminate throughout the network, particularly influencing nodes directly connected to these matched addresses. Therefore, despite the small number of direct matches, the indirect influence of these connections via semantic information propagation would significantly contribute to the overall performance enhancement observed in our results.

Reviewer comment (Novelty):
The insight that Twitter-matched addresses are more active in Ethereum is rather expected since Twitter is known to be popular (as the authors also confirm) by main actors of the Ethereum ecosystem.

Author response:
We appreciate your observation regarding the increased activity of Twitter-matched addresses in the Ethereum ecosystem. Indeed, while this may appear intuitive given Twitter's popularity among key Ethereum participants, we would like to highlight the absence of empirical studies that have previously explored and validated this correlation. Our research represents the first systematic effort to analyze and substantiate this hypothesis with concrete evidence. This undertaking not only confirms a widely held perception but also significantly enriches our understanding of the interplay between social media behavior and blockchain activity.
    
    Furthermore, a principal contribution of our work is we propose a pioneering dataset that matches Twitter and Ethereum entities. This resource not only facilitates the verification of prevalent insights but also opens avenues for future explorations in the domain of blockchain and social media analytics.

Reviewer comment (Experiments):
There are some technical things that I discuss below, but these seem to be more minor in nature. As an example, the set of features degree (3 features) and neighbor (3 features) seem to be highly linearly correlated (as also suggested by their statistics in Table 3). Keeping only one of them, i.e., one from the "degree" family, and one from the "neighbor" family, would potentially improve the performance of the networks. I provide a couple of more minor comments below.

Author response:
We greatly appreciate your technical observations and suggestions. Regarding the features of 'degree' and 'neighbor' that you mentioned, we understand your point about their potential linear correlation as suggested in Table 3. We would like to clarify that while these features are related, they represent distinct aspects of our dataset. Neighbors and degrees provide different perspectives: the number of neighbors highlights direct connections, whereas degrees can encapsulate the intensity of these connections, especially when multiple edges exist between two neighbors. To address your suggestion, we have conduct additional Ethereum link prediction to evaluate the impact of retaining only one feature from each of these categories – one from the 'degree' family and one from the 'neighbor' family. The results are as below.
    
    
    
    **Ethereum Address Link Prediction Without Twitter Features: Original vs. Simplified Feature Sets (Original: Complete 8-dimension Structural Features; Simplified: Selected One Feature Each from 'Degree' and 'Neighbor' Families)**

    | **Method**     | **Feature Set**           | **AUC-ROC** | **Precision** | **Recall** | **F1 Score** |
    |----------------|---------------------------|-------------|---------------|------------|--------------|
    | **Cluster-GCN** | Original                  | 0.82±0.01   | 0.78±0.01     | 0.70±0.01  | 0.74±0.01    |
    |                | Simplified                | 0.79±0.00(↓)| 0.70±0.02(↓)  | 0.78±0.02(↑)| 0.73±0.00(↓)|
    | **GraphSage**   | Original                  | 0.84±0.00   | 0.77±0.01     | 0.77±0.01  | 0.78±0.00    |
    |                | Simplified                | 0.84±0.00(-)| 0.78±0.01(↑)  | 0.75±0.01(↓)| 0.76±0.01(↓)|
    | **APPNP**       | Original                  | 0.82±0.01   | 0.80±0.03     | 0.68±0.02  | 0.74±0.01    |
    |                | Simplified                | 0.80±0.01(↓)| 0.74±0.02(↓)  | 0.73±0.01(↑)| 0.74±0.00(↑)|
    | **TAGCN**       | Original                  | 0.81±0.01   | 0.82±0.01     | 0.64±0.02  | 0.72±0.01    |
    |                | Simplified                | 0.79±0.01(↓)| 0.80±0.03(↓)  | 0.60±0.02(↓)| 0.73±0.00(↑)|


    


    **Ethereum Address Link Prediction With Twitter Features: Original vs. Simplified Feature Sets (Original: Complete 8-dimension Structural Features; Simplified: Selected One Feature Each from 'Degree' and 'Neighbor' Families)**

    | **Method**     | **Feature Set**           | **AUC-ROC** | **Precision** | **Recall** | **F1 Score** |
    |----------------|---------------------------|-------------|---------------|------------|--------------|
    | **Cluster-GCN** | Original                  | 0.86±0.00   | 0.81±0.00     | 0.75±0.01  | 0.78±0.00    |
    |                | Simplified                | 0.82±0.02(↓)| 0.78±0.02(↓)  | 0.67±0.01(↓)| 0.73±0.01(↓)|
    | **GraphSage**   | Original                  | 0.86±0.00   | 0.81±0.01     | 0.79±0.01  | 0.80±0.00    |
    |                | Simplified                | 0.85±0.00(↓)| 0.80±0.01(↓)  | 0.76±0.02(↓)| 0.78±0.01(↓)|
    | **APPNP**       | Original                  | 0.89±0.02   | 0.83±0.00     | 0.78±0.00  | 0.81±0.00    |
    |                | Simplified                | 0.88±0.01(↓)| 0.81±0.01(↓)  | 0.81±0.01(↓)| 0.81±0.00(-)|
    | **TAGCN**       | Original                  | 0.88±0.00   | 0.85±0.01     | 0.73±0.01  | 0.78±0.00    |
    |                | Simplified                | 0.85±0.01(↓)| 0.84±0.01(↓)  | 0.66±0.01(↓)| 0.74±0.00(↓)|

     
    
    
    The results from our experiments provided insightful findings. When we limited the features to just one from the degree family and one from the neighbor family, we generally observed a decrease in prediction performance across the two experimental settings, particularly in the predictions involving Twitter features. This trend highlights the significance of utilizing a comprehensive set of features in our model to effectively capture the intricate details and nuances of our dataset. Your suggestion played a crucial role in leading us to this valuable understanding. We deeply appreciate this opportunity to explore these aspects further, as it has been instrumental in refining and strengthening our approach and methodology.

Reviewer comment (Experiments):
Below Figure 2: "The total collection accounts for 30,387 Twitter profiles". I was confused here: why is this lower than the initial number (since, according to the previous lines more data was added) and why are Followers and Friends useful here? Friends are not discussed elsewhere if I am not wrong.

Author response:
Thank you for highlighting the discrepancy regarding the total number of Twitter profiles mentioned below Figure 2. The figure of 30,387 represents the active Twitter accounts at the time of constructing our graph. This number is lower than the initial count as it accounts for only those profiles that were accessible and active during our data collection process. We have updated this figure to 30,387 in blue color in our revised paper's section 3.1. We appreciate your attention to this detail.

Reviewer comment (Writing):
Should it be Twitter or X :)?

Author response:
We are thankful for the chance to clarify the importance of incorporating 'Followers' and 'Friends' data in our study. These metrics are crucial for evaluating the influence and network span of Twitter accounts. In our research, 'Friends' specifically denotes the accounts followed by a Twitter user. We have meticulously gathered data on the followers and followed accounts of Twitter users linked to Ethereum addresses, playing a vital role in forming a comprehensive and directed Twitter graph. Additionally, the quantity of 'Friends' a Twitter account maintains is indicative of its engagement level, thereby enhancing our analysis with valuable insights into user engagement and interaction trends within the Twitter network. In our updated paper, we have provided detailed explanations about 'Followers' and 'Friends' in Section 3.1, highlighted in blue for easy reference. Thank you once more for this perceptive inquiry.

Reviewer comment (Writing):
Is it 19 billion or 190 billion as of September 23, the market value for Ethereum (page 1)?

Author response:
In response to your query about the correct term - Twitter or X - the data we collected was valid until August 2022, and it referred to Twitter at that time. We appreciate your attention to this detail and confirm that the term 'Twitter' is used accurately in our context.

Reviewer comment (Experiments):
Though the title says ‘bridging Ethereum and Twitter’, the proposed dataset is focused on NFT transactions and related Twitter accounts. The author should either prove that with this dataset, GNN would have the ability to find other kinds of matching links (i.e., phish-hack EOA nodes and corresponding Twitter accounts), or modify the title to be more precise.

Author response:
Thank you for your thoughtful feedback on our paper titled “Bridging Ethereum and Twitter.” We understand your concerns about the focus on NFT transactions and corresponding Twitter accounts. This title was chosen to showcase our pioneering research of the integration between Ethereum and Twitter data, considering that the NFT data was extracted Ethereum transactions. Although we recognize the benefits of a more specific title, the ICLR 2024 guidelines limit modifications to the title and abstract during the rebuttal phase. Should our paper be accepted, we will revise the title to “ETGraph: A Pioneering Dataset Bridging Ethereum NFTs and Twitter” for the camera-ready version, in line with ICLR’s policy for minor adjustments. We appreciate your valuable suggestion!

Reviewer comment (Reproducibility):
In Section 3.3, the authors claim, ‘we obtain embeddings for all Twitter accounts using the DeepWalk algorithm in the Twitter graph’. Figure 2 also shows structural features given by deep walk. However, as stated in Appendix, only the handcrafted 8 features from C.1 are used.

Author response:
We are grateful for your attention to detail in highlighting the discrepancy between Section 3.3 and the Appendix C.1 of our paper. You are correct in noting that we initially neglected to include the details regarding the use of the DeepWalk algorithm for obtaining Twitter features in the Appendix C.1. We have addressed this oversight and updated Appendix C.1 to correctly depict the use of DeepWalk, now highlighted in blue, ensuring consistency with the methodology outlined in Section 3.3 and Figure 2. Thank you for bringing this to our attention, allowing us to improve the clarity and consistency of our paper!

Reviewer comment (Experiments):
For task 1, the authors only use matched addresses. Since GNN can pass node features via message-passing mechanism, the semantic information is able to spread around. Whether does the semantic information benefit or harm the prediction of non-matched addresses?

Author response:
We greatly appreciate your insightful query regarding the use of matched addresses in Task 1 and the potential effect of semantic information on the prediction of non-matched addresses! In response, we conducted an additional experiment focusing on Ethereum link prediction among non-matched nodes using multiple GNN models, where the training, validation, and test sets were devoid of any matching link addresses. The results are attached below:
    
    **Results for non-matching Ethereum addresses' Link Prediction**  
    *(Without: without Twitter semantic information; With: with Twitter semantic information)*

    | **Method**    | **Experimental Setting**                  | **AUC-ROC** | **Precision** | **Recall** | **F1 Score** |
    |---------------|---------------------------------------|-------------|---------------|------------|--------------|
    | GCN           | Without         | 0.84±0.01   | 0.76±0.01     | 0.75±0.01  | 0.75±0.00    |
    |               | With             | 0.84±0.00(↑)| 0.76±0.00(↑)  | 0.75±0.02(↓)| 0.76±0.01(↑)|
    | GAT           | Without         | 0.78±0.02   | 0.72±0.01     | 0.68±0.03  | 0.70±0.02    |
    |               | With            | 0.85±0.01(↑)| 0.76±0.00(↑)  | 0.79±0.02(↑)| 0.77±0.01(↑)|
    | APPNP         | Without         | 0.84±0.01   | 0.80±0.00     | 0.70±0.01  | 0.74±0.00    |
    |               | With            | 0.87±0.00(↑)| 0.79±0.00(↓)  | 0.77±0.00(↑)| 0.78±0.00(↑)|
    | Cluster-GCN   | Without         | 0.88±0.00   | 0.81±0.00     | 0.78±0.01  | 0.80±0.01    |
    |               | With              | 0.90±0.00(↑)| 0.82±0.00(↑)  | 0.82±0.01(↑)| 0.82±0.00(↑)|
    | TAGCN         | Without         | 0.84±0.00   | 0.81±0.00     | 0.71±0.00  | 0.76±0.00    |
    |               | With           | 0.85±0.00(↑)| 0.80±0.00(↓)  | 0.74±0.01(↑)| 0.77±0.00(↑)|

    *Note: '↑' indicates an increase, '↓' indicates a decrease.*
    
    The results of this experiment indicate that in most cases, the semantic information indeed benefits the prediction of non-matched addresses. This outcome not only corroborates the capability of GNNs to disseminate node features through the message-passing mechanism but also underscores the utility of our dataset in enhancing Ethereum link predictions, even in the absence of direct links to Twitter. Your feedback has been instrumental in guiding this valuable extension of our research. We have added this experiment to our revised Appendix I.4, highlighting its subtitle in blue color. Thank you again for your insight!

Reviewer comment (Experiments):
For task 3 is to predict whether one Ethereum address is connected to one Twitter account. In Appendix K, the authors describe "For Twitter accounts that do not have a matched Ethereum address, we ... appending an eight-dimensional zero-vector to the existing structural features." While in Section 4.4, the negative samples are pick non-existing connections between Ethereum addresses and all their matched Twitter accounts. If the negative samples are $\{e_k=(v_i, v_j)| v_i \in V_\text{eth}, v_j\in V_{\text{twitter with profile}}, e_k\notin E_\text{match}\}$, why are there Twitter accounts with no Ethereum addresses? If the negative samples are $\{e_k=(v_i, v_j)| v_i \in V_\text{eth}, v_j\in V_{\text{twitter}}, e_k\notin E_\text{match}\}$, then does the input contain bias already?

Author response:
Thank you for your observations on Task 3 of our study. We appreciate this opportunity to clarify our methodology:
    
    **Objective of Task 3**: Our objective in Task 3 is to accurately determine if a specific Ethereum address (A) is connected to its real matching Twitter account (B). All the edges we analyze are actual matching links between Ethereum and Twitter accounts we have collected.

    **Unmatched Twitter Accounts**: We try our best to collect the graph as large as possible to collect more informative data from both Twitter and Ethereum, and collect matching link information as much as possible. Then, on both of them, we conduct GNN to calculate node representations based on our collected sufficient data. Though many Twitter accounts do not have matching link information, they are important because they contribute to the node representation calculation by utilizing the GNN to facilitate the message passing between Twitter nodes, in order to enhance the prediction of matching links.
    
    For Twitter accounts that do not have a corresponding Ethereum address, our methodology involves appending an eight-dimensional zero-vector to their respective feature sets. This procedure is vital for preserving the dimensional uniformity among all nodes within the most extensive connected Twitter graph, which is a critical factor for the optimal performance of GNN. It is important to highlight that, although these zero-vector features do not represent active attributes, they play a crucial role in upholding the structural integrity necessary for GNN analysis.

    **Negative Sample Selection**: In our study, we carefully curated negative samples comprising hypothetical, non-existent connections between Ethereum addresses and Twitter accounts. This methodology was intentionally adopted to enhance our model's ability to discern between genuine and artificial connections, thereby improving its overall predictive precision. It's important to note that these negative samples were meticulously selected from the pool of nodes involved in actual matching links within our dataset. To address concerns about potential biases, especially between matching and non-matching Twitter accounts, we consciously avoided choosing Twitter nodes that do not have corresponding matches in Ethereum for our negative samples. This decision aligns with the second scenario you outlined, ensuring that our experimental design remains balanced and free from unintended biases.
    
    **Furthermore, in response to your concerns, we have conducted a supplementary experiment to investigate whether selecting non-matching Twitter accounts introduces bias in our analysis. The experiment results are attached below.**

Reviewer comment (Novelty):
As the paper suggests a new dataset, it is acknowledged that the technical novelty is not strong enough.

Author response:
Thank you for bringing up this point. We acknowledge your concerns about the technical novelty of our work. Nonetheless, it's important to emphasize that the development of our dataset transcends simple data aggregation. Our process entailed intricate data collection from Ethereum and Twitter, followed by sophisticated integration and preprocessing tasks, such as graph construction and feature engineering. These elements collectively enhance the technical complexity of our project, setting it apart in the landscape of machine learning research.