Paper: Todyformer: Towards Holistic Dynamic Graph Transformers with Structure-Aware Tokenization (ICLR-2024)

Reviewer comment (Theory):
To the best of my knowledge, there is no study pointing out existing well-known temporal graph neural networks (TGNN) have the problem of over-smoothing. I also do not see any theoretical or experimental evidence on the over-smoothing problem of TGNN. Therefore, the motivation of this paper may be not solid.

Author response:
In the realm of continuous-time dynamic graphs, the structure resembles that of regular graphs, but with the added dimension of temporal edge features. Notably, over-smoothing becomes a pronounced issue in scenarios involving temporal edges, where any pair of nodes can engage in multiple interactions over time. It is a nontrivial task to assert that the challenges associated with over-smoothing in non-temporal graphs persist in temporal graphs. The primary focus of this work, however, is not to delve into a theoretical or empirical investigation of over-smoothing in temporal graphs. Instead, the aim is to propose strategies for enhancing model performance in the context of temporal graphs. Our empirical findings lend support to the hypothesis that this improvement is closely tied to mitigating the effects of over-smoothing and over-squashing phenomena.

Reviewer comment (Experiments):
The inputs of TodyFormer are edges within a temporal window. How to set the size of this window? Does it mean that the edges in the previous time will see the edges in the latter time in the window (information leakage)? How do you prevent the information leakage problem?

Author response:
There is no concern regarding information leakage in the link prediction task. In this context, for a target edge at a specific time, all preceding edges until that time are considered as the historical context for the target (anchor) edge. As a result, there is no inadvertent data leakage. The model outputs node embeddings for the source and destination nodes in the target edges based on their historical context. The historical context can be constrained to a recent time window, which determines the size of the input graph received by the model. This input graph is then partitioned into equally-sized subgraphs to form patches. Subsequently, each patch is locally encoded by a message-passing Graph Neural Network (GNN) tailored for dynamic graphs. These subgraphs are then globally encoded at the node level using Transformers, as detailed in the paper.

Reviewer comment (Experiments):
More sensitivity analysis on other hyper-parameters should be conducted.

Author response:
Additional sensitivity analyses on positional encoding, as well as the number of layers and blocks, have been incorporated into the supplementary materials in Appendix A.5 and A.3.3.

Reviewer comment (Experiments):
There is no training/testing time comparison with other baselines.

Author response:
The qualitative and quantitative comparison of time complexity has been appended to Appendix A4. In A.4.1, an in-depth exploration of the model's time complexity concerning the number of nodes, edges, and patches is presented. In A.4.2, an analysis of Figure 4 is provided, depicting the performance versus inference time across three sizable datasets. Considering the delicate trade-off between performance and complexity, our models surpass all others in terms of Average Precision (AP) while concurrently positioning in the left segment of the diagrams, denoting the lowest inference time. Notably, as depicted in Figure 4, Todyformer remains lighter and less complex than state-of-the-art (SOTA) models like CAW across all datasets.

Reviewer comment (Experiments):
The authors claim that Transformers demonstrate exceptional computational capacity, compared with Graph Neural Networks. However, the computational complexity of GNNs and traditional Transformers is $O(N)$ [1] and $O(N^2)$ [2] respectively, where $N$ is the number of nodes in the input graph. I suggest reporting the complexity and runtime in experiments.

Author response:
Thank you for your appreciative remarks and constructive feedback. It is noteworthy to highlight that our transformer architecture diverges from the conventional method of calculating attention maps, which involves attention across all pairs of nodes, resulting in a computational complexity of O(N^2). Instead, our approach involves attention exclusively across embeddings of the same nodes generated in different patches. Consequently, the total pairs of nodes for which attention is calculated is at most P squared, where P denotes the number of patches. This innovative approach is achieved through the unique combination of patchifier, local dynamic graph encoder, and Transformer, facilitating the pooling of embeddings for the same node.
A more comprehensive analysis of computational complexity has been appended to Appendix A.4.1, providing a detailed calculation of the computational complexity of Todyformer based on the number of patches, edges, and nodes.
It is also worth mentioning we have not stated the todyformer has computational capacity over  Graph Neural Networks (GNNs). In the Appendix A.4.2, we conducted a quantitative comparison of inference time versus performance for our model against baseline models. The results reveal that the new inference time is slightly higher than TGN but significantly lower than CAW, while our model's performance consistently outperforms all the compared models.

Reviewer comment (Experiments):
Todyformer is similar to Vision Transformer, while the authors do not provide the necessary analysis in terms of graph learning.

Author response:
Our approach shares commonalities with Vision Transformer (ViT) up to the stage where both architectures follow a sequence of patchifying, tokenization, and Transformer utilization. However, distinctions arise in the patchifying process; we generate patches across the temporal domain, whereas ViT does so across the spatial domain. Moreover, tokenization in our model is achieved through local message-passing, in contrast to ViT's method of linear projection applied to flattened patches.

Reviewer comment (Theory):
The authors may want to analyze the expressiveness of Todyformer in terms of sub-structure identification and the Weisfeiler-Leman (WL) test.

Author response:
We have conducted a comprehensive analysis and established certain levels of expressiveness through the lens of Weisfeiler-Lehman (WL) tests. The detailed proof and additional findings will be presented in the extended version of this work and in our forthcoming research. Our intention is to draw comparisons with the PINT model \cite{}, which has been theoretically established as expressive. This comparative analysis aims to contribute to a deeper understanding of the expressive capabilities of our model in relation to existing theoretical frameworks.

Reviewer comment (Evaluation):
Please explain why the baseline results in Table 2 are different from those from Table 3 in [3].

Author response:
They have recently revised their pre-print from version 1 to version 2, as indicated by the ArXiv versioning system. In Table 2 and Table 10 , we have accordingly adjusted the main baseline values by incorporating the newly reported values from their revised paper. Notably, even with the updated baseline values, the performance of Todyformer remains superior to that of the other baselines.

Reviewer comment (Experiments):
The authors claim that the significantly low performance of Todyformer on the Wiki and Review datasets is due to the insufficient hyperparameter search. However, in my opinion, hyperparameter tuning is difficult to improve 3% accuracy.

Author response:
The observed improvement in validation performance for two small datasets following the table update underscores the sensitivity of the final performance to random seeds. This reiterates the importance of comparing average performance across multiple trials to provide a more robust and representative assessment of the model's capabilities. Evaluating the mean performance helps mitigate the impact of random variations and provides a more reliable estimate of the model's overall effectiveness.  For TGBWiki and TGBReview, Todyformer achieves the second and third-best positions, respectively, on the test set. The observed performance suggests the possibility of overfitting on these smaller datasets. To address this, further exploration through hyperparameter search is recommended. Conducting an extensive search for optimal hyperparameters can help mitigate overfitting issues and enhance the model's generalization across diverse datasets.

Reviewer comment (Evaluation):
Please report the standard deviation in Table 2 following the baselines. Moreover, I suggest reporting a statistically significant difference.

Author response:
We have updated our results by incorporating the standard deviation of the multi-trial outcomes into both Table 2 and Table 10. These values are based on 5 trials, utilizing the same negative samples provided by the leaderboard. The updated results have been seamlessly integrated into the paper and are thoroughly discussed to provide a more comprehensive understanding of the experimental outcomes. The protocols employed in the leaderboard adhere to the utilization of identical seeds and negative samples across methods. In the ranking process, the mean and standard deviation are pivotal metrics. Notably, our method demonstrates a significant distinction, particularly showcasing the lowest standard deviation for the three big datasets of TGBL. This highlights the robustness and consistency of our approach across multiple trials and reinforces its favorable performance characteristics.

Reviewer comment (Writing):
Some presentations need improvement. Some notations need to be clear and some formulas need to be written for clarity. For example, what is the notation a,b means in E = |{e_a,...,e_b}| in section 3.2? What is the specific formula for positional encoding o() in section 3.4? For the transformer formula (6), could you specify whether there are LayerNorm and Feed-Forward modules as transformer? Could you give the formula of Q, K, V, and their dimension for clarity?

Author response:
Acknowledged in full. We have revised the formulations within the main body of the text, denoting the modifications with the color blue for clarity and easy tracking. Additionally, the notation pertaining to edges in section 3.2 has been rectified. To enhance transparency, the positional encoding formulation has been added to the appendix and referenced in the methodology section (3.4). Detailed configurations of the transformer are now shown in the paper, revealing the presence of two transformer layers within each block, incorporating layer normalization and feed-forward modules. Notably, the dropout rate and number of heads for the transformer component are specified as 0.1 and 2 respectively. The dimensions of Q, K, and V for the transformer have been incorporated, accompanied by an explanation of how the remainder of the architecture is fed into the transformer component. To enhance clarity, we have included the formulas for Q, K, and V, along with their corresponding dimensions.

Reviewer comment (Evaluation):
I wonder about the results of Wikipedia and Reddit you mentioned in Section 4.1 datasets as it’s not shown in Table 1. As for the results shown in the Appendix, it seems they are not strong enough in the Inductive setting, especially for the Reddit dataset, which makes the statement in section 4.2 “Todyformer consistently outperforms the baselines’ results on all datasets.” misleading.

Author response:
we appreciate your acknowledgment of the nuanced aspect requiring additional clarification in the paper. In the experimental setup, we have explicitly differentiated the names of the datasets employed in the first round of experiments from Reddit and Wikipedia. Furthermore, we have addressed and rectified potentially misleading terms in section 4.2, providing clarification and utilizing the color blue for emphasis and easy identification. Moreover, the discourse on the model's performance across the extended set of datasets has been incorporated into Appendix A.3.1.

Reviewer comment (Writing):
Section 3.5 “This issue is magnified in dynamic graphs when temporal long-range dependencies intersect with the structural patterns.” Could you please give some analysis of it or give some examples or literature to show the importance of over-smoothing problems in dynamic graphs? This can make this paragraph more convincing.

Author response:
The problem definition for over-smoothing have been added in the paper. we have also added a section (A.5) to provide the results of experiments for over-smoothing and over-squashing.

Reviewer comment (Experiments):
Could you please show your efficiency comparison against other methods, especially CAW and Dyg2Vec? In my opinion, the computation of graph transformer on each node could have high time complexity, could you please analyze it?

Author response:
The qualitative and quantitative comparison of time complexity has been appended to Appendix A4. In A.4.1, an in-depth exploration of the model's time complexity concerning the number of nodes, edges, and patches is presented. In A.4.2, an analysis of Figure 4 is provided, depicting the performance versus inference time across three sizable datasets. Considering the delicate trade-off between performance and complexity, our models surpass all others in terms of Average Precision (AP) while concurrently positioning in the left segment of the diagrams, denoting the lowest inference time. Notably, as depicted in Figure 4, Todyformer remains lighter and less complex than state-of-the-art (SOTA) models like CAW across all datasets.

Reviewer comment (Evaluation):
Could you give more analysis for the experiments weaker than the baseline? For example, in the second set of experiments, why did this method fail in these two smaller datasets?

Author response:
We have updated our results by incorporating the standard deviation of the multi-trial outcomes into both Table 2 and Table 10. These values are based on 5 trials, utilizing the same negative samples provided by the leaderboard. The observed improvement in validation performance for two small datasets following the table update underscores the sensitivity of the final performance to random seeds. This reiterates the importance of comparing average performance across multiple trials to provide a more robust and representative assessment of the model's capabilities. Evaluating the mean performance helps mitigate the impact of random variations and provides a more reliable estimate of the model's overall effectiveness.  For TGBWiki and TGBReview, Todyformer achieves the second and third-best positions, respectively, on the test set. The observed performance suggests the possibility of overfitting on these smaller datasets. To address this, further exploration through hyperparameter search is recommended. Conducting an extensive search for optimal hyperparameters can help mitigate overfitting issues and enhance the model's generalization across diverse datasets.

Reviewer comment (Theory):
This paper does not provide the mathematical form of the overall loss function, which leads to an incomplete explanation of the model in Section 3.

Author response:
To elucidate the precise loss functions employed in our study for various downstream tasks, we have incorporated the mathematical formulations of these losses into the appendix of the main manuscript. Furthermore, to provide a comprehensive understanding of our approach, we have included references to prior works that have utilized similar loss functions. This ensures transparency in our methodology and facilitates a contextual understanding of the chosen loss functions in the broader scientific discourse.

Reviewer comment (Novelty):
In the detail of three main components, many ideas are not novel. For instance, in encoding Module, the Transformers is very basic model. The window-based encoding paradigm is from DyG2Vec. The positional-encoding idea is also from previous work.

Author response:
In response to the concern regarding the perceived lack of novelty in our work, we have explicitly emphasized the innovative aspects of our research both within the revised manuscript and in this rejoinder. While it is acknowledged that the individual components we employ share nomenclature with existing literature in fields such as Computer Vision and Natural Language Processing, our study marks the inaugural amalgamation of these components. This unique integration is specifically tailored to enhance the expressive capabilities of dynamic graph encoders. Each constituent has been purposefully incorporated to address the inherent limitations of dynamic graph encoders, thus contributing novel insights to the field.

Reviewer comment (Reproducibility):
Could you add mathematical form of the overall loss function?

Author response:
Noted, the mathematical formulations have been added to the appendix and it's referenced in the main text.