Paper: PRES: Toward Scalable Memory-Based Dynamic Graph Neural Networks (ICLR-2024)

Reviewer comment (Experiments):
The trained graphs seem small. It is unclear how PRES performs on large-scale graphs.

Author response:
- MDGNN primarily employs a link prediction task, which makes dataset size mostly dependent on the number of events rather than vertices. The datasets and baselines used in our empirical study adhere to the standard practices within the MDGNN field (e.g., [1][2]).
- The analysis and methods presented in the paper primarily focus on the size of the temporal batch. They are independent of the size of the dataset. As a result, they are applicable to datasets of any scale.
- We have included an additional dataset, GDELT, containing approximately 17k vertices and 2 million events in the main experiment (page 8). The results from this dataset align with our predictions and expectations.

| GDELT | TGN | JODIE | APAN |
| --- | --- | --- | --- |
| without PRES | 96.8%, 50 Epoches, 1325 Second/Epoch | 95.1%,  50 Epoches, 1123 Second/Epoch | 96.7%,   50 Epoches, 1215 Second/Epoch |
| with PRES | 96.0%,   50 Epoches, , 490 Second/Epoch,  | 94.3%,  50 Epoches, 401 Second/Epoch | 96.0%,  50 Epoches, 506 Second/Epoch |

Reviewer comment (Evaluation):
No absolute execution time is reported.

Author response:
- Thank you for the suggestions. We have included absolute execution time in the revised version (Table 1 Page 8). The absolute execution time reported aligns with the value of the observed speed-up.

Reviewer comment (Evaluation):
In many cases, PRES still sacrifices some precision for performance gains.

Author response:
- We want to clarify that the slight drop in model accuracy shown in Table 1 is very small, around 1%. This trade-off between making the training process faster and maintaining high accuracy is quite beneficial, especially during the phase when developers are experimenting with different settings for their models (like trying different model types or regularization techniques). When we achieve a speedup of around 2 to 3 times, it significantly speeds up the development process, saving a lot of time and possibly money if we're using cloud services. This speed boost allows developers to try out more configurations and methods, which can help them find the best settings and make up for the tiny accuracy loss.
- It's important to note that the broader community also values training efficiency, as seen in various studies that explore methods to improve efficiency even if it means a small drop in accuracy (like the line of research on using staleness in training [3]).

Reviewer comment (Experiments):
PRES is mainly evaluated on four graph datasets (Reddit, Wiki, Mooc, and LastFM). It seems these graphs are not very large with around 1K to 10K vertices and 400K to 1.3M edges. It would be helpful to justify that these graphs are large enough or PRES’s performance is not affected by the graph size.

Author response:
- MDGNN primarily employs a link prediction task, which makes dataset size mostly dependent on the number of events rather than vertices. The datasets and baselines used in our empirical study adhere to the standard practices within the MDGNN field (e.g., [1][2]).
- The analysis and methods presented in the paper primarily focus on the size of the temporal batch. They are independent of the size of the dataset. As a result, they are applicable to datasets of any scale.
- We have included an additional dataset, GDELT, containing approximately 17k vertices and 2 million events in the main experiment (page 8). The results from this dataset align with our predictions and expectations.

| GDELT | TGN | JODIE | APAN |
| --- | --- | --- | --- |
| without PRES | 96.8%, 50 Epoches, 1325 Second/Epoch | 95.1%,  50 Epoches, 1123 Second/Epoch | 96.7%,   50 Epoches, 1215 Second/Epoch |
| with PRES | 96.0%,   50 Epoches, , 490 Second/Epoch,  | 94.3%,  50 Epoches, 401 Second/Epoch | 96.0%,  50 Epoches, 506 Second/Epoch |

Reviewer comment (Evaluation):
It would be helpful to report the absolute execution time as well rather than relative speedup only.

Author response:
- Thank you for the suggestions. We have included absolute execution time in the revised version (Table 1 Page 8). The absolute execution time reported aligns with the value of the observed speed-up.

Reviewer comment (Experiments):
It would be helpful to discuss if this method can be extended to multi-nodes with more GPUs.

Author response:
- Our method and analysis apply to the temporal batch size of MDGNN, which can be viewed as the global batch size in distributed deep neural network training. Therefore, our method can naturally be applied in a multi-node setting to enable a larger "temporal (global) batch size" (data parallelism) and better leverage the abundant computational resources available in such configurations.
- It is worth noting that in the current state, MDGNN struggles to fully harness the abundant computational resources offered in a multi-node setup, due to the limited temporal batch size. Addressing this limitation forms a central motivation behind our work, and we aspire to enhance MDGNN's efficiency in capitalizing on computational resources in the future.

Reviewer comment (Evaluation):
It seems Table 1 shows that PRES still sacrifices some precision for performance gains in many cases. Please correct me if I have any misunderstanding here.

Author response:
- We want to clarify that the slight drop in model accuracy shown in Table 1 is very small, around 1%. This trade-off between making the training process faster and maintaining high accuracy is quite beneficial, especially during the phase when developers are experimenting with different settings for their models (like trying different model types or regularization techniques). When we achieve a speedup of around 2 to 3 times, it significantly speeds up the development process, saving a lot of time and possibly money if we're using cloud services. This speed boost allows developers to try out more configurations and methods, which can help them find the best settings and make up for the tiny accuracy loss.
- It's important to note that the broader community also values training efficiency, as seen in various studies that explore methods to improve efficiency even if it means a small drop in accuracy (like the line of research on using staleness in training [3]).

Reviewer comment (Evaluation):
The tradeoff of improved speed at the cost of lower accuracy does not seem to be appealing.

Author response:
**Q1 (W1). Usefulness of improved training efficiency and the scale of dataset (tradeoff of improved speed)**

**Usefulness and Significance of Improve Training Efficiency**

- Enhancing training efficiency is particularly crucial in the development phase, where it is necessary to experiment with various hyperparameter settings (e.g., different model configurations, and different regularizations). Achieving a speedup of approximately 2x to 3x can significantly accelerate the development process, resulting in substantial savings of computation, waiting time, and potentially monetary cost when utilizing cloud services. This acceleration empowers developers to explore a broader range of configurations and methodologies, ultimately leading to the potential discovery of optimal settings that can offset the minor accuracy trade-offs (~1%).
- The importance of training efficiency is also well-recognized and of interest to the community, as evidenced by numerous studies exploring techniques to increase efficiency despite some loss in accuracy (e.g., the extensive line of research in using staleness in training [1]).

**Scale of the Tested Dataset and Report of Training Time** 

- We have reported the training time of a complete epoch for each dataset and model (Table 1 of Page 8) in the revised version.  For instance, when training TGN with default settings (50 epoches) on the LASTFM dataset can take up to 9 hours within our computational environment (NVIDIA Tesla V100). Therefore, achieving the observed speed improvements can result in substantial savings (~5.8 hours) in computation and waiting time.
- MDGNN employs a link prediction task for training, making the size of the dataset dependent on the number of events rather than vertices. The datasets utilized in our study encompass event ranges from 10k (WIKI) to 1.3 million (LASTFM), which are reasonably large for academic purposes. Additionally, we introduced an additional dataset, GDELT, consisting of approximately 17k vertices and 2 million events in the main experiment (as shown in Table 1 on Page 8 in the revised paper). The results for this dataset align with the observations from the other datasets:

| GDELT | TGN | JODIE | APAN |
| --- | --- | --- | --- |
| without PRES | 96.8%, 50 Epoches, 1325 Second/Epoch | 95.1%,  50 Epoches, 1123 Second/Epoch | 96.7%,   50 Epoches, 1215 Second/Epoch |
| with PRES | 96.0%,   50 Epoches, , 490 Second/Epoch,  | 94.3%,  50 Epoches, 401 Second/Epoch | 96.0%,  50 Epoches, 506 Second/Epoch |

Reviewer comment (Experiments):
Comparison with prior work on increasing temporal batch size is insufficient.

Author response:
**Q3 (W2,W3). Comparison with previous baselines and how does PRES differ from [4] [5]?**

- As previously mentioned, the impact of temporal batch size represents a relatively uncharted territory within MDGNN research, forming a central motivation for our work. We are not aware of any existing works that have explicitly focused on increasing the temporal batch size in MDGNN training.
- The primary goal of our empirical study is to demonstrate that our method can expedite the training of existing MDGNN models. The baseline methods we employ are standard and representative within the field of MDGNN, as evidenced by prior research [2][3].

Thank you for pointing out the potential omission of related works. We have now included a discussion of these two papers in the extended related works section in the revised paper. Here is a brief summary of the distinctions:

- EDGE [4]: EDGE focuses on accelerating computation by being selective to the event updates while our focus is to enlarge temporal batch size. Therefore, EDGE's focus and objectives differ from ours. Furthermore, we are unable to rapidly implement EDGE for direct comparison as they did not provide the public code.
- DistTGL [5]: Our method and problem are orthogonal to the issues addressed by DistTGL. Therefore, our method can naturally be used in conjunction with DistTGL.
    - DistTGL endeavours to construct an efficient pipeline for a distributed MDGNN training system. It focuses on developing solutions to handle dependencies that emerge across different batches during distributed training. Additionally, DistTGL addresses the challenge of efficient scaling concerning the number of GPUs.
    - In contrast, our analysis is centred on the interdependencies of events within a batch and investigates the implications of temporal batch size. Our method aims to enlarge the size of the temporal batch without suffering significant performance degradation.
    - We will include the experiment that combines our method and DistTGL to explore whether the performance can be further improved.

Reviewer comment (Evaluation):
Concern about the usefulness of gaining speed at the cost of accuracy. At first, it seemed the goal was to achieve SOTA accuracy, but instead, the approach gains a speedup of around 2x to 3x at the cost of decreased accuracy (~1.0%). This is questionable given the community’s effort to improve accuracy, especially since the tested datasets have only a few thousand vertices, which would not take long to train. This concern is compounded by the lack of reported training time (only speedup is reported) and insufficient baselines. A strong justification is needed.

Author response:
First of all, thank you for the response, suggestions, and the chance to further clarify and address your concerns.

> I am little bit more, but still not fully convinced on the claim between accuracy vs speedup. This is because many GNN (and TGN) papers propose around 1% improvement in each publication. The authors mention that PipeGCN [1] utilizes staleness, but PipeGCN does not sacrifice accuracy at all.

We would like to further address the significance of improved training efficiency with the following aspects:

Motivating setting and scenario

- We agree that accuracy is of central importance to machine learning models under unlimited resources. However, different settings might have different preferences. For instance,  a setting with limited resources (time or computation) might prefer better training efficiency.
    
    One typical application of MDGNN, the mode considered in our paper,  is an E-commerce platform where MDGNN is used for real-time personalization of user experience [2]. In such a scenario, models must be trained within tight time constraints to adapt to evolving user behaviour and preferences, ensuring that recommendations remain relevant and engaging.
    

PipeGCN

- First of all, we would like to point out that it takes a large epoch number (e.g., training Reddit for 2000 epochs, a significantly higher number compared to the typical 100 to 200 epochs) for PipeGCN to reach the same performance as the baselines. In our experimental setting, we follow the default setup as TGN which uses 50 epochs. As illustrated by Fig.4 [1], PipeGCN also experiences minor performance degradation (~1%) if we look at the region where the vanilla baseline has converged when compared to the vanilla baseline in the convergence region (e.g., around 250 epochs for ogbn-products and 1000 epochs for Reddit).
    
    As our proposed method does not affect the expressive power of the underlying models,  we believe our method can also achieve similar experimental results as PipeGCN if we allow the training session to be long enough. We are working hard on showing such an experiment and will have you updated once we have the data ready.
    

> Maybe one way to make this work shine is to provide a 2D plot of speed-accuracy comparison among a series of other works to show the provided trade-off is meaningful. That is, something we could find from pruning/quantization/nas papers a few year ago (although those were usually comparing the inference time with accuracy). For example, Fig1 of Efficientnet.'

- Thank you for suggesting a 2D plot to showcase the trade-off between speed and accuracy among various works. We are diligently working on creating such a plot and will notify you as soon as we have both the data and plot ready.

> I could not find LastFM result from Appendix F of the revision. Could you provide a precise pointer?

- The figures of the LastFM dataset are included in Figure 13 Page 24 of the updated version.

We sincerely appreciate your engagement and feedback, which contribute to the improvement of our work.

Reviewer comment (Experiments):
Question regarding the consensus on the ‘optimal’ temporal batch size. In Figure 3, baselines are shown with batch sizes up to 100, where ‘small batch size’ seems to be ~50, but the ‘optimal’ size in Figure 4 appears to be ~500. If most MDGNNs already use ~500, the insight from theorem 1 loses impact. A comprehensive overview of currently used batch sizes is needed.

Author response:
**Q2. What is the consensus on the ‘optimal’ temporal batch size? and significance of Theorem 1**

- First, we would like to point out that the batch size of 500 is still small relative to the size of the dataset. Furthermore, the choice of temporal batch size in practical applications should not diminish the significance of Theorem 1. The main contribution of Theorem 1 is to provide an alternative perspective to the conventional wisdom regarding the use of large batches. The prevailing belief has been that employing a large temporal batch size is generally detrimental due to the interdependence of events within that batch. However, Theorem 1 uncovers a surprising advantage of using a large temporal batch size, which results in a smaller variance in the gradient.
- The impact of temporal batch size represents a relatively uncharted territory within MDGNN research, which constitutes a central motivation for our work. Due to the trade-off between training efficiency and sensitivity to accuracy, determining the "optimal batch size" is likely to vary for different datasets and tasks. While establishing a unified criteria for determining the optimal temporal batch size is an interesting research topic, it falls beyond the primary scope of this paper.

Reviewer comment (Experiments):
Clarification on how PRES differs from other baselines. Two related works are missing from the paper: “Efficient Dynamic Graph Representation Learning at Scale” (arXiv 2021) and “DistTGL: Distributed Memory-Based Temporal Graph Neural Network Training” (arXiv 2023). Both aim to increase temporal batch size without accuracy loss. The former uses prediction for data-parallelism, and the latter pushes batch size for distributed GPU clusters. These and other similar baselines should be compared methodologically and speedup-wise in the current setting. Additionally, it would be interesting to see if they can benefit from PRES.

Author response:
**Q3 (W2,W3). Comparison with previous baselines and how does PRES differ from [4] [5]?**

- As previously mentioned, the impact of temporal batch size represents a relatively uncharted territory within MDGNN research, forming a central motivation for our work. We are not aware of any existing works that have explicitly focused on increasing the temporal batch size in MDGNN training.
- The primary goal of our empirical study is to demonstrate that our method can expedite the training of existing MDGNN models. The baseline methods we employ are standard and representative within the field of MDGNN, as evidenced by prior research [2][3].

Thank you for pointing out the potential omission of related works. We have now included a discussion of these two papers in the extended related works section in the revised paper. Here is a brief summary of the distinctions:

- EDGE [4]: EDGE focuses on accelerating computation by being selective to the event updates while our focus is to enlarge temporal batch size. Therefore, EDGE's focus and objectives differ from ours. Furthermore, we are unable to rapidly implement EDGE for direct comparison as they did not provide the public code.
- DistTGL [5]: Our method and problem are orthogonal to the issues addressed by DistTGL. Therefore, our method can naturally be used in conjunction with DistTGL.
    - DistTGL endeavours to construct an efficient pipeline for a distributed MDGNN training system. It focuses on developing solutions to handle dependencies that emerge across different batches during distributed training. Additionally, DistTGL addresses the challenge of efficient scaling concerning the number of GPUs.
    - In contrast, our analysis is centred on the interdependencies of events within a batch and investigates the implications of temporal batch size. Our method aims to enlarge the size of the temporal batch without suffering significant performance degradation.
    - We will include the experiment that combines our method and DistTGL to explore whether the performance can be further improved.