Paper: Temporal Generalization Estimation in Evolving Graphs (ICLR-2024)

Reviewer comment (Experiments):
One major concern is the lack of baseline methods. The proposed method is only compared with a linear regression, which turns out to be a naive baseline. The methods mentioned in the related work D.2 are not compared. Moreover, only the MAPE with standard error is evaluated. More evaluation metrics are expected.

Author response:
Thanks for your valuable suggestions. We have conducted additional two baselines (DoC and Supervised) with 4 classical evaluation metrics for regression task (MAPE, RMSE, MAE and Standard Error). The performance comparison is shown in Table 1 of the general response. Our SMART consistently outperforms both Linear and DoC, and closely approach the supervised methods. We provide more detailed information in Appendix H.3 in our revised paper. Regarding the methods discussed in related work in Appendix D.2, these methods focus on how to improve model performance on evolving graphs, instead of on how to characterize the model performance dynamics in graph evolution. Since the target is different,  these methods therefore can not be used for estimating generalization dynamics.

Reviewer comment (Experiments):
It will be better if the methods are evaluated on a few more state-of-the-art backbones to further enhance the effectiveness of the proposed method.

Author response:
Thanks for your comment. To fully validate and compare the effectiveness of different methods, we add Graph Transformer backbone besides GCN, GAT and GraphSAGE. As shown in Table 5, first of all, our SMART shows consistent best performance among different architectures. For different graph neural network architectures, they tend to capture the structure and feature information from different aspects. Due to the simple model structure of GCN, our SMART shows advanced prediction performance among other architectures, which is also consistent with the theory of classical statistical machine learning.

Table 5. Performance comparison of different GNN architectures

| Dataset | Model | GCN | GAT | GraphSAGE | TransformerConv |
| --- | --- | --- | --- | --- | --- |
| OGB-arXiv | Linear | 10.5224 | 12.3652 | 19.5480 | 14.9552 |
|  | DoC | 13.5277±1.4857 | 12.2138±1.222 | 20.5891±0.4553 | 10.0999±0.1972 |
|  | SMART | 2.1897±0.2211 | 3.1481±0.4079 | 5.2733±2.2635 | 3.5275±1.1462 |
| DBLP | Linear | 16.4991 | 17.6388 | 23.7363 | 18.2188 |
|  | DoC | 4.3910±0.1325 | 13.8735±4.1744 | 11.9003±1.8249 | 9.0127±2.6619 |
|  | SMART | 3.4992±0.1502 | 6.6459±1.3401 | 9.9651±1.4699 | 6.4212±1.9358 |
| Pharmabio | Linear | 32.3653 | 29.0404 | 31.7033 | 31.8249 |
|  | DoC | 8.1753±0.0745 | 7.4942±0.0702 | 6.6376±0.0194 | 5.3498±0.2636 |
|  | SMART | 1.3405±0.2674 | 1.2197±0.2241 | 3.1448±0.6875 | 2.7357±1.1357 |

Reviewer comment (Theory):
The assumptions made to conduct theoretical analysis and bound derivation could be a concern as they are often too strong. Evolving graphs also contain nodes disappearing in many cases (e.g. sensor networks with possible malfunction over time), which means the assumption of an ever-growing network can be often invalid. Moreover, the zero-mean requires a non-min-max preprocessing of features. The authors need to be specific about the claims.

Author response:
Thanks for your valuable suggestions. We generalize these two assumptions in the following way. (1) Regarding the ever-growing assumption of graphs, we modify it to the setting where nodes appearing after deployment have a probability of disappearing (Assumption 3-3). (2) Regarding the zero-mean normalized feature, we modify it to the setting where the feature vector has a constant mean conditioned on all previous graph states (Assumption 3-4). The completed assumptions please refer to Assumption 3 in Appendix C.2. Based on this assumption, we further extend the unavoidable representation distortion theorem to Theorem 4, and the proof of Theorem 4 can be found in Appendix C.4 and C.5.

Reviewer comment (Writing):
It would be better to define some notations such as the function l, and generalization error beforehand for better understanding of the idea.

Author response:
Thanks for your advice. In our paper, $I$ denotes the mutual information. For two random variable $X$ and $Y$, the mutual information $I$ is denoted as follows:

$$I(X ; Y)=\sum_{x \in X} \sum_{y \in Y} p(x, y) \log \frac{p(x, y)}{p(x) p(y)}$$

Generalization error refers to the performance of a trained model when making predictions on new, unseen data compared to the performance on the data it was trained on. In our work, the generalization error we estimate is the performance of the trained GNN model on the test samples during the graph evolution process. We have added the above definition to the revised paper, and we believe that this will be more helpful for readers to understand our work.

Reviewer comment (Experiments):
The data augmentation in graph is too naive. The authors have only considered to randomly add or drop edges, which should only work for graphs without edge labels. While the random data augmentation technique works for barabasi-albert random graphs, it should not be effective for the cases where edge labels are also changing (for example, people change their relationship status with other people). The authors should consider more complex data augmentation techniques in graphs.

Author response:
We have added three graph data augmentation methods to generate different views for self-supervised graph reconstruction in a contrastive manner, i.e., DropEdge [5], DropNode [6] and Feature Mask [7]. Table 3 shows the experimental results, and our SMART shows the best performance on three datasets. We observe that different data augmentation methods on performance have similar performance.

Table 3. Performance comparison with different graph data augmentation methods.

| Method | OGB-arXiv | DBLP | Pharmabio |
| --- | --- | --- | --- |
| Linear | 10.5224 | 16.4991 | 32.3653 |
| DoC | 9.5277±1.4857 | 4.3910±0.1325 | 8.1753±0.0745 |
| SMART (DropEdge) | 2.1897±0.2211 | 3.4992±0.1502 | 1.3405±0.2674 |
| SMART (DropNode) | 1.7163±0.3783 | 3.8166±0.5124 | 0.9050±0.2606 |
| SMART (Feature Mask) | 1.8733±0.2729 | 3.0183±0.4073 | 0.7352±0.4001 |

Reviewer comment (Theory):
Theorem 1 is evaluated only for one layer GNN with a Leaky ReLU activation. Is Theorem 1 adjustable to multi-layer GNN and different GNN backbone structures such as GCN/GAT/GraphSage?

Author response:
Thanks for your comments. As we mentioned in the general response, we generalize Theorem 1 to a more general class of multi-layer graph neural network architectures (i.e., Theorem 3). Specifically, we consider the model where a feedforward neural network of an arbitrary depth is concatenated with the graph convolution network. The detailed theoretical analysis please refer to Appendix C.1 and the proof can be found in Appendix C.3.

Reviewer comment (Evaluation):
Figure 3, what is the y-axis prediction loss? Why not use other evaluation metrics to showcase the GNN prediction performances deterioration?

Author response:
Thanks for your comment. Y-axis indicates the prediction loss of GNN model on test datasets during graph evolution. It is also possible to use other performance indicators, while the loss on the test sample is just a simple and intuitive indicator.

Reviewer comment (Experiments):
Is there a particular reason that RNN is used instead of other time-series model to capture the temporal variation?

Author response:
To capture the temporal dynamics of GNN generalization variation, we propose an RNN-based method (i.e. LSTM) and achieve satisfied estimation performance. In our paper, we used the vanilla LSTM model to demonstrate the importance of temporal encoding via time series models. We do agree that under a more complex scenario, advanced time series model may achieve better performance. 

To further investigate this problem, we additionally replace the LSTM with other two time series model Bi-LSTM and TCN. As depicted in Table 4, our SMART achieves the best performance among three different time series models on both OGB-arXiv and Pharmabio datasets. On DBLP dataset, SMART achieves the best performance using LSTM, while SMART equipped with Bi-LSTM and TCN show comparable performance with DoC. In order to maintain consistency in the experiment, we adopted the same hyperparameter settings. From the perspective of parameter size, the parameter quantity of Bi-LSTM is nearly twice that of LSTM, while the parameter quantity of TCN is very small. We observe that as the parameter size increases, the prediction error generally decreases first and then increases. From these preliminary results, it is indeed important to choose appropriate models and parameter settings based on the complexity of the data.

Table 4. Performance comparison with different time series model.

| Method | OGB-arXiv | DBLP | Pharmabio |
| --- | --- | --- | --- |
| Linear | 10.5224 | 16.4991 | 32.3653 |
| DoC | 9.5277±1.4857 | 4.3910±0.1325 | 8.1753±0.0745 |
| SMART (LSTM) | 2.1897±0.2211 | 3.4992±0.1502 | 1.3405±0.2674 |
| SMART (Bi-LSTM) | 4.3601±0.5816 | 4.2402±0.6996 | 1.1219±0.0397 |
| SMART (TCN) | 1.5725±0.2500 | 4.5970±0.2607 | 1.8830±0.4161 |

Reviewer comment (Experiments):
Will the contrastive graph reconstruction improve the feature extractor during the pre-deployment phase? Why just limit the contrastive learning and reconstruction to post-deployment?

Author response:
Thanks for your advice. Sorry for the vague explanation we may have in the previous version. In SMART, we also conduct contrastive learning and graph reconstruction during training in pre-deployment phase, as shown in Algorithm 1 of Appendix A. The self-supervised loss function plays an important role to optimize a better estimator. We have further clarified the use of contrastive learning and graph reconstruction in our revised paper.

Reviewer comment (Experiments):
How will the proposed algorithm be used towards more complex real-time graphs? For example, how to deal with graphs that contain changing node labels and changing edge labels? How to apply the methods to heterogeneous graphs or spatial temporal graphs?

Author response:
Thanks for your valuable comments. These are all promising directions for future work. But we also would like to point out that the setting considered in this paper, where only the graph topology and node feature evolve, is one of the most basic settings in analyzing graph evolution. In our paper, we establish both  theoretical and experimental results showing that even without change of labels and in graphs without requiring heterogeneity, representation distortion is inevitable. We do believe that our proposed SMART can be fused with more powerful contrastive learning or time series techniques and  adapted  to more complicated graph evolution scenarios.

Reviewer comment (Experiments):
The authors only compared to a single linear regression baseline. While other methods for generalization prediction are not specific to temporal graph data, it should still be possible to consider some. Maybe some from [1] could be adapted? In this vein, I think there should be more citations to other generalization error predictors and a dedicated related works section in the main paper.

Author response:
Thank you very much for pointing out a relevant paper [1] in the field of computer vision. After in-depth reading, we adopt the DoC method proposed in this paper as a baseline for performance comparison. We present some additional experimental results in Q2 of the general response, where 4 evaluation metrics are shown in Table 1. We observe that in our experimental settings,  SMART shows consistent better performance than DoC. This may be due to the reason that DoC method is optimized for the setting where the testing distribution slightly differs from training data. However, in our setting, graph evolution is very fast and the evolving graphs are usually very different from their initial states. We also provide more detailed discussion in Appendix H.3 in our revised paper.

[1] Devin Guillory, et al. "Predicting with Confidence on Unseen Distribution." ICCV, 2021.

Reviewer comment (Theory):
The theorems need to be discussed more. For example, in theorem 1, I don't think that beta has been defined. Furthermore, while the authors make the strong but acceptable assumption that the graph will only add edges, I was wondering if it was also assumed that the underlying graphs were homophilous. Do the authors expect the proposed method to work on heterophilous graphs as well?

Author response:
Thanks for your valuable advice. In our theoretical analysis, $\beta$ is the slope ratio for negative values instead of a flat slope in ReLU. We have added an explanation of $\beta$ in our revised paper. Regarding the homogeneity of graphs, we do not make any assumptions in both theoretical analysis and empirical study. Meanwhile, we conduct a data analysis of the changes of homogeneity ratio over time on all the datasets as shown in Figure 10 of Appendix G. We observe that different datasets exhibit diverse variations in the homogeneity ratio of graphs and that SMART consistently outperforms the other two self-supervised baselines in all cases.

Reviewer comment (Writing):
The writing could be improved. For example, its not clear why the approach is referred to as "constrastive." From my understanding the authors do augment the graph, but the overall loss is purely reconstruction based.

Author response:
Constrastive self-supervised learning is a popular and effective learning strategy to enrich the supervised signals. It can be divided into two stages: first is the graph augmentation and then is the contrastive pretext tasks. Therefore, a general learning objective of contrastive self-supervised learning approaches is denoted as follows:

$$\theta^*, \varphi^* = \arg \min_{\theta, \varphi} \mathcal{L}(p_{\varphi}(f_{\theta}(\tilde{A}^{(1)}, \tilde{X}^{(1)}), f_{\theta}(\tilde{A}^{(2)}, \tilde{X}^{(2)})))$$

where $\tilde{A}^{(1)}$ and $\tilde{A}^{(2)}$ are two augmented graph adjacency matrices, $\tilde{X}^{(1)}$ and $\tilde{X}^{(1)}$ are two node feature matrices under different augmentations. In our setting, We perform data augmentation (e.g., randomly dropout node, dropout edge, mask feature) on the graph data with a probability of 0.5, keeping it unchanged with a probability of 0.5. Afterwards, we define a pretext task is to minimize the distance of two augmented graphs via link prediction and node feature reconstruction. We note that our proposed SMART is actually a special setting in the general contrastive learning framework.

Reviewer comment (Experiments):
The baseline only includes simple linear regression models. However, a better baseline should be a model that continuously acquires new node labels and gets retrained on the new data, to examine whether the generalization curve of such retraining is close to the prediction of the proposed method. This can more intuitively demonstrate the advantages of the method.

Author response:
Thanks for your suggestion. We have added a supervised baseline and compared it with our SMART. Please refer to Q2 in general response. We also provide more detailed information in Appendix H.3 in our revised paper. Overall, SMART achieves comparable performance on dataset OGB-arXiv, while achieving slightly worse performance on dataset Pharmabio, when compared to the supervised baseline.