From: ICLR.cc/2023/Conference/Paper1696/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper1696/Authors
title: Dear Reviewer 1y45:
comment: May we know if our response resolved your concerns? Thanks.

From: ICLR.cc/2023/Conference/Paper1696/Authors
title: Thank you for your reply
comment: Thanks for your reply. We are sorry for not making it clear, and we would like to further clarify your concern as follows.

Response to 1: actually it is true that different nodes have diverse distributions in the spatio-temporal data. This point has verified by several impactful papers [1-4]. For example in AGCRN [3] (262 citations), the authors point out that "traffic series exhibit diversified patterns, they may appear similar, dissimilar, and even contradictory owning to the distinct attributes across a variety of data sources."

[1] Matrix Factorization for Spatio-Temporal Neural Networks with Applications to Urban Flow Prediction. CIKM 2019.

[2] Urban traffic prediction from spatio-temporal data using deep meta learning. KDD 2019.

[3] Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting. NeurIPS 2020.

[4] Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting. KDD 2022.


Response to 2: currently in the paper, the parameter size of SimST is less than the other GNN-based methods. For example, SimST-CT has 150k parameters, and GWNET has around 300k parameters. Therefore, the reviewer may attribute our faster converange rates to the size of parameters. But we have conducted experiments before on setting the parameter size of SimST-CT to 400k, which is larger than GWNET. The results show that the learning curves of SimST-CT with 150k and 400k parameters are basically the same. Thus, the parameter size will not influence our learning process much, and our proposed method still has superior convergence rates than the other baselines. By the way, since you have already run our code, the aforementioned results can be validated by changing init_dim from 64 to 128 (in the code). And then you can compare the curves with different parameter sizes.

Moreover, we would like to argue that using learning curve to empirically reflect the difficulty of learning process is a common way in the literature. For example, in the Figure 1-3 of the Adam [5] approach, the authors use the learning curve to demonstrate the supriority of the proposed method empirically.

[5] Adam: A Method for Stochastic Optimization. ICLR 2015.

From: ICLR.cc/2023/Conference/Paper1696/Reviewer_4rep
title: Still confusing
comment: Thank you for the detailed answers. However, A1 is not convincing to me.

1. The logic of A1 is here, "GNN-based models require all nodes to be involved in the message passing step, and thus the model needs to forecast the future for every node simultaneously." so "such joint forecasts over all nodes make it extremely difficult to optimize the model parameters". This seems illogical. One situation that this statement is true is that different nodes have different distributions. If so, the further experiment should be conducted.

2. I don't think a learning curve could demonstrate the point of the response. The parameter size also significantly influences the learning curve.

From: ICLR.cc/2023/Conference/Paper1696/Authors
title: Dear Area Chair:
comment: We thank all reviewers for their thoughtful and constructive reviews, and we have added related content and experiments in the revised manuscript. Meanwhile, we have provided thorough responses to each reviewer and hope you can look through them. We sincerely expect AC to prompt reviewers to actively participate in the discussion to help us make the paper stronger. We are also open to further discussion if the concerns have not been fully addressed. Thanks.

From: ICLR.cc/2023/Conference/Paper1696/Authors
title: Response to all reviewers
comment: Dear reviewers,

We would like to thank you all for your valuable feedback and insightful comments. We are really encouraged to see that some positive aspects of our paper have been appreciated, such as an interesting idea to work on (reviewer 1y45, FBm5) and achieving SOTA performance with much faster inference speed (reviewer 1y45, 4rep, FBm5). Your expert knowledge helps us to strengthen the manuscript significantly. In the following parts, we endeavor to provide responses to each reviewer. We sincerely hope that a revision (see the updated paper) is still considered.

From: ICLR.cc/2023/Conference/Paper1696/Authors
title: Response to Reviewer FBm5
comment: Thank you very much for appreciating the idea of removing GNN in traffic forecasting and the potential on intriguing future studies.

[C1] The proposal is a heuristic without performance guarantees (although most of the existing works on this topic share the same
issue).

[A1] Thanks for pointing this out. We agree with your comment that most existing works in this field fall short in theoretical analysis and interpretability. However, we also observe that the community is now putting more efforts in improving this aspect. For example, TAMP-S2GCNETS [1] utilizes the mathematical tools of multi-persistence to capture hidden time-conditioned properties in spatio-temporal data. Thank you for this helpful comment, we expect to equip our solution with theoretical guarantees in the future.

[1] TAMP-S2GCNETS: Coupling Time-Aware Multipersistence Knowledge Representation with Spatio-Supra Graph Convolutional Networks for Time-Series Forecasting. ICLR 2022.

[C2] Some extra experimental results may be reported: the model training efficiency, and the impact of 𝑘.

[A2] Thanks for pointing this out. We added the training efficiency results in Appendix A.3. We also added a figure of the impact of 𝑘 in Appendix A.4.

Thank you again for the valuable feedback. Please let us know if you have any other comments/questions.

From: ICLR.cc/2023/Conference/Paper1696/Authors
title: Response to Reviewer 4rep
comment: Thanks for your positive comments on our paper writing and superior model performance. We further add more clarifications corresponding to the given comments below.

[C1] The reason why such a simple model outperforms other baselines is not explained. The model only utilizes the spatial information in the Spatial Proximity Injector, but it outperforms other GNN-baselines based on not only the original matrix but also the adaptive matrix. This should be further explained.

[A1] Thanks for this comment. First, we would like to clarify that the spatial information is provided by both the spatial position injector and spatial proximity injector. Second, we apologize for not clearly stating why our method can outperform other baselines. The reasons are as follows. GNN-based models require all nodes to be involved in the message passing step, and thus the model needs to simultaneously forecast the future for every node. Since traffic forecasting is a challenging task, such joint forecasts over all nodes make it extremely difficult to optimize the model parameters. In contrast, SimST can be seen as a single-node model, which means that each node is processed individually by the model. This makes the optimization process easier. The learning curve figure in Appendix A.5 validates our hypothesis, where SimST exhibits significantly faster convergence rates than other baselines.

[C2] Adding the proposed module based on these vanilla models and showing the performance should be more convincing.

[A2] Thanks for pointing this out. Following your suggestion, we have included the vanilla models’ results in Figure 3 of the main paper. We also modified the text (in blue color) to reflect this change.

[C3] The reproducibility is not good, since the provided code is not reproducible. I run the code by default setting on PEMS-04 on 3080ti torch 1.10.0, and only get MAPE above 0.11.

[A3] We believe that this comment may be based on a misunderstanding. Actually, the results achieved by the reviewer, especially this line: "Average Test MAE: 18.7109, Test RMSE: 30.2952, Test MAPE: 0.1272" is basically the same as our reported results in the last row of Table 2 (18.68, 30.30, 12.72%). Thus, we believe that your reproduced results actually verify the reproducibility of our code. We would like you to please consider updating your score if our reproducibility is not a problem.

Thank you again for the valuable feedback. Please let us know if you have any other comments/questions.

From: ICLR.cc/2023/Conference/Paper1696/Authors
title: Response to Reviewer 1y45
comment: Thank you very much for appreciating the novelty of the proposed idea and the quality of our presentation. We address the questions as below.

[C1] The model actually utilizes the adjacency information and the spatial proximity injector works the same way as graph convolution. The whole framework is not that GNN-free.

[A1] Sincere thanks for raising this concern. We would like to clarify it as follows. The core component of GNNs is the message passing step, where features are aggregated to the anchor node via a weighted summation operation. And typically, a stack of GNN layers are required to propagate the information. In contrast, our spatial proximity injector only concatenates the raw time series data from neighbors to the anchor node. We do not need the process of feature aggregation (or weighted sum). Thus, we argue that it would be safe to say that our framework is GNN-free.

[C2] In the paragraph under Eq(4), the author says ‘Note that this injector operates in the data preprocessing stage, so it has no influence on model training and inference.’ It seems that the time consumption of Spatial Proximity Injector is not taken into consideration while comparing efficiency, which could be unfair for baselines.

[A2] Thanks for pointing this out. In fact, some baselines also require data preprocessing. For example, STGODE requires the construction of a semantic adjacency matrix by using the dynamic time warping algorithm to precompute the similarity of two time series. We do not count its cost in this stage either. Moreover, we would like to argue that it is common practice in the community to only compare efficiency during training and inference. For example, GWNET compares the training and inference time per epoch with baselines in Table 4 of its paper. AGCRN compares the training time per epoch in Table 3 of its main paper. Lastly, we would like to provide you with the preprocessing time. The time are around 40.3 seconds and 248.7 seconds on the PeMSD4 and LA datasets, respectively. Considering that the preprocessed inputs can be stored and reused, we think that the cost of a few minutes could be cheap.

[C3] Since the motivation of the work falls on efficiency, the author should compare more metrics. TPS is not as straightforward as training and inferring time per epoch.

[A3] Thanks for your advice. We have attached the details of the training and inference time per epoch in Appendix A.3. We also modified the main paper (in blue text) to point out this change. For the inference efficiency, the conclusions of using inferring time are similar to that of using TPS. For the training time, SimST variants are generally slower than GWN and AGCRN, due to the usage of a small actual batch size.

Thank you again for the valuable feedback. Please let us know if you have any other comments/questions.

From: ICLR.cc/2023/Conference/Paper1696/Authors
title: Response to Reviewer k8Ws
comment: [C1] The proposed approach’s main contribution is to add spatial information as new features through manual processing. No new machine learning model/algorithms are proposed. The contribution of the paper is very limited.

[A1] We respectfully disagree with the reviewer. Our contributions go significantly beyond adding spatial information as new features. We have listed our three contributions at the end of the introduction section, but their full depth bears further emphasis.

Thus, we would like to clarify our contributions. Most existing STGNNs approaches rely on GNNs to model spatial dependencies, leading to high computational cost and poor scalability. In our work, we demonstrate that it is feasible to remove GNNs from traffic forecasting tasks altogether and replace them with our new SimST framework design. The SimST framework confirms that GNNs are unnecessary for spatial modeling in traffic forecasting. Our approach achieves two significant benefits: with almost always better accuracy, and several times faster inference speed (e.g., SimST-WN is 11× - 17× and 30× - 39× faster than STGODE and STGNCDE, respectively.)

Moreover, we believe that simplicity is not in itself a flaw: simple methods are more easily adopted and built upon, which is an important contributor to research progress and impact. In the computer vision and graph domains, MLP-Mixer [1], SGC [2] and GLNN [3] showed that simple MLP or MLP-like architectures achieved competitive performance to conventional architectures, and were influential in questioning conventional wisdom and sparking further research into more diverse architectures.

[1] MLP-Mixer: An all-MLP Architecture for Vision. NeurIPS 2021. [2] Simplifying Graph Convolutional Networks. ICML 2019. [3] Graph-Less Neural Networks: Teaching Old MLPs New Tricks via Distillation. ICLR 2022

[C2] The evaluation results show very marginal improvements over existing approaches. No statistical significance provided.

[A2] Thanks for raising this concern. However, we believe that SimST’s performance improvements are not marginal, but rather substantial. For example, in Table 2 of the main paper, we can see that the earlier STGODE improves the performance from 21.19 (achieved by STSGCN) to 20.84 on the PeMSD4 dataset (a 1.7% improvement) while GMSDR further enhances the performance to 20.49 (also a 1.7% improvement). Both STGODE and GMSDR were published at top-tier conferences, i.e., KDD. Compared to the latest papers GMSDR and STGNCDE in this year, our method advances the result from 20.49/19.21 to 18.68 (a 8.8/2.8% improvement). Thus, we argue that our method achieves a significant performance gain over existing arts.

This result is even more compelling considering that the goal of our SimST method is to show that a GNN-free model can be competitive. Thus, we achieved beyond expectations that the performance is not only competitive, but actually beats existing methods. In terms of statistical significance, we performed the corresponding tests, as mentioned in the caption of Table 2.


[C3] The way the adjacency matrix is defined lacks generalizability.

[A3] In fact, using road network distance to describe the proximity between nodes is the mainstream method in the traffic forecasting research field. For example, DCRNN (a pioneering work in traffic forecasting) defines the adjacency matrix based on the road network distance. Many follow-ups (including ours) use the same matrix definition as DCRNN, e.g., STGCN, GWNET, STSGCN, STGODE, and so on. Thus, it is common practice to define the matrix based on the road network distance.

Thank you again for the valuable feedback. Please let us know if you have any other comments/question

From: ICLR.cc/2023/Conference/Paper1696/Reviewer_k8Ws
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a spatiotemporal prediction model for traffic forecasting to reduce the computational cost. The new model is built based on a multi-layer perceptron with spatial information explicitly modeled as additional features. The method is evaluated on real-world traffic datasets against major baselines in this field.  
strength_and_weaknesses: Strengths:

1. Spatiotemporal prediction is an important topic.
2. Incorporating spatiotemporal autocorrelation in the model is a reasonable idea. 

Weaknesses:

1. The proposed approach's main contribution is to add spatial information as new features through manual processing. No new machine learning model/algorithms are proposed. The contribution of the paper is very limited.
2.  The evaluation results show very marginal improvements over existing approaches. No statistical significance provided. 
3. The way the adjacency matrix is defined lacks generalizability. 

clarity,_quality,_novelty_and_reproducibility: The novelty of the work is weak. The proposed method is a simple preprocessing over spatial data and is not a major contribution.
summary_of_the_review: The paper proposes a spatiotemporal forecasting model for traffic network. The model is built on a simple MLP with spatial information manually integrated as new features. Experimental results on several real-world datasets are provided. The paper investigates an important problem but lacks sufficient technical contributions. The proposed ideas are appear to be trivial data processing and feature engineering practices. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: None.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1696/Reviewer_1y45
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper focuses on the problem of traffic forecasting, and proposes a straightforward and simple GNN-free framework. The author argues that GNNs, which are widely used in existing works to model spatial correlations, are inefficient due to their message passing scheme. For replacing GNNs, two spatial context injectors are designed to provide neighboring and positional information. A new training strategy is applied for training the proposed framework, and the performance and efficiency of the model is evaluated on five widely used datasets.
strength_and_weaknesses: Strengths:
S1. The idea of GNN-free traffic forecasting model is interesting.
S2. The proposed model achieves SOTA performance and has better efficiency.
S3. Spatial Proximity Injector advances message passing operations to the data preprocessing stage, which makes the model GNN-free and saves a lot of training time.

Weaknesses:
W1. As in S3, the model actually utilizes the adjacency information and the Spatial Proximity Injector works the same way as graph convolution. The whole framework is not that GNN-free.
W2. In the paragraph under eq(4), the author says ‘Note that this injector operates in the data preprocessing stage, so it has no influence on model training and inference.’ It seems that the time consumption of Spatial Proximity Injector is not taken into consideration while comparing efficiency, which could be unfair for baselines.
W3. Since the motivation of the work falls on efficiency, the author should compare more metrics. TPS is not as straightforward as training and inferring time per epoch.


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. 
The overall idea is interesting. 
The used datasets are public, and the code is provided, which makes it easier to reproduce the results. 
summary_of_the_review: Generally, I like the idea of removing GNN for spatiotemporal forecasting. If the stated weaknesses are solved, I will raise my score. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1696/Reviewer_4rep
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper presents a temporal model-agnostic and GNN-free spatial-temporal learning framework. Two spatial modules: Spatial Position Injector and Spatial Proximity Injector are used to inject spatial information and the output embedding is put into the temporal model to make predictions. The experiment shows the presented model is effective and efficient.

strength_and_weaknesses: pros:
a. The presented model is clear and easy to understand
b. The result shows the model outperforms the traffic forecasting baselines not only effectively but also efficiently.
c. The structure of the paper is clear and the writing is good.

cons:
a. The experiment shows the good performance of the proposed simple model. However, the reason why such a simple model outperforms other baselines is not explained. The model only utilizes the spatial information in the Spatial Proximity Injector, but it outperforms other GNN-baselines based on not only the original matrix but also the adaptive matrix. This should be further explained.
b. The ablation study is not convincing. Without the two proposed modules, the model degenerates to CT, WN, and GRU. Adding the proposed module based on these vanilla models and showing the performance should be more convincing.

clarity,_quality,_novelty_and_reproducibility: The presentation, figures, and tables of this paper are clear. The novelty is fair. The reproducibility is not good, since the provided code is not reproducible. I run the code by default setting on PEMS-04 on 3080ti torch 1.10.0, and only get MAPE above 0.11. Here are the results.
Horizon 1, Test MAE: 16.6053, Test RMSE: 26.7830, Test MAPE: 0.1124
Horizon 2, Test MAE: 17.4781, Test RMSE: 28.1616, Test MAPE: 0.1191
Horizon 3, Test MAE: 17.9478, Test RMSE: 28.9799, Test MAPE: 0.1226
Horizon 4, Test MAE: 18.2494, Test RMSE: 29.5336, Test MAPE: 0.1246
Horizon 5, Test MAE: 18.5514, Test RMSE: 30.0432, Test MAPE: 0.1257
Horizon 6, Test MAE: 18.7712, Test RMSE: 30.4357, Test MAPE: 0.1277
Horizon 7, Test MAE: 18.9878, Test RMSE: 30.8132, Test MAPE: 0.1291
Horizon 8, Test MAE: 19.1742, Test RMSE: 31.1383, Test MAPE: 0.1304
Horizon 9, Test MAE: 19.3937, Test RMSE: 31.4713, Test MAPE: 0.1314
Horizon 10, Test MAE: 19.5566, Test RMSE: 31.7213, Test MAPE: 0.1322
Horizon 11, Test MAE: 19.7641, Test RMSE: 32.0313, Test MAPE: 0.1345
Horizon 12, Test MAE: 20.0507, Test RMSE: 32.4302, Test MAPE: 0.1366
Average Test MAE: 18.7109, Test RMSE: 30.2952, Test MAPE: 0.1272

summary_of_the_review: This paper presents a simple ST learning framework with good performance. However, more explanation of how the simple model works should be addressed and reproduciable code should be provided. Thus, this paper is below the acceptance threshold.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1696/Reviewer_FBm5
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a spatio-temporal learning framework named SimST for traffic forecasting. The core proposal is to avoid using graph neural networks for spatial correlation modelling, and hence to improve the model forecasting efficiency. To achieve this, the paper proposes to use two modules to encode the spatial correlations: a spatial position injector and a spatial proximity injector. The spatial position injector is an MLP that starts with random position encodings and uses back propagation at training to learn spatial position encodings for the nodes (e.g., sensors or locations of interest). The spatial proximity injector simply concatenates each node's traffic observation with those of its top-k neighbours. 

Experimental results on five real datasets comparing against 11 baseline methods confirm the effectiveness of the proposed framework. 
strength_and_weaknesses: Strength:
1. The proposal to get rid of graph neural networks for spatial correlation learning in traffic forecasting is interesting and may intrigue a series of follow-up studies. 

2. Although the proposed model structure is simple, it achieves state-of-art forecasting accuracy while being much faster in model inference.

Weakness:
1. The proposal is a heuristic without performance guarantees (although most of the existing works on this topic share the same issue). 
clarity,_quality,_novelty_and_reproducibility: The paper is very well written and easy to follow. None of the model components are particularly new, but they are put together in a way that is shown to produce the state-of-the-art accuracy. 

Some extra experimental results may be reported:
1. Table 4 has shown the inference efficiency. How about the model training efficiency which may be more important as most models are very fast at inference already. 

2. What is the impact of k (as in the top-k neighbours of the spatial proximity injector)?
summary_of_the_review: The paper proposes a simple model structure that uses MLPs to replace graph neural networks for spatial correlation modelling in traffic forecasting. Like most exisiting models on this topic, there is no performance guarantee for the proposed model. However, it has been shown the achieve state-of-the-art accuracy on multiple real datasets while being much faster in model inference. It has the potential to intrigue a series of follow-up studies.

=== Update after rebuttal ==

I appreciate the efforts to prepare the response. The point on theoretical performance guarantees cannot be addressed in the current formed of the paper. Given that I already gave a positive rating, I'll keep my current score. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold