Paper: TESTAM: A Time-Enhanced Spatio-Temporal Attention Model with Mixture of Experts (ICLR-2024)

Reviewer comment (Novelty):
In terms of spatial and temporal modeling, this manuscript does not seem to bring new techniques.

Author response:
Thank you for your feedback! We would like to emphasize that this paper proposes novel context-aware spatial modeling (i.e., different spatial modeling method selections), which reveals that spatio-temporal modeling with multiple spatial modeling methods could lead to better forecasting results. 

Furthermore, Time-enhanced attention and temporal information embedding are our approach’s novel temporal modeling methods, which enable better information transfer from the input domain to the output domain. As shown in Table 2 in Sec. 4, without time-enhanced attention, TESTAM experiences a performance degradation from 6.40 to 6.64 in terms of MAE with EXPY-TKY. Please see Table 2 in Sec. 4 and check the ablation study results.

Reviewer comment (Writing):
There are multiple sets of terminologies describing the same thing without an explicit connection. For example, when categorizing traffic patterns, both "recurring vs non-recurring" and "normal vs abnormal" (in Conclusion) are used; when describing the three experts, "temporal modeling, spatio-temporal modeling with static graph, spatio-temporal modeling with dynamic graph" (in Abstract) and "identity experts, adaptive experts, attention experts" (in Figure 1). Clarifying these key concepts will make the presentation clearer.

Author response:
Thank you for your feedback! We have revised the inconsistent use of terms and clarified them in our manuscript.

We additionally provide information in Fig. 1, as "Identity, adaptive, and attention experts are responsible for temporal modeling, spatial modeling with a learnable static graph, and with a dynamic graph (i.e., attention), respectively." We would like to kindly inform you that we additionally provide the description of the spatial modeling methods in Sec. 3.2 (in the paragraph  "spatial modeling layer").

Reviewer comment (Evaluation):
Only the overall metrics for predicting accuracy are reported. It is unclear what recurring and non-recurring patterns that TESTAM captured while other methods miss. Some in-depth experimental analysis (see Q4 for details) will better support the claim of the first contribution.

Author response:
Thank you for your valuable comment! We additionally provide the newly updated and detailed experimental results in Appendix C. Based on the experiment results for roads that have difficult conditions–isolated roads, intersections, or sudden traffic jams caused by events (e.g., accidents)–TESTAM achieves a consistent performance improvement of 4% to 7% compared to other models. We also provide qualitative results in Appendix C with descriptions that showcase how TESTAM improves performance under difficult conditions.

Thank you for your comment! The two scenarios, (I), and (E), have been chosen with visual investigation and based on news articles, respectively, so we do not have statistical information for them. However, we provide a qualitative analysis of them in Appendix C.1, especially in the paragraphs "Spatially Isolated Roads (I)" and "Event-Aware Forecasting Case."

For (H), we additionally mentioned that we selected 1) the intersections or ramps based on our visual investigation and 2) the roads with the top 10\% entropy-based time-series complexity.

Thank you for your comment! If we have simultaneously visualized the location of the selected roads and line charts associated with the roads, it may reduce readability with smaller line charts and too much information provided at once.

We provide the selection processes for each scenario (I), (H), and (E) in Appendix D, including the number of selected roads. For (I), we have chosen 162 roads, which are about 10\% of the whole roads. In the case of (H), we have selected 184 roads using time-series complexity and include 34 hard-to-predict roads, such as intersections, ramps, and highway entrances. For (E), we have selected two circumstances: 1) Christmas and Eve in the hard-to-predict roads (H) and 2) traffic control and construction reported from the official announcements. 

Thank you for your suggestion! In addition to the selection process, we have updated the locations in Fig. 6 and 7. We especially provide the locations of the roads, including Road 1111, 1165, 1166, and 1349. Please note that we did not report the location of 1349 since it is the normal metropolitan expressway with a sudden event (i.e., traffic control).

Reviewer comment (Writing):
Why the proposed method is abbreviated as TESTAM?

Author response:
TESTAM is the abbreviation for “time-enhanced spatio-temporal attention model.” We clarified this in the Introduction.

Reviewer comment (Writing):
What does "in-situ" mean in the context of this manuscript?

Author response:
We use the term “in-situ” to better describe the dynamic and context-aware selection of spatial modeling methods.

Reviewer comment (Experiments):
The temporal information embedding and time-enhanced attention seem novel for traffic forecasting. Do they affect the performance? Some discussion or ablation study is needed.

Author response:
We newly updated our ablation study results (i.e., Table 2 in Sec. 4) for TIM and time-enhanced attention. Without time-enhanced attention, the model indicates high-performance degradation from 6.40 to 6.64 in terms of MAE for the EXPY-TKY dataset.

Reviewer comment (Experiments):
In the abstract, the authors state that TESTAM can model isolated nodes, highly related nodes, and recurring and non-recurring events. Can you provide some experimental results and analysis for this claim, e.g. case study?

Author response:
Thank you for your valuable comment! We additionally provide the newly updated and detailed experimental results in Appendix C. Based on the experiment results for roads that have difficult conditions–isolated roads, intersections, or sudden traffic jams caused by events (e.g., accidents)–TESTAM achieves a consistent performance improvement of 4% to 7% compared to other models. We also provide qualitative results in Appendix C with descriptions that showcase how TESTAM improves performance under difficult conditions.

Reviewer comment (Presentation):
Please add comments to the meaning of results in bold and underlined in Table 1.

Author response:
We clarified the meaning of the results in bold and underlined in Table 1. Results in bold indicate the best performance and those underlined indicate the second-best performance

Reviewer comment (Presentation):
Why are some equations in section 3 not numbered? I believe some of them are proposed by the authors, e.g. time-enhanced attention, and thus should be numbered.

Author response:
We revised manuscripts for the equations to be numbered

Reviewer comment (Experiments):
The effectiveness of Time-enhanced Attention has not been verified through ablation experiments, and readers cannot determine whether it truly enhances the model's predictive performance.

Author response:
W2. The effectiveness of time-enhanced attention should be verified through ablation study  
> We newly updated our ablation study results (i.e., Table 2 in Sec. 4) for time-enhanced attention. Without time-enhanced attention, the model indicates high-performance degradation from 6.40 to 6.64 in terms of MAE for the EXPY-TKY dataset.

Reviewer comment (Writing):
The distinction between best-route selection and worst routing avoidance is not clear. When introducing best-route selection, the authors mention node-wise routing and node-wise pseudo-label but haven't explained them. Additionally, changing 'smaller than q-th quantile' to 'smaller than (1-q)-th quantile' alone is considered best-route selection, but the reason for this change has not been provided.

Author response:
W3. Need detailed information for the best-route selection loss function  
> We updated the detailed information on the best-route selection in Appendix A. In our paper, we set q = 0.7. In this setting, an error greater than 70% of the error distribution, which is equal to the top 30% MAE, will be the worst route. In this setting (q=0.7), the bottom 30% MAE will be the best route, which means an error smaller than 1 - q-th quantile will be the best route. We revised our manuscripts and added detailed explanations in Appendix A.

Reviewer comment (Evaluation):
The experimental results are inconsistent. In Table 2, the results for TESTAM in the ablation experiment are for a 30-minute forecast, but they do not match the results in Table 1. Additionally, in Table 2, the PEMS-BAY dataset performs better in the 'w/o gating' condition than the complete TESTAM model in Table 1.

Author response:
W4-1. About experimental results in Table 2  
> Our results are actually consistent. To further clarify this, the results in Table 2 are reported across all forecasting windows. This means that the averages from short-term to long-term forecasting results. We have clarified the missing information in our manuscript.  

W4-2. Concerning computational costs  
> TESTAM uses a small number of parameters with fewer layers compared to other models, as a result, the computational costs are not expensive. We report a detailed time complexity analysis in Appendix B. For your information, we briefly summarize the results here. TESTAM requires a similar time as DCRNN in training. In the inference phase, TESTAM shows the second fastest computation (about 7 seconds) with the smallest number of the parameters, which is faster than that of the other models except Graph-WaveNet (about 4 seconds).

Reviewer comment (Evaluation):
The magnitude of the performance improvement is minimal. Table 1 shows that many metrics only slightly improve in terms of percentiles, at the cost of three times the computational time and memory for building three expert models.

Author response:
W4-3. Marginal improvements in Table 1, require a case study and investigation  
> As we described in our manuscripts, we focus on in-situ selection in spatial modeling methods to improve situation awareness of the model. We additionally provide the newly updated and detailed experimental results in Appendix C. Based on the experiment results for roads that have difficult conditions–isolated roads, intersections, or sudden traffic jams caused by events (e.g., accidents)–TESTAM achieves a consistent performance improvement of 4% to 7% compared to other models. We also provide qualitative results in Appendix C with descriptions that showcase how TESTAM improves performance under difficult conditions.

Reviewer comment (Presentation):
The title doesn’t well match with this study itself. Actually, this work mostly focuses on traffic forecasting, but the title suggesting a new general Spatio-Temporal Attention Model with MoE seems much more general and broader.

Author response:
W5. The title isn't well-matched to the manuscripts  
> Thank you! We will add the traffic forecasting in the title in the final version. It will be "TESTAM: A Time-Enhanced Spatio-Temporal Attention Model with Mixture of Experts for Traffic Forecasting"

Reviewer comment (Theory):
In memory-based gating networks, why is the similarity between the model's input and output used as the routing probability? This lacks a certain theoretical foundation (or some analysis) and explanation.

Author response:
Q3. Why do the authors utilize the similarity between input and output as routing probability?  
> Thank you for your feedback. I guess there exists a misunderstanding. In short, the similarity between $z_e$ and $O_i$ is equivalent to that between the output representation of experts and input-conditioned memory units trained to be in the same space as $z_e$.  

> In our design, we first make query $Q_i$ by projecting input to the query domain. In this process, the projection learns the input-output mapping, which converts input $X$ (in the input domain) into the $Q$ (in the same domain as final hidden states). Afterward, it queries the memory unit by attention mechanism. Then, the queried memory $O$ is an input-conditioned memory unit that will be trained to be in the same space as the output representations of experts. Therefore, in fact, the similarity between $z_e$ and $O_i$ is equivalent to one between the output representation of experts and the input-conditioned output representation. In this design, the routing probability will be interpreted as the finding of the most feasible solution from experts by using input-conditioned typical output representation.

Reviewer comment (Experiments):
The main issue of the paper is perhaps the relatively weak experimental results. As shown in Table 1, the proposed model TESTAM has very similar performance to MegaCRN (Jiang et al., 2023) and sometimes PM-MemNet (Lee et al., 2022).

Author response:
As we described in our manuscripts, we focus on in-situ selection in spatial modeling methods to improve situation awareness of the model. We additionally provide the newly updated and detailed experimental results in Appendix C. Based on the experiment results for roads that have difficult conditions–isolated roads, intersections, or certain events, such as sudden traffic jams by accidents–TESTAM achieves a consistent performance improvement of 4% to 7% compared to other models, including MegaCRN, which is a state-of-the-art model. We also provide qualitative results in Appendix C with descriptions that showcase how TESTAM improves performance under difficult conditions.

Reviewer comment (Evaluation):
There are no model training time results. I wonder if TESTAM is even slower than MegaCRN and/or PM-MemNet to train.

Author response:
TESTAM uses a small number of parameters with fewer layers compared to other models, as a result, the computational costs are not expensive. We report a detailed time complexity analysis with the METR-LA dataset in Appendix B. For your information, we briefly summarize the results here. TESTAM requires a similar time as DCRNN in training. In the inference phase, TESTAM shows the second fastest computation (about 7 seconds) with the smallest number of the parameters, which is faster than that of the other models except Graph-WaveNet (about 4 seconds).

Reviewer comment (Experiments):
The discussion of the results has focused on comparisons with GMAN and StemGNN which are from 2020 and are relatively "old" methods already.

Author response:
As we described in our manuscripts, we focus on in-situ selection in spatial modeling methods to improve situation awareness of the model. We additionally provide the newly updated and detailed experimental results in Appendix C. Based on the experiment results for roads that have difficult conditions–isolated roads, intersections, or certain events, such as sudden traffic jams by accidents–TESTAM achieves a consistent performance improvement of 4% to 7% compared to other models, including MegaCRN, which is a state-of-the-art model. We also provide qualitative results in Appendix C with descriptions that showcase how TESTAM improves performance under difficult conditions.

Reviewer comment (Writing):
Minor presentation issues: Typo: "is noisy sensitive", "ahve", "Let define"; Check "where f(X(t), θ)" below Equation 1; Grammar: "traffic data is noisy and contains many non-stationary, make the best route selection hard."

Author response:
We first revised the manuscript with authors to fix the typos related to mathematical equations and worked with native English speakers to improve our manuscripts.