Paper: Continual Traffic Forecasting via Mixture of Experts (ICLR-2024)

Reviewer comment (Presentation):
The Structure of the paper could be improved - Section 4 seems to be a bit too long, while part of the experiments, especially the settings and principles have to be left in the appendix. As I am not an expert in this area, section 4 is a little hard to follow, many complex modules are introduced in this section which makes it easy to lose.

Author response:
Firstly, I apologize for any inconvenience caused while reading the paper. The contents of Section 4 are considered essential components of our model, which made it challenging to simplify the content. To aid in understanding our model we have added various sections and discussions in the appendix for a more detailed explanation of the model's architecture. Please refer to the updated paper for this information.

Reviewer comment (Novelty):
As far as I know, clustering seems to be a common technique in the field of machine learning, while seldom reference is about the utilization of clustering in the traffic area. I think a discussion of clustering in this area is needed. Besides, the clustering operation seems to cost extra resources, in the sense of time or memory. Do the authors consider that?

Author response:
Thank you for your constructive advice. In response to your comments, we have created a new section titled 'Appendix J: Exploring Clustering in Traffic Forecasting and Revisiting the Contributions of Our Proposed Model.' In this section, we examine how clustering methodologies are used in existing traffic forecasting, their purposes, and how our modeling aligns or differs from these methods. We also agree that the time and space complexity of the clustering operation is an important consideration. Reflecting this, we have added 'Appendix L: Complexity Analysis,' where we analyze the time and space complexity of the entire model, including the clustering operation. Please refer to the updated paper for this information.

Reviewer comment (Experiments):
The paper does not discuss the generalizability of the proposed method to other traffic datasets, as there is only one dataset in the paper. I understand the hard of acquiring real-world data. Can we just use the real-world roadnet but generate several new traffic flows? In this way we can evaluate the methods on multiple datasets and see its generalization ability.

Author response:
Thank you for your valuable feedback. One limitation of our research, as you pointed out, is its generalization ability. The scarcity of datasets in the field of continual traffic forecasting, a relatively new and under-researched area, poses challenges in fully assessing the generalizability of our model. The dataset we currently use represents a real-world long-term streaming network, encompassing a lengthy period of seven years with new sensors added annually. Our model architecture is specifically tailored for such long-term streaming networks. If we were to create a new dataset suitable for long-term streaming networks, it would likely be challenging to do so quickly in a short period. Most existing datasets used in traffic forecasting are collected over short durations and thus are not directly applicable for evaluating our model tailored for long-term streaming networks. Despite these challenges, we have conducted extensive visualization and experimentation to demonstrate that our model is well-adapted and effectively designed for continual traffic forecasting, exhibiting significant performance improvements over existing models. We are hopeful that as continual traffic forecasting becomes more recognized and researched, a wider variety of datasets and benchmarks will become available, energizing the field and enabling more comprehensive future studies and comparative analyses. This discussion will be included in the newly created 'Appendix M: Limitation and Future Work' section.

Reviewer comment (Novelty):
As mentioned in the weaknesses, it is suggested that the authors discuss more about clustering in the literature and try to improve their experiments.

Author response:
Thank you for your constructive advice. In response to your comments, we have created a new section titled 'Appendix J: Exploring Clustering in Traffic Forecasting and Revisiting the Contributions of Our Proposed Model.' In this section, we examine how clustering methodologies are used in existing traffic forecasting, their purposes, and how our modeling aligns or differs from these methods. We also agree that the time and space complexity of the clustering operation is an important consideration. Reflecting this, we have added 'Appendix L: Complexity Analysis,' where we analyze the time and space complexity of the entire model, including the clustering operation. Please refer to the updated paper for this information.

Reviewer comment (Reproducibility):
Although the paper provides a comprehensive explanation of the methodology, further technical insights regarding the implementation and specific algorithms within the TFMoE method would be beneficial.

Author response:
I agree with your opinion that, although technical insights regarding each methodology are provided in the Introduction and Proposed Method sections of the paper, further efforts to clarify these points are necessary. In response, we have created a new section titled ‘Appendix J: Exploring Clustering in Traffic Forecasting and Revisiting the Contributions of Our Proposed Model’ to reorganize and clarify our contributions. Please refer to the updated paper for this information.

Reviewer comment (Evaluation):
The paper falls short in providing a detailed analysis of the limitations of the proposed TFMoE, a factor which could be significant for future research and practical applications.

Author response:
In response to your feedback, we have created a new section in the appendix titled 'Appendix M: Limitation and Future Work' where we analyze the current limitations of our model. Please refer to the updated paper for this information.

Reviewer comment (Theory):
The computational complexity of the TFMoE algorithm, especially in the phases of continual training and forecasting, which could be a concern for large-scale datasets, is not discussed in the manuscript.

Author response:
Thank you for the valuable feedback. In response, we have added a new section titled 'Appendix L: Complexity Analysis' where we conduct an analysis of both time complexity and space complexity. As mentioned in the Introduction, the reason for using continual learning is that retraining the entire network is inefficient. Instead, we use continual learning methods to maintain high performance while training only a few newly added nodes. However, in our methodology, when we train the first task, we use the entire dataset, which can be problematic. We have addressed this discussion in a newly created section 'Appendix M: Limitation and Future Work'. Please refer to the updated paper for this information.

Reviewer comment (Novelty):
Although the paper employs sound-good methodology and achieves competitive performance, further efforts regarding the technical innovation and methodological novelty would be beneficial.

Author response:
I agree with your opinion that, although technical insights regarding each methodology are provided in the Introduction and Proposed Method sections of the paper, further efforts to clarify these points are necessary. In response, we have created a new section titled ‘Appendix J: Exploring Clustering in Traffic Forecasting and Revisiting the Contributions of Our Proposed Model’ to reorganize and clarify our contributions. Please refer to the updated paper for this information.

Reviewer comment (Experiments):
A more detailed exposition of the dataset and the experimental settings used in the evaluation, including their characteristics and potential biases, would enrich the manuscript.

Author response:
We have described the dataset in a manner that is typical in the field of traffic forecasting. This can be verified in 'Appendix A: Dataset.' Additionally, details about the experimental settings, such as dataset preprocessing, can be found in 'Appendix C: Evaluation Protocol.' Information on the model's hyperparameter settings is available in 'Appendix D: Implementation Details.' Please refer to these sections for more information.

Reviewer comment (Writing):
How does the proposed method address the heterogeneous graph structures over time?

Author response:
In our research, we focus on a homogeneous graph structure that aligns with the nature of our 1-dimensional traffic flow data. Each node in our graph represents a data point in traffic flow, and our structure does not incorporate heterogeneous elements. The primary aim of our study is to model the changes in traffic flow over time, capturing the patterns and variations within this specific context. As such, the question of addressing heterogeneous graph structures does not directly apply to our work, since our methodology and data are inherently homogeneous. While our current approach is specialized for uniform data types, we recognize the potential for future research to explore and incorporate more complex, heterogeneous graph structures, expanding the scope and applicability of our work.

Reviewer comment (Experiments):
Does the model's performance have a dependence on the pre-trained clustering results?

Author response:
If 'pre-trained clustering' refers to the clustering methodology in the pre-training stage, then such an experiment would indeed be a good way to support our main idea. We appreciate the suggestion and have created a new section titled 'Appendix K: Analysis of the Effects of Clustering in the Pre-training Stage' in response to your comments. In this section, we have conducted experiments and analyzed the results. Please refer to the updated paper for this information.

Reviewer comment (Experiments):
Does the model have sensitivity to the parameters?

Author response:
We have presented experimental results for the core hyperparameters that we believe are crucial in our modeling. Experiments on the number of Experts can be found in the appendix 'Appendix G: Optimizing Expert Selection.' Experiments on the replay ratio for 'Reconstruction-Based Replay' are available in 'Appendix H: Replay Method Analysis: Reconstruction-Based Replay vs Random Sampling.' Experiments on the sampling ratio for 'Forgetting-Resilient Sampling' are detailed in 'Appendix I: Evaluating the Impact of Sampling Ratio on Model Performance.' However, as you suggest, experiments on the sensitivity of various other hyperparameters are also important. We plan to add a section related to parameter sensitivity and will update it for inclusion in the final version of the paper.

Reviewer comment (Experiments):
Experiments are only done in one dataset. It is better to extend the scope of experiments to validate the method.

Author response:
As you have mentioned, our research was conducted on a single dataset. Continual traffic forecasting is a relatively new research field with limited datasets available and a scarcity of baselines. We acknowledge these limitations and have discussed them in the 'Appendix M: Limitation and Future Work' section of our paper. We also address this topic in our response to reviewer p8v4, which we invite you to consult for further details. To mitigate these issues, we have conducted a variety of experiments and visualizations for every component and module proposed in our study. We hope this approach will be considered in your assessment of our work.

Reviewer comment (Novelty):
While the components are well combined, most of them are existing techniques.

Author response:
If by 'existing techniques' you are referring to methods like Clustering, Mixture of Experts (MoE), VAE sampling, Learning without Forgetting (LwF), and Graph Learning that we employed to prevent catastrophic forgetting in continual traffic forecasting, then indeed, these are established methods, and we have duly cited them in our paper. The novelty of our work lies in the exploration and analysis of how these methodologies can be appropriately and effectively adapted to the field of continual traffic forecasting. From this perspective, if you are aware of any other papers employing similar techniques in this context, we would greatly appreciate your sharing them with us. This would further enrich our understanding and the scope of our research.