Paper: Spatio-Temporal Few-Shot Learning via Diffusive Neural Network Generation (ICLR-2024)

Reviewer comment (Evaluation):
The time consumption of this framework needs to be provided.

Author response:
Thanks for this suggestion. In response, we present a detailed comparison of the computational cost of our proposed model against baselines, as the following table:


| Model              | RegionTrans | DASTNet  | MAML   | TPB   | STGFSL | Ours   | 
| :----:             | :----:      | :----:   | :----: |:----: | :----: | :----: |
| Computational cost |~1.5min      | ~1h      | ~2h    | ~1h   | ~2h    | ~3h    |


Our model demonstrates efficient training, completed within 3 hours on a single GPU (RTX-2080Ti), which we believe is an acceptable time consumption. It is important to note that the slightly longer time consumption in our model is attributed to the step-by-step denoising process implemented in the DDPM approach. 


We chose DDPM because it is a classic, simple but effective diffusion framework.  Notably, our framework is designed to be compatible with more efficient diffusion models, such as DDIM [1]. Employing these models has the potential to significantly reduce the current computational cost. We want to underscore that our framework has the ability to achieve remarkably better performance with a relatively lower overhead. We have added discussions on the model's efficiency in the revised manuscript (Appendix C.2). 

[1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising Diffusion Implicit Models." International Conference on Learning Representations. 2020.

Reviewer comment (Evaluation):
Some details require further clarification. For example, how exactly is the improvement value of 9.8% calculated? As in the last part of the Introduction part, i.e., “an improvement of 9.8%”. Similar cases are in the description text of Table 2.

Author response:
The reported improvement of 9.8% is calculated by averaging the error reductions presented in Table 1 (previous version). The calculation is as follows:

$$Improvment = (4.32\\% + 4.62\\% + 17.1\\% + 14.9\\% + 13.6\\% + 7.26\\% + 8.17\\% + 8.62\\%)/8=9.82\\%$$

 
We have revised the presented tasks and datasets in Table 1, so the specific improvement value has also been changed to 7.87\% by averaging these four datasets. In response, we have revised the description to include information on the specific scenarios or tasks for which this improvement is observed.

Reviewer comment (Experiments):
I think the rationality behind the excellent performance of GPDiff is easy to understand. But I still expect some further explanations by designing some more pertinent experiments. For example, which kind of regions in target cities would be much easier to benefit from knowledge transfer in this pretraining framework?

Author response:
We appreciate the insightful suggestion.  In response, we have added **fine-grained evaluations**. The conclusion is that **regions sharing similar patterns are more amenable to knowledge transfer.** 

Upon careful analysis, we first observed performance variances among different regions in the target city. To further elucidate this phenomenon, we performed clustering on the source city time series and investigated how the target regions align with the clustering results. Our findings reveal that regions with relatively better performance seamlessly fit into one of the clusters in the source cities, whereas regions with lower performance struggle to align with any clusters. As for the detailed experiments and results, please see **Appendix C.3** in the revised manuscript. 

Importantly, our framework offers a valuable opportunity for highly developed cities with abundant data to assist less developed cities with limited data. We believe that leveraging more data-rich cities as source cities can lead to further performance improvements.

Reviewer comment (Experiments):
Case Study only wrote for different regions to generate a different distribution of parameters, indicating that the parameters can be generated for different data, but can explain the parameters are effective?

Author response:
Thanks for your constructive question. In the revised version, we have designed and conducted experiments on two synthetic datasets, allowing us to control spatio-temporal pattern similarities between regions. We carefully analyzed temporal variations and spatial information. On the one hand, we observe that regions exhibiting highly similar spatio-temporal patterns showcase similar distributions of parameters.  On the other hand,  we observe that the generated parameter maps for the same regions closely align with their corresponding ground-truth distributions.


To construct the synthetic datasets, we first construct spatial connections between regions within each city. Then, we generate time series data for these regions following specific patterns. This approach ensured that regions have distinct spatio-temporal patterns.

We first explore the parameters of well-trained models for regions in the source city.  Subsequently, in the parameter generation results for the target city, we selected two regions characterized by similar time series patterns and symmetry and symmetrical spatial locations. Our analysis includes the depiction of the ground-truth parameter maps and generated parameter maps for these regions.




For a more detailed analysis of the synthetic data experiments, please refer to **Appendix C.4** in the revised manuscript.

Reviewer comment (Experiments):
Insufficient related work and baselines: The examination of related works in the paper appears to be extremely insufficient, with a noticeable lack of comparison to existing studies. From my understanding, numerous studies [1,2,3,4, 5] have delved into the application of meta-learning or hypernetworks to tailor prediction models for each node or region, especially [1] and [2] with high impacts in the field of traffic forecasting. It seems that implementing these directly could address the issue of varying data distributions across regions or cities. Therefore, I find it necessary for this paper to provide a compelling justification for the utilization of the framework proposed in this manuscript. What's more, they should be considered as baselines for empirical comparison.

Author response:
Thanks for this comment. We appreciate your thorough examination of related work and baselines, and we would like to further clarify the distinctions between our research goals and the referenced prior work.



**1. For related work**, the referenced papers [1-5] indeed share certain correlations with our framework, particularly in their use of non-shared parameters for different nodes. **It's important to clearly explain the key differences between our approach from these works.** 
- Firstly, our framework embraces the non-shared parameter concept without imposing restrictions on the spatio-temporal graph model, ensuring compatibility with state-of-the-art models. In contrast, the referenced papers propose specific model designs for spatio-temporal prediction. 
- Secondly, our approach serves as a pre-training framework, capable of acquiring knowledge from multiple cities and effectively transferring it to few-shot scenarios. In contrast, the methods you referenced lack a dedicated focus on transfer learning and have certain limitations in data-scarce urban environments, e.g., these methods necessitate training data for each city's prediction.

In response to your comments, we have revisited the related work section in our initial submission. In Section 2.1, we categorized urban transfer learning works based on the granularity of transferred knowledge, distinguishing between coarse-grained and fine-grained transfer.  We acknowledge that our previous version missed a specific discussion on hypernetworks for spatio-temporal graph models [1-5]. **Your feedback prompted us to rectify this, and we have added discussions on these works in Section 2.3.**


**2. For baselines**, It is important to note that our research addresses a different problem compared to the studies you referenced. Those papers you listed primarily focus on designing spatio-temporal graph prediction models with non-shared network parameters for individual nodes. In contrast, our work centers on the challenge of spatio-temporal graph **transfer** learning. We introduce **a general transfer learning framework** that is designed to be compatible with state-of-the-art spatio-temporal graph models.  This framework serves as a general approach rather than a specific model design. 

To ensure a fair comparison with other urban transfer learning methods, we implement a consistent base model for all baselines, aligning with common practices in existing transfer learning research [1]. Given the overarching nature of our framework, which is not tied to a particular model architecture, it's not appropriate to directly compare it with the specific models mentioned in your comment. While the referenced models share a similar concept (non-shared parameters) with our employed baselines like [1], our chosen baseline is more aligned with STG transfer learning, yielding state-of-the-art performance. 

We hope this clarification explains the motivation behind not including the mentioned papers as baselines.

[1] Lu, Bin, et al. "Spatio-Temporal Graph Few-Shot Learning with Cross-City Knowledge Transfer." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.

Reviewer comment (Novelty):
Limited technical novelty: Compared to existing meta-learning-based methods, this paper has no significant novelty and technical contributions. The primary difference lies in the utilization of a DDPM (along with some prompts) to generate model parameters for each region, without theoretical contributions from the model side. This approach doesn't seem to surpass the acceptance threshold set by ICLR. The originality and technical value of this study could benefit from further enhancement.

Author response:
Thanks for your comments, we would like to clarify the novelty of our paper. 

To address your concern about limited technical novelty, it's essential to highlight the distinct focus of our research. Unlike meta-learning-based model design, our primary objective centers around urban **transfer** learning, specifically addressing the challenge of transferability between source and target cities. Existing works in this field mainly fall into two categories: (1) feature-based transfer learning [1,2,3], which focuses on common and transferrable features, and (2) parameter-based transfer learning [4,5], utilizing meta-learning for model initialization in the target city. Comparing our work to studies within the urban transfer learning domain is more appropriate than traditional meta-learning approaches.


The  originality and technical value of our paper are in the following aspects:

**1. Fundamental and Novel Knowledge Transfer**: Unlike feature or parameter-based approaches, we propose the transfer of more fundamental knowledge—specifically, the knowledge of obtaining personalized model parameters. This represents a unique contribution to spatio-temporal graph transfer learning.

**2. Pre-training Paradigm**: We successfully employ a generative pre-training paradigm for effective spatio-temporal knowledge transfer across diverse cities. 
A key departure from traditional pre-training paradigms lies in our focus on the parameter space rather than the data space.  This pre-training paradigm facilitates to acquire knowledge from data-rich cities.


[1] Tang, Yihong, et al. "Domain adversarial spatial-temporal network: a transferable framework for short-term traffic forecasting across cities." Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022.
[2] Liu, Zhanyu, Guanjie Zheng, and Yanwei Yu. "Cross-city Few-Shot Traffic Forecasting via Traffic Pattern Bank." Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 2023.
[3] Jin, Yilun, Kai Chen, and Qiang Yang. "Selective cross-city transfer learning for traffic prediction via source city region re-weighting." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.
[4] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. "Model-agnostic meta-learning for fast adaptation of deep networks." International conference on machine learning. PMLR, 2017.
[5] Lu, Bin, et al. "Spatio-Temporal Graph Few-Shot Learning with Cross-City Knowledge Transfer." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.

Reviewer comment (Experiments):
Weak evaluation: Firstly, while the paper asserts that the proposed framework is model-agnostic, this claim is substantiated solely with evaluations on two outdated baselines, published before 2019, while recent methodologies have been overlooked, e.g., [6, 7, 8], see more in [9]. Secondly, the affirmations made in Section 4.4 lack persuasiveness. Figure 4 doesn't offer any insightful revelations, except reporting their differences. For instance, is there any proximity amongst various regions based on their respective POI information? These elements need to be elaborated on for a comprehensive understanding. Thirdly, the paper lacks a discussion on the model's efficiency. Customizing models for each region always brings considerable computational overhead. By the way, the efficiency of the proposed method should be benchmarked against the existing approaches as mentioned in W1.

Author response:
**Q4: While the paper asserts that the proposed framework is model-agnostic, this claim is substantiated solely with evaluations on two outdated baselines, published before 2019, while recent methodologies have been overlooked, e.g., [6, 7, 8], see more in [9].**

**Response**: Here we would like to clarify the rationality of our experiments regarding model-agnostic characteristics.

**1. As for the base model selection**, we use STGCN and GWN  because the two models have been widely recognized for their robustness and reliability.  Specifically, we adopt a consistent base model in our framework and baselines for a fair comparison, aligning with existing practices in the field [1]. STGCN and GWN were deliberately chosen to provide a strong foundation for evaluating the model-agnostic nature of our framework. Choosing them is not meant to overlook recent methodologies but rather to ensure a consistent benchmark. 

**2. As for the framework flexibility**, we want to emphasize that our framework extends beyond STGCN and GWN. It is adaptable to any state-of-the-art models mentioned in your review. Specifically, we implement our framework on STID [7] you referenced to demonstrate its flexibility. STID, given its relatively simpler design and potential limitations in modeling complex spatio-temporal relationships, lags slightly behind STGCN and GWN. Please see the revised section 4.3 for more details.


In summary, the rationale behind our choice of STGCN and GWN as baselines is grounded in their well-established effectiveness, providing a robust foundation for evaluating the model-agnostic nature of our framework. In the meantime, we have also demonstrated our framework's compatibility with state-of-the-art models.


[1] Lu, Bin, et al. "Spatio-Temporal Graph Few-Shot Learning with Cross-City Knowledge Transfer." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.


**Q5: The affirmations made in Section 4.4 lack persuasiveness. Figure 4 doesn't offer any insightful revelations, except reporting their differences.**


**Response:** Thanks for your constructive question. In the revised version, we have designed and conducted experiments on two synthetic datasets, allowing us to control spatio-temporal pattern similarities between regions. We carefully analyzed temporal variations and spatial information. On the one hand, we observe that regions exhibiting highly similar spatio-temporal patterns showcase similar distributions of parameters.  On the other hand,  we observe that the generated parameter maps for the same regions closely align with their corresponding ground-truth distributions.


For a more detailed analysis of the synthetic data experiments, please refer to **Appendix C.4** in the revised manuscript.


**Q6: The paper lacks a discussion on the model's efficiency. Customizing models for each region always brings considerable computational overhead. By the way, the efficiency of the proposed method should be benchmarked against the existing approaches as mentioned in W1.**

**Response:** We appreciate the reviewer's comment regarding the necessity for a discussion on the model's efficiency. In response, we present a detailed comparison of the computational cost of our proposed model against baselines, as outlined in the table below:


| Model              | RegionTrans | DASTNet  | MAML   | TPB   | STGFSL | Ours   | 
| :----:             | :----:      | :----:   | :----: |:----: | :----: | :----: |
| Computational cost |~1.5min      | ~1h      | ~2h    | ~1h   | ~2h    | ~3h    |


Our model demonstrates efficient training, completed within 3 hours on a single GPU (RTX-2080Ti), which we believe is an acceptable time consumption. It is important to note that the slightly longer time consumption in our model is attributed to the step-by-step denoising process implemented in the DDPM approach. 


We chose DDPM because it is a classic, simple but effective diffusion framework.  Notably, our framework is designed to be compatible with more efficient diffusion models, such as DDIM [1]. Employing these models has the potential to significantly reduce the current computational cost. We want to underscore that our framework has the ability to achieve remarkably better performance with a relatively lower overhead. We have added discussions on the model's efficiency in the revised manuscript (Appendix C.2). 

[1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising Diffusion Implicit Models." International Conference on Learning Representations. 2020.

Reviewer comment (Experiments):
Some SOTA baselines are missing, such as CrossTReS, MetaST, etc.

Author response:
**Q7: Some SOTA baselines are missing, such as CrossTReS, MetaST, etc.**

**Response:** Thanks for your observation. Regarding the absence of CrossTReS in our comparisons, it's important to note that we have no human mobility road network data for the specific cities targeted in our study. Applying CrossTReS without these crucial datasets would not result in a fair and meaningful comparison. Therefore, we did not include CrossTReS due to the incompatibility of our available data.

As for MetaST, its utilization of CNN to model spatial relationships is not directly compatible with our dataset. Our dataset is structured as a graph, unlike the image-based format required by MetaST. Thus, integrating MetaST into our comparisons is not directly applicable.

Despite these specific reasons for not including them as baselines in our experiments, it's crucial to clarify that we did not overlook these models. In the previous version, we did provide discussions on these state-of-the-art urban transfer learning solutions. This was strategically done in the introduction and related work sections to reveal the key contributions of our approach.

Reviewer comment (Presentation):
The presentation of experimental results can be improved. For example, in Figure 4, why only show the temporal variations? What about spatial information? Visualization of model parameters needs to be more apparent.

Author response:
Thanks for your constructive question. In the revised version, we have designed and conducted experiments on two synthetic datasets, allowing us to control spatio-temporal pattern similarities between regions. We carefully analyzed temporal variations and spatial information. On the one hand, we observe that regions exhibiting highly similar spatio-temporal patterns showcase similar distributions of parameters.  On the other hand,  we observe that the generated parameter maps for the same regions closely align with their corresponding ground-truth distributions.

To construct the synthetic datasets, we first construct spatial connections between regions within each city. Then, we generate time series data for these regions following specific patterns. This approach ensured that regions have distinct spatio-temporal patterns.

We first explore the parameters of well-trained models for regions in the source city.  Subsequently, in the parameter generation results for the target city, we selected two regions characterized by similar time series patterns and symmetry and symmetrical spatial locations. Our analysis includes the depiction of the ground-truth parameter maps and generated parameter maps for these regions.

For a more detailed analysis of the synthetic data experiments, please refer to **Appendix C.4** in the revised manuscript.

Reviewer comment (Evaluation):
I find that the presented results on the traffic speed prediction task seem much less than those of the crowd flow prediction task.

Author response:
Thanks for your comments. In the previous version, due to the page limit, we only presented most results on the crowd flow prediction task. We acknowledge the need for a more comprehensive representation of our model's performance on the traffic speed prediction task.

In the revised version, we have added the experimental results on the traffic speed prediction task to the main paper and moved some traffic speed prediction results to the Appendix. The updated version now includes additional experimental results on the traffic speed prediction task, providing a more balanced result presentation. Please refer to the revised Table 1 and Figure 7, and the corresponding analysis in the main paper.

Reviewer comment (Theory):
While the proposed method is sound, it is not clear what is the theoretical foundation behind GPDiff.

Author response:
**Response:**    Thank you for raising this question. The theoretical foundation of GPDiff lies in its generative framework, where a hypernetwork is pre-trained to generate non-shared parameters for different regions in the city. The effectiveness of our approach is grounded in two key assumptions:

***Assumption 1:** Regions with distinct spatio-temporal patterns correspond to different optimized model parameters*

$$P_i^* \neq P_j^* \quad \text{if} \quad D(X_i, X_j) > \epsilon,$$

$$D(P_i^*, P_j^*)>D(P_m^*, P_n^*) \quad \text{if} \quad D(X_i, X_j) > D(X_m, X_n)$$

where $P_i^*$ represents the optimized model parameters for region $i$, $X_i$ denotes the spatio-temporal pattern of the region $i$, and $D$ signifies the dissimilarity metric between the patterns of two regions.

***Assumption 2:** It is conceivable to find $f_\theta$ capable of capturing the mapping relation from the spatio-temporal patterns of each region ($C_i$) to the optimized model parameters ($P_i^*$). The similarity between parameters is indicative of pattern similarity.*

$$P_i = f_\theta(C_i) \quad \text{where} \quad \theta = \underset{\theta}{\text{argmin}} \sum_i \text{Distance}(P_i^*, f_\theta(X_i)),$$



To empirically validate these assumptions, in the revised version, we designed and conducted experiments using two synthetic datasets representing source and target cities. These datasets allow us to control spatio-temporal pattern similarities between regions.


To confirm assumption 1, we analyzed the parameters of well-trained models for regions in the source city. Our observations revealed that regions with highly similar spatio-temporal patterns exhibit analogous parameter distributions, while regions with distinctly different patterns showcase divergent parameter distributions. This finding supports the notion that regions with distinct spatio-temporal patterns correspond to different optimized model parameters.


To validate assumption 2, we utilized the trained generative model in the target city. We selected two regions with similar spatio-temporal patterns and observed that the generated parameter maps closely aligned with their corresponding ground-truth distributions. This indicates that our framework effectively captures the mapping relation between spatio-temporal patterns and parameter distributions.


For a more detailed analysis of the synthetic data experiments, please refer to **Appendix C.4** in the revised manuscript.

Reviewer comment (Experiments):
Why choose diffusion for parameter generation? How much time does it consume in the denoising process?

Author response:
**Response:**  We choose diffusion for parameter generation based on the following considerations:

**1. Structural Similarity with Images.** There exists a structural similarity between parameter maps and images. Extensive research has demonstrated the superior performance of diffusion models in the context of image generation. Leveraging this similarity, we find diffusion models to be well-suited for capturing the intricate patterns and relationships within parameter maps.

**2. Relevance of Diffusion Process to Parameter Optimization.** The denoising process, as employed in our framework, closely mirrors the progression from random initialization to trained parameters. This progression aligns with the optimization journey, making the diffusion process an appropriate and meaningful mechanism for generative pre-training of model parameters.


**Regarding the time consumption during the denoising process,** our current implementation takes less than 1 minute to complete a single denoising iteration. Notably, the denoising process for different nodes can be computed in parallel, contributing to the overall efficiency of our approach. Thus, the use of the diffusion model demonstrates an acceptable time consumption. Meanwhile, it's essential to note that the required time can vary based on the choice of diffusion models. 

In our implementation, we use DDPM simplicity. Although DDPM serves our purposes effectively, it's recognized as relatively less efficient due to the step-by-step denoising process. If prioritizing time efficiency in denoising, alternatives like DDIM [1] could be explored.

[1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising Diffusion Implicit Models." International Conference on Learning Representations. 2020.

Reviewer comment (Experiments):
It seems that the method uses a newly proposed network. It is unclear whether the performance gain comes from the Diffusion or the network. Could the framework generalized to other networks?

Author response:
**Response:**  Given that our approach and baselines employ the same base model, the performance gain comes from the novel framework, and the core idea is a generative hypernetwork based on the diffuson model.  The baselines cannot leverage the power of generative modeling and the capability of diffusion probabilistic model.


Our framework can be generalized to other networks. We have implemented different base models, including STGCN, GWN, and STIN (see Section 4.3 and Appendix C.1). Our framework consistently achieves superior performance, demonstrating the adaptability and effectiveness of our framework. This underscores the adaptability and effectiveness of our framework across various network architectures.