Paper: Spatio-Temporal Graph Learning with Large Language Model (ICLR-2024)

Reviewer comment (Novelty):
The contributions of this study are weakly linked with the claimed challenges, and the proposed method does not sufficiently address these challenges.

Author response:
For weaknesse W1,  challenges of our paper are (1) \textbf{Long-Range Spatio-Temporal Dependencies}. (2) \textbf{Data Sparsity and Noise} and (3) \textbf{Dynamic and Evolving Nature}. To solve these challenges, our proposed method integrates a LLM-based spatio-temporal knowledge learner with a cross-view mutual information maximization paradigm, which further improves the representation ability of our method.

Reviewer comment (Theory):
Intuitively, LLM is able to provide more general semantic context about geospatial location information, but using semantic information for a spatio-temporal prediction task is not well formulated.

Author response:
For weaknesse W2, LLM provides general semantic context about geospatial location information, which captures the global-view spatial-temporal graph information and improves representatio ability of method on spatial-temporal graph.

Reviewer comment (Experiments):
The contribution of using LLM is not fully investigated.

Author response:
Q1: In our paper, we do not pretrain the LLM. We adopt LLM to obtain node (region) embeddings of the spatial-temporal graph. This process includes two step. The first step is to generate region profiles via LLM (gpt-3.5-turbo). The second step is to encode text representations via the text-embedding-ada-002) model. Besides, we adopt the time-evolving traffic features as node features of the spatial-temporal graph. Thus, these region representation vectors effectively capture the dynamic nature of traffic patterns over time, allowing us to extract essential time-evolving traffic features. By incorporating these features into our model, we ensure that our spatio-temporal predictions are accurate and reliable. This enables us to make informed predictions and effectively address the challenges posed by the ever-changing traffic conditions. Overall, our approach of leveraging region representation vectors enhances the performance of our model in predicting spatio-temporal traffic patterns.  (re-do)

Reviewer comment (Reproducibility):
Detailed information of the applied LLM is not fully disclosed, weakening the reproducibility of the paper.

Author response:
Q7: We have uploaded a collection of prompt files that play a crucial role in facilitating the capture of spatial-temporal graph semantic information by the LLM (Language-Region Model). These prompt files contain valuable data and structured information that enable the model to understand and incorporate the complex relationships and dynamics present in the spatial-temporal graph. By providing this essential input to the LLM, we ensure that it can effectively learn and represent the semantic information embedded in the spatial-temporal graph. This, in turn, enhances the model's ability to make accurate predictions and perform various spatial-temporal analysis tasks. The availability of these prompt files is a valuable resource that empowers the LLM to capture and utilize the spatial-temporal graph's semantic information efficiently.

Reviewer comment (Experiments):
LLM is pretrained in most cases. How to make sure the information used to train LLM is within the same range of time as the data used to evaluate the model performance? The knowledge of LLM can be stable—how to make sure the information is robust and updated for spatio-temporal prediction?

Author response:
During our extensive experimentation with various training datasets for the LLM (Language-Region Model), we have observed that the output of the LLM, represented as region representation vectors, remains stable across different training data. Interestingly, this stability has had minimal impact on the performance of the downstream tasks. This indicates that the LLM's ability to generate consistent region representations is robust and reliable, thereby ensuring the reliability and effectiveness of the LLM for subsequent tasks.

Reviewer comment (Reproducibility):
Which LLM is used here?

Author response:
Q2: We adopt Chatgpt-3.5 Turbo.

Reviewer comment (Experiments):
Can you compare your model performance between using and without using LLM?

Author response:
Q3: We provide crime predcition w/o using LLM as follows:

Chi-Crime: 1.0781 0.5008 
NYC-Crime: 2.5543 0.5518

Reviewer comment (Experiments):
Does different LLM impact your model performance?

Author response:
Q4: Yes. We have tired other LLM models, like LLama2. However, we find that ChatGPT-3.5 Turbo achieves the best performance, which make it the ideal choice for our needs.

Reviewer comment (Experiments):
Does different training data used for LLM impact your model performance?

Author response:
During our extensive experimentation with various training datasets for the LLM (Language-Region Model), we have observed that the output of the LLM, represented as region representation vectors, remains stable across different training data. Interestingly, this stability has had minimal impact on the performance of the downstream tasks. This indicates that the LLM's ability to generate consistent region representations is robust and reliable, thereby ensuring the reliability and effectiveness of the LLM for subsequent tasks.

Reviewer comment (Evaluation):
Could you elaborate more on how semantic information generated by LLM provides meaningful contribution to improve the model performance?

Author response:
Q6: This process includes two step. The first step is to generate region profiles via LLM (gpt-3.5-turbo). The second step is to encode text representations via the text-embedding-ada-002) model. In the provided prompt, we have taken into account two crucial components: temporal semantic information and spatial semantic information of each node on the spatial-temporal graph. By incorporating these components, our aim is to enable the LLM (Language-Region Model) to capture the semantic information of the spatial-temporal graph in a region representation format. This approach significantly enhances the model's performance by effectively leveraging the spatial-temporal relationships and characteristics present in the data. The inclusion of both temporal and spatial semantic information contributes to a more comprehensive and accurate understanding of the underlying spatial-temporal graph, leading to improved model performance and results.

Reviewer comment (Novelty):
Although the paper has an interesting contribution in combining information maximisation with LLM, the contribution is limited.

Author response:
For W1: The paper tackles three key challenges that are prevalent in the analysis of spatial-temporal data: (1) Long-Range Spatio-Temporal Dependencies (2) Data Sparsity and Noise (3) Dynamic and Evolving Nature. To address these challenges, the proposed method can capture long-range dependencies in the data via leveraging the LLM-based learner, allowing for more accurate modeling of complex spatial-temporal patterns.

Furthermore, the incorporation of the cross-view mutual information maximization paradigm enhances the representation ability of the method. This paradigm enables the model to learn meaningful representations by maximizing the mutual information between different views of the data, effectively addressing the issues of data sparsity and noise.

By combining these components, the proposed method overcomes the challenges posed by long-range spatio-temporal dependencies, data sparsity and noise, as well as the dynamic and evolving nature of the data. The resulting approach provides an improved representation capability and enhances the accuracy and robustness of spatial-temporal analysis.

Reviewer comment (Novelty):
I disagree with the author that the LLM part is a main contribution. The proposed method seems only LLM-related part is the summarization part in Section 3.2.2. There is no further interaction between the model and the LLM.

Author response:
For W2: By incorporating LLM, our method gains a deeper understanding of the spatial-temporal graph structure, allowing it to capture and encode the underlying semantic context. This improved representation capability leads to more robust and accurate predictions in various spatial-temporal tasks.

Reviewer comment (Experiments):
In the Introduction, the contribution part claims that incorporating LLM-based knowledge could handle the distribution shifts. However, it seems that there is no experiment to support/justify this claim.

Author response:
For W3: We provide case study in experiments to provide the claim that incorporats LLM-based knowledge could handle the distribution shifts.

Reviewer comment (Reproducibility):
Another minor issue: providing the code repository is good. However, there are some comments given in Chinese, which is not friendly to other readers.

Author response:
For W5:  We will edit it in the code.

Reviewer comment (Reproducibility):
Is the proposed method trained end-to-end? Or the ChatGPT summarization part should be pre-processed before the entire model training?

Author response:
Q1: Yes. To achieve region representation learning vectors, it is recommended to pre-process the summarization part of ChatGPT before training the entire model. This pre-processing step involves transforming the output of the summarization part into region representation vectors. These region representation vectors capture the essential semantic information and characteristics of the summarized text.
By pre-processing the summarization part in this manner, we can ensure that the subsequent training of the entire model incorporates the region representation learning vectors. This approach enables the model to leverage the region-specific information effectively and enhances its ability to understand and generate responses that are contextually relevant and aligned with the desired semantic representations.
In summary, by pre-processing the summarization part to obtain region representation learning vectors, we enhance the model's comprehension of the input data and subsequently improve its performance in generating meaningful and accurate responses.

Reviewer comment (Experiments):
How would you ensure that the LLM knowledge is actually helping the model? This isn't shown in the ablation study.

Author response:
Q2: We provide ablation study on crime prediction task w/o LLM, the results are provided as follows:

Chi-Crime: 1.0781 0.5008 
NYC-Crime: 2.5543 0.5518

Reviewer comment (Evaluation):
How would you ensure if the potential bias from the sparse spatiotemporal data especially in crime and house price cases are not exacerbated with the use of LLM?

Author response:
Q3: We provide a method via data augmentation method via CL loss to alleviate data sparsity issue. Besides, LLM provides spatial and temporal information as external information to alleviate spatial and temporal sparsity issue.

Reviewer comment (Reproducibility):
The paper's main innovation is the use of LLM to summarize region Spatio-temporal information, but Appendix 5 only provides a simple and incomplete example. For an LLM like ChatGPT, the design of the prompt significantly impacts the final result. Therefore, the complete prompt needs to be provided to improve reproducibility.

Author response:
Thanks for your suggestions. We have provided a set of prompt files to enhance the reproducibility of our model. These prompt files particularly focus on improving the understanding and implementation of Equation (6).

The first sub-equation in Equation (6) involves calculating the InfoNCE between the embedding vectors of the mobility graph obtained from the LLM and the GCN (Graph Convolutional Network). This comparison helps measure the similarity and alignment of the embeddings generated by both models for the mobility graph.

The second sub-equation in Equation (6) calculates the InfoNCE between the embedding vectors of the distance graph from the LLM and the GCN. This step provides insights into the similarity and alignment of the embeddings generated for the distance graph by both models.

The third sub-equation in Equation (6) computes the cosine similarity between the embedding vectors of the mobility graph generated by the LLM and the GCN. This similarity measurement captures the degree of alignment and correspondence between the embeddings produced by both models for the mobility graph.

Lastly, the fourth sub-equation in Equation (6) determines the cosine similarity between the embedding vectors of the mobility graph derived from the initial embedding of the first-layer GCN output and the GCN itself. This comparison evaluates the similarity and alignment between the initial GCN embeddings and the subsequent GCN outputs for the mobility graph.

By providing these sub-equations and their corresponding prompt files, we aim to enhance the transparency and reproducibility of our model, allowing researchers to better understand and replicate our findings.

Reviewer comment (Writing):
In Equation (6), the differences between $h$, $h^M$ and $h^D$ (also for f) are not clearly stated.

Author response:
Thanks for your suggestions. We have provided a set of prompt files to enhance the reproducibility of our model. These prompt files particularly focus on improving the understanding and implementation of Equation (6).

The first sub-equation in Equation (6) involves calculating the InfoNCE between the embedding vectors of the mobility graph obtained from the LLM and the GCN (Graph Convolutional Network). This comparison helps measure the similarity and alignment of the embeddings generated by both models for the mobility graph.

The second sub-equation in Equation (6) calculates the InfoNCE between the embedding vectors of the distance graph from the LLM and the GCN. This step provides insights into the similarity and alignment of the embeddings generated for the distance graph by both models.

The third sub-equation in Equation (6) computes the cosine similarity between the embedding vectors of the mobility graph generated by the LLM and the GCN. This similarity measurement captures the degree of alignment and correspondence between the embeddings produced by both models for the mobility graph.

Lastly, the fourth sub-equation in Equation (6) determines the cosine similarity between the embedding vectors of the mobility graph derived from the initial embedding of the first-layer GCN output and the GCN itself. This comparison evaluates the similarity and alignment between the initial GCN embeddings and the subsequent GCN outputs for the mobility graph.

By providing these sub-equations and their corresponding prompt files, we aim to enhance the transparency and reproducibility of our model, allowing researchers to better understand and replicate our findings.

Reviewer comment (Theory):
The author pointed out that the framework can preserve both short-term and long-range dependencies. There is a need to better explain which part of the approach captured the long-range information on the spatial-temporal graph.

Author response:
For W2 (d) LLM plays a crucial role in capturing the broader context of geospatial location information. By considering the global perspective, it effectively incorporates long-range dependencies and interactions between various spatial locations and time points. This holistic view allows our method to capture the overall semantic context of the spatial-temporal graph, which is essential for accurate modeling and analysis.  

The integration of LLM within our method enables the extraction of meaningful features and representations that go beyond local interactions. By considering the global-view spatial-temporal graph information, our approach gains a deeper understanding of the complex relationships and dependencies within the data.  

For Q1, LLM serves as a mechanism to capture the relationships and dependencies that extend beyond local neighborhoods. By considering the entire spatial-temporal graph, it enables our method to understand the complex interplay between different locations and time points, thereby capturing the broader context of the data. This comprehensive perspective is particularly important in spatial-temporal analysis, as it allows for a deeper understanding of the underlying patterns and trends.  

By incorporating LLM, our method gains the ability to capture long-range dependencies and interactions, which may not be apparent when considering only local information. This is especially valuable in scenarios where spatial and temporal influences span across large distances and time intervals.

Reviewer comment (Experiments):
The loss function comprises many terms and different combinations of the four loss weights are likely to affect the model performance. It’s advisable to analyze their effects.

Author response:
For W2 (c) In order to explore the potential effects of weight values assigned to different terms of the loss function, we will conduct grid-search experiments. Grid-search involves systematically trying out different combinations of weight values for each term and evaluating the performance of the model under each configuration.  

By varying the weights, we can examine how the model responds to different emphasis on each term of the loss function. This allows us to understand the impact of each term on the overall optimization process and the resulting performance of the model.  

The grid-search experiments will involve defining a range of weight values for each term and systematically iterating through all possible combinations. For each combination, we will train the model and evaluate its performance using appropriate evaluation metrics. This process will be repeated for multiple iterations to ensure robustness and reliability of the results.

Reviewer comment (Experiments):
If constructing the graph using POI feature similarity, similar to [1], what would be the results in the case study?

Author response:
For Q2, our method denoises the POI constructed graph via CL loss. If constructing the graph using POI feature similarity, region relations are presented only via region functions, ignoring the spatial-temporal correlations.

Reviewer comment (Experiments):
What is the workflow for using the framework? Is it first learning region representations with STLLM, and then using these representations as inputs for a downstream predictive model? If so, how will it perform if directly using the LLM-based representation as inputs without the GNN part?

Author response:
For Q3, yes. The incorporation of a Graph Neural Network (GNN) component is essential for capturing high-order spatial information and effectively modeling long-range spatial relations among regions. Without the utilization of a GNN, it becomes challenging to capture the complex dependencies and interactions that exist between distant regions.  

GNNs are specifically designed to operate on graph-structured data, making them well-suited for spatial modeling tasks. By propagating information across the graph, GNNs can capture the spatial relationships between regions, even when they are far apart. This allows our method to capture the long-range spatial dependencies that are critical for comprehensive spatial understanding.  

The GNN component enables our model to learn and incorporate the spatial context of neighboring regions, which in turn provides a more holistic representation of the overall spatial structure. By leveraging high-order spatial information, our method can capture not only the direct relationships between nearby regions but also the indirect relationships between distant regions.  

By dropping the GNN component, the ability to capture long-range spatial relations is severely limited. The model would rely solely on local information, neglecting the broader spatial context. This omission could lead to incomplete spatial representations and a loss of critical information necessary for accurate spatial modeling.

Reviewer comment (Reproducibility):
What LLM do you use for the LLM-based spatio-temporal knowledge learner? There is only a mention in related work that: “This study applies decoder-only LLM (GPT-3.5) to enhance the quality of the spatio-temporal graph with effective augmentation.” However, if the GPT-3.5 is used, how can GPT-3.5 obtain the latent representation vectors F (in Section 3.2.2)? Is it to use the OpenAI’s text embeddings? What is the cost of conducting training and prediction for experiments?

Author response:
For Q4, ChatGPT-3.5 Turbo. The process includes twp steps. The first step is to generate region profiles via LLM (gpt-3.5-turbo). The second step is to encode profiles representations via the text-embedding-ada-002) model.  For training, the cost of conducting training and prediction for experiments is around 24 dollars.