Paper: KITS: Inductive Spatio-Temporal Kriging with Increment Training Strategy (ICLR-2024)

Reviewer comment (Evaluation):
Reconstructing a single node at a time would remove the "graph gap", this would be the proper way of carrying out the evaluation.

Author response:
> **W1, W2, W3**: The sparsity of training graph.

As mentioned in the first paragraph of introduction, the sensors are often **sparsely** deployed in real applications for saving cost. We refer to some datasets used in our experiments, which are both from real-world applications.  
- PEMS07 comes from the Caltrans Performance Measurement System (PeMS) deployed in California (in use). It covers a large discrict, but the average node degree is 1.79 for the full graph (883 nodes), which means the graph is indeed locally sparse.  
- Similarily, AQI-36 dataset is collected from an air quality monitoring application, where only 36 monitoring stations are installed for the whole Beijing City.

In addition, many existing kriging studies assume sparsely distributed sensors, including IGNNK (Wu et al., 2021a), LSJSTN (Hu et al., 2021) and INCREASE (Zheng et al., 2023). We follow this line of research and target similar problem settings in our paper.

Reviewer comment (Experiments):
After removing nodes for training, graphs can become sparse. However, this issue is only caused by the removal of nodes for evaluation, i.e., it is an issue of the training/evaluation procedure and not an inherent issue of kriging methods. This would not be a problem in any real-world application as you would eventually train the model on the full graph.

Author response:
> **W1, W2, W3**: The sparsity of training graph.

As mentioned in the first paragraph of introduction, the sensors are often **sparsely** deployed in real applications for saving cost. We refer to some datasets used in our experiments, which are both from real-world applications.  
- PEMS07 comes from the Caltrans Performance Measurement System (PeMS) deployed in California (in use). It covers a large discrict, but the average node degree is 1.79 for the full graph (883 nodes), which means the graph is indeed locally sparse.  
- Similarily, AQI-36 dataset is collected from an air quality monitoring application, where only 36 monitoring stations are installed for the whole Beijing City.

In addition, many existing kriging studies assume sparsely distributed sensors, including IGNNK (Wu et al., 2021a), LSJSTN (Hu et al., 2021) and INCREASE (Zheng et al., 2023). We follow this line of research and target similar problem settings in our paper.

Reviewer comment (Experiments):
The considered datasets are quite small, removing many nodes at random might result in disconnected graphs that would explain the poor performance for some of the baselines.

Author response:
> **W4**: The considered datasets are quite small, removing many nodes at random might result in disconnected graphs that would explain the poor performance for some of the baselines.

We collect some statistics of 4 datasets, including (1) the total number of nodes, (2) the number of isolated nodes (in the inference graph) and (3) the number of isolated nodes (in the training graph after dropping 50% nodes), as shown in the table below.

|Dataset|#Node|#Isolated (inference)|#Isolated (train)|
|-|-|-|-|
|METR-LA|207|1|4|
|PEMS-BAY|325|6|4|
|SEA-LOOP|323|0|38|
|PEMS07|883|27|135|

Based on the statistics, we find that while dropping nodes for training, as adopted by baselines, might introduce some isolated nodes (more or less depending on the datasets), but the inferior performance of baselines cannot be majorly attributed by this phenomenon. This is because even in cases few or no isolated nodes are introduced (e.g., for the META-LA and PEMS-BAY datasets), their performance is inferior to ours by a clear margin (please refer to Table 2). We believe it is the graph gap issue that majorly affects the performance, which motivates us to propose our new increment training strategy to insert virtual nodes instead. In addition, we note that these datasets have been widely used in existing studies of spatio-temporal kriging.

Reviewer comment (Novelty):
The paper has good methodological novelty, although it is not framed in the proper way as there is no inherent "graph gap" issue in spatio-temporal kriging. The paper should either be rewritten targeting a specific operational setting (heavily scaling back the current claims and significance of the work) or presenting the method as a data augmentation strategy on graphs that are not made artificially sparse.

Author response:
Thanks for recognizing the postive aspects of our work (i.e., "methodologically significant", "the introduced method is significant (the combination of the increment strategy with NCR is very appealing)", etc.). We would like to respond to your follow-up comments as follows.

First, we would make our claims very concrete for ease of our discussions. We believe **we agree on the following claims**, but in case you would disagree with any of them, please let us know, and we shall discuss further.

**Claim 1**: Existing studies (and their training strategies), including IGNNK (Wu et al., 2021a), LSJSTN (Hu et al., 2021) and INCREASE (Zheng et al., 2023), all suffer from the "graph gap" issue.

**Claim 2**: Our KITS method proposed in this paper largely mitigates the "graph gap" issue.

**Claim 3**: An alternative training method (which you suggested in the reivew of our previous submission) does not suffer from the "graph gap" issue. But as we have explained and also verified via experiments (Appendix H.8), it does not work as good as our KITS method for the setting targetted in this paper (i.e., when we have sparsely distributed sensors). It might work for some other settings (e.g., when we have densely distributed sensors), but they are out of the scope of our paper.

In addition, we emphasize that we **did NOT claim** that the "graph gap" issue is one of the kriging task, as you have commented - *"I do not think the mismatch between the graph used for training and evaluation is the relevant issue here as it can easily be avoided as previously discussed."* We agree that it is not an inherent issue of the kriging task. We only claim that it is **an issue of existing inductive kriging methods (and their training strategies)** (i.e., the above Claim 1). 

Further, we agree that some graph augmentation elements are involved in our methodology (e.g., the way of inserting virtual nodes and constructing the edges is essentially a graph augmentation procedure). However, this graph augmentation procedure is only one of a few components of our methodology and there are alternative ways for the graph augmentation procedure. Therefore, we tend not to over emphasize the graph augmentation aspect of our methodology and frame it as a graph augmentation technique as you have suggested.

Once again, we appreciate the comments you have made on our paper. We hope that you would give our response some further thoughts. In case you have further questions, please let us know and we will be more than happy to explain further. For your information, the author response period for our submissions has been extended until the end of December 1st. Thank you very much for your time and efforts on reviewing our paper and discussing with us!

Reviewer comment (Theory):
The main position of this paper is grounded in the spatio-temporal setting but the proposed treatment does not seem to have anything specific to the temporal aspect. The aggregation of temporal feature within +/- m steps is kind of a random treatment to me. I do not see a very particular reasoning why doing so makes sense. What if such aggregation erases important small-scale variation patterns in the time-series data? Furthermore, m is a value dependent on the nature of the data so even though the ablation studies conclude that m = 2 gives best result, it is still specific to the few set of experimental data and that should not be a generic guideline to choose m.

Author response:
> **W1, Q2**: Intuition behind Spatio-Temporal Graph Convolution (STGC) Module.

Thanks for raising this good question! We reply to this question as follows:
1. For Q2 (why the setting is spatio-temporal), we claim that for the $i$-th node $X_i^t$ obtained from timestamp $t$:
    - if it aggregates information from other nodes $X_j^t$ under the same timestamp $t$, where $j \neq i$, then the **spatial** aspect will be included in the setting;
    - if it obtains information from different timestamps, e.g., $t-1$, $t+1$, then the **temporal** aspect will be captured in the setting.
2. More details about STGC module:
    - Firstly, we explain the differences between normal GCN (spatial only) and STGC (spatial + temporal).
        - In Figure 3(a), for node-1 under timestamp $T_i$, normal GCN would only aggregate node-1 with node-2,3,5's (node-1's neighbors) information under **the same timestamp $T_i$**;
        - Differently, STGC would aggregate not only **neighbor information within the same timestamp**, but also **information from each node's neighbors across time**.
    - Then, we explain the design and rationale behind STGC module:
        - **Forbid each node to aggregate information from itself (including the same and different timestamps)**:
            - As introduced in the paper, the nodes can be divided into 2 groups, namely observed nodes (with sensors) and target nodes (no sensors). Observed nodes have sensor readings all the time, while target nodes can never have sensor readings. The essence of kriging is to learn the pattern only on observed nodes during training, and directly apply the learned pattern to target nodes for value estimation. Due to the fact that observed nodes not only have sensor readings but also have loss regulations, their kriging patterns can be easily learned, i.e., their learned representations/features can be quite good.
            - With the above viewpoint in mind, if we allow self-loops, then the observed nodes would keep aggregating *good-quality* features, while the target nodes would keep aggregating *bad-quality* features from itself across time. Therefore, the pattern learned on observed nodes cannot suit target nodes well, and the performance would be suboptimal.
            - Therefore, we (1) remove self-loop in the adjacency matrix (**verified in row 1&3 of Table 11**); and (2) do not utilize RNN-like modules to aggregate standard temporal information (**verified in row 1&2 of Table 11**). For example, in Figure 3, node-1 ($T_i$) will not receive information from node-1($T_i$) and node-1($T_{i+1}, T_{i-1}$).
        - **Aggregate each node with its neighbors information (from the same timestamp)**: The intuition is the same as that of the normal GCN.
        - **Aggregate each node with its neighbors information (from different timestamps)**: This design is less common yet effective (**verified in row 4-8 of Table 11**). We provide a few examples to explain the intuition as follows:
            - Consider that we have road 1 and road 2, and the vehicles would drive from road 1 to road 2. Suppose at 8:00 am, a traffic jam occurs at road 2, and affected by road 2, road 1  gets crowded at 8:05 am. Therefore, the data distributions (e.g., vehicle speed data) of road 1 (8:05) have high correlations with those of road 2 (8:00).
            - Similarily, suppose a group of vehicles drive on road 1 at 8:00 am, and the same group drive to road 2 at 8:05 am. The driving behavior of the same group might be uniform, thus, the data distributions (e.g., vehicle speed data) of road 2 (8:05) have high correlations with those of road 1 (8:00).
    - Finally, the hyperparameter $m$ is used to define the scope, e.g., if $m=2$, then we consider each node at $T_i$ has high correlations with its neighbors at $\{T_{i-2}, T_{i-1}, T_i, T_{i+1}, T_{i+2}\}$, and would aggregate the former with the latter.

Reviewer comment (Experiments):
Although comparison with recent pure graph-based approaches has been well presented, I am still curious to know how well the proposed approach improves over a hybrid approach that integrates GCN with traditional krigging techniques. For example, the series of work on graph convolutional Gaussian processes. There is quite a substantial volume on that & I think the authors have not discussed that in the literature review. Such techniques can be used to address this krigging problem. After all, if I remember correctly, Gaussian processes were referred to as krigging in the past (in geo-statistics).

Author response:
> **W2, Q1, Q3**: Discussion about Graph Convolutions with Gaussian Process.

Many thanks for this comment! Our responses are as follows:
1. You are right that "Gaussian processes were referred to as krigging in the past (in geo-statistics)", and **we have included it in our baselines, which is called OKriging**. However, this method requires GPS coordinates as inputs, and some datasets do not have such information, therefore, the results of OKriging are missing in some datasets.
2. To the best of our knowledge, there is no existing GCN+GP method particulary designed for the kriging task, but we do agree it's a natural and good idea to combine GCN and GP for kriging. Therefore, we have followed your advice to include one of the latest GCN+GP methods published in ICLR'23 called GNN-GP (https://openreview.net/forum?id=flap0Bo6TK_), and conducted some experiments as follows:
    - We first clarify the differences between the data used in GNN-GP and our paper:
        - GNN-GP focuses on graph regression datasets like Chameleon to perform semi-supervised regression task. This dataset has **1 data sample**, 2,277 nodes and **3,132 input features**. The data is transformed from **$2,277 \times 3,132$** to **$2,277 \times 1$**.
        - For the dataset METR-LA, adopted in this paper, it has **6,872 test samples**, 207 nodes and **1 input features**. The data is transformed from **$207 \times 1$** to **$207 \times 1$**.
    - Then, we highlight the adaptions we've made to apply GNN-GP to our kriging task, and show the best MAE scores (after grid search) in the table below:
        - **Option 1**: Forward GNN-GP 6,872 times, the shape of input is $207 \times 1$ (speed features of current timestamp), the shape of output is $207 \times 1$.
        - **Option 2**: Forward GNN-GP 6,872 times, the shape of input is $207 \times 6,872$ (speed features from all timestamps), the shape of output is $207 \times 1$.
        - **Option 3**: Forward GNN-GP 1 time, the shape of input is $207 \times 6,872$ (speed features from all timestamps), the shape of output is $207 \times 6,872$.
        - **Option 4**: Forward GNN-GP 287 times, and process 24 samples together in each forward operation. This option is aligned with baselines and our KITS, i.e., in each data sample, we would use data from 24 consecutive timestamps for aggregating temporal information. The shape of input is $207 \times 24$ (speed features from 24 timestamps), the shape of output is $207 \times 24$.
    - We could observe that GNN-GP does not perform well in this task (we've already tuned the hyperparameters), e.g., it has the worst behavior, compared with all methods in Table 2 (e.g., the MAE score of our KITS is 6.1276). We discuss the possible reasons as follows:
        - GNN-GP originally requires 3,132 diverse input features, but there is only 1 input feature available in kriging. Although we can take data from different timestamps to enrich the input features, such input features are different from those used in GNN-GP.
        - Unlike the learnable parameters in baselines and KITS which can witness all data samples, and could learn and remember some intrinsic attributes of each node, the Gaussian kernels in GNN-GP cannot have such ability.
        - As mentioned in one of the baselines INCREASE (WWW'23, Section 2.2), the Gaussian assumption is quite restrictive, as the data used in kriging may not follow Gaussian distributions.
3. Based on the responses above, we think GCN+GP is an interesting idea for kriging, but the current GCN+GP methods have not been well adapted for kriging, and we think how to design GCN+GP particularly for kriging can be regarded as a possible future direction.

|Input shape|Output shape|#Forward|Time cost|MAE|
|-|-|-|-|-|
|$207 \times 1$|$207 \times 1$|6,872|890s|10.6211|
|$207 \times 6,872$|$207 \times 1$|6,872|893s|10.6214|
|$207 \times 6,872$|$207 \times 6,872$|1|3s|**10.5634**|
|$207 \times 24$|$207 \times 24$|287|40s|10.5935|

Reviewer comment (Presentation):
In terms of the presentation, I believe it would be better if the authors spend some space on summarizing GCN which will highlight better the essence of transferability in the inductive setting of krigging. Otherwise, while the current presentation is fine for people who are familiar with graph neural network, it will still leave a lot of gap for generic readers.

Author response:
> **W3**: Summarize GCN in the draft.

We shall include some representative GNNs in related work.

Reviewer comment (Novelty):
Could the authors position this work with the other literature on graph convolutional GPs, which is an integrative approach combining both elements of GCN & a traditional non-graph krigging method (i.e., GPs)?

Author response:
> That being said, I do want to push back a little on the statement that "there is no existing GCN+GP method particulary designed for the kriging task". Could the authors discuss this paper on graph GP: https://arxiv.org/pdf/1809.04379
It seems like it has not been used in the exact context here but I am curious if it can be repurposed for that?
This question is meant for constructive feedback only (my support will not change) so please feel free to expand a detailed discussion as necessary.

Thank you for your follow-up comment. Our responses are as follows.
- The suggested baseline (which we refer to as GGP) is designed originally for the semi-supervisied **node classification** task. It was adapted for the semi-supervised **node regression** task in the GNN-GP paper (ICLR'23). Therefore, this adapted method can also be used for the kriging task targeted in this paper. Note that we have included the discussion on the GNN-GP method in our previous response.
- We checked that the GGP code (its adapted version for the semi-supervised node regression task) is available in the github respository of GNN-GP (https://github.com/niuzehao/gnn-gp). Therefore, we used the code and conducted experiments (with Option 1 specified in our previous response) on the METR-LA dataset. The best MAE score we have obtained for GGP (by exploring different hyperparameters) is 10.5997. As a comparison, the MAE score of our KITS is 6.1276. According our experimental results, GGP, just like GNN-GP, does not perform well on spatio-temporal kriging task. The reasons that we have discussed for explaining why GNN-GP does not work well for the kriging task in our previous response should remain applicable for GGP.

Reviewer comment (Novelty):
Could the authors elaborate more on why this paper is specifically positioned in the temporal-spatial setting even though the proposed technique does not really have any new innovations for temporal modeling? I might have missed something important here.

Author response:
> **W1, Q2**: Intuition behind Spatio-Temporal Graph Convolution (STGC) Module.

Thanks for raising this good question! We reply to this question as follows:
1. For Q2 (why the setting is spatio-temporal), we claim that for the $i$-th node $X_i^t$ obtained from timestamp $t$:
    - if it aggregates information from other nodes $X_j^t$ under the same timestamp $t$, where $j \neq i$, then the **spatial** aspect will be included in the setting;
    - if it obtains information from different timestamps, e.g., $t-1$, $t+1$, then the **temporal** aspect will be captured in the setting.
2. More details about STGC module:
    - Firstly, we explain the differences between normal GCN (spatial only) and STGC (spatial + temporal).
        - In Figure 3(a), for node-1 under timestamp $T_i$, normal GCN would only aggregate node-1 with node-2,3,5's (node-1's neighbors) information under **the same timestamp $T_i$**;
        - Differently, STGC would aggregate not only **neighbor information within the same timestamp**, but also **information from each node's neighbors across time**.
    - Then, we explain the design and rationale behind STGC module:
        - **Forbid each node to aggregate information from itself (including the same and different timestamps)**:
            - As introduced in the paper, the nodes can be divided into 2 groups, namely observed nodes (with sensors) and target nodes (no sensors). Observed nodes have sensor readings all the time, while target nodes can never have sensor readings. The essence of kriging is to learn the pattern only on observed nodes during training, and directly apply the learned pattern to target nodes for value estimation. Due to the fact that observed nodes not only have sensor readings but also have loss regulations, their kriging patterns can be easily learned, i.e., their learned representations/features can be quite good.
            - With the above viewpoint in mind, if we allow self-loops, then the observed nodes would keep aggregating *good-quality* features, while the target nodes would keep aggregating *bad-quality* features from itself across time. Therefore, the pattern learned on observed nodes cannot suit target nodes well, and the performance would be suboptimal.
            - Therefore, we (1) remove self-loop in the adjacency matrix (**verified in row 1&3 of Table 11**); and (2) do not utilize RNN-like modules to aggregate standard temporal information (**verified in row 1&2 of Table 11**). For example, in Figure 3, node-1 ($T_i$) will not receive information from node-1($T_i$) and node-1($T_{i+1}, T_{i-1}$).
        - **Aggregate each node with its neighbors information (from the same timestamp)**: The intuition is the same as that of the normal GCN.
        - **Aggregate each node with its neighbors information (from different timestamps)**: This design is less common yet effective (**verified in row 4-8 of Table 11**). We provide a few examples to explain the intuition as follows:
            - Consider that we have road 1 and road 2, and the vehicles would drive from road 1 to road 2. Suppose at 8:00 am, a traffic jam occurs at road 2, and affected by road 2, road 1  gets crowded at 8:05 am. Therefore, the data distributions (e.g., vehicle speed data) of road 1 (8:05) have high correlations with those of road 2 (8:00).
            - Similarily, suppose a group of vehicles drive on road 1 at 8:00 am, and the same group drive to road 2 at 8:05 am. The driving behavior of the same group might be uniform, thus, the data distributions (e.g., vehicle speed data) of road 2 (8:05) have high correlations with those of road 1 (8:00).
    - Finally, the hyperparameter $m$ is used to define the scope, e.g., if $m=2$, then we consider each node at $T_i$ has high correlations with its neighbors at $\{T_{i-2}, T_{i-1}, T_i, T_{i+1}, T_{i+2}\}$, and would aggregate the former with the latter.

Reviewer comment (Experiments):
It will be good to run some comparative experiments with a few representative graph convolutional GPs.

Author response:
Thank you again for your recognition of our work! We will include the discussion in our experiments.

Reviewer comment (Experiments):
My main concern is that the introduction of virtual nodes does not add additional information to the dataset. Consequently, my intuition is that this strategy is more effective when the dataset size is small. In the case of the benchmark datasets, which contain a limited number of spatial data points (ranging from a maximum of 883 to a minimum of 80), the approach seems advantageous. However, in scenarios where there is a larger denser spatial dataset, it's worth considering whether the issue of sparsity remains as significant. Would the gap in sparsity still present a challenge in extensive populated spatial data environments? But again, I am not a domain expert for inductive spatio-temporal kriging methods based on GNN, maybe this is indeed the usual dataset size for this line of work. I am happy to change my score if this gets justified.

Author response:
> **W1**: The sparsity of deployed sensors.

Thanks for this comment. Our responses are as follows:

As written in the 1st paragraph of introduction ("Nevertheless, due to the **high cost of devices and maintenance expenses**, the actual **sparsely deployed sensors**..."), we target the kriging setting where the sensors are sparsely deployed (mainly for cost saving considerations). This setting has been studied in many existing kriging methods, including IGNNK (Wu et al., 2021a), LSJSTN (Hu et al., 2021) and INCREASE (Zheng et al., 2023). For this setting, the **graph gap issue** would remain valid. We further provide a few **practical application scenarios** for justifying the targetted setting.
   1. The PEMS07 dataset comes from the Caltrans Performance Measurement System (PeMS) deployed in California (in use). It covers a large district, but the average node degree is 1.79 for the full graph (883 nodes), which means the graph is indeed locally sparse.
   2. Similarily, the AQI-36 dataset is collected from an air quality monitoring application, where only 36 stations are installed for the whole Beijing City.
   3. In our own business data of traffic, only 71 traffic sensors (to measure traffic flow) are sparsely installed to a large region, centered at Jinghu Amusement Park, Shaoxing, Zhejiang, China. The average node degree is less than 3.
   4. In many water quality management systems, the sensors are sparsely installed. For example, the Rhein river in Europe only has 2-3 sensors in each city it flows through.
   5. In a landslides monitoring application, there are only 7 landside tilt sensors installed in Yushan mountain in Taiwan.

We are not aware of any existing studies/datasets/practical scenarios, which target the kriging task with desnsely distributed sensors. We believe the kriging with densely distributed sensors would naturally be an easier task than the one targeted in this paper since each node would have more neighbors to leverage for the kriging task.

Reviewer comment (Experiments):
When aggregating spatio-temporal features from neighboring nodes, is it worth exploring some attention mechanism to gain more global information (especially with so many timesteps in the datasets)?

Author response:
> **Q1**: Use attention mechanism for global spatio-temporal features aggregation.

Thank you for your suggestion! Our understanding of the suggested idea is as follows - in case we misunderstood the idea, please kindly let us know. Suppose node-1's neighbors are node-2, 3, 4, and their data from 24 consecutive timestamps are available in a data sample. Your suggested idea is to use attention to aggregate node-1's features (in 1 of 24 timestamps) with node-2, 3, 4's features (from all 24 timestamps), so as to capture global dependency (in the temporal dimension).

Our responses are as follows:
- **Temporal scope (from local to global)**: We have studied the effects of the temporal scope, which we represent with a parameter $m$ (i.e., the information over $2 \times m + 1$ timestamps is aggregated). According to the results shown in Table 11, we can observe that larger $m$ (more global temporal information) cannot guarantee better performance.
- **GCN v.s. GAT/Attention**: When we started this work, we also thought about which to use among GCN, GAT or standard Self-Attention (without adjacency matrix). We finally chose GCN, and we explain as follows:
    - GAT and Self-Attention would firstly perform projection operations to project input features to $Q$, $K$ and $V$ in attention blocks. However, we think such projections cannot work well in spatio-temporal kriging. This is because in kriging, observed nodes would consistently have loss supervisions, while target nodes do not have loss supervisions all the time. Hence, the complex projections learned on observed nodes may not be suitable for target nodes during inference since it would get overfitting on observed nodes.
    - We conduct experiments by replacing GCN (M-0 of Table 5) with GAT and Self-Attention under the same condition. We show the MAE scores on METR-LA dataset in the table below (we've already tuned the hyperparameters). We could see that attention-like methods cannot outperform GCN-like methods.

|Method|MAE|
|-|-|
|GCN|**6.7597**|
|GAT|7.3220|
|Attention|8.2679|