Paper: Biased Temporal Convolution Graph Network for Time Series Forecasting with Missing Values (ICLR-2024)

Reviewer comment (Experiments):
missing principled baseline: forecast based on the time series **and the imputation mask**.

Author response:
Thanks for your suggestion. We have **changed the experimental setting to make the imputation mask available to all baseline methods**. This can indeed improve the performance of most methods, especially for those Transformer-based ones. We have **revised the manuscript** to include the corresponding analysis and discussion.

Reviewer comment (Experiments):
In the main experiment in table 2 the authors compare several forecasting models for completely observed time series after imputing zeros ("Transformer_0") and after imputing with a specific imputation method ("Transformer_t"). However, this way the information which observations have been missing gets lost, and this might be relevant ("informed missingness"). The default approach is to impute a zero **and** to add a channel carrying the imputation mask (i.e., as "1" if the value was observed and a "0" if it originally was missing and now has been imputed). This way, the information about missingness is not lost. It would be really important to add these principled baselines to make sure that the specific way the proposed models deal with the missing values really is causing the observed differences.

Author response:
Thanks for your suggestion. We have **changed the experimental setting to make the imputation mask available to all baseline methods**. This can indeed improve the performance of most methods, especially for those Transformer-based ones. We have **revised the manuscript** to include the corresponding analysis and discussion.

Reviewer comment (Novelty):
Generally speaking, BiaTCGNet appears to be a result of (1) an almost direct application of a TCN with a straightforward modification to account for partial observations (Liu et al., 2018), and (2) leveraging a conventional GCN for learning of an adjacency matrix relying on two node embedding sets so as to capture asymmetric spatial correlations. In that regard, the novelty of this work may be considered incremental. Therefore, I would encourage the authors to further clarify and/or elaborate on the novelty of the two modules within BiaTCGNet.

Author response:
We would like to **highlight the innovations** of our proposed model from three aspects. 
- For the PartialTCN module, we introduce the **instance PartialTCN** that focuses on capturing the temporal dependencies whereas it leaves the instance correlation for Biased GCN, this stands in contrast to the PartialCNN which relies on the convolution to capture the correlation from the two dimensions, and thus allowing for the **exploration of temporal patterns that are invariant to instances**. 
- For the Biased GCN module, we innovate the GCN by **adding a bias term (prior knowledge) to account for different missing patterns** (illustrated by Equation-9 in the revised manuscript), which will then be used to **correct the message passing strength** in the graph information diffusion. Our ablation study shows that such innovation is critical to the forecasting performance. 
- We stack multiple blocks of the two modules and design a hierarchical architecture, and propose **a new mask updating mechanism** in both modules (as shown in Equation-6 and Equation-10) to **ensure a corrected mask** when information flows from bottom to up.

We have **updated both the content and Figure-1 (b)** in the revised manuscript to highlight these innovations.

Reviewer comment (Experiments):
There seems to be a fairly recent work [C1] on attention-based memory networks for joint modeling of local spatio-temporal features and global historical patterns in multivariate time series with missing values. Moreover, the forecasting problem formulation in [C1] seems to be consistent with the one considered in this work. Therefore, I would suggest that the authors consider comparing the GCN-M method from [C1] with BiaTCGNet. Alternatively, I would ask the authors to provide the specific reason as to why this method has not been included among the baselines?

[C1] Zuo, J., Zeitouni, K., Taher, Y., & Garcia-Rodriguez, S. (2023). Graph convolutional networks for traffic forecasting with missing values. Data Mining and Knowledge Discovery, 37(2), 913-947.

Author response:
Thanks for pointing us the missing work. The GCN-M [1] is now **included as a baseline method** in our revised manuscript and its results are available in **Section 5.3 and Sections B, C, and G**. Note that as GCN-M requires location information, it is only applicable to the Metr-LA, PEMS, and Beijing Air datasets.

References:

[1] Jingwei Zuo, Karine Zeitouni, Yehia Taher, and Sandra Garcia-Rodriguez. Graph convolutional networks for traffic forecasting with missing values. Data Mining and Knowledge Discovery, 2023.

Reviewer comment (Evaluation):
For MAE and MAPE, both are evaluation metrics for the median estimate, and for MAPE the denominator, without clarification, is assumed to be the same for all methods. However, for example in Table 2, under these two metrics, the ranking of methods can differ. This raises concern about whether the experiment setup or the metric calculation is different from what is assumed.

Author response:
Thanks for pointing us this ambiguity. We actually adopt the **MAE** as our **training loss**, and we have revised the Equation-2 to make it clear.  Regarding the ranking difference of MAE and MAPE, they are indeed the same metric up to a normalization (denominator), however, the **denominators are not a fixed constant** and change against different points (y_t). As a consequence, the **ranking of the two metrics can differ** even for the same method and dataset, and this is also observed in prior work [1, 2].

Reviewer comment (Reproducibility):
An interval is implied for all the reported results, and only in the caption from Table 2 it is mentioned that 'the results are averaged over 5 runs'. It is unclear what these 5 runs refer to. Do they refer to different random seeds for training only? It is assumed so given that the paper mentioned the train/test/validate split is based on ratio chronologically, which would fix the dataset so there is no variation from the data.

Author response:
Yes, the five-runs mean **different random seeds for training only**, i.e., we run a method for 5 rounds with different random seeds and calculate the averaged values from the 5 rounds. We have revised the paper to make it clear now.

Reviewer comment (Experiments):
Introducing missing masks to the feature map, motivated by vision tasks, makes sense; however, it would be good to compare against introducing the missing masks as input features directly so the model can be trained with the knowledge of what inputs are missing.

Author response:
Thanks for your suggestion. We have **changed the experimental setting for all baseline methods** such that they all take the missing masks as direct input features on all datasets. This can indeed improve the performance of most methods, especially for those Transformer-based ones. We have **revised the manuscript** to include the corresponding analysis and discussion.