Paper: BayOTIDE: Bayesian Online Multivariate Time series Imputation with functional decomposition (ICLR-2024)

Reviewer comment (Experiments):
The evaluation approach remains weak: I am not sure the considered data sets are varied enough, as missing data patterns considered in the paper are quite limited.

Author response:
We agree that more missing rate settings will help to better evaluate the performance of the proposed method. However, we argue that the main contribution of **BayOTIDE**, the first Bayesian online imputation model, is mainly on the novel methodological and theoretical aspects, instead of the performance gain over large-scale datasets under different missing rate settings. Similar method-driven work in this area, like **CSDI** [ Tashiro, Neurips 2021],  **CSBI** [Chen, ICML 2022], and **TIDER** [Liu, ICRL 2023] all only evaluate the performance under no-more-three missing rate settings.  

Thus, we only evaluate the performance under two missing rate settings in the paper. We will add statements to clarify this point in the revised version.

Reviewer comment (Evaluation):
I cannot give a rating above 2 for contribution because of limited evaluation (i.e., it is unclear how well the proposed approach performs in a more general setting). The same holds for the general rating.

Author response:
We agree that more missing rate settings will help to better evaluate the performance of the proposed method. However, we argue that the main contribution of **BayOTIDE**, the first Bayesian online imputation model, is mainly on the novel methodological and theoretical aspects, instead of the performance gain over large-scale datasets under different missing rate settings. Similar method-driven work in this area, like **CSDI** [ Tashiro, Neurips 2021],  **CSBI** [Chen, ICML 2022], and **TIDER** [Liu, ICRL 2023] all only evaluate the performance under no-more-three missing rate settings.  

Thus, we only evaluate the performance under two missing rate settings in the paper. We will add statements to clarify this point in the revised version.

Reviewer comment (Experiments):
The approach considers trend and seasonality explicitly. I am not sure any of the compared benchmarks explicitly consider these channels of the time series. Therefore, I would at least include a standard multi-output GP framework with linear + periodic kernels or a spectral kernel (the implementation of the latter should be available in gpytorch). In particular, on the example represented in Figure 1 I would expect these standard methods to perform well.

Author response:
Good point. We do agree that the multi-output GP with linear + periodic kernels can work well in the simulation study. However, we argue that even with linear + periodic kernels,**multi-output GP can not handle the real-world long-time series. It's because the multi-output GP is a full GP, which has $O(T^2N^2)$ space cost and $O(T^3N^3)$ time cost**. where $T$ is channel number and $N$ is the number of timestamps. Thus, it's not scalable to the real-world long time series.

 To make the multi-output GP work on the real-world long time series, we have to **patch the long time series into short segments**, and then train the multi-output GP on each segment (**The segment size is $30 \times 30$ in our experiments**). The patching makes the kernel can not capture any long-term dependency, like seasonal dependency, and the performance will be similar or worse than the multi-output GP with RBF kernels. To verify this, we compare the performance of multi-output GP with RBF kernels and multi-output GP with linear + periodic kernels on the *traffic-guangzhou* dataset. The results are shown in the following table:

| Observed-ratio=70% | RMSE | MAE | CRPS |
| ------ | ---- | --- | --- | 
| **BayOTIDE** | 3.724 | 2.611 |  0.053 | 
| **MT-GP-RBF** | 4.471 | 3.223 |  0.082  | 
| **MT-GP-linear+periodic** | 8.673 | 7.224 |  0.252 |

Reviewer comment (Writing):
There are typos and indentation issues in the paper.

Author response:
Thanks for the suggestion! We will polish the presentation quality in the revised version.

Reviewer comment (Experiments):
Although there are no existing online multivariate imputation model, the comparison with online univariate & probabilistic imputation models, e.g. state-space model with Kalman filtering, can be included in the experiment.

Author response:
Good suggestion! We add the comparison with a single state-space model (**single-SS**) with Kalman filtering on *traffic-guangzhou* The results are shown in the following table:

| Observed-ratio=70% | RMSE | MAE | CRPS |
| ------ | ---- | --- | --- | 
| **BayOTIDE** | 3.724 | 2.611 |  0.053 | 
| **single-SS** | 5.152 | 4.691 |  0.132 | 

| Observed-ratio=50% | RMSE | MAE | CRPS |
| ------ | ---- | --- | --- | 
| **BayOTIDE** | 3.820 | 2.687 |  0.055 | 
| **single-SS**| 5.558 | 4.821 |  0.173 | 

It's clear that the **single-SS** model is much worse than the proposed method. The reason is that the **single-SS** model is not able to capture the cross-channel dependency, which is crucial for multivariate time series imputation.

Reviewer comment (Evaluation):
The main advantage of considering multivariate time series is that the correlation between dimensions can be captured. It seems that all the evaluation metrics are univariate. I suggest the authors to also include multivariate metrics (e.g., energy score [1] and sum CRPS [2]) in the experiments to evaluate if the propose method can better capture the correlations than baselines.

Author response:
We do agree that the multivariate metrics are more informative than univariate metrics, but we clarify that the *energy score*[1] will reduce to the *CRPS* ($\beta=m=1$) and *RMSE* ($\beta=2$), which are already included in the paper. For the *sum CRPS* used in [2], it seems to be the sum over the CRPS overall evaluation points. As we report the mean of the CRPS overall evaluation points, the *sum CRPS* will have the same effect as the *CRPS* in our case. However, we do agree that multivariate metrics may help to better understand the performance under multivariate setting and will investigate it in future work. 


[1] Gneiting, Tilmann, and Adrian E. Raftery. "Strictly proper scoring rules, prediction, and estimation." Journal of the American Statistical Association 102.477 (2007): 359-378.


[2] Kan, K., Aubet, F. X., Januschowski, T., Park, Y., Benidis, K., Ruthotto, L., & Gasthaus, J. (2022, May). Multivariate quantile function forecaster. In International Conference on Artificial Intelligence and Statistics， 2022

Reviewer comment (Experiments):
The model size (e.g., number of parameters) is not reported in the experimental results. It is recommended to also report model size. The proposed model seems to be outperforming, but could it be because that it is using a larger model?

Author response:
Good point! We take the *traffic-guangzhou* dataset(size: 213 channels, 500 timestamps) as an example, and show the model size comparison of *BayOTIDE* and *CSDI*[1]—the state-of-the-art probabilistic imputation method.  

 As *BayOTIDE* is a factorization-based method, the model size is determined by the number of components. Corresponding to the hyper-parameter setting in Table 4, we set $D_r + D_s = 40$ components. Then the weights U's size is $40 \times 213$, and the components V's size is $40 \times 500 \times 2$(the mean and diag var). Thus, the total model size is $40 \times (213+500 + 500) = 48520$.

*CSDI* is a diffusion-based method, the model size training memory is determined by the number of diffusion steps and the structure of the deep-based score estimator. The default structure *CSDI*'s(Figure 6 in CSDI paper) includes a spatial-temporal transformer with 5 layers and 8 heads and fully connected layes. The parameters also include the embedding(dim=128) of timestamps and features, and the diffusion step embedding. Thus, the total model size is at least $5 \times 16 \times 2 \times (500\times3+128\times3) + 128 \times (213 + 500) =  388224$. 

Thus, the model size of *BayOTIDE* is much smaller than *CSDI*. The performance gain of *BayOTIDE* over *CSDI* is mainly due to the factorization-based formulation, not the model size. In the *TIDER* paper[2], which takes a similar factorization formulation with *BayOTIDE*, the authors reported the memory usage of TIDER and CSDI (Figure 4 in TIDER paper). The memory usage of TIDER is much smaller than CSDI.

[1]: Tashiro, Yusuke, et al. "Csdi: Conditional score-based diffusion models for probabilistic time series imputation." Advances in Neural Information Processing Systems 34 (2021): 24804-24816.

[2]: LIU, SHUAI, et al. "Multivariate Time-series Imputation with Disentangled Temporal Representations." The Eleventh International Conference on Learning Representations. 2022.

Reviewer comment (Writing):
The writing is rough, with some unclear and insufficient descriptions.

Author response:
Could you please specify which part is unclear and insufficient? We will gladly add statements to clarify the unclear and insufficient parts.

Reviewer comment (Theory):
Unclear theoretical support.

Author response:
The factorization-based formulation of the proposed method is aligned with Bayesian PCA[1], which is a well-known and classical Bayesian method with theoretical support.  The state-space GP prior[2] is known to be highly flexible/expressive for function estimation and has been used in numerous applications, including time series modeling [3]. The EP-based inference[4] is also a well-known approximate inference method with theoretical gaurentees on provable convergence. Thus, the proposed method is well-supported by the theoretical results of the above methods.

Ref:

[1] Bishop, Christopher. "Bayesian PCA." Advances in neural information processing systems 11 (1998).


[2] Rasmussen, Carl Edward. "Gaussian processes in machine learning." Summer school on machine learning. Berlin, Heidelberg: Springer Berlin Heidelberg, 2003. 63-71.


[3] Roberts, Stephen, et al. "Gaussian processes for time-series modeling." Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 371.1984 (2013): 20110550.


[4]:Minka, Thomas P. "Expectation propagation for approximate Bayesian inference." arXiv preprint arXiv:1301.2294 (2013).

Reviewer comment (Presentation):
Section 3.2 does not seem to play any role. Perhaps this paper has been rejected before, and the authors just wanted to add a "modern"-looking section. I got it. But it looks hardly related.

Author response:
The state-space GP we introduced in section 3.2  actually plays a key role in the proposed method. We use that state-space GP to model the trend and seasonal components in a continuous and functional manner with linear time cost. The chain structure of the state-space GP also enables online learning with KF. Specifically, we have to use the chain-structured prior (eq 2 in section 3.2) to replace the full GP prior (eq 7 in section 4.2) in the proposed method (the paragraph under eq7). We apologize for the unclear presentation and will add statements to highlight the role of the state-space GP in the proposed method.

Reviewer comment (Writing):
Very poor proof-reading quality. Basic latex commands such as \eqref, \cite, etc. are not properly used. I know this might be re-re… submission, but PLEASE be respectful to the reviewers by meticulously proof-reading the manuscript before submission.

Author response:
Thanks for the suggestion! We will polish the presentation quality in the revised version. Actually, it's the first time we submit this paper to a peer-reviewed conference, so we are pretty grateful to get your valuable feedback and will continue to improve the presentation quality in the future :)

Reviewer comment (Theory):
How does the well-known non-identifiability with respect to the unitary transformation in the UV-factorization form (1) take effect on the result?

Author response:
Good point! We do acknowledge that the non-identifiability of unitary transformation will make learned components(V) and weights(U) lose the nice properties of uniqueness. However, we argue that the loss of non-identifiability is not a big issue in our case. The reason is that the components(V) is assigned with functional priors (GP), and are doomed to be non-linear and periodic. Then, even the non-identifiability effect takes place, saying the components(V) do not match the right "scale" but still have the right "shape", the weights(U) will be able to compensate for the "scale" effect. Actually, in our simulation study(Section 5.1), we do observe the non-identifiability effect, but when combining learned components(V) and weights(U), we could still get the right trend and seasonal components with the weights (Figure 1, (b)-(e)). We will add statements to clarify this point in the revised version.

Reviewer comment (Theory):
I am not clear how the almost linear trend could be separated in the result presented in Fig.1. How did the kernel expansion with the GPs produce the linear-looking trend component? What is the intuition behind it?

Author response:
Good question! The short answer is that the separation of linear trend is not learned by the GP, but by the weights(U). Specifically, we set four GP components for $V(t)$ in the simulation study, and the first component is a Matern kernel, and the last three components are periodic kernels. Thus, the separated linear trends shown in Figure 1 (b)-(e) are actually from one GP kernel (Matern kernel), but with four different weights(U). The intuition behind it is that the different channels of dynamics may share similar trend components but with different weights. The message-passing algorithm in the proposed method is able to learn the channel-wise weight precisely with conditional moment-matching.

Reviewer comment (Novelty):
Although I support accepting this paper, I am not 100% sure about the novelty. Bayesian PCA-based imputation is well-established. I know the main novelty comes from the time-series part, but your paper was not very clear about the "delta" from the pre-deep learning imputation works. To defend your work, please elaborate on the novelty in light of existing works. I just want to help you --- I suspect many ICLR reviewers do not have a strong understanding of the machine learning basics such as Bayesian PCA, and hence, papers like this one tend to receive unfairly low ratings.

Author response:
Great point! We just follow your suggestion and add general statements to highlight the novelty of the proposed method in the response overview. Please refer to sections **Novelty and contribution on time series imputation side** and  **Novelty and contribution on Bayesian learning side** in the response overview.