Paper: Conditional Information Bottleneck Approach for Time Series Imputation (ICLR-2024)

Reviewer comment (Theory):
In equation 4, both the entropy term and the KL term are dropped to compute the lower bound. Will this bound be too loose? After dropping, the lower bound looks like a reconstruction loss rather than mutual information.

Author response:
Please allow us to briefly summarize the discussion in [1] about the boundeness of (4). The first term is the entropy of the complete features at time step $t$. This quantity is independent from the parameters of our interest, so does not have any effect on the boundness. The second term is the KL-divergence between two distribution, $p(x_t|z_t)$ and $p_\theta(x_t|z_t)$, where $p_\theta(x_t|z_t)$ is a variational approximation of $p(x_t|z_t)$. The KL-divergence term will be small, resulting in a tight bound, if the assumption on the prior distribution $p(x_t|z_t)$ (in our case, a Gaussian distribution) is reasonable and the model achieves good reconstruction performance. For the detailed derivation and interpretation of (4), please refer to [1].

Reviewer comment (Reproducibility):
The paper doesn't mention the number of learnable parameters for each methods.

Author response:
We have provided a more detailed specification of hyperparameters, including the number of learnable parameters, in Appendix C of the revised version (page 20). To ensure a fair comparison, we ensure that the magnitudes of the number of parameters were set to be equivalent among the evaluated deep learning methods throughout our experiments.

Reviewer comment (Experiments):
I'd like to see comparison with another two baselines mTAN [1] and CSDI [2].

[1] Multi-Time Attention Networks for Irregularly Sampled Time Series  
[2] CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation

Author response:
We have included the following **four** cutting-edge deep learning-based time series imputation  methods: two predictive methods including an RNN-based method, BRITS [1], and a transformer-based methods, SAITS [2], and two generative methods including an autoencoder-based method, mTANs [3], and a diffusion-based method, CSDI [4]. 

Due to the limited rebuttal period, we have provided results for the Beijing, USLocal, and Physionet2012 datasets in the revised manuscript. Please find the result table on the general response and the revised manuscript. To summarize, our method achieves the best performance on the USLocal dataset, which has the longest time steps (T=168) and clear periodicity. For the Beijing dataset, our method performs slightly better than or comparably to state-of-the-art predictive methods. Here, CSDI (with n_samples=25) demonstrates the best performance but with a computation time approximately 2000 times slower than ours due to the required sampling process. For Physionet dataset, our method outperforms the cutting-edge benchmarks in terms of AUROC suggesting that our approach recovers semantically meaningful information for downstream tasks.

Reviewer comment (Experiments):
The paper exclusively focuses on Variational Autoencoder (VAE) models grounded in Information Bottleneck (IB) theory. While this approach is well-articulated, it is notable that other kinds of generative models, for instance, ODE-based models and diffusion models, have demonstrated remarkable performance in the field of time series imputation. The absence of a discussion or comparison with SOTA models in the related work and experimental sections is conspicuous, making it less convincing regarding its contribution to the irregular time series imputation task.

Author response:
### **[Q1, W1] Comparison with other methods**

Please allow us to first compare our method with predictive methods (including transformers) and diffusion methods, respectively. Comparing with predictive methods, our method supports multiple imputations and uncertainty quantification. This is extremely important in real-world data, because the confidence on individual imputed datapoint helps real-world decision-making, such as prescription for patients or weather forecasting. Comparing with diffusion methods, our method is much faster and allows us to introduce inductive bias about the underlying temporal dynamics, which is essential for real-time decision-makings and long-term periodic imputation. In our experiment, our method is approximately 2000 times faster than CSDI (with nsamples=25) under the same order of magnitude for the number of parameters. Our method could be more useful on the field that requires real-time imputation, such as weather forecasting or healthcare. Moreover, as shown in our experiments, we can inject inductive bias in terms of temporal kernels. This could provide more effective way to impute extremely long time series which has periodicity such as weather data, and our experiments on the USLocal dataset also supports this.

We have included the following **four** cutting-edge deep learning-based time series imputation  methods: two predictive methods including an RNN-based method, BRITS [1], and a transformer-based methods, SAITS [2], and two generative methods including an autoencoder-based method, mTANs [3], and a diffusion-based method, CSDI [4]. 

Due to the limited rebuttal period, we have provided results for the Beijing, USLocal, and Physionet2012 datasets in the revised manuscript. Please find the result table on the general response and the revised manuscript. To summarize, our method achieves the best performance on the USLocal dataset, which has the longest time steps (T=168) and clear periodicity. For the Beijing dataset, our method performs slightly better than or comparably to state-of-the-art predictive methods. Here, CSDI (with n_samples=25) demonstrates the best performance but with a computation time approximately 2000 times slower than ours due to the required sampling process. For Physionet dataset, our method outperforms the cutting-edge benchmarks in terms of AUROC suggesting that our approach recovers semantically meaningful information for downstream tasks.

Reviewer comment (Novelty):
While the paper introduces a novel regularization method in the context of time series imputation, the innovation appears to be somewhat incremental. The approach can be perceived as a clever technique or ‘Trick’ rather than a substantial paradigm shift.

Author response:
### **[Q2,3] Discussions on Regularization**

Before answer to these, we would like to clarify the difference between IB and CIB, in terms of how the CIB can alleviate regularization effect. It is true that CIB optimizes smaller amount of information, but it does not mean that the CIB imposes loose regularization. Rather, the CIB **guides** the direction of optimization, towards to preserve temporal dynamics, while having concise information.

As illustrated in Figure 1(A) and (5), the absolute quantity of information that CIB optimizes is smaller than what the traditional IB does, i.e., ($I(Z_t; X^o_{1:T}) = I(Z_t; X^o_{1:T}) - I(Z_t;X^o_{\setminus t}$), $I(\cdot;\cdot) ≥ 0$). However, the more crucial aspect is the direction in which regularization guides the optimization, rather than the value itself. Since the traditional IB focuses on having concise information as illustrated in Figure 1(A), the direction of regularization does not consider preserving information about temporal dynamics from other time steps. Contrarily, the regularization using CIB effectively explicitly maintains information from the remaining time steps as derived in (5). 

We can provide the measured information $I(Z_t;X^o_{\setminus t}$) to demonstrate the conservation of temporal dynamics information. However, that metric may not offer useful insights as it is the objective we optimize. Instead, to showcase how well the temporal information is preserved and affects the downstream task, we use results on the RotatedMNIST dataset, where arbitrary timestep is totally missing such that the model must infer from the remaining time series. Therefore, we believe performance improvement on interpolation and extrapolation provides a proxy metric / criteria on how CIB effectively addresses this issue.

Reviewer comment (Experiments):
What are the unique aspects of the CIB approach compared to other methodologies, including ODE, transformers, and diffusion models? Can you provide a more comprehensive evaluation, including comparisons with state-of-the-art models like ODE-based models and diffusion models, to demonstrate the performance of your model?

Author response:
### **[Q1, W1] Comparison with other methods**

Please allow us to first compare our method with predictive methods (including transformers) and diffusion methods, respectively. Comparing with predictive methods, our method supports multiple imputations and uncertainty quantification. This is extremely important in real-world data, because the confidence on individual imputed datapoint helps real-world decision-making, such as prescription for patients or weather forecasting. Comparing with diffusion methods, our method is much faster and allows us to introduce inductive bias about the underlying temporal dynamics, which is essential for real-time decision-makings and long-term periodic imputation. In our experiment, our method is approximately 2000 times faster than CSDI (with nsamples=25) under the same order of magnitude for the number of parameters. Our method could be more useful on the field that requires real-time imputation, such as weather forecasting or healthcare. Moreover, as shown in our experiments, we can inject inductive bias in terms of temporal kernels. This could provide more effective way to impute extremely long time series which has periodicity such as weather data, and our experiments on the USLocal dataset also supports this.

We have included the following **four** cutting-edge deep learning-based time series imputation  methods: two predictive methods including an RNN-based method, BRITS [1], and a transformer-based methods, SAITS [2], and two generative methods including an autoencoder-based method, mTANs [3], and a diffusion-based method, CSDI [4]. 

Due to the limited rebuttal period, we have provided results for the Beijing, USLocal, and Physionet2012 datasets in the revised manuscript. Please find the result table on the general response and the revised manuscript. To summarize, our method achieves the best performance on the USLocal dataset, which has the longest time steps (T=168) and clear periodicity. For the Beijing dataset, our method performs slightly better than or comparably to state-of-the-art predictive methods. Here, CSDI (with n_samples=25) demonstrates the best performance but with a computation time approximately 2000 times slower than ours due to the required sampling process. For Physionet dataset, our method outperforms the cutting-edge benchmarks in terms of AUROC suggesting that our approach recovers semantically meaningful information for downstream tasks.

### **[Q3] Generalizability**

For generalizability, CIB provides more robust way to consider temporal dynamics. GP-VAE, which uses Gaussian Process under IB framework, lacks generalizability since it requires prior knowledge on the temporal smoothness (hyperparameters of Gaussian processes) and optimizes remaining parameters depending on that setting. Therefore, GP-VAE is vulnerable to input with totally different temporal dynamics. However, uniform CIB which uses no temporal kernels, provides effective way to impute time series without assumptions on the temporal dynamics of inputs (please refer to performances of (uniform) versions).

Reviewer comment (Evaluation):
The paper claims that the CIB approach can mitigate the excessive regularization associated with IB. However, there seems to be a lack of quantitative analysis to substantiate this claim. It would be beneficial if the authors could provide explicit metrics or criteria to measure the extent of regularization and demonstrate how CIB effectively addresses this issue.

Author response:
### **[Q2,3] Discussions on Regularization**

Before answer to these, we would like to clarify the difference between IB and CIB, in terms of how the CIB can alleviate regularization effect. It is true that CIB optimizes smaller amount of information, but it does not mean that the CIB imposes loose regularization. Rather, the CIB **guides** the direction of optimization, towards to preserve temporal dynamics, while having concise information.

As illustrated in Figure 1(A) and (5), the absolute quantity of information that CIB optimizes is smaller than what the traditional IB does, i.e., ($I(Z_t; X^o_{1:T}) = I(Z_t; X^o_{1:T}) - I(Z_t;X^o_{\setminus t}$), $I(\cdot;\cdot) ≥ 0$). However, the more crucial aspect is the direction in which regularization guides the optimization, rather than the value itself. Since the traditional IB focuses on having concise information as illustrated in Figure 1(A), the direction of regularization does not consider preserving information about temporal dynamics from other time steps. Contrarily, the regularization using CIB effectively explicitly maintains information from the remaining time steps as derived in (5). 

We can provide the measured information $I(Z_t;X^o_{\setminus t}$) to demonstrate the conservation of temporal dynamics information. However, that metric may not offer useful insights as it is the objective we optimize. Instead, to showcase how well the temporal information is preserved and affects the downstream task, we use results on the RotatedMNIST dataset, where arbitrary timestep is totally missing such that the model must infer from the remaining time series. Therefore, we believe performance improvement on interpolation and extrapolation provides a proxy metric / criteria on how CIB effectively addresses this issue.

## Rebuttal 4 (ID: cOsZ8TF9V9)
We appreciate Reviewer yyLt, wRxf, and Vm5N on the comment.

In addition to the qualitative assessment on $I(Z_t;X^o_{\setminus t})$ shown in Figure 1 and empirical performances in Section 5, we compare the approximated quantity on $I(Z_t;X^o_{\setminus t})$ using (9) to assert that regularization term in our CIB objective indeed preserves more temporal information. Please note that, as derived in (A.4), the lower bound is given by $I(Z_t;X^o_{\setminus t}) \ge \log(N) - L^3_\phi$, which depends on the batch size $N$ and the objective $L^3_\phi$. Table below shows the values of  $L^3_\phi$, indicating that a lower value corresponds to more preserved information. 

| Dataset  | VAE | HI-VAE | GP-VAE | Ours (Uniform) | Ours (Kernel) |
|--|--|--|--|--|--|
| RotatedMNIST | $3.619 \pm 0.005$ | $3.577 \pm 0.008$ | $3.435 \pm 0.001$ | $\mathbf{3.428} \pm 0.006$ | $\mathbf{3.418} \pm 0.002$ | 
| Beijing | $3.950 \pm 0.015$ | $3.822 \pm 0.012$ | $3.762 \pm 0.012$ | $\mathbf{3.700} \pm 0.008$ | $\mathbf{3.686} \pm 0.006$ | 


This results demonstrates that regularization in our CIB objective preserves more information in $I(Z_t;X^o_{\setminus t})$ based on the approximation. We also added the Appendix B.3 in the revised manuscript, illustrating above experiments and results.

Again, thank you for the feedback and we look forward to further discussions if you have additional questions or suggestions!

Reviewer comment (Theory):
The motivation of using conditional term $X^o_t | X^o_{\t}$ should be further explained. Why using the conditional term can tackle the issue of conventional IB?

Author response:
Please allow us to explain the motivation behind the conditional term using a compelling example illustrated in Figure 1.

The conventional IB comprises two terms: one reconstruction and one regularization. As illustrated in Figure 1A left, a simple extension of the conventional IB to time series imputation aims at maximizing the information related to $X_t$ (red) while minimizing the information from the entire time series input $X^{o}_{1:T}$ (blue). However, such regularization can be overly strong, impeding the model's ability to accurately impute missing values by capturing the underlying temporal dynamics. As depicted in Figure 1B, the conventional IB may fail in interpolation or extrapolation scenarios where the model needs to gather information about the temporal dynamics

We tackle the problem of \textit{loss of temporal dynamics} by modifying the regularization term in the IB framework. Specifically, we propose CIB to preserve information from the remaining timesteps. As illustrated in Figure 1A right, the CIB does not minimize the information from $X_{\setminus t}^o$, when maximizing information that is required for reconstructing $X_t$. This is mathematically confirmed by (5), where applying the chain rule gives that the conditional regularization encourages conciseness by minimizing $I(Z_{t}; X^o_{1:T}$), while capturing the underlying temporal dynamics by maximizing $I(Z_{t}; X^{o}_{\setminus t}$).

Reviewer comment (Presentation):
A figure of the whole framework will be helpful for understanding this work.

Author response:
We have added a figure illustrating our framework in Appendix C (page 19).  We hope this helps readers to better understand our work.

Reviewer comment (Experiments):
The compared baselines are out of date. There exist massive works on time series imputation, the authors should compare their method with more recent works.

Author response:
We have included the following **four** cutting-edge deep learning-based time series imputation  methods: two predictive methods including an RNN-based method, BRITS [1], and a transformer-based methods, SAITS [2], and two generative methods including an autoencoder-based method, mTANs [3], and a diffusion-based method, CSDI [4]. 

Due to the limited rebuttal period, we have provided results for the Beijing, USLocal, and Physionet2012 datasets in the revised manuscript. Please find the result table on the general response and the revised manuscript. To summarize, our method achieves the best performance on the USLocal dataset, which has the longest time steps (T=168) and clear periodicity. For the Beijing dataset, our method performs slightly better than or comparably to state-of-the-art predictive methods. Here, CSDI (with n_samples=25) demonstrates the best performance but with a computation time approximately 2000 times slower than ours due to the required sampling process. For Physionet dataset, our method outperforms the cutting-edge benchmarks in terms of AUROC suggesting that our approach recovers semantically meaningful information for downstream tasks.

Reviewer comment (Reproducibility):
It would be appreciated if the authors could provide code.

Author response:
We will upload the entire source code on Githhub upon acceptance.

Reviewer comment (Writing):
As many papers dealing with temporal data, sometimes notation becomes a bit complex and makes the reading more difficult.

Author response:
[W1] We provide a notation table in Appendix E.

Reviewer comment (Reproducibility):
I believe more information about the experimental section could be provided. That, is, for HealingMNIST and RotatedMNIST, which kind of CNN-based encoder/decoder architecture is being used; for Physionet, which kind of networks are used? There is no information about this, especially for the proposed method, since for baselines this can be accessed in their referred papers. And this is quite important when asking why the proposed method outperforms any other baseline. This leads me to the following point.

Author response:
### **[W2] Architecture specification**
In our original submission, we specified architecture specifications in Appendix C.1 due to the page limit. We have added a guiding sentence in the revised manuscript. 

Here, we briefly summarize implementation details. To highlight the contribution of our novel objective, our implementation adopts most of the settings used in GP-VAE [1]. For the HealingMNIST and RotatedMNIST datasets, we employ one layer of a CNN with a kernel size of 3, followed by a stochastic encoder consisting of two-layer LSTM. The decoder is a two-layer feed-forward network. The primary architectural distinction of our method from GP-VAE is the stochastic encoder; GP-VAE utilizes dimension-wise encoding to employ a Gaussian process prior, while our method utilizes latent variables at each time step similar to conventional VAEs.

Reviewer comment (Reproducibility):
There is no reference for code availability to reproduce the experiments. Will you upload the code to Git Hub or any available repository? This is rather important for the sake of reproducibility. And given that the results are (fairly always) better than baselines, this should be a must-have for this paper.

Author response:
[W4] We will upload the whole source code to Github.

Reviewer comment (Novelty):
I find the paper rather novel in terms of the proposal of the CIB objective. However, in terms of the VAE-based model, I believe there is much inspiration coming from GPVAE [1], due to the idea of including an extra kernel to further enhance the learning of the temporal dynamics. Is this the case? Should the authors be more worried about properly referencing this work?

Author response:
### **[W5] Relation to GP-VAE**
We appreciate the reviewer for bringing this important point to our attention and apologize for the unclear comparison with GP-VAE. As you mentioned, we admit that our method and GP-VAE share some philosophy on how to tackle time series imputation and use temporal kernels. However, there are key differences between our work and GP-VAE. Under the IB framework, GP-VAE models the smooth temporal evolution of latent variables by replacing the conventional unit Gaussian prior with a GP prior specified by temporal kernels.
However, our work is motivated by the inherent limitation of the IB in discarding temporal information (as discussed in Section 3.1) and proposes a novel CIB principle that alleviates the strict regularization of IB. In our approach, temporal kernels are **optionally** adopted to introduce an inductive bias about the underlying temporal dynamics. This is different from GP-VAE, where the temporal kernel is introduced specifically for the GP prior. 

We have included this detailed discussion about the relation to GP-VAE in the Related Work of the revised manuscript (page 7).

Reviewer comment (Experiments):
An ablation study should be included to determine the importance of each contribution in the paper. That is, i) how the different terms in the proposed objective influence the optimization ii) how important is it to introduce the kernel — without the kernel, would the results approach the ones from GPVAE, or does the proposed model still outperform other baselines? I think it is important to analyze each contribution of the paper to obtain a clear and detailed view of the behaviour of the proposed model.

Author response:
### **[W6 (ii)] Regarding the introduction of temporal kernel**

As mentioned in (Relation to GP-VAE), we optionally adopt temporal kernels to introduce an inductive bias about the underlying temporal dynamics. So, the main novelty of our work comes from how we model the time series imputation using the conditional IB framework, rather than introducing the temporal kernel. The imputation performance of our method without kernel (denoted as uniform) still demonstrates significant improvement, supporting our claim.

### **[Q3, W6 (i)] Discussions on beta and gamma - Sensitivity analysis and optimization**

As the reviewer suggested, we have included sensitivity analysis and optimization strategy  on beta and gamma. Before presenting our new results, please allow us to redefine optimization hyperparameters as the form of $\min L^1_{\phi, \theta} + \beta’ L_\phi^2 + \gamma’ L_\phi^3$, so that we can isolate effects of each regularization terms given fixed reconstruction. (Please refer to Appendix B3 for details)

**Sensitivity on $\beta’$.** Manipulating $\beta’$ is related to the regularization trade-off that a low $\beta’$ (weak regularization) can cause a lack of time series-wise information, which may result in overfitting solely on observed values while a high $\beta’$ (strong regularization) can cause loss of timepoint-wise information since all distribution will become equivalent to unit Gaussian. We have included a qualitative analysis on HealingMNIST, showing these behaviors in Figure B2(a) in in Appendix B3 (page18). 

**Sensitivity on $\gamma’$.** Since $\gamma’$ basically works as a regularization term, sensitivity on $\gamma’$ is similar to $\beta’$, but the behaviour is slightly different. For a low $\gamma’$, there is no driving force on conserving temporal dynamics (see Section 3.1), thereby causing a loss of temporal dynamics and would eventually become equivalent to HI-VAE when $\gamma’=0$. A high $\gamma’$ also can cause a loss of timepoint-wise information, since $\mathcal{L}_3$ will map all $z_t$ from the same time series into a single point (achieving maximum alignment), which means that we cannot distinguish individual timepoints. We have included a qualitative analysis on HealingMNIST, showing these behaviors in Figure B2(b) in in Appendix B3 (page18). 

**Optimization Strategy.** It is true that simultaneous optimization of $\beta$ and $\gamma$ could be difficult. However, based on the above sensitivity analysis, we provide a practical suggestion: First, we set $\gamma’=0$ and find optimal $beta’$ (which is HI-VAE). Second, under a similar range of order of magnitude of $\beta’$, find the optimal $\gamma’$. 

We have included above discussions in Appendix B3 of the revised version.