Paper: TSGM: Regular and Irregular Time-series Generation using Score-based Generative Models (ICLR-2024)

Reviewer comment (Writing):
The procedure of generating irregular time series is not clearly written. (See questions for more details.)

Author response:
**Q3.As for irregular time series generation, how is the irregular-sampled timestamp generated in your sampling procedure?**  

We randomly drop 30\%, 50\%, and 70\% of observations from regular time-series. We followed the procedure of how to make irregular-sampling timestamps by GTGAN for fair comparison. We add a description of the procedure to the problem formulation section, Section 3.1.

Reviewer comment (Theory):
What is the time complexity of TSGM compared to baseline.

Author response:
**Q1.What is the time complexity of TSGM compared to baseline?**  

We compared the sampling time of TSGM to TimeGAN and GTGAN in Appendix H. Please see Table 13.

Reviewer comment (Reproducibility):
What is the diffusion procedure for generating irregular time series using continuous-time models as encoder?

Author response:
**Q2.What is the diffusion procedure for generating irregular time series using continuous-time models as encoder?**  

It is conceptually the same at the one for regular time-series. The only difference is whether its time interval is fixed or irregular. It is well-known that Neural CDEs are continuous RNNs [1]. So, we use Neural CDEs to deal with irregular time-series. Therefore, we train our score network by using latent variables, which is encoded by our Neural CDE-based encoder, and generate recursively latent variables as we have done in the regular time-series generation.  

[1] Patrick Kidger. Neural Controlled Differential Equations for Irregular Time Series. NeurIPS20.

Reviewer comment (Reproducibility):
As for irregular time series generation, how is the irregular-sampled timestamp generated in your sampling procedure?

Author response:
**Q3.As for irregular time series generation, how is the irregular-sampled timestamp generated in your sampling procedure?**  

We randomly drop 30\%, 50\%, and 70\% of observations from regular time-series. We followed the procedure of how to make irregular-sampling timestamps by GTGAN for fair comparison. We add a description of the procedure to the problem formulation section, Section 3.1.

Reviewer comment (Theory):
Please make a more clear explanation of how Eq. (12) is calculated.

Author response:
**Q4.Please make a more clear explanation of how Eq. (12) is calculated.  
Q5.How does the loss function related to time series data generation?**  

Unlike image generation, time-series must be generated by considering past observations. But previous researches did not consider such a conditional generation, so we had to derive objective for conditional score function, ${\nabla}log p({\textbf{x}}^s_{1:n}|\textbf{x}^0_{1:n-1})$.  

But calculating an exact value of ${\nabla}log p({\textbf{x}}^s_{1:n}|\textbf{x}^0_{1:n-1})$ is computationally prohibitive [1]. As ${\nabla}log p({\textbf{x}}^s_{1:n}|\textbf{x}^0_{1:n})$ has a definite form and follows a specific SDE (e.g., VE, VP, subVP) [2], we drive that $L_1$ is same with $L_{score}$ by constant term.  

Considering that using $\textbf{x}_{1:n}$ is computationally expensive, we use an autoencoder to encode it to latent variable.  

Once we generate $\textbf{h}_{1:N}$, we acquire total sequence by using a trained decoder.  

As a result, our loss function for the score network is $L_{score}^{\mathcal{H}}$.  

[1] Aapo Hyv{{\"a}}rinen. Estimation of Non-Normalized Statistical Models by Score Matching. Journal of Machine Learning Research, 2005.  

[2] Simo Särkkä. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.

Reviewer comment (Theory):
How does the loss function related to time series data generation?

Author response:
**Q4.Please make a more clear explanation of how Eq. (12) is calculated.  
Q5.How does the loss function related to time series data generation?**  

Unlike image generation, time-series must be generated by considering past observations. But previous researches did not consider such a conditional generation, so we had to derive objective for conditional score function, ${\nabla}log p({\textbf{x}}^s_{1:n}|\textbf{x}^0_{1:n-1})$.  

But calculating an exact value of ${\nabla}log p({\textbf{x}}^s_{1:n}|\textbf{x}^0_{1:n-1})$ is computationally prohibitive [1]. As ${\nabla}log p({\textbf{x}}^s_{1:n}|\textbf{x}^0_{1:n})$ has a definite form and follows a specific SDE (e.g., VE, VP, subVP) [2], we drive that $L_1$ is same with $L_{score}$ by constant term.  

Considering that using $\textbf{x}_{1:n}$ is computationally expensive, we use an autoencoder to encode it to latent variable.  

Once we generate $\textbf{h}_{1:N}$, we acquire total sequence by using a trained decoder.  

As a result, our loss function for the score network is $L_{score}^{\mathcal{H}}$.  

[1] Aapo Hyv{{\"a}}rinen. Estimation of Non-Normalized Statistical Models by Score Matching. Journal of Machine Learning Research, 2005.  

[2] Simo Särkkä. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.

Reviewer comment (Theory):
How does the proposed loss function in Eq. (12) solve the computation issue? Does it accelerate the training process or the sampling procedure?

Author response:
**Q6.How does the proposed loss function in Eq. (12) solve the computation issue? Does it accelerate the training process or the sampling procedure?**  

As we mentioned above, calculating ${\nabla}log p({\textbf{x}}^s_{1:n}|\textbf{x}^0_{1:n-1})$ is computationally prohibitive. So we derive that train ${\nabla}log p({\textbf{x}}^s_{1:n}|\textbf{x}^0_{1:n})$ is the same as in [1]. It is not related to accelerating, but training.  

[1] Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661–1674, 2011.

Reviewer comment (Reproducibility):
Which encoder are used for encoding the irregular time series in your experiment?

Author response:
**Q7.Which encoder are used for encoding the irregular time series in your experiment?**  

As we explained Q2, we use Neural CDE for irregular time-series. The detailed architecture is described in Appendix I and J.

Reviewer comment (Reproducibility):
The expressions for $\mathbf{f}$ and $g$ used in Eq. 1 are missing. These are crucial components of the Langevin sampling that these score models are based on. While the work mentions VE, VP, and subVP from previous works, it does not provide the expressions used in this work.

Author response:
**Q1. What are the \textbf{f} and \textbf{g} used in Eq. 1?**

In the Introduction section, we mentioned the type of SDEs which we used : "Our TSGM can be further categorized into two types depending on the used stochastic differential equation type: VP and subVP."

We give additional section in Appendix N to explain the 3 types of SDEs. Especially, we described the detailed forms of **f** and g. They are called a drift and a diffusion respectively, which constitute a Brownian motion of adding noises into data samples.

Reviewer comment (Writing):
A specific paragraph adds no new information and introduces a variable $D$ that is never used again in the paper. It is likely obvious to readers that a multivariate time series would be represented as a matrix, yet this is stated in detail while more vital model information (e.g., $\mathbf{f}$ and $g$ expressions) is omitted.

Author response:
**Q2. "For synthesizing regular time series, ~ (see Section 3.2 and Appendix I)." & "When synthesizing regular time-series only, ~ and our proposed design is more general than this approach." A  one or half-sentence description would be ideal for the first paragraph. The second paragraph is the primary issue. It adds no new information and introduces a variable that is never used in the paper again.**

We modify the passage you mentioned. Especially, the second paragraph seems to be unnecessary so we remove it and add some phrases describing **f** and g.

Reviewer comment (Theory):
The theorems and corollary are presented as statements followed by either more statements or a description of terms, without proofs or explanation of why they are valid. This lack of explanation or motivation confuses the reader, and the authors should look at other works for proper examples.

Author response:
**Q3. The theorems and corollary are statements followed by either more statements or a description of the terms. At no point are they proven nor do the authors give some sense of why they may hold.**

We add intuitive descriptions for our loss function prior to our theorem in Section 3.3. We mention the reason of why we have to use the two loss functions and the motivation of our main theorem.

Reviewer comment (Presentation):
The text references many appendices, but the appendices are missing. Even if included, the main text should still provide brief descriptions of what is happening, rather than simply referencing an appendix and moving on without context.

Author response:
**Q4. The text references many appendices, but they are all missing.**

We already included our appendices in the supplementary zip file in our initial submission. Please check it and we also attach them to our revised main paper for your convenience.

Reviewer comment (Novelty):
The authors claim uniqueness and being first-of-its-kind without adequate comparisons to previous works using diffusion models, even making false statements that dismiss previous works without citing them.

Author response:
**Q5. The authors falsely state "Although there exist diffusion-based time-series forecasting and imputation methods, our target score function and its denoising score matching loss definition are totally different from other baselines." to dismiss previous works without citing any of them. The authors do not follow up with detail about how their results are unique so the reader is left confused. In this light both the above. "Despite the previous efforts to generate time-series using GANs and VAEs, according to our survey, there is no research using SGMs for this purpose. Therefore, we extend SGMs into the field of synthesis1."**

We have justified the differences between them and ours on Appendix B. Since time series generation is different from imputation and forecasting, we did not write it in the main paper but explain it in the appendix with several experiments. We add appropriate citations in the our revised main paper. Since you missed our appendix in the supplementary zip file in our initial submission, all these confusions are caused. Sorry for not mentioning that our appendix is in the zip file in our initial submission.

Reviewer comment (Novelty):
The paper omits mention of Song et al.'s "SCORE-BASED GENERATIVE MODELING THROUGH STOCHASTIC DIFFERENTIAL EQUATIONS," a highly related work that disproves some statements in the paper, including the claim that their denoising score matching loss is totally different from baselines.

Author response:
**Q5. The authors falsely state "Although there exist diffusion-based time-series forecasting and imputation methods, our target score function and its denoising score matching loss definition are totally different from other baselines." to dismiss previous works without citing any of them. The authors do not follow up with detail about how their results are unique so the reader is left confused. In this light both the above. "Despite the previous efforts to generate time-series using GANs and VAEs, according to our survey, there is no research using SGMs for this purpose. Therefore, we extend SGMs into the field of synthesis1."**

We have justified the differences between them and ours on Appendix B. Since time series generation is different from imputation and forecasting, we did not write it in the main paper but explain it in the appendix with several experiments. We add appropriate citations in the our revised main paper. Since you missed our appendix in the supplementary zip file in our initial submission, all these confusions are caused. Sorry for not mentioning that our appendix is in the zip file in our initial submission.

Reviewer comment (Novelty):
Statements in the paper, such as “there is no research using SGMs for this purpose,” are inaccurate since time-series generation using diffusion models does exist, and need modifying to reflect this.

Author response:
**Q5. The authors falsely state "Although there exist diffusion-based time-series forecasting and imputation methods, our target score function and its denoising score matching loss definition are totally different from other baselines." to dismiss previous works without citing any of them. The authors do not follow up with detail about how their results are unique so the reader is left confused. In this light both the above. "Despite the previous efforts to generate time-series using GANs and VAEs, according to our survey, there is no research using SGMs for this purpose. Therefore, we extend SGMs into the field of synthesis1."**

We have justified the differences between them and ours on Appendix B. Since time series generation is different from imputation and forecasting, we did not write it in the main paper but explain it in the appendix with several experiments. We add appropriate citations in the our revised main paper. Since you missed our appendix in the supplementary zip file in our initial submission, all these confusions are caused. Sorry for not mentioning that our appendix is in the zip file in our initial submission.

Reviewer comment (Theory):
The first line under Figure 2 is important to the argument but lacks sufficient evidence and explanation as to why $\mathbf{h}_n \sim \mathbf{x}_n$.

Author response:
**Q7. The first line under Figure 2 is important to your argument and needs more evidence and explanation to why $\textbf{h}_{n}  $ and $\textbf{x}_{1:n}$ are substituted**

Since we use RNN and Neural CDE as autoencoder, we can substitute total sequence with a latent variable, as we mentioned in related work section, Section 2.2. In fact, the relation above is the main design philosophy (or goal) of RNNs. We add more explanations about in Section 3.3.

Reviewer comment (Reproducibility):
The appendices are missing, making it very difficult to fully review the work since many crucial details are located there.

Author response:
**Q4. The text references many appendices, but they are all missing.**

We already included our appendices in the supplementary zip file in our initial submission. Please check it and we also attach them to our revised main paper for your convenience.

Reviewer comment (Writing):
There are typos in the paper, such as in Eq. 5 where $\hat{\mathbf{x}}_n$ is also an RNN and should have the same arguments as $\mathbf{h}_n$, and the redundant phrase “Eq. equation 12.”

Author response:
**Q8. Some typos (1) In Eq. 5, $\hat{\textbf{x}}_n$ is also an RNN so shouldn't it have the same arguments as $\textbf{h}_n$?
(2) "Eq. equation 12"**

(1) You can intuitively think that total sequence is accumulated in latent feature and each total sequence is restored by subtracting past data from latent feature. This is derived by the autoencoder loss function, $L_{ed}$ in equation (6).

The above approach is also used in TimeGAN[1].

[1] Yoon, Time-series generative adversarial networks.
Neurips19.

(2) We modified it in our revised main paper.

Reviewer comment (Experiments):
The authors did not clarify why time series generation is significant. The authors mentioned in the introduction that “In many cases, however, time-series samples are incomplete and/or the number of samples is insufficient”, which is served as the motivation for this work. However, for incomplete cases, time series imputation can be used, which has been investigated in the previous works mentioned in the paper, such as TimeGrad and CSDI. For the insufficient sample number case, the authors did not show whether generated samples can increase the quality or robustness of learning.

Author response:
**Q1.The authors did not clarify why time series generation is significant. The authors mentioned in the introduce that “In many cases, however, time-series samples are incomplete and/or the number of samples is insufficient”, which is served as the motivation for this work. However, for incomplete cases, time series imputation can be used, which has been investigated in the previous works mentioned in the paper, such as TimeGrad and CSDI. For insufficient sample number case, the authors did show whether generation samples can increase the quality or robustness of learning.**

There are many articles on real-world applications of time series synthesis. A selected list of citations is as follow [1,2,3,4], and we put all the citations in the revised paper with appropriate discussions.

Time-series synthesis serves a crucial role in data augmentation for insufficient or imbalanced training data. In addition, we can also address the privacy issues in releasing real data. For instance, Physionet[5], a famous dataset for time series classification, deliberately removed 90\% of observations to protect the privacy of patients, posing challenges for learning and analysis. We can release synthesize time series instead.

The task of generating data while considering missing values, i.e., irregularity, and feature interdependences presents a considerable challenge. Our research is specifically tailored to confront this complexity.

[1] S.Dash. Medical time-series data generation using generative adversarial networks. AIME, 2020.

[2] M.Dogariu. Generation of Realistic Synthetic Financial Time-series. ACM, 2022.

[3] Y. Kang. GRATIS: Generating Time series with diverse and controllable characteristics. ACM, 2020.

[4] Q. Wen. Time series data augmentation for deep learning: A survey. ISCAI, 2021.

[5] Ikaro Silva. Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In 2012 Computing in Cardiology. IEEE, 245–248.

Reviewer comment (Novelty):
The proposed method lacks novelty when compared to existing works like TimeGrad and CSDI. In the appendix, the authors differentiate their approach from these existing works by highlighting that TSGM is designed for synthesizing time series data from scratch, while the other works do not offer this capability. However, the authors did not discuss whether these works can be easily extended for generation of time series.

Author response:
**Q2.The proposed method lacks novelty when compared to existing works like TimeGrad and CSDI. In the appendix, the authors differentiate their approach from these existing works by highlighting that TSGM is designed for synthesizing time series data from scratch, while the other works do not offer this capability. However, the authors did not discuss whether these works can be easily extended for generation time series.**

Please read Appendix B and C. Not only did we discuss the differences between previous works and TSGM in Appendix B, but we also conducted several experiments for extending TimeGrad and CSDI toward generation tasks in Appendix C. For TimeGrad, it failed to generate Energy, which is a high dimensional dataset. CSDI also synthesized poor data on Energy and AI4I.

Reviewer comment (Experiments):
The experiment part needs to be strengthened. Can the authors compare with these methods in terms of forecast and imputation? Forecast and imputation of time series can be considered as conditional generation, which is more challenging and important.

Author response:
**Q3.The experiment part needs to be strengthened. Can the authors compare with these methods in terms of forecast and imputation? forecast and imputation of time series can be considered as conditional generation, which is more challenge and important.**

It needs significantly modifying the architecture to conduct forecasting and imputation. Instead, we modified the methods of forecasting and imputation toward generation tasks in Appendix B and C. We demonstrate that TSGM outperforms other domains' models. Also, as we explained in Q1, time-series generation is not only a hard and significant task comparable to forecasting and imputation but also an independent task distinguished from other domains. Therefore, we leave the application of TSGM to other domains as future work.

Reviewer comment (Theory):
The Corollary 3.2 seems useless. The operation of removing the expectation regarding to x^0_{n} is redundant, as in L2 we have taken the expectation regarding to x^0_{1:N}.

Author response:
**Q4.The Corollary 3.2 seems useless. The operation of removing the expectation regarding to $\textbf{x}^0_{n}$ is redundant, as in L2 we have taking the expectation regarding to $\textbf{x}^0_{1:N}$.**

Please note that the expectation of $\textbf{x}^0_{n}$ of Theorem 3.1 and the expectation of $\textbf{x}^0_{1:N}$ are independent. Therefore, we have to independently sample $\textbf{x}^0_{n}$ after sampling $\textbf{x}^0_{1:N}$. That is why $\textit{the law of total expectation}$ is used in such a case and our corollary can be viewed as an application of the law (c.f. Appendix A). Thanks to corollary, it is sufficient to sample $\textbf{x}^0_{1:N}$ only once.

Reviewer comment (Presentation):
The index of theorems and corollaries should be unified.

Author response:
**Q5.The index of theorems and corollaries should be unified.**

Thanks for your indication. We unify the index.