Paper: iHyperTime: Interpretable Time Series Generation with Implicit Neural Representations (ICLR-2024)

Reviewer comment (Reproducibility):
The author claims that "a single layer SIREN has been shown to be structurally similar to a Fourier mapped perception (FMP)". I am not clear whether the author use SIREN (equation 4) or FMP (equation 5) to implement fr(t) in the paper.

Author response:
We apologize for the lack of clarity in the description of the method. TSNet uses a SIREN to model the Residual component, and a Fourier decomposition to model the Seasonality component. We do not actually use a Fourier mapped perceptron (FMP), and we corrected the related section. In particular, we have rewritten all Section 3.1.1 to provide a more concise and clear explanation of the TSNet architecture.

Reviewer comment (Reproducibility):
If f(t) is multidimensional, does the author model each dimension as ftr(t) + fs(t) + fr(t) independently?

Author response:
We thank the reviewer for raising this question. If f(t) is multidimensional, iHT models each dimension as $f_T(t) + f_S(t) + f_R(t)$ independently. Following an $m$-dimensional time series, the trend block outputs $m$ trends, the seasonality block outputs $m$ time series that correspond to the seasonality of each feature, and the same is for the residual block. Each of the features are then added together obtaining an $m$-dimensional time series.  

We apologize for the lack of clarity in our previous reply. Across most of the network building blocks (embeddings, decoders, TSNet), the Trend-Seasonality-Residual (TSR) components $f_T(t)$, $f_S(t)$, $f_R(t)$ of the time series are processed separately, but the different dimensions (in the case of multivariate data) are computed and encoded together. Furthermore, in the first block of the architecture (set encoder) both the TSR components and the channels are processed together. This enables the network to generate embeddings that encode interactions between the different components, as well as between different signal channels. This is in contrast to other SOTA methods such as Fourier Flows [1], where individual channels are processed independently in multivariate time series. We will add a clarification of this point in the final version of the paper.

[1] A. Alaa, A. Chan, M. van der Schaar. Generative time-series modeling with fourier flows. In ICLR, 2021.

Reviewer comment (Novelty):
It would be better to cite Deep Sets when discussing the set encoder.

Author response:
We thank the reviewer for the suggested reference, we have now cited Deep Sets in the revised paper version, in the set encoder subsection in Section 3.2.

Reviewer comment (Writing):
The writing is currently very unclear and is unable to convey the critical ideas of the paper.

Author response:
We thank the reviewer for the suggestions. We acknowledge that the paper presentation was not clear enough. In the new version of the paper we extensively rewrite section 3, and we have colored the rewritten parts in blue to highlight the modifications/additions. 
As suggested, we have also evaluated our approach on three real datasets, showing that the model is able to disambiguate between season, trend, and residual in an interpretable manner for real world data. 

I hope the reviewer finds our results useful in assessing the quality of our work. We are happy to answer any additional questions.

Reviewer comment (Writing):
How is the model used to perform generation? Based on the writing, I am unable to understand how this formulation, which requires taking data as input with the set encoder, can be used for unconditional generation of new time series.

Author response:
We apologize for the lack of clarity in the description of the generative method. In the main paper we have now added a subsection at the end of section 3.2 explaining how we perform generation. Generation of new time series is produced by randomly selecting pairs of time series from the training-set, and performing a linear interpolation between their embeddings $Z^{(1)}$ and $Z^{(2)}$:
    $$ Z^{\\text{gen}} = Z^{(1)} + \\lambda (Z^{(2)} - Z^{(1)}) $$
where $\\lambda$ is also sampled randomly. Optionally, the interpolation can be performed on individual components $Z_T$, $Z_S$, $Z_R$ of the embeddings, enabling the conditional generation of time series.

Reviewer comment (Writing):
The paragraph on __Residual blocks__ in Section 3.1.1 does not make sense at all. Why is it describing the seasonal component, "In this work, we propose to model the seasonal component of the time series as a ...", which was already addressed in the previous paragraph? The notation is suddenly changed from $t$ to $x$? Why talk about random Fourier features if the paper proposes to use SIREN? I do not understand what is the implemented/proposed model for the residual component after reading this paragraph.

Author response:
We apologize for the mistake in explaination of the seasonal component: when we said "seasonal" we should have said "residual". 

**Change of notation** In the trend and seasonality networks, we used $t$ because there is a single layer. And in residual, we used $x$ to indicate the outputs of the successive layers. We should have clarified this, also stating that $x_0 = t$. We have now rewritten this section ensuring consistency of the notation.

**Random Fourier Features** We apologize for the lack of clarity in this aspect. Our initial goal was to convey that using SIRENs or random Fourier Features was equivalent w.r.t. having a good representation of frequencies in the signal, following [1]. However, we agree that this leads to confusion. We have now revised Section 3.1.1 to clarify this and we removed any misleading statements about Fourier Features Networks (FFNs).

[1] Benbarka, Nuri et al. “Seeing Implicit Neural Representations as Fourier Series.” 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (2021).

Reviewer comment (Reproducibility):
What are the details of the hypernetwork component/iHyperTime architecture? More details with mathematical description needs to be given for this part. How is the Set Encoder and SIREN layers used here? Details of the hypernetwork are not given.

Author response:
We thank the reviewer for pointing out an important missing aspect. We have now included two subsections in Section 3.2 detailing how the set encoder and hypernetworks are used. For the Set encoder we have provided a mathematical description and how it is parameterized using SIREN layers. We have provided implementation details in Section D.2 of the appendix, which is now referred in the main paper. Furthermore, in section D.2 we have also added more information on how the Set Encoder is used, and provided more details about the hypernetwork architecture.

Reviewer comment (Reproducibility):
Important details on problem formulation and training are not included. Concretely, what are the time coordinates $t_i$? For long sequences, is the whole time series encoded into a set, or divided into windows? Section 4.2 states "Performance results for regularly sampled time series are provided in Table 1, for univariate and multivariate datasets of varying lengths." -- which datasets are univariate, which are multivariate? What are the varying lengths? How do I interpret 24/72/360 in Table 1?

Author response:
We thank the reviewer for pointing out aspects that were not clearly explained in the paper.

**Time coordinate.** In our model, we use a single floating point value as temporal coordinate (time $t$). As data pre-processing, all the values are scaled to the interval $[-1, 1]$, with a common global factor for all time series of a dataset. In all cases, regardless of sequence length, the time series is fed to the set encoder as a single set, and is hence converted into a single embedding $Z$. We have now included this information in Appendix D.2.
    
**Datasets.** Following previous methods [1, 2, 3], we used the Stock and Energy datasets [5] sliced into 24 time-steps windows. These datasets are multivariate, with 6 and 28 features, respectively. Furthermore, in order to make a fair comparison with  [3] and  [4], who show performances in longer time series of Stock data, we generated an univariate version of the Stock dataset by selecting one single feature (Close Price), and slicing it in two ways: using a window of 72 time steps (Stock72), and a window of 360 time steps (Stock 360).

Additionally, in Section 4.4 we consider 4 univariate datasets with longer real-world time-series from Monash repository [6], namely FRED-MD, NN5 Daily, Temperature Rain, and Solar Weekly. The first three datasets have time series of approx. $750$ time steps, while the last one contains time series of length $52$. This information is summarized in Appendix B, where Table 8 presents the main characteristics of each dataset. We have improved this table to clarify the difference between the different datasets, in terms of sequence length and number of channels. And we refer this table in Section 4.1 of the main paper.
    
Finally, in Section 4.3 we evaluate our method on irregular data. Following existing literature [2], we create irregular time-series datasets by randomly dropping 30, 50, 70\% of observations from the original datasets. We select each observation to drop uniformly at random, and independently for each time series $\{(t_i, y_i)\}_{i=0}^N$ in the original dataset. 

- [1] J. Yoon, D. Jarrett, M. van der Schaar. Time-series generative adversarial networks. In NeurIPS, 2019.

- [2] J. Jeon, J. Kim, H. Song, S. Cho, N. Park. GT-GAN: General purpose time series synthesis with generative adversarial networks. In NeurIPS, 2022.

- [3] A. Coletta, S. Gopalakrishan, D. Borrajo, S. Vyetrenko. On the constrained time-series generation problem. In NeurIPS, 2023.

- [4] A. Alaa, A. Chan, M. van der Schaar. Generative time-series modeling with fourier flows. In ICLR, 2021.

- [5] L. Candanedo, V. Feldheim, D. Deramaix. Data driven prediction models of energy use of appliances in a low-energy house. Energy and buildings, 140:81–97, 2017.

- [6] R. Godahewa, C. Bergmeir, G. Webb, R. Hyndman, P. Montero-Manso. Monash time series forecasting archive. In NeurIPS, 2021.

Reviewer comment (Evaluation):
Is the model actually able to disambiguate between season, trend, and residual in an interpretable manner for real world data? Can you visualize the seasonal-trend decomposition on real world datasets with strong seasonality/trend?

Author response:
We thank the reviewer for suggesting an interesting new experiment. We have now conducted an analysis on three real world datasets to show that iHT is able to disambiguate between trend, season and residual components after training.

The new experiments are included in Section 1.6 of a new supplementary material, submitted along the main paper. We will add the experiments in the final version of the paper.
    
**Datasets** 

We used two well known real-world time series datasets that exhibit strong seasonality and trend: Atmospheric CO2 and Airline Passenger, and used the Stock72 dataset described in section B of the supplementary material that exhibits strong trend but low seasonality.  

The Atmospheric CO2 dataset corresponds to monthly data (January 1959 to December 1987) with a total of 348 observations with a seasonality period of 12 [1]. We sliced the data into time series of sequence length 60, obtaining a total of 288 time series. 

The Airline Passenger dataset corresponds to monthly data (January 1948 to December 1960) with a total of 144 observations with a seasonality period of 12 [2]. We sliced the data into time series of sequence length 36, obtaining a total of 108 time series. 

To estimate the presence of trend and seasonality of each dataset we use two metrics defined in [3]: the strength of trend and strength of seasonality. As these datasets do not include ground-truth trend and seasonality, we estimate them using STL [1]. Given a time series with additive decomposition in trend, seasonality and residual:
    $
        y_t = T_t+ S_t + R_t
   $
 , we can define the strength of trend as:  
$$
        F_T = \max\left(0, 1 - \frac{\text{Var}(R_t)}{\text{Var}(T_t+R_t)}\right)
$$

This measures the relative variance of the residual component $R_t$ to the variance of the trend with the residual component. The value ranges from 0 to 1, with 0 indicating no trend and 1 indicating a strong trend.  

Strength of seasonality is defined by:    
$$
        F_S = \\max\\left(0, 1 - \\frac{\\text{Var}(R_t)}{\\text{Var}(S_t+R_t)}\\right)
$$
 Which measures the relative variance of the residual component $R_t$ to the variance of the seasonality with the residual component. A value close to 0 indicates little to no seasonality, and a value close to 1 indicates strong seasonality.

Table below shows these metrics for the three datasets. As expected, Stock72 doesn't exhibit strong seasonality (as stock data usually have not such component), while the other two datasets exhibit strong trends and seasonality, with all values on average above 0.98.
    
$$
\\begin{array}{cccc}
\\hline
\\text{Component} & \\text{CO2} & \\text{Air Pass} & \\text{Stock 72} \\\\ \\hline 
    \\text{Trend strength}   & 0.99 \\pm 0.01 & 0.98 \\pm0.01 & 0.99 \\pm 0.01 \\\\
        \\text{Seasonal strength} &  0.99 \\pm 0.00 & 0.99 \\pm0.01  & 0.32 \\pm 0.09 \\\\    
\\hline
\\end{array}
$$

**Results**

We trained iHT on each dataset and we show in Figures 8 and 9 of the Supplementary the reconstructed time series and the output of each of the individual blocks for the CO2 and Air Passanger datasets, respectively. For the output of the trend, seasonality and residual block, we compare with the STL method. We observe that there is a good agreement in the trend and seasonality components in both datasets. The STL method requires as parameter the period of the seasonality, which is known for both datasets, while our approach is able to estimate the seasonality from the data. In the case of the Stock72 dataset, for the STL comparison, we estimated the period using a Fourier transform and retrieving the dominant frequency. Figure 10 of the supplementary shows the decomposition generated by iHT and compared against STL. We can see in this case that the seasonal component is quite small in magnitude compared to the residuals.
    
We will include this new analysis on real-world data in the revised version of the manuscript. 
    
[1] Cleveland, R. B., Cleveland, W. S., McRae, J. E. & Terpenning, I. (1990). STL: A Seasonal-Trend Decomposition Procedure Based on Loess (with Discussion). Journal of Official Statistics.

[2] Downloaded from https://www.kaggle.com/datasets/rakannimer/air-passengers

[3] Wang, X., Smith, K. A., & Hyndman, R. J. (2006). Characteristic-based clustering for time series data. Data Mining and Knowledge Discovery, 13(3), 335–364.

Reviewer comment (Experiments):
W.2. The results are good at highlighting that the generated time series is faithful to the original distribution through the \textit{predictive}, \textit{discriminative}, and \textit{marginal} scores, but there is no qualitative or quantitative experiment on the diversity of the generated time series. My concerns comes from the generation procedure. If $z^{(gen)} = 0.9 z^{(1)} + 0.1 z^{(2)}$, the new time series $x^{(gen)}$ would be very likely $x^{(1)}$ leading to good fidelity scores but not really generating novel time series. Moreover it is standard in generation to split the time series into train/test datasets before the generation phase. Then, generate some new $x^{(gen)}$ according to $z^{(i)}$'s from the train dataset and then compute the \textit{predictive}, \textit{discriminative}, and \textit{marginal} scores between the $x^{(gen)}$ and the $x^{(j)}$'s from the test dataset. From the evaluation metrics paragraph, it is unclear if the dataset is split before generation.

Author response:
**Train-Test Split.**
 We agree that splitting the time series into train/test is the best approach for a fair evaluation of the generative models. However, we found that some of state-of-art approaches train and test the model on the whole dataset for Stock and Energy data[1,2,3]; and to perform a consistent comparison w.r.t. the performance reported in their original paper, we decided to keep the same approach in our paper.
We however agree with the reviewer's concern, and thus we are happy to report in the table below (also Table 1 of supplementary) our initial effort in evaluating approaches while randomly splitting data between training set (i.e., 80$\\%$ of dataset) and test set (i.e., 20$\\%$ of dataset). The Table reports the quantitative metrics for the regular Stock data with length 24. While Figure 5 in the supplementary material shows the t-SNE visualization and the data distribution. These results confirm the findings from the our paper. Notice that, for the Monash datasets we already divided the dataset in training ($80\%$) and test ($20\%$) set, similar to the benchmark model LS4 [4]. 

**Diversity.** Finally, we also agree that diversity of generated time-series plays a key role in the evaluation of generative models. We can qualitatively evaluate the diversity and coverage from t-SNE of Figure 5 (as well as in t-SNE plots of the main paper). The t-SNE plots show that iHyperTime synthetic data mostly overlap with real data: i.e., iHyperTime is able to have enough diversity to reconstruct the real data distribution. Following reviewer suggestion, we also use a novel metric to quantitatively evaluate the diversity of time-series, namely $sym$-Recall[6] which extends the $\\beta$-Recall from[5]. Such metric quantifies the diversity of time-series as the extent to which synthetic samples cover the full variability of real samples. Initial results are reported in the Table below. 

$\\begin{array}{l|ccccccc}
\\hline
\\text{Metric} & \\text{iHT (Ours)} & \\text{GT-GAN} & \\text{TimeGAN} &  \\text{DiffTime} & \\text{LS4} & \\text{FFlows} \\\\ \\hline
\\text{Discr-score} \\downarrow & 0.054\\pm0.028 &  0.273\\pm0.046 & 0.068\\pm0.018 & 0.079\\pm0.014 & 0.154\\pm0.080 & 0.426\\pm0.032 \\\\
\\text{Pred-score} \\downarrow & 0.037\\pm0.000 & 0.046\\pm0.001 & 0.043\\pm0.001& 0.044\\pm0.001 & 0.039\\pm0.000 & 0.055\\pm0.003 \\\\
\\text{Marginal Score} \\downarrow  &  0.355 & 0.403 & 0.434 & 0.359 & 0.513 & 0.335 \\\\
\\text{sym-Recall} \\uparrow &  0.594 & 0.500 & 0.413 & 0.787 & 0.206 & 0.000 \\\\
\\hline
\\end{array}$


We will add a more extensive discussion, and new experiments in the final version of the paper. 


**References**

[1] - Yoon, Jinsung, Daniel Jarrett, and Mihaela Van der Schaar. "Time-series generative adversarial networks." Advances in neural information processing systems 32 (2019).


[2] - Jeon, Jinsung, et al. "GT-GAN: General Purpose Time Series Synthesis with Generative Adversarial Networks." Advances in Neural Information Processing Systems 35 (2022): 36999-37010.

[3] - Alaa, Ahmed, Alex James Chan, and Mihaela van der Schaar. "Generative time-series modeling with fourier flows." International Conference on Learning Representations. 2020.

[4] - Zhou, Linqi, et al. "Deep latent state space models for time-series generation." International Conference on Machine Learning. PMLR, 2023.

[5] - Alaa, Ahmed, et al. "How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models." International Conference on Machine Learning. PMLR, 2022.

[6] - Khayatkhoei, Mahyar, and Wael AbdAlmageed. "Emergent Asymmetry of Precision and Recall for Measuring Fidelity and Diversity of Generative Models in High Dimensions." arXiv preprint arXiv:2306.09618 (2023).

Reviewer comment (Experiments):
MW1. The point arguing that SIREN outperforms Fourier Features Network (FFN) for learning and representing high frequencies is at my knowledge not true for time series. Indeed, classical Fourier Features where frequencies are sampled in the linear scale can suffer from spectral bias. But if the frequencies are sampled in the logarithm scale or drawn from a Gaussian distribution as in [2] the spectral bias doesn't stand anymore. In the context of DeepTime [3], the authors demonstrate the superiority of FFN over SIREN in time series forecasting.

Author response:
We have now revised Section 3.1.1 to clarify and remove any misleading statements about SIREN outperforming Fourier Features Networks (FFNs) in terms of spectral bias. Following Q4, we will provide an ablation study shortly comparing SIREN and FFN and will add a more extensive discussion in the final version of the paper.   
**[UPDATE]** We have added the additional ablation studies in the supplementary material, and we discuss them in a comment below.

Reviewer comment (Presentation):
MW2. Figure 4 is not completely clear. More details are needed to understand how this figure was generated. At first glance, even if the method is efficient for different sequence lengths, the training and inference time should increase with the sequence length, at least due to the loading of the data. Moreover, it is quite peculiar to see a method, Fourier Flows, having a convex curve in time with the sequence length. This makes the argument on the independence with respect to the sequence length less convincing, even though the model still is efficient at training time as shown in Table 12.

Author response:
We acknowledge that Figure 4 is missing essential details and a more accurate discussion of the results. We also agree with the reviewer that the main goal of our analysis is showing a relatively low computational time for our approach w.r.t. existing methods, as also shown in Table 12. In re-writing the final paper, we are making sure of not claiming any independence with respect to the sequence length.

**More details** Our experiments use the same synthetic data-set of 8192000 data-points (e.g. number of samples $\times$ sequence length), organized in 5 datasets of different sequence lengths $l \\in \\{80, 320, 1280, 5120, 20480\\}$. Thus, we obtain 5 datasets of $\\{102400, 25600, 6400, 1600, 400\\}$ number of samples, which we process using $\\{1024, 256, 64, 16, 4\\}$ as batch-size, respectively.  Thanks to this batch-size, we can fairly train each model for 100 iteration and always loading to GPU the same amount of data-points, for each sequence length and iteration.For inference, we only sample a single batch of data.

**Impact of data loading** All the times are computed directly in the training and testing loop, after the model has been already loaded in the GPU memory, using a NVIDIA T4 GPU. Moreover, we found that the data loading (both for training and inference) is negligible w.r.t. model computational time. We measure the loading being around $ 0.0145$ sec for training (100 iterations) and inference $0.001$ sec (1 epoch). 

**Fourier Flow** Finally, we repeated the experiments with Fourier Flows with the new computational times reported in Table 2 of supplementary. However, we still have some convexing. We believe the slightly drop likely comes from not optimized code to handle large batch-sizes: longer time-series (e.g., 320 and 1280) worsen the computational time, but the model is also relieved by the smaller batch-size (e.g., 256 and 64).

Reviewer comment (Experiments):
MW3. In Figures 14-20, the distribution of the original data is different for each baseline, this limits the conclusion that can be drawn from these experiments.

Author response:
We thank the reviewer for spotting this issue. We are improving the charts in the final version of the paper with the same scale for both the x and y axes. Figure 5 of the supplementary material already shows some of the new charts.

Reviewer comment (Reproducibility):
MW4. The interpolation process for generating new time series is clear when reading the code, however it is not really described in the paper. This would be important for better clarity of the paper to have this description at least in the appendix.

Author response:
We apologize for the lack of clarity in the description of the method. We are incorporating a new detailed description of our time series generation procedure.

Reviewer comment (Experiments):
W.3. One key feature of the IHyperTime architecture is the permutation invariant set encoder which allows to encode time series with different sampled grids. However the set encoder is known for underfitting [1] because of the naive aggregation mechanism. And there is no metrics in the experiments allowing to understand the quality of the reconstruction. Moreover, the dimension of $Z_{T}, Z_{S}$ and $Z_{R}$ seems to be crucial and are not discussed in the paper (they are set to 10, 15, 15). It would be interesting to understand the trade-off between the quality of reconstruction and the quality of generation according to the dimension of the $Z's$ (please see Q.3).

Author response:
We thank the reviewer for the suggestion. 

**Reconstruction.** In the supplementary material we now report both the quantitative and qualitative metrics to assess the quality of the reconstruction in train and test set. In Table 5 of supplementary we report the reconstruction error for the iHyperTime for the Energy24, Stock24, Stock72, and Stock360 datasets. While Figure 6 and Figure 7 show the actually quality of reconstructed time-series for Stock 72 and Stock 360, respectively. 

**Latent dimension**
The model results at varying of the latent dimensions $Z_T$, $Z_S$ and $Z_R$ are shown in the supplementary material on Table 3 and Table 4 for stock 24 and stock 72, respectively. As suggested we evaluate both the reconstruction error and the generation metrics, for different z\_{T}\_{S}\_{R} values (e.g., z\_4\_6\_8 represents $Z_T=4$, $Z_S=6$ and $Z_R=8$). The results show comparable results for Stock 72. For Stock 24 we have slightly better generative performance for small dimensions at the cost of higher reconstruction error, and vice versa.

Reviewer comment (Experiments):
It is unclear which data is missing in experiments with irregular timeseries. Is this missing data in training or testing? The procedure of removing data needs to be defined.

Author response:
Following existing literature [1,2,3] we create irregular time-series datasets by randomly dropping 30, 50, 70\% of observations from the original datasets. We select each observation to drop uniformly at random, and independently for each time series  {$(t_i, y_i)$}$_{i=0}^N$ in the original dataset.  The randomly removed data is the same for every model and every repeat, and used both in training and testing. We will add citations and more details for the irregular time-series dataset generation in the final version of the paper.

Reviewer comment (Evaluation):
It is unclear how the method performs in terms of clustering/classification score.

Author response:
We thank the reviewer for pointing out two interesting tasks that were not sufficiently discussed.  While our work primarily focuses on learning an interpretable trend-seasonality time series representation for time-series generation, we agree with the reviewer that this latent representation could be easily used and leveraged in broader applications, like cluster or classification. In particular, these applications could extensively benefit from the learned interpretable decomposition that separates trend, seasonality and residuals. We will discuss the possible different applications in the final version of the paper.

Reviewer comment (Experiments):
The results are not compared to GNN based generation methods.

Author response:
We thank the reviewer for the suggestion. In our current effort, we are improving the state-of-art section discussing related GNN work [4,5,6,7]. Among this work we note that: 

- the work in [4] is the only one directly addressing the generation of time-series using a Graph-based model. Although contemporaneous, we plan to compare our proposed approach against this model, and we contacted the authors to access their code. 

- the work in [5] proposes an interesting graph neural network approach to embed irregularly sampled and multivariate time-series. While the work focuses on forecasting, it can be extended to other applications including generation. We are evaluating the complexity and effort of adapting such model for a comparison.

Reviewer comment (Experiments):
Recent works show that RNN encoder-decoder with various training strategies such as weak decoder or contrastive or attractive losses can both generate clustered interpretable latent representation and generate sequences. How these models compare with iHyperTime?

Author response:
We thank the reviewer for suggesting an interesting discussion on RNN-based architectures for intepretable latent representations and sequence generation. In our current effort we are improving the state-of-art review to better discuss RNN-based approaches against iHyperTime. 

To the best of our knowledge, classical RNN approaches have shown lower performance in generative tasks (e.g., against GAN-based approaches like TimeGAN [1] used in our comparison). Even when they are trained using teacher-forcing (T-Forcing) [2] as well as professor-forcing (P-Forcing) [3] to regularize the generation. In fact, autoregressive approaches condition the model using previously generated samples, and even small prediction errors can compound over the sequences.  Therefore, we believe that a first major difference w.r.t. iHyperTime is that our approach can generate realistic time-series for longer sequences over 700 time steps. 

As suggested by reviewer we are also surveying approaches using contrastive loss, attractive losses, and weak decoder which have shown to improve performance compared with other losses [4, 5]. We will discuss the main differences between our intepretable latent representation for trend, seasonality, and residuals, against clustered interpretable latent representation [4, 6, 7, 8].

Finally, to the best of our knowledge, existing work do not target directly time-series representation for generation. Instead, our approach achieves state-of-art performance in such generation task with also an interpretable time-series representation. We are happy to include any useful references the reviewer may suggest.