Paper: GENERATIVE TIME SERIES LEARNING WITH TIME-FREQUENCY FUSED ENERGY-BASED MODEL (ICLR-2024)

Reviewer comment (Presentation):
They explained their model well, but the placement of Figures and Tables is not appropriate.

Author response:
● We rearranged some tables and figures in the updated paper .

Reviewer comment (Presentation):
The table mentioned in the ablation study is difficult to see because it is not in the main paper.

Author response:
● This is unfortunate due to the space limit of the main paper.

Reviewer comment (Presentation):
The overall structure of the paper is not friendly to readers.

Author response:
● We rearranged some tables and figures to make the updated paper more reader friendly. However, we were unable to include all tables and figures to the main paper due to the space limit imposed by ICLR.

Reviewer comment (Theory):
Isn't there a process in Equation 6 to invert from the fequency domain back to the time domain? If so, I don't understand how the fequency domain feature and time domain feature are added in Equation 7.

Author response:
● No, we do not invert the frequency domain back to the time domain. As the goal of the training the energy based model is to get well estimation of the unnormalized probatlity density for temoral path,  we extract the time domain and freqency domain feature for each layer, then add the time domain and frequency domain features, which yields better results in our experiments that concating them together.

Reviewer comment (Experiments):
The residual dilated convolutional network part in the 4.4 ablation study does not seem to have any meaning. There is no doubt that performance increases as the number of layers increases because the capacity of the model increases. Rather, it seems meaningful to use the same number of layers but subtract the residual part.

Author response:
● We try to show in this ablation study that the multi-layer rather than a single-layer dilated convolutional neural network is necessary to filter out the noise in the time series. It is well-known that some high frequency modes could be noise instead of valuable signal. Since our TF-EBM model keeps all the frequency modes out of the FFT, which is different from some other works (FedFormer, Fast Fourier Convolution, etc. ) that keeps a portion of them, we reply on such multi-layer dilated CNN to automatically extract useful modes.

Reviewer comment (Evaluation):
It's unclear how significant the resulting improvements are, the numbers. Can the authors quantify if the improvements are significant? (instead of just reporting means, report the errors too). Also, this might be a typo but Table 3 first row FEDformer MSE is the lowest.

Author response:
Q1: It's unclear how significant the resulting improvements are, the numbers. Can the authors quantify if the improvements are significant? (instead of just reporting means, report the errors too). Also, this might be a typo but Table 3 first row FEDformer MSE is the lowest.

● Our TF-EBM model, being a probabilistic time series model, is evaluated using the CRPS metric to assess its distribution modeling capabilities. To strengthen the comparison, we have incorporated the TimeGrad model, another prominent EBM-based time series model, into Table 1. Additionally, we have included extended CRPS results for longer forecast horizons of 336 and 720 time steps in Table 8 of the Appendix. The improvements over state-of-the-art probabilistic models can be substantial, with a 30% reduction in CRPS for traffic data. Furthermore, we present both MAE and MSE metrics based on the error between the mean value of the forecasted distribution and the ground truth. Employing the mean value as the point estimate allows for a fair comparison with recent transformer-based models.

● The typo in the MSE field inTable 3 for FEDformer is fixed in the updated paper.

Reviewer comment (Experiments):
The paper mentions that deterministic methods generally don't help with uncertainty quantification, that sets the reader up to seeing uncertainty quantification being addressed by the paper, but it doesn't seem like it has been addressed in text or in experiments.

Author response:
Q3: The paper mentions that deterministic methods generally don't help with uncertainty quantification, that sets the reader up to seeing uncertainty quantification being addressed by the paper, but it doesn't seem like it has been addressed in text or in experiments.

● Our TF-EBM model's uncertainty quantification capability stems from its probabilistic nature. As a probabilistic time series model, TF-EBM generates probability distributions for future values, allowing us to sample from these distributions to obtain the empirical quantiles of the uncertainty of our predictions. This probabilistic approach enables us to readily quantify the uncertainty associated with our predictions. For example, we can output the 70% confidence interval along with the mean value as shown in Figure 3 in the appendix. In contrast, point forecast models like TimesNet and AutoFormer only provide single-valued predictions for each time step, lacking the ability to capture the inherent uncertainty in time series data. 

● To further demonstrate the uncertainty quantification capability, we performed an experiment in Section G in the appendix of the updated paper. The synthetic ground truth is injected with different levels of noise. We output the 99% confidence interval along with the mean forecasts. One can observe that the model forecasts in the time steps with larger noise variance show wider confidence interval than those in the low variance time stpe. The uncertainty of the forecasts is consisitent with the level of the noise.

Reviewer comment (Evaluation):
More analysis on why the time-frequency modeling outperforms just time or just frequency features could strengthen this contribution.

Author response:
● We consider the particularity of time series as seqence data which can be descirbed in the time and freqency demension. As we laid out in Section 3.5 and references therein, the time and frequency dimension mostly characterize the local and global patterns of the time series, respectivley, that are both crucial for long-term time series forecasting. Therefore, it is a natural choice to combine both patters to enhance the model performance.  

● We added additinoal ablation study on the exchange rate and traffic datasets with results included in Table 6 in the Appendix. The results on the full datasets show that combing both time and frequency features contribute better forecasting ability in our model design.

Reviewer comment (Experiments):
The pre-training evaluation is limited. More exhaustive experiments on the transfer learning capabilities could be done.

Author response:
● We evaluated the transfer learning capabilities of our model by pre-training it on the electricity dataset and fine-tuning it on the traffic and four ETT datasets. The results, summarized in Table 9, demonstrate that the fine-tuned model outperforms or matches the performance of other models.

Reviewer comment (Experiments):
The comparison to autoregressive models like LSTMs is missing. This could reveal advantages over common recurrent approaches.

Author response:
● We showed DeepAR model results in Table 1 for the CRPS metric. DeepAR is a strong autoregressive model based on LSTM. Additionally, in both Table 1 and 8, we added results from the TimeGrad model, which is a strong autoregressive EBM model.  

● We appended additional column in Table 3 for the MAE/MSE metric obtained using LSTM model. They are also taken directly from TimesNet paper.  

● TF-EBM outperforms both DeepAR and TimeGrad on most of the datasets.

Reviewer comment (Experiments):
The synthetic ablation study focuses on noise removal. Ablations on modeling long-term dependencies could be more insightful.

Author response:
● We added additional ablation studies for long-term forecasts in Figure 4 in the Appendix.  Long-term pattern mainly construct low frequency components in time series. In the ablation study, we forecast the future time steps using one layer, two layers and three layers of the Time-Freq Block separately and apply the FFT for the different forecast. The experiment results show that three layers of Time-Freq Block can well charaterize the low-frequency part and indicate that stack of Time-Freq Block help to capture long term pattern.

Reviewer comment (Experiments):
All datasets are regular time series. Applying to irregularly sampled data from healthcare etc. could reveal robustness.

Author response:
● TF-EBM is a time series model that is currently designed for synchronous time series data. However, there are several approaches to dealing with irregularly sampled or asynchronous time series data, such as converting the asynchronous timestamps to synchronous ones by binning. This approach could be implemented in future work to expand the capabilities of TF-EBM.

Reviewer comment (Reproducibility):
Hyperparameter tuning details are not provided clearly. It's unclear if suboptimal settings affect comparisons.

Author response:
● Other model results in the experiments were directly taken from the paper TimesNet, i.e., we did not run those models. We just used the same training and testing sets for each dataset for a fair comparison. We will release our code implenentation with hyperpameter settings for TF-EBM once the paper is accepted.

Reviewer comment (Presentation):
The text just above Eq 6 defines the operator $g = MLP(SiLU(Conv1D(\cdot)))$, while Figure 1 and earlier text indicate that Dilated Conv and Conv1d layers were used. Am I understanding correctly that the Dilated Conv and Conv layers in the figure represent the $g$ operator? If so, can this inconsistency be resolved? Otherwise, can the figure or text be amended to address this point of confusion?

Author response:
● a)The Dildated Conv and Conv layers shown inside the Time-Freq Block (TFB) in Figure 1 do not represent the g operator. Note that we named it as DSC (DilatedConv -> SiLU -> Conv1d) network in the updated paper to make it clearer. The g operator (g_time and g_freq) is only applied in the Time Feat Moudule and Frequency Feat Module in Figure 1 which is formalized in the Eq 6.  The Time-Freq Block shown in Figure 1 comprises two compents: one is the DSC network, and the other is the time-frequency (TF) domain features extracting network. The Figure 1, Eq.6 and Section 3.5 are all updated in the revised paper.

Reviewer comment (Presentation):
I find the layout of the sub-blocks within the TFB on the left side of Figure 1 confusing. The inputs to the DC -> SiLU -> Conv1D block are the time feats or frequency feats, right? This is not apparent to me from the organization of the figure, which has arrows pointing from the DC -> SiLU -> Conv1D block to the time/freq feat modules. And why is there an arrow pointing up from the Conv1D block?

Author response:
● b)The inputs to the DSC network shown in the sub-blocks within TFB on the left side of Figure 1 are not the time feats or frequency feats. The inputs to the DSC network of the L-th layer are the sum of the inputs and outputs of DSC of the  (L-1)th layer (defined by Eq 5 in the updated paper). The inputs to the time and frequency feats are the outputs of the DSC network (red arrow in Figure 1). The outputs of the time and frequency feats are added together to produce the ultimate energy scalar defined by Eq 7.

Reviewer comment (Reproducibility):
How were missing data values (e.g., as in imputation settings) handled in computation of the FFT?

Author response:
● For both imputation and forecasting tasks, missing values are replaced with random Gaussian noise prior to the FFT transform, followed by Langevin sampling to generate model-derived samples.

Reviewer comment (Evaluation):
I don't understand the motivation for using a probabilistic score CRPS for a distributional forecast in the short-term forecasting problem, and the MAE/MSE for a point prediction in the long-term forecasting and imputation problems. It seems to me that it would be valuable to examine measures of point forecast skill and distributional forecast skill in both forecasting settings. This is particularly salient since the second sentence of the abstract reads, "However, existing models primarily focus on deterministic point forecasts, neglecting generative long-term probabilistic forecasting and pre-training models across diverse time series analytic tasks, which are essential for uncertainty quantification and computational efficiency." The abstract states that long-term probabilistic forecasting is an important problem, but we do not see CRPS results for the long-term forecasting example. Could a more complete suite of results be added?

Author response:
● We included CRPS results for the long-term (forecast horizon equals 336 and 720) forecasting in Table 8 in the Appendix. We compared against three models: DeepAR, DeepState, and TImeGrad, but not the RSGLS-ISSM or ARSGLS because their code implementations were not publicly available. We did not test the Wiki dataset as it only has 792 data points that are not enough for the long-term forecast.

Reviewer comment (Experiments):
The ablation study looking at the contribution of time and frequency extraction only looks at a subset of the data sets. I think it would be interesting to provide results for all data sets used in the paper, and to provide some examination to indicate settings in which it is more or less helpful to use both feature sets. For example, could columns be added to Table 6 giving something like the lag 1 autocorrelation of the series and the power of the first few dominant frequencies? With the hypothesis being that the relative value of using both feature groups may depend on characteristics of the time series being modeled?

Author response:
● We performed the same ablation study on the exchange rate and traffic datasets as well. The results are included in Table 6 in the Appendix, which also includes the average of the lag-1 autocorrelation, standard deviation, and four dominant power spectrums. However, such characteristics of the timer series does not seem to have strong correlation with the relative value of using both feature groups.  Actually,the model equipped with time and frequency feature extracting module performs the best on almost all datasets and forecasting ranges compared with only equipped with either time or frequency extracting module. The three exceptions on the ETTm1 and traffic datasets do not seem to have strong statistical significance.

Reviewer comment (Novelty):
The author missed some pioneering literature about basic EBM, such as:  
[1] "A theory of generative convnet." International Conference on Machine Learning. PMLR, 2016.  
[2] "Implicit generation and modeling with energy based models." Advances in Neural Information Processing Systems 32 (2019).  
[3] "Cooperative training of descriptor and generator networks." IEEE transactions on pattern analysis and machine intelligence 42.1 (2018): 27-45.

Author response:
● We have added the pioneering literature mentioned about basic EBM in the updated paper.