Generative Time Series Forecasting with Diffusion, Denoise and Disentanglement
Abstract
Time series forecasting has been a widely explored task that is of great importance in many applications.
However, it is common that real-world time series data are recorded in a short time period, which results in a big gap between the deep model and the limited and noisy time series.
In this work, we propose to address the time series forecasting problem with generative modeling and propose a bidirectional variational auto-encoder (BVAE) equipped with diffusion, denoise, and disentanglement, namely D 3 VAE.
Specifically, a coupled diffusion probabilistic model is proposed to augment the time series data without increasing the aleatoric uncertainty contributing to the data and implement a more tractable inference process with the BVAE.
To ensure the generated series move toward the true target, we further propose to adapt and integrate the multiscale denoising score matching into the diffusion process for time series forecasting.
In addition, to enhance the interpretability and stability of the prediction, we treat the latent variable in a multivariate manner and disentangle them on top of minimizing total correlation.
Extensive experiments on both synthetic and real-world data show that D 3 VAE outperforms competitive algorithms with remarkable margins. Our implementation is available at https://github.com/ramber1836/d3vae .
Introduction
Time series forecasting is of great importance for risk-averse and decision-making.
Traditional
RNN-based methods capture temporal dependencies of the time series to predict the future.
Long short-term memories (LSTMs) and gated recurrent units (GRUs) [105, 35, 32, 80] introduce the gate functions into the cell structure to handle long-term dependencies effectively.
The models based on convolutional neural networks (CNNs) capture the complex inner patterns of the time series through convolutional operations [60, 8, 7].
Recently, the Transformer-based models have shown great performance in time series forecasting [101, 106, 55, 61] with the help of multi-head self-attention.
However, one big issue of neural networks in time series forecasting is the uncertainty [31, 1] resulting from the properties of the deep structure.
The models based on vector autoregression (VAR) [12, 24, 51] try to model the distribution of time series from hidden states which could provide more reliability to the prediction, while the performance is not satisfactory [58].
Interpretable representation learning is another merit of time series forecasting.
Variational autoencoders (VAEs) have shown not only the superiority in modeling latent distributions of the data and reducing the gradient noise [75, 54, 63, 89] but also the interpretability of time series forecasting [26, 27].
However, the interpretability of VAEs might be inferior due to the entangled latent variables.
There have been efforts to learn representation disentangling [50, 6, 39], which show that the welldisentangled representation can improve the performance and robustness of the algorithm.
Moreover, real-world time series are often noisy and recorded in a short time period, which may result in overfitting and poor generalization issues [30, 81, 107, 93] 1 .
To this end, we address the time series forecasting problem with generative modeling.
Specifically, we propose a bidirectional variational auto-encoder (BVAE) equipped with diffusion, denoise, and disentanglement, namely D 3 VAE.
More specifically, we first propose a coupled diffusion probabilistic model to remedy the limitation of time series data by augmenting the input time series, as well as the output time series, inspired by the forward process of the diffusion model [85, 41, 70, 74].
Besides, we adapt the Nouveau VAE [89] to the time series forecasting task and develop a BVAE as a substitute for the reverse process of the diffusion model.
In this way, the expressiveness of the diffusion model plus the tractability of the VAE can be leveraged together for generative time series forecasting.
Though the merit of generalizability is helpful, the diffused samples might be corrupted, which results in a generative model moving toward the noisy target.
Therefore, we further develop a scaled denoising score-matching network for cleaning diffused target time series.
In addition, we disentangle the latent variables of the time series by assuming that different disentangled dimensions of the latent variables correspond to different temporal patterns (such as trend, seasonality, etc.).
Our contributions can be summarized as:
• We propose a coupled diffusion probabilistic model aiming to reduce the aleatoric uncertainty of the time series and improve the generalization capability of the generative model.
• We integrate the multiscale denoising score matching into the coupled diffusion process to improve the accuracy of generated results.
• We disentangle the latent variables of the generative model to improve the interpretability for time series forecasting.
• Extensive experiments on both synthetic datasets and real-world datasets demonstrate that D 3 VAE outperforms competitive baselines with satisfactory margins.
Methodology 2.1 Generative Time Series Forecasting
Problem Formulation.
Given an input multivariate time series X = { x 1 , x 2 , · · · , x n | x i ∈ R d } and the corresponding target time series Y = { y n +1 , y n +2 , · · · , y n + m | y j ∈ R d ′ } ( d ′ ≤ d ).
We assume that Y can be generated from latent variables Z ∈ Ω Z that can be drawn from the Gaussian distribution Z ∼ p
( Z | X ) .
The latent distribution can be further formulated as p ϕ
( Z | X ) =
g ϕ ( X ) where g ϕ denotes a nonlinear function.
Then, the data density of the target series is given by:
[Equation]
where f θ denotes a parameterized function.
The target time series can be obtained directly by sampling from p θ ( Y ) .
In our problem setting, time series forecasting is to learn the representation Z that captures useful signals of X , and map the low dimensional X to the latent space with high expressiveness.
The framework overview of D 3 VAE is demonstrated in Fig.
Before diving into the detailed techniques, we first introduce a preliminary proposition.
Proposition 1.
Given a time series X and its inherent noise ϵ X , we have the decomposition:
X = ⟨ X r , ϵ X ⟩ , where X r is the ideal time series data without noise.
X r and ϵ X are independent of each other.
Let p ϕ
( Z | X ) =
p ϕ ( Z | X r , ϵ X ) , the estimated target series (cid:98) Y can be generated with the distribution p θ ( (cid:98) Y | Z ) =
p θ ( (cid:98) Y r | Z ) · p θ ( ϵ (cid:98) Y | Z ) where (cid:98) Y r is the ideal part of (cid:98) Y and ϵ (cid:98) Y is the estimation noise.
Without loss of generality, (cid:98) Y r can be fully captured by the model.
That is, ∥ Y r − (cid:98) Y r ∥ −→where Y r is the ideal part of ground truth target series Y .
In addition, Y can be decomposed as Y = ⟨ (cid:98) Y r , ϵ Y ⟩ ( ϵ Y denotes the noise of Y ).
Therefore, the error between ground truth and prediction, i.e., ∥ Y − (cid:98) Y ∥ =
∥
ϵ Y − ϵ (cid:98) Y ∥ > 0 , can be deemed as the combination of aleatoric uncertainty and epistemic uncertainty. First, both the input and output series are augmented simultaneously with the coupled diffusion process .
Then the diffused input series are fed into a proposed BVAE model for inference, which can be deemed as a reverse process .
To make the estimated target move toward the true target series, a denoising score-matching mechanism is further applied.
Meanwhile, the latent states in BVAE are leveraged for disentangling such that the model interpretability and reliability can be improved. .
2.2 Coupled Diffusion Probabilistic Model
The diffusion probabilistic model (diffusion model for brevity) is a family of latent variable models aiming to generate high-quality samples. To equip the generative time series forecasting model with high expressiveness, a coupled forward process is developed to augment the input series and target series synchronously. Besides, in the forecasting task, a more tractable and accurate prediction is expected. To achieve this, we propose a bidirectional variational auto-encoder (BVAE) to take the place of the reverse process in the diffusion model.
We present the technical details in the following two parts, respectively. 2.2.1 Coupled Diffusion Process The forward diffusion process is fixed to a Markov chain that gradually adds Gaussian noise to the data [85, 41]. To diffuse the input and output series, we propose a coupled diffusion process, which is demonstrated in Fig.
Specifically, given the input X = X
(0) ∼ q ( X (0) ) , the approximate posterior q ( X (1: T ) | X (0) ) can be obtained as
[Equation]
where a uniformly increasing variance schedule β = { β 1 , · · · , β T | β t ∈ [0 , 1) } is employed to control the level of noise to be added.
Then, let α t = 1 − β t
and ¯ α t = (cid:81) ts =1 α s , we have
[Equation]
Furthermore, according to Proposition 1 we decompose X (0) as X (0) = ⟨ X r , ϵ X ⟩ .
Then, with Eq.
(3), the diffused X ( t ) can be decomposed as follows:
[Equation]
where δ X denotes the standard Gaussian noise of X .
As α can be determined when the variance schedule β is known, the ideal part is also determined in the diffusion process.
Let (cid:101) X ( t ) r = √ ¯ α t X r
and δ ( t ) (cid:101) X = √
¯ α t ϵ X
+ (1 − ¯ α t ) δ X , then, according to Proposition 1 and Eq. (4), we have
[Equation]
where δ ( t ) (cid:98) Y denotes the generated noise of (cid:98) Y ( t ) .
To relieve the effect of aleatoric uncertainty resulting from time series data, we further apply the diffusion process to the target series Y
= Y (0) ∼ q ( Y (0) ) .
In particular, a scale parameter ω ∈ (0 , 1) is adopted, such that β ′ t =
ωβ t , α ′ t = 1 − β ′ t and ¯ α ′ t =
(cid:81) ts =1 α ′ s .
Then, according to Proposition 1, we can obtain the following decomposition (similar to Eq.
[Equation]
Consequently, we have q ( Y ( t ) )
=
q ( (cid:101) Y ( t ) r ) q ( δ ( t ) (cid:101) Y ) .
Afterward, with Proposition 1 and Eqs.
(5) and (6) we can draw the following conclusions. The proofs can be found in Appendix B.
Lemma 1.
∀ ε > 0 , there exists a probabilistic model f ϕ,θ :
= ( p ϕ , p θ ) to guarantee that
[Equation]
Y ( t ) r ))
< ε , where (cid:98) Y ( t ) r = f ϕ,θ
( X ( t ) ) .
Lemma 2.
With the coupled diffusion process, the difference between diffusion noise and generation noise will be reduced, i.e., lim t →∞ D KL ( q ( δ ( t ) (cid:101) Y )
||
p θ ( δ ( t ) (cid:98) Y | Z ( t ) ))
< D KL ( q ( ϵ Y ) ||
p θ ( ϵ (cid:98) Y )) .
Therefore, the uncertainty raised by the generative model and the inherent data noise can be reduced through the coupled diffusion process.
In addition, the diffusion process augments the input series, as well as the target series, simultaneously, which can improve the generalization capability for (esp.
short) time series forecasting.
2.2.2 Bidirectional Variational Auto-Encoder
Traditionally, in the diffusion model, a reverse process is adopted to generate high-quality samples [85, 41].
However, for the generative time series forecasting problem, not only the expressiveness but also the supervision of the ground truths should be considered.
In this work, we employ a more efficient generative model, i.e., bidirectional variational auto-encoder (BVAE) [89], to take the place of the reverse process of the diffusion model.
The architecture of BVAE is described in Fig.where Z is treated in a multivariate fashion Z
= { z 1 , · · · , z n } ( z i ∈ R m , z i = [ z i, 1 , · · · , z i,m ] ) and z i +1 ∼ p ( z i +1 |
z i , X ) .
Then, n is determined in accordance with the number of residual blocks in the encoder, as well as the decoder.
Another merit of BVAE is that it opens an interface to integrate the disentanglement for improving model interpretability (refer to Section 2.4).
2.3 Scaled Denoising Score Matching for Diffused Time Series Cleaning
Although the time series data can be augmented with the aforementioned coupled diffusion probabilistic model, the generative distribution p θ ( (cid:98) Y ( t ) ) tends to move toward the diffused target series
Y ( t ) which has been corrupted [65, 87].
To further “clean” the generated target series, we employ the Denoising Score Matching (DSM) to accelerate the de-uncertainty process without sacrificing the model flexibility.
DSM [90, 65] was proposed to link Denoising Auto-Encoder (DAE) [91] to Score
Matching (SM) [43].
Let (cid:98) Y denote the generated target series, then we have the objective
[Equation]
where p σ 0 ( (cid:98) Y , Y ) is the joint density of pairs of corrupted and clean samples ( (cid:98) Y , Y ) , ∇ (cid:98) Y log( q σ 0 ( (cid:98) Y | Y )) is the derivative of log density of the single noise kernel which is dedicated to replacing the Parzen density estimator: p σ 0 ( (cid:98) Y )
=
(cid:82) q σ 0 ( (cid:98) Y | Y )
p ( Y ) dY in score matching, and E ( (cid:98) Y ; ζ ) is the energy function.
In the particular case of Gaussian noise, log( q σ 0 ( (cid:98) Y | Y ))
=
− ( (cid:98) Y − Y ) 2 / [Equation] σ 02 + C .
Thus, we have
[Equation]
Then, for the diffused target series at step t , we can obtain
[Equation]
To scale the noise of different levels [65], a monotonically decreasing series of fixed σ values { σ 1 , · · · , σ T | σ t = 1 − ¯ α t } (refer to the aforementioned variance schedule β in Section 2.2) is adopted.
Therefore, the objective of the multi-scaled DSM is
[Equation]
where σ ∈ { σ 1 , · · · , σ T } and l ( σ t ) = σ t .
With Eq.
(10), we can ensure that the gradient has the right magnitude by setting σ 0 .
Note that, in the generative time series forecasting setting, the generated samples will be tested without applying the diffusion process.
To further denoise the generated target series (cid:98) Y , we apply a single-step gradient denoising jump [78]:
[Equation]
The generated results tend to possess larger distribution space than the true target, and the noisy term in Eq.
(11) approximates the noise between the generated target series and the “cleaned” target series.
Therefore, σ 20 ∇ (cid:98) Y E ( (cid:98) Y ; ζ ) can be treated as the estimated uncertainty of the prediction.
2.4 Disentangling Latent Variables for Interpretation
The interpretability of the time series forecasting model is of great importance for many downstream tasks [88, 38, 44].
Through disentangling the latent variables of the generative model, not only the interpretability but also the reliability of the prediction can be further enhanced [64].
To disentangle the latent variables
Z = { z 1 , · · · , z n } , we attempt to minimize the Total Correlation (TC) [94, 50], which is a popular metric to measure dependencies among multiple random variables,
[Equation]
where m denotes the number of factors of z i that need to be disentangled.
In general, lower TC means better disentanglement if the latent variables preserve useful information.
However, a very low TC can still be obtained when the latent variables carry no meaningful signals.
Through the bidirectional structure of BVAE, such issues can be tackled without too much effort.
As shown in Fig.
1, the signals are disseminated in both the encoder and decoder, such that rich semantics are aggregated into the latent variables.
Furthermore, to alleviate the effect of potential irregular values, we average the total correlations of z 1: n , then the loss w.r.t.
the TC score of BVAE can be obtained:
[Equation]
1: repeat 2:
X (0) ∼ q ( X (0) ) ,
Y (0) ∼ q ( Y (0) ) , δ X ∼ N (0 , I d ) , δ Y ∼ N (0 , I d ) 3:
Randomly choose t ∈ { 1 , · · · , T } and with Eqs.
(4) and (6), 4:
X ( t ) =
√ ¯ α t X (0)
+ (1 − ¯ α t ) δ X ,
Y ( t ) = (cid:112) ¯ α ′ t Y (0) + (1 − ¯ α ′ t ) δ Y 5:
Generate the latent variable Z with BVAE, Z ∼ p
ϕ
( Z | X ( t ) ) 6:
Sample (cid:98) Y ( t ) ∼ p θ ( (cid:98) Y ( t ) | Z ) and calculate D KL ( q ( Y ( t ) )
||
p θ ( (cid:98) Y ( t ) )) 7:
Calculate DSM loss with Eq.
Calculate total correlation of Z with Eq.
Construct the
L
(14) 10: θ,
L ) 11: until Convergence 2.5 Training and Forecasting Training Objective. To reduce the effect of uncertainty, the coupled diffusion equipped with the denoising network is proposed without sacrificing too much generalizability.
Then we disentangle the latent variables of the generative model by minimizing the TC of the latent variables.
Finally, we reconstruct the loss with several trade-off parameters, and with Eqs.
(10), (11) and (13) we have
[Equation]
where L mse calculates the mean square error (MSE) between (cid:98) Y ( t ) and Y ( t ) .
We minimize the above objective to learn the generative model accordingly. Algorithms.
Algorithm 1 displays the complete training procedure of D 3 VAE with the loss function in Eq. For inference, as described in Algorithm 2, given the input series X , the target series can be generated directly from the distribution p θ which is conditioned on the latent states drawn from distribution p ϕ .
Experiments 3.1 Experiment Settings
Datasets.
We generate two synthetic datasets suggested by [23],
[Equation]
where w t ∈ R 2 and 0 ≤ w t, 1 , w t, 2 ≤ 1 ( t = 1 , 2 , 3 ), F ∈ R 2 × k ∼ U
[ − 1 , 1] , k denotes the dimensionality and N is the number of time points, a, b are two constants.
We set a = 0 .
9 , b = 0 .
2 , k = 20 to generate D 1 , and a = 0 .
5 , b = 0 .
5 , k = 40 for D 2 , and N = 800 for both D 1 and D 2 .
Six real-world datasets with diverse spatiotemporal dynamics are selected, including Traffic [58],
Electricity 2 , Weather 3 , Wind (Wind Power) 4 , and ETTs [106] (ETTm1 and ETTh1).
To highlight the uncertainty in short time series scenarios, for each dataset, we slice a subset from the starting point to make sure that each sliced dataset contains at most 1000 time points.
Subsequently, we [Table]
[Table] W i nd [Table] obtained 5% -Traffic, 3% -Electricity, 2% -Weather, 2% -Wind, 1% -ETTm1, and 5% -ETTh1.
The statistical descriptions of the real-world datasets can be found in Appendix [-F.
-]C.1.
All datasets are split chronologically and adopt the same train/validation/test ratios, i.e. 7:1:2.
Baselines.
We compare D 3 VAE with one GP (Gaussian Process) based method (GP-copula [76]), two auto-regressive methods (DeepAR [77] and TimeGrad [74]), and four VAE-based methods, i.e., vanilla VAE, NVAE [89], factor-VAE (f-VAE for short) [50] and β -TCVAE [15].
Implementation Details.
An input- l x -predict- l y window is applied to roll the train, validation, and test sets with stride one time step, respectively, and this setting is adopted for all datasets.
Hereinafter, the last dimension of the multivariate time series is selected as the target variable by default.
We use the Adam optimizer with an initial learning rate of 5 e − 4 .
The batch size is 16 and the training is set to 20 epochs at most equipped with early stopping.
The number of disentanglement factors is chosen from { 4 , 8 } , and β T ∈ β is set to range from 0 .
01 to 0 .
1 with different diffusion steps T ∈ [100 , 1000] , then ω is set to 0 .
The trade-off hyperparameters are set as ψ = 0 .
05 , λ = 0 .
1 , γ = 0 .
001 for ETTs, and ψ = 0 .
5 , λ = 1 . 0 , γ = 0 .
01 for others.
All the experiments were carried out on a Linux machine with a single NVIDIA P40 GPU.
The experiments are repeatedtimes and the average and variance of the predictions are reported.
We use the Continuous Ranked Probability Score (CRPS) [68] and Mean Squared Error (MSE) as the evaluation metrics.
For both metrics, the lower, the better.
In particular, CRPS is used to evaluate the similarity of two distributions and is equivalent to Mean Absolute Error (MAE) when two distributions are discrete.
3.2 Main Results
Two different prediction lengths, i.e., l y ∈ { 8 , 16 } ( l x = l y ), are evaluated.
The results of longer prediction lengths are available in Appendix D.
Toy Datasets.
In Table 1, we can observe that D 3 VAE achieves SOTA performance most of the time, and achieves competitive CRPS in D 2 for prediction length 16.
Besides, VAEs outperform
VARs and GP on D 1 , but VARs achieve better performance on D 2 , which demonstrates the advantage of VARs in learning complex temporal dependencies.
Real-World Datasets.
As for the experiments on real-world data, D 3 VAE achieves consistent SOTA performance except for the prediction length 16 on the Wind dataset (Table 2).
Particularly, under the input-8-predict-8 setting, D 3 VAE can provide remarkable improvements in Traffic, Electricity,
Wind, ETTm1, ETTh1 and Weather w.r.t.
MSE reduction (90%, 71%, 48%, 43%, 40% and 28%).
Regarding the CRPS reduction, D 3 VAE achieves a 73% reduction in Traffic, 31% in Wind, and 27% in Electricity under the input-8-predict-8 setting, and a 70% reduction in Traffic, 18% in Electricity, and 7% in Weather under the input-16-predict-16 setting.
Overall, D 3 VAE gains the averaged 43% MSE reduction and 23% CRPS reduction among the above settings.
More results under longer prediction-length settings and on full datasets can be found in Appendix D.1.
Uncertainty Estimation.
The uncertainty can be assessed by estimating the noise of the outcome series when doing the prediction (see Section 2.3).
Through scale parameter ω , the generated distribution space can be adjusted accordingly (results on the effect of ω can be found in Appendix D.3).
The showcases in Fig.
3 demonstrate the uncertainty estimation of the yielded series in the Traffic dataset, where the last six dimensions are treated as target variables.
We can find that noise estimation can quantify the uncertainty effectively.
For example, the estimated uncertainty grows rapidly when extreme values are encountered. [Table]
Disentanglement Evaluation.
For time series forecasting, it is difficult to label disentangled factors by hand, thus we take different dimensions of Z as the factors to be disentangled:
z i =
[ z i, 1 , · · · , z i,m ] ( z i ∈ Z ).
We build a classifier to discriminate whether an instance z i,j belongs to class j such that the disentanglement quality can be assessed by evaluating the classification performance.
Besides, we adopt the Mutual Information Gap (MIG) [15] as a metric to evaluate the disentanglement in a more straightforward way.
Due to the space limit, the evaluation of disentanglement with different factors can be found in Appendix E.
3.3 Model Analysis
Ablation Study of the Coupled Diffusion and Denoising Network.
To evaluate the effectiveness of the coupled diffusion model (CDM), we compare the full versioned D 3 VAE with its three variants: i) D
3 VAE − (cid:101) Y , i.e. D 3 VAE without diffused Y , ii)
D 3 VAE − (cid:101) X , i.e. D 3 VAE without diffused X , and iii) D 3 VAE − CDM , i.e. D 3 VAE without any diffusion.
Besides, the performance of D 3 VAE without denoising score matching (DSM) is also reported when the target series is not diffused, which are denoted as D 3 VAE − (cid:101)
Y − DSM and D 3 VAE − CDM − DSM .
The ablation study is carried out on Traffic and Electricity datasets under the settings of input-16-predict-16 and input-32-predict-32.
In Table 3, we can find that the diffusion process is effective in augmenting the input or the target.
Moreover, when the target is not diffused, the denoising network would be deficient since the noise level of the target cannot be estimated by then.
Variance Schedule β
and The Number of Diffusion Steps T .
To reduce the effect of the uncertainty while preserving the informative temporal patterns, the extent of the diffusion should be configured properly.
Too small a variance schedule or inadequate diffusion steps will lead to a meaningless diffusion process. Otherwise, the diffusion could be out of control 5 .
Here we analyze the effect of the variance schedule β and the number of diffusion steps T .
We set β 1 = 0 and change the value of β T in the range of [0 . 01 , 0 . 1] , and T ranges from 100 to 4000 .
As shown in Fig.
4, we can see that the prediction performance can be improved if proper β and T are employed.
Discussion
Sampling for Generative Time Series Forecasting.
The Langevin dynamics has been widely applied to the sampling of energy-based models (EBMs) [21],
[Equation]
where k ∈ { 0 , · · · , K } , K denotes the number of sampling steps, and ρ is a constant.
With K and ρ being properly configured, samples with high quality can be generated.
The Langevin dynamics has been successfully applied to image reconstruction and natural language processing [56, 20].
In this work, we employ a single-step gradient denoising jump to generate the target series.
The experiments hat were carried out demonstrate the effectiveness of such single-step sampling.
To investigate whether it is worth taking more sampling steps for further performance improvement of time series forecasting, we conduct an extra empirical study.
We showcase the prediction results under different sampling strategies in Fig.
By omitting the additive noise in Langevin dynamics, we employ the multi-step denoising for D 3 VAE to generate the target series and plot the generated results in Fig. 5a.
Then, with the standard Langevin dynamics, we can implement a generative procedure instead of denoising and compare the generated target series with different ρ (see Figs. 5b to 5d).
We can observe that, more sampling steps might not be helpful in improving prediction performance for generative time series forecasting (Fig. 5a).
Besides, larger sampling steps would lead to high computational complexity.
On the other hand, different configurations of Langevin dynamics (with varying ρ ) cannot bring indispensable benefits for time series forecasting (Figs. 5b to 5d).
[Figure]
[Figure]
With the coupled diffusion probabilistic model, the aleatoric uncertainty of time series can be reduced though, new bias is brought into the series to mimic the distribution of the input and target.
However, as a common issue in VAEs that any introduced bias in the input will result in bias in the generated output [92], the diffusion steps and variance schedule need to be chosen cautiously, such that this model can be applied to different time series tasks smoothly.
The proposed model is devised for general time series forecasting, it should be used properly to avoid the potential negative societal impacts, such as any kind of illegal applications.
In time series predictive analysis, disentanglement of the latent variables has been of great importance for interpreting the prediction to provide more reliance.
Due to the lack of prior knowledge of the entangled factors in generative time series forecasting, only unsupervised disentanglement learning can be done, which has been proven theoretically feasible for time series [64].
Despite this, for boarder applications of disentanglement and better performance, it is still worth exploring the ways of labeling the factors of time series in the future.
Moreover, because of the uniqueness of time series data, it is also a promising direction to explore more generative and sampling methods for the time series generation task.
Conclusion
In this work, we propose a generative model with the bidirectional VAE as the backbone.
To further improve the generalizability, we devise a coupled diffusion probabilistic model for time series forecasting.
Then a scaled denoising network is developed to guarantee the prediction accuracy.
Afterward, the latent variables are further disentangled for better model interpretability.
Extensive experiments on synthetic data and real-world data validate that our proposed generative model achieves SOTA performance compared to existing competitive generative models. Acknowledgement We thank Longyuan Power Group Corp. Ltd. for supporting this work. A Related Work A.1 Time Series Forecasting
We first briefly review the related literature of time series forecasting (TSF) methods as below.
Complex temporal patterns can be manifested over short- and long-term as the time series evolves across time.
To leverage the time evolution nature, existing statistical models, such as ARIMA [9] and Gaussian process regression [10] have been well established and applied to many downstream tasks [40, 42, 3]. Recurrent neural network (RNN) models are also introduced to model temporal dependencies for TSF in a sequence-to-sequence paradigm [35, 13, 95, 58, 67, 73, 77]. Besides, temporal attention [72, 86, 82] and causal convolution [4, 8, 79] are further explored to model the intrinsic temporal dependencies. Recent Transformer-based models have strengthened the capability of exploring hidden intricate temporal patterns for long-term TSF [101, 61, 99, 106].
On the other hand, the multivariate nature of TSF is another topic many works have been focusing on.
These works treat a collection of time series as a unified entity and mine the inter-series correlations with different techniques, such as probabilistic models [77, 73], matrix/tensor factorization [79 ? ], convolution neural networks (CNNs) [4, 58], and graph neural networks (GNNs) [37, 62, 104, 100, 11].
To improve the reliability and performance of TSF, instead of modeling the raw data, there exist works inferring the underlying distribution of the time series data with generative models [103, 18].
Many studies have employed a variational auto-encoder (VAE) to model the probabilistic distribution of sequential data [29, 34, 16, 14, 69]. For example, VRNN [16] employs the VAE to each hidden state of RNN such that the variability of highly structured sequential data can be captured. To yield predictive distribution for multivariate TSF, TLAE [69] implements nonlinear transformation by replacing matrix factorization with encoder-decoder architecture and temporal deep temporal latent model. Another line of generative methods for TSF focus on energy-based models (EBMs), such as TimeGrad [74] and ScoreGrad [102].
EBMs do not place a restriction on the tractability of the normalizing constants [102].
Though flexible, the unknown normalizing constant makes the training of EBMs particularly difficult.
This paper focuses on TSF with VAE-based models.
Besides, as many real-world time series data are relatively short and small [84], a coupled probabilistic diffusion model is proposed to augment the input series, as well as the output series, simultaneously, such that the distribution space can be enlarged without increasing the aleatoric uncertainty [48].
Moreover, to guarantee the generated target series moving toward the true target, a multi-scaled score-matching denoising network is plugged in for accurate future series prediction.
To the best of our knowledge, this is the first work focusing on generative TSF with the diffusion model and denoising techniques. A.2 Time Series Augmentation Both traditional methods and deep learning methods can deteriorate when limited time series data are encountered. Generating synthetic time series is commonly adopted for augmenting short time series [17, 25, 103].
Transforming the original time series by cropping, flipping and warping [46, 19] is another kind of approach dedicated to TSF when the training data is limited.
Whereas the synthetic time series may not respect the original feature relationship across time, and the transformation methods do not change the distribution space.
Thus, the overfitting issues cannot be avoided.
Incorporating the probabilistic diffusion model for TSF differentiates our work from existing time series augmentation methods.
A.3 Uncertainty Estimation and Denoising for Time Series Forecasting
There exist works aiming to estimate the uncertainty [48] for time series forecasting [71, 96, 36] by epistemic uncertainty.
Nevertheless, the inevitable aleatoric uncertainty of time series is often ignored, which may stem from error-prone data measurement, collection, and so forth [97].
Another line of studies focuses on the detection of noise in time series data [66] or devising suitable models for noise alleviation [33].
However, none of the existing works attempts to quantify the aleatoric uncertainty, which further differentiates our work from priors.
It is necessary to relieve the effect of noise in real-world time series data [22].
[5, 57] propose to preprocess the time series with smoothing and filtering techniques.
However, such preprocessing methods can only be applied to the noise raised by the irregular data of time series.
Neural networks are also introduced to denoise the time series [28, 83, 33, 47], while these deep networks can only deal with specific types of time series as well.
A.4 Interpretability of Time Series Forecasting
A number of works put effort to explain the deep neural networks [98, 49, 2] to make the prediction more interpretable, but these methods often lack reliability when the explanation is sensitive to factors that do not contribute to the prediction [52].
Several works have been proposed to increase the reliability of TSF tasks [44, 45]. For multivariate time series, the interpretability of the representations can be improved by mapping the time series into latent space [27]. Besides, recent works have been proposed to disentangle the latent variables to identify the independent factors of the data, which can further lead to improved interpretability of the representation and higher performance [39, 59, 50]. The disentangled VAE has been applied to time series to benefit the generated results [64].
However, the choice of the latent variables is crucial for the disentanglement of time series data.
We devise a bidirectional VAE (BVAE) and take the dimensions of each latent variable as the factors to be disentangled. B Proofs of Lemma 1 and Lemma
With the coupled diffusion process and Eqs.
(5) and (6), as well as Proposition 1, introduced in the main text, the diffused target series and generated target series can be decomposed as (cid:101) Y ( t ) = ⟨ (cid:101) Y ( t ) , δ ( t ) (cid:101) Y ⟩ and (cid:98) Y ( t ) = ⟨ (cid:98) Y ( t ) , δ ( t ) (cid:98) Y ⟩ . Then, we can draw the following two conclusions:
Lemma 1.
∀ ε > 0 , there exists a probabilistic model f ϕ,θ :
= ( p ϕ , p θ ) to guarantee that D KL ( q ( (cid:101) Y ( t ) r ) ||
p
θ ( (cid:98) Y ( t ) r )) < ε , where (cid:98) Y ( t ) r = f ϕ,θ ( X ( t ) ) .
Proof. According to Proposition 1, (cid:98) Y r can be fully captured by the model. That is, ∥ Y r − (cid:98) Y r ∥ −→where Y r is the ideal part of ground truth target series Y .
And, with Eq. (6) (in the main text), (cid:101) Y ( t ) r = (cid:112) ¯ α ′ t Y r . Therefore, ∥ (cid:101) Y ( t ) r − (cid:98) Y ( t ) r ∥ −→ 0 when t → ∞ . Lemma 2.
With the coupled diffusion process, the difference between diffusion noise and generation noise will be reduced, i.e., lim t →∞
[Equation]
Proof.
According to Proposition 1, the noise of Y consists of the estimation noise ϵ (cid:98) Y and residual noise δ Y , i.e., ϵ Y = ⟨ ϵ (cid:98) Y , δ Y ⟩ where ϵ (cid:98) Y and δ Y are independent of each other, then q ( ϵ Y ) =
q ( ϵ (cid:98) Y ) q ( δ Y ) .
Let ∆ =
D KL ( q ( ϵ Y ) ||
p θ ( ϵ (cid:98) Y )) − D KL ( q ( ϵ (cid:98) Y ) || p θ ( ϵ (cid:98) Y )) , we have
[Equation]
Y )
||
p θ ( ϵ (cid:98) Y ))
=
D KL ( q ( δ Y ) ||
p θ ( ϵ (cid:98) Y )) > 0 , which leads to D KL ( q ( ϵ Y ) ||
p θ ( ϵ (cid:98) Y ))
> D KL ( q ( ϵ (cid:98) Y )
||
p θ ( ϵ (cid:98) Y ) > 0 .
Moreover, both δ ( t ) (cid:101) Y and δ ( t ) (cid:98) Y are Gaussian noises, when t → ∞ , ∃ ε ′ > 0 , we have D KL ( q ( δ ( t ) (cid:101) Y ) || p θ ( δ ( t ) (cid:98) Y | Z ( t ) )) ≤ ε ′ <
D KL ( q ( ϵ Y ) ||
p θ ( ϵ (cid:98) Y )) . C Extra Implementation Details C.1 Experimental Settings Datasets Description.
The main descriptive statistics of the real-world datasets adopted in the experiments of this work are demonstrated in Table 5. Input Representation. We adopt the embedding method introduced in [106] and feed it to an RNN to extract the temporal dependency.
Then we concatenate them as follows:
[Equation] [Table] where X is the raw time series data and E ( · ) denotes the embedding operation.
Here, we use a two-layer gated recurrent unit (GRU), and the dimensionality of the hidden state and embeddings are 128 and 64 , respectively. Diffusion Process Configuration. Besides, the diffusion process is configured to be β t ∈ [0 , 0 . 1] and
T = 100 for the Weather dataset, β t ∈ [0 , 0 . 1] and T = 1000 for the ETTh1 dataset, β t ∈ [0 , 0 .
08] and T = 1000 for the Wind dataset, and β t ∈ [0 , 0 . 01] and T = 1000 for the other datasets.
C.2 Implementation Details of Baselines
We select previous state-of-the-art generative models as our baselines in the experiments on both synthetic datasets and real-world datasets.
Specifically, 1) GP-copula [76] is a method based on the Gaussian process which is devoted to high dimensional multivariate time series, 2) DeepAR [77] combines traditional auto-regressive models with RNNs by modeling a probabilistic distribution in an auto-encoder fashion, 3) TimeGrad [74] is an auto-regressive model for multivariate probabilistic time series forecasting with the help of an energy-based model, 4) Vanilla VAE (VAE for short) [53] is a classical statistical variational inference method on top of auto-encoder, 5) NVAE [89] is a deep hierarchical VAE built for image generation using depth-wise separable convolutions and batch normalization,
6) factor-VAE (f-VAE for short) [50] disentangles the latent variables by encouraging the distribution of representations to be factorial and independent across dimensions, and 7) βTCVAE
[15] learns the disentangled representations with total correlation variational auto-encoder algorithm. To train DeepAR, TimeGrad, and GP-copula in accordance with their original settings, the batch is constructed without shuffling the samples.
That is, the instances (sampled with the input- l x -predict- l y rolling window and l x = l y , as illustrated in Fig. 6) are fed to the training procedure of these three baselines in chronological order. Besides, these three baselines employ the cumulative distribution function (CDF) for training, so the CDF needs to be reverted to the real distribution for testing.
For f-VAE, β -TCVAE, and VAE, since the dimensionality of different time series varies, we design a preprocess block to map the original time series into a tensor with the fix-sized dimensionality which can further suit the VAEs well. The preprocess block consists of three nonlinear layers with the sizes of the hidden states to be: { 128 , 64 , 32 } . For NVAE, we keep the original settings suggested in [89] and use Gaussian distribution as the prior.
All the baselines are trained using early stopping and the patience is set to 5.
[Table]
E l ec
[Table] r i c it [Table] W ea
[Table] h e r [Table] [Table]
ETT h
[Table] W i nd [Table] [Table]
Longer-Term Time Series Forecasting. To further inspect the performance of our method, we additionally conduct more experiments for longer-term time series forecasting.
In particular, by configuring the output length with 32 and 64 6 , we compare D 3 VAE to other baselines in terms of MSE and CRPS, and the results (including short-term and long-term) are reported in Table 6. We can conclude that D 3 VAE also outperforms the competitive baselines consistently under the longer-term forecasting settings. Time Series Forecasting in Full Datasets.
Moreover, we evaluate the predictive performance for time series forecasting in two “full-version” datasets, i.e. 100%-Electricity and 100%-ETTm1.
The split of train/validation/test is 7:1:2 which is the same as the main experiments.
The comparisons in terms of MSE and CRPS can be found in Table 7.
With sufficient data, compared to previous state-of-the-art generative models, the MSE and CRPS reductions of our method are also satisfactory under different settings (including input-16-predict-16 and input-32-predict-32).
For example, in
Electricity dataset, compared to the second best results, D 3 VAE achieves 52% (0.680 → 0.330) and 54% (0.727 → 0.336) MSE reductions, and 34% (0.675 → 0.445) and 36% (0.692 → 0.444) CRPS reductions, under input-16-predict-16 and input-32-predict-32 settings, respectively.
D.2 Case Study
We showcase the prediction results of our model and seven baseline models on the Traffic and
Electricity datasets in Figs.
7 and 8.
Our model can provide the most accurate forecasting results in terms of trends and variations.
D.3 The Effect of Scale Parameter ω
We demonstrate the forecasting results with different values of ω on Electricity and Traffic datasets, and the results are plotted in Figs.
9 and 10.
It can be depicted that larger or smaller ω would lead to deviated prediction which is far from the ground truth.
Therefore, the value of ω does affect the prediction performance, which should be tuned properly.
D.4 Sensitivity Analysis of Trade-off Parameters in Reconstruction Loss L
To examine the effect of the trade-off hyperparameters in loss L , we plot the mean square error (MSE) against different values of trade-off parameters, i.e. ψ , λ and γ , in the Traffic dataset.
Note that the relative value of MSE is plotted to ensure the difference is distinguishable.
This experiment is conducted under different settings: input-8-predict-8, input-16-predict-16, and input-32-predict-32.
For ψ , the value ranges from 0 to 0 .
8 , λ ranges from 0 to 1 . 6 , and γ ranges from 0 to 0 .
The results are shown in Fig.
We can see that the performance of the model varies slightly as the trade-off parameters take different values, which shows that our model is robust enough against different trade-off parameters.
[Table]
[Table]
D.5 Scalability Analysis of Varying Time Series Length and Dataset Size
We additionally investigate the scalability of D 3 VAE against different lengths of the time series and varying amounts of available data.
The experiments are conducted on the Electricity and Traffic datasets, and the results are reported in Tables 8 and 9, respectively.
We can observe that the predictive performance of D 3 VAE is relatively stable under different settings.
In particular, the longer target series to predict the worse performance might be obtained.
Besides, when the amount of available data is shrunk, D 3 VAE performs more stable than expected.
Note that on the 20%-Electricity dataset, the performance of D 3 VAE is much worse than other subsets of the Electricity dataset, this is mainly because the sliced 20%-Electricity dataset involves more irregular values.
E Disentanglement for Time Series Forecasting
Fig.
13 illustrates the disentanglement of latent variable Z for time series forecasting.
It is difficult to choose suitable disentanglement factors under the unsupervised learning of disentanglement.
Therefore, we attempt to inspect the TSF performance against different numbers of factors to be disen-
[Figure]
tangled.
We implement a simple classifier as a discriminator to further evaluate the disentanglement quality in Fig.
12 (and Algorithm 3 demonstrates the training procedure of the discriminator).
To be specific, we take different dimensions of Z as the factors to be disentangled: z i = [ z i, 1 , · · · , z i,m ] ( z i ∈ Z ), then an instance consisting of factor and label ( z i,j , j ) is constructed. We shuffle these m examples for each z i and attempt to classify them with a discriminator, then the disentanglement can be evaluated by measuring the loss of the discriminator.
The learning curve of the discriminator can be leveraged to assess the disentanglement and the discriminator is implemented by an MLP with six nonlinear layers and 100 hidden states.
The results of prediction, discriminator loss and the total correlation w.r.t.
different numbers of factors are plotted in Fig.
12, respectively.
As shown in Fig.
12, the number of factors does affect the prediction performance, as well as the disentanglement quality.
On the other hand, the learning curves can be converged when different number of factors are adopted, which validates that the disentanglement of the latent factors is of high quality.
In addition to the above method evaluating the disentanglement indirectly, we adopt another metric named Mutual Information Gap (MIG) [15] to evaluate the quality of disentanglement in a more straightforward way.
Specifically, for a latent variable z i ∈ Z , the mutual information between z i,j , and a factor v k ∈ [1 , m ] can be calculated by
[Equation]
where d denotes the sample of z i,j and S v k is the support set of v k .
Then, for z i,j
[Equation]
[Figure]
where submax means the second max value of I d ( z i,j , v k ) , then the MIG of Z can be obtained as
[Equation]
We evaluate the quality of disentanglement in terms of MIG on ETTm1 and ETTh1 datasets, respectively, which can be seen in Fig.
From Figs. 14a and 14b, when the weight of disentanglement (i.e., γ in Eq.
(14) of the main text) grows, the disentangled factors are of higher quality.
In other words, the latent variables can be disentangled with the help of the disentanglement module in D 3 VAE.
In addition, we examine the changes in MIG against different numbers of factors.
We can observe that the difficulty of disentanglement climbs up as the number of factors increases.
F Model Inspection: Coupled Diffusion Process
To gain more insights into the coupled diffusion process, we demonstrate how a time series can be diffused under different settings in terms of variance schedule β and the max number of diffusion steps T .
The examples are illustrated in Fig.
It can be seen that when larger diffusion steps or a wider variance schedule is employed, the diffused series deviates far from the original data gradually, which may result in the loss of useful signals, like, temporal dependencies.
Therefore, it is important to choose a suitable variance schedule and diffusion steps to ensure that the distribution space is deviated enough without losing useful signals.
G Necessity of Data Augmentation for Time Series Forecasting
Limited data would result in overfitting and poor performance.
To demonstrate the necessity of enlarging the size of data for time series forecasting when deep models are employed, we implement a two-layer RNN and evaluate how many time points are required to ensure the generalization ability.
A synthetic dataset is adopted for this demonstration.
According to [23], we generate a toy time series dataset with n time points in which each point is a d -dimension variable:
[Equation]
where w t ∈ R 2 , F ∈ R 2 × d ∼ U [ − 1 , 1] , ϵ ∼ N (0 , I ) , v ∼ N (0 , 0 .
5 I ) , and d = 5 .
An input-8predict-8 window is utilized to roll this synthetic dataset.
We split this synthetic dataset into training and test sets with a ratio of 7:3.
We train the RNN in 100 epochs at most, and the MSE loss of training and testing are plotted in Fig.
It can be seen that the inflection points of the loss curves move back gradually and disappear as increasing the size of the dataset.
Besides, with fewer time points, like, 400, the model can be overfitted more easily.
[Figure]
[Figure]
T =
T =
T =
T =
T =
T =
[Figure]