Learning Latent Seasonal-Trend Representations for Time Series Forecasting
Abstract
Forecasting complex time series is ubiquitous and vital in a range of applications but challenging.
Recent advances endeavor to achieve progress by incorporating various deep learning techniques (e.g., RNN and Transformer) into sequential models.
However, clear patterns are still hard to extract since time series are often composed of several intricately entangled components.
Motivated by the success of disentangled variational autoencoder in computer vision and classical time series decomposition, we plan to infer a couple of representations that depict seasonal and trend components of time series.
To achieve this goal, we propose LaST, which, based on variational inference, aims to disentangle the seasonal-trend representations in the latent space.
Furthermore, LaST supervises and disassociates representations from the perspectives of themselves and input reconstruction, and introduces a series of auxiliary objectives.
Extensive experiments prove that
LaST achieves state-of-the-art performance on time series forecasting task against the most advanced representation learning and end-to-end forecasting models.
For reproducibility, our implementation is publicly available on Github 1 .
Introduction
Time series forecasting plays a significant role in plethora of modern applications, ranging from climate analysis [1], energy production [2], traffic flows [3] to financial markets and various industrial systems [4].
The ubiquity and importance of time series data have recently attracted researcher efforts resulting in a myriad of deep learning forecasting models [5, 6] ameliorating the time series forecasting.
Based on advanced techniques such as RNN and Transformer [7]), these methods usually learn latent representations to epitomize every instant of the signals and then derive forecasting results by a predictor, achieving great progress on forecasting tasks.
However, these models have difficulties to extract exact/clear information related to temporal patterns (e.g., seasonality, trend, and level), especially in supervised end-to-end architecture without any constraint on representations [8].
As a consequence, efforts have been made to apply the variational inference into time series modeling [9, 10, 11], where improved guidance for latent representations with probabilistic form has been proved beneficial to downstream time series tasks [12].
However, when various intricately co-evolving constituents exist a time series data, analyzing with a single representation will result in superficial variable and models’ non-reusablity and lack of interpretability, due to the highly entangled nature of neural networks [13, 14].
Thus, while providing efficiency and effectiveness, existing approaches with a single high-dimensional representation sacrifice the information utilization and explainability, which may further lead to overfitting and degenerated performance.
To address the above limitations and seek a new disentangled time series learning framework, we leverage the ideas from the decomposition strategy [15, 16] to split time series data into several components, each of which captures an underlying pattern category.
The decomposition assists the analysis process and reveals underlying insights more consistently with human intuition.
This insight motivates us to produce a couple of latent representations that respond to different time series characteristics (in our case the seasonal and trend), from which we predict the results by formulating sequence as the sum of these characteristics.
The representations should be as independent as possible to avoid a model prone to feature entanglement, while also have sufficient information to the input sequence.
Towards that, we propose a novel framework LaST to learn the La tent S easonal- T rend representations for time series forecasting.
LaST exploits an encoder-decoder architecture and follows variational inference theory [17] to learn a couple of disentangled latent representations that describes seasonal and trend of time series.
To achieve this goal, LaST enforces the representations into disentanglement under two constraints: (1) from the input reconstruction, we dissect intrinsic seasonal-trend patterns which can be readily obtained from raw time series and off-the-shelf measurement methods, and accordingly design a series of auxiliary objectives; (2) from representations themselves, we minimize the mutual information (MI) between seasonal and trend representations on the premise that grantee the consistency between input data and each of them.
Our main contributions are threefold:
• We start with the variational inference and information theory to design the seasonal-trend representations learning and disentanglement mechanisms, and practically demonstrate their effectiveness and superiority (over the existing baselines) on time series forecasting task.
• We propose LaST, a novel latent seasonal-trend representations learning framework, which encodes input as disentangled seasonal-trend representations and provides a practicable approach that reconstructs seasonal and trend separately to avoid chaos.
• We introduce MI terms as a penalty and present a novel tractable lower bound and an upper bound for their optimizations.
The lower bound ameliorates the biased gradient issue in prevalent MINE approach and ensures informative representations.
The upper bound provides the feasibility to further reduce the overlapping of seasonal-trend representations.
Related work
Most of the deep learning methods for time series forecasting are designed as an end-to-end architecture.
Various basic techniques (e.g., residual structure [18, 19], autoregressive network [20, 21], and convolutions [22, 23]) are exploited to produce expressive non-linear hidden states and embeddings that reflect the temporal dependencies and patterns.
There is also a body of works that apply
Transformer [7] structure into time series forecasting tasks [24, 25, 26, 6], aiming to discover the relationships across the sequence and focus on the important time points.
Deep learning methods have achieved superior performance in comparison to the classical algorithms such as ARIMA [27] and VAR [28], and have become prevalent in multiple applications.
Learning flexible representations has been demonstrated to be beneficial for downstream tasks by numerous researches [12, 29].
In time series representations domain, early methods, employing the variational inference, jointly train an encoder and corresponding decoder that reconstructs raw signals to learn approximate latent representations [10, 30, 31].
Recent efforts have improve these variational methods [32, 33] by establishing more complex and flexible distributions using techniques such as copula [32] and normalizing flow [34, 35].
Another group of works exploited the burgeoning contrastive learning to obtain invariant representations from augmented time series [36, 37, 38], which avoids the reconstruction process and improves representations without additional supervision.
Time series decomposition [15, 16] is a classical method that splits complex time series into several components to obtain the temporal patterns and interpretability.
Recent works have applied machine learning and deep learning approaches [39, 40, 41] to robustly and efficiently achieve the decomposition on large-scale datasets.
There are also research results which tackle forecasting with the assistance of decomposition.
For example, Autoformer [26] decomposes time series into seasonal-trend parts by average pooling and introduces an autocorrelation mechanism to empower
Transformer [7] for better relations discovery.
CoST [38] encodes signals into seasonal and trend representations in frequency and temporal domain, respectively, and introduces contrastive learning to supervise their learning.
Different from our work, these methods exploit simple average pooling decomposition mechanism which may provide incompatible periodical assumptions, or intuitively disentangle the representations by processing in different domains.
Meanwhile, LaST adaptively epitomizes the seasonality and trend by disentangled representations and boosts their disassociation from a probabilistic perspective in the latent space.
Latent seasonal-trend representations learning framework
We now formalize the problem definition and introduce the proposed LaST framework.
We note that in LaST we use seasonal and trend characteristics for disentanglement learning but our framework can be easily extended to adapt situations that have more than two components to dissociate.
Problem definition.
Consider a time series dataset D consisting of N i.i.d.
sequences denoted as X ( i ) 1: T = { x ( i ) 1 , x ( i ) 2 , · · · , x ( i ) t , · · · , x ( i ) T } , where i ∈ { 1 , 2 , . . .
, N } , and each x ( i ) t is univariate or multivariate value representing the current observation(s) at time instant t (e.g., price and rainfall).
We aim to derive a model that outputs expressive representations Z 1: T , suitable for predicting future sequence(s) Y =
ˆ X T +1: T
+ τ .
Hereafter, when there is no ambiguity we omit the superscripts and the subscript 1: T .
A model that infers the likelihood between observation X and future Y with latent representation Z can be formulated as follows:
[Equation]
From the perspective of variational inference (cf.
[17]), the likelihood P ( X | Z ) is calculated by a posterior distribution Q ϕ ( Z | X ) and maximized by the following evidence lower bound (ELBO
[Equation]
where Θ is composed of ψ , ϕ , and θ denotes learned parameters
[Equation] [Figure] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table]
+gEeegS1dEtaqAmokihZ/SK3hxwXpx352PWuuQUMwfoD5zPH6iCkYI=</latexit> ˜ Z t <latexit sha1_base64="blo2+iWLmYnB0p/MRZ06BwhpEnw=">AAAB6HicbVDLTgJBEOzFF+IL9ehlIjHxRHbVqEeiF4+QyMPAhswODYzMzm5mZk3Ihi/w4kFjvPpJ3vwbB9iDgpV0UqnqTndXEAuujet+O7mV1bX1jfxmYWt7Z3evuH/Q0FGiGNZZJCLVCqhGwSXWDTcCW7FCGgYCm8Hoduo3n1BpHsl7M47RD+lA8j5n1Fip9tAtltyyOwNZJl5GSpCh2i1+dXoRS0KUhgmqddtzY+OnVBnOBE4KnURjTNmIDrBtqaQhaj+dHTohJ1bpkX6kbElDZurviZSGWo/DwHaG1Az1ojcV//Paielf+ymXcWJQsvmifiKIicj0a9LjCpkRY0soU9zeStiQKsqMzaZgQ/AWX14mjbOyd1k+r12UKjdZHHk4gmM4BQ+uoAJ3UIU6MEB4hld4cx6dF+fd+Zi35pxs5hD+wPn8AbnZjOU=</latexit> Y
[Figure]
However, as pointed out in Sec.
1, this faces and entanglement problem and cannot clearly extract complicated temporal patterns.
To ameliorate this limitation, we incorporate the decomposition strategy into our LaST that learns a couple of disentangled representations to depict seasonal and trend dynamics.
Specifically, we formulate the temporal signals X and Y as the sum of seasonal and trend components, i.e., X =
X s + X t .
Accordingly, the latent representation Z is factorized into Z s and Z t , assumed to be independent of each other – i.e., P ( Z ) =
P ( Z s ) P ( Z t ) .
Figureillustrates the two parts of the LaST framework: (a) representations learning, producing disentangles seasonal-trend representations (separate reconstructions and MI constraints); and (b) prediction based on learned representations.
Theorem 1.
With the decomposition strategy, Eq.
(2) (i.e., the ELBO) naturally has the following factorized form:
[Equation]
The detailed inference process of the above formula is provided in Appendix A.1.
The ELBO is split into three main units, i.e., Eqs.
(3), (4), and (5).
The predictor makes forecasting and measures the accuracy (e.g,.
L1 or L2 losses), reconstruction and KL divergence are served as regularization terms aiming to improve the learned representations.
The three units are described in the following.
Predictor: The predictor (cf.
Eq.
(3) and Figure 1(b)) can be regarded as the sum of two independent parts: log
[Equation]
Here we introduce two specialized approaches to harness the seasonal-trend representations combining their own characteristics.
Given the seasonal latent representation Z s ∈ R T × d , the seasonal predictor first employs the discrete Fourier transform (DFT) algorithm to detect the seasonal frequencies, i.e.,
Z s F = DFT ( Z s ) ∈ C F × d , where F =
⌊ T +1 2 ⌋ due to the Nyquist theorem [42].
Then, we inverse the frequencies back to the temporal domain to extend the representation to the future part, i.e., ˜
Z s = iDFT ( Z s F ) ∈ R τ × d .
More details of the DFT and iDFT functions can be found in Appendix
B.1.
Given Z t , trend predictor provides a feed forward network (FFN)
f : T → τ to produce a predictable representation
˜ Z t ∈ R τ × d .
We end the predictor with two FFNs to map
˜ Z s and ˜ Z t into
Y s and Y t , respectively, and obtain the forecasting result Y by their sum.
Reconstruction and KL divergence:
Among these two terms, the KL divergence can be easily estimated by Monte Carlo sampling with prior assumptions.
Here we take a widely used setting that priors both follow N (0 , I ) for efficiency, more discussions of our priors can be found in Appendix
C. As for reconstruction term, it cannot be directly measured owing to the unknown X s and X t .
Besides, merging these two terms into E Q ϕs,ϕt ( Z s ,Z t | X ) [log P θ s ,θ t
X | Z s , Z t )] will result in chaos since the decoder is prone to reconstruct the intricate time series from every representation.
Theorem 2.
With the Gaussian distribution assumption, the reconstruction loss L rec can be estimated without leveraging X s and X t , and Eq.
(4) can be replaced with the following formula:
[Equation] [Equation]
where A XX ( κ ) = (cid:80) T − κ i {+=
+} ( X t − ¯ X )( X t + κ − ¯ X ) is the autocorrelation coefficient with lagged value κ (we employ an efficient implementation in frequency domain [43], details are in Appendix B.2),
CORT ( X, ˆ X t ) is the temporal correlation coefficient, and ∆ X i =
X i − X
i − 1 is the first difference.
The proof is provided in Appendix A.2.
According to Eq.
(6), the reconstruction loss now can be estimated and, conversely, used to supervise disentangled representation learning.
However, we find that this framework still holds certain drawbacks: (1) The KL divergence tends to narrow down the distance between posterior and prior.
The modeling choice tends to sacrifice the variational inference vs. data fit when modeling capacity is not sufficient to achieve both [44].
The posterior may become almost non-informative for the inputs, which causes the forecastings irrelevant to the observations.
The disentanglement of the seasonal-trend representations is boosted indirectly by the separate reconstruction, where we need to impose a direct constraint on the representations themselves.
We alleviate these limitations by introducing additional mutual information regularization terms.
Specifically, we increase the mutual information between Z s , Z t and X to alleviate the divergence narrowing problem [44, 45], while decreasing mutual information between Z s and Z t to further dissociate their representations.
The maximizing objective of LaST becomes
[Equation]
where I ( · , · ) denotes the mutual information between two representations.
However, the two mutual information terms are untraceable [46, 47, 48].
We address this problem in the next section.
We now address the traceable mutual information bounds, maximizing I ( X, Z s ) and I ( X, Z t ) , and minimizing I ( Z s , Z t ) in Eq.
(8), and provide lower and upper bounds for the model optimization.
Lower bound for I ( X, Z s ) or I ( X, Z t ) .
We omit the superscript s or t when analyzing lower bound.
Among the prior approaches exploring the lower bounds for MI [49, 50, 51], MINE [51], for example, employs KL divergence between the joint distribution and marginals and defines an energybased variational family to achieve a flexible and scalable lower bound.
This can be formulated as I ( X, Z ) ≥ E Q ϕ ( X,Z )
[ γ α ( X, Z )] − log E Q ( x ) Q ϕ ( z ) [ e γ α ( X,Z ) ]
= I MINE , where γ α is a learned normalized critic with parameters α .
However, this bound suffers from the biased gradient owing to the parametric logarithmic term (see Appendix A.3 for a proof).
Inspired by [47], we substitute the logarithmic function by its tangent family to ameliorate the above biased bound:
[Equation]
(9) where η denotes the different tangent points.
The first inequality relies on the concave negative logarithmic function – the values on the curve are upper bounds for that on the tangent line, and is tight when the tangent point overlaps the independent variable, i.e., the true value of E Q ( x ) Q ( z )
[ e γ ( X,Z ) ] .
The closer the distance between tangent point and independent variable, the greater the lower bound.
Therefore, we set η as the variational term E Q ( x ) Q ϕ ( z ) [ e γ α ( X,Z ) ] that estimates the independent variable to obtain as great lower bound as possible.
In the second inequality, γ α ( x, z ) – a critic function activated by Sigmoid – is limited within [0 , 1] and thus − (log η − 1) ≥ 0 .
This inequality is tight only if E Q ( x ) Q ϕ ( z ) [ γ α ( X, Z )] = 1 , which means γ α can discriminate whether a pair of variables ( X, Z ) is sampled from the joint distribution or marginals.
Similarly to MINE, this consistency problem can be addressed by the universal approximation theorem for neural networks [52].
Thus,
Eq.
(9) provides a flexible and scalable lower bound for I ( X, Z ) with unbiased gradient.
For the evaluation, we exploit a traceable manner [53, 51] that draws joint samples ( X ( i ) , Z ( i ) ) by
Q ( Z ( i ) |
X ( i ) )
P D ( X ( i ) ) .
As for the marginal Q ϕ ( Z ) , we randomly select a datapoint j and then sample it from Q ϕ
( Z | X ( j ) )
P D ( X ( j ) ) .
Details of the optimization are shown in Algorithm 1.
Upper bound for I ( Z s , Z t ) .
Few efforts have been made that explore the traceable upper bound for mutual information [54, 47, 55].
Existing upper bounds (listed in Appendix D.1) are traceable with known probabilistic density of joint or conditional distributions here being Q ( Z s | Z t ) , Q ( Z t | Z s ) or
Q ( Z s , Z t ) .
However, these distributions lack the interpretability and can hardly be directly modeled, which leads to untraceable estimations of the above upper bounds.
To avoid the direct estimation of unknown probabilistic densities, we introduce an energy-based variational family for Q ( Z s , Z t ) that uses a normalized critic γ β ( Z s , Z t ) like Eq.
(9) to establish a traceable upper bound.
Specifically, we incorporate the critic γ β into the upper bound I CLUB [55] to obtain a traceable Seasonal-Trend Upper Bound (STUB) for I ( Z s , Z t ) , which is defined as:
[Equation]
The derivation details of this formula are provided in Appendix D.2.
The inequality in Eq.
(10) is tight only if Z s and Z t are a pair of independent variables [55].
This is exactly a sufficient condition for I STUB , since MI and Eq.
(11) are both zeros on the independent situation, which is our seasonaltrend disentanglement optimal objective.
The critic γ β , similar to γ α , takes on the discriminating responsibility but provides converse scores, constraining the MI to a minimum.
However, Eq.
(11) may get negative values during the learning of parameter β , resulting an invalid upper bound for MI.
To alleviate this problem, we additionally introduce a penalty term ∥
I neg STUB ∥ 2 to assist the model optimization, which is a L2 loss of the negative parts in I STUB .
For the evaluation, we take the same sampling manner as the one in the lower bound and optimization details are also shown in Algorithm 1.
[Table] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Experiments
We now present the results of our extensive experimental evaluations comparing LaST with state-ofthe-art baselines and report a series of empirical results, along with ablation study and visualizations of seasonal-trend representations.
Further details and results are provided in Appendix F.
5.1 Settings
Datasets and Baselines.
We conducted our experiments on seven real-world benchmark datasets from four categories of mainstream time series forecasting applications: (1) ETT 2 [25]: Electricity
Transformer Temperature consists of the target value “oil temperature” and six “power load” features, recorded hourly (i.e., ETTh1 and ETTh2) and every 15 minutes (i.e., ETTm1 and ETTm2) over two years.
(2) Electricity , from the UCI Machine Learning Repository 3 and preprocessed by [56], is composed of the hourly electricity consumption of 321 clients in kWh from 2012 to 2014.
Exchange [56] with daily exchange rates of eight countries from 1990 to 2016.
(4) Weather 4 contains 21 meteorological indicators (e.g., temperature and humidity) and is recorded every 10 minutes in 2020.
We compare our LaST with latest state-of-the-art methods on time series modeling and forecasting tasks from two categories: (1) representation learning techniques, including COST [38], TS2Vec [37], and TNC [36]; (2) end-to-end forecasting models, including VAE-GRU [10], Autoformer [26], Informer [25], and TCN [22].
Further descriptions and settings of these baselines are provided in appendix F.1.
Evaluation setup.
Following the prior work, we run our model on both univariate and multivariate forecasting settings.
In multivariate forecasting, LaST accepts and forecasts all variables in datasets.
In univariate forecasting, LaST only consider a specific feature in each dataset.
We employ the standard normalization and set input length T = 201 for all datasets.
For the dataset split, we follow standard protocol that categorizes all datasets into training, validation and test set in chronological order by the ratio of 6:2:2 for all datasets.
We report the evaluation results on the test set while the model achieves the best performance on the validation set.
Implementation details.
As for the network structure of LaST, we use a single-layer fully connected network as the feed forward network (FFN), which is applied in the modeling of posterior, reconstruction, and predictor.
Besides, we employ the 2-layer MLP for the critic γ in MI bound estimations.
Dimensions of seasonal and trend representations are consistent.
We set them as 32 in univariate forecasting and as 128 in multivariate forecasting on other datasets.
MAE loss is used to measure [Table] the forecasting derived from the predictor.
For the training strategy, we use the Adam [57] optimizer and training process is early stopped within 10 epoches.
We initialize the learning rate with 10 - 3 and decay it with 0 .
95 weight every epoch.
5.2 Performance comparisons and model analysis
Effectiveness.
Tables 1 and 2 summarize the results of univariate and multivariate forecastings respectively.
LaST achieves state-of-the-art performance against the advanced representation baselines on five real-world datasets.
The relative improvements on MSE and MAE are 25 .
6% and 22 .
1% against the best representation learning method CoST
and are 22 . 0% and 18 .
9% against the best end-to-end models Autoformer.
We note that Autoformer achieves better performance on long horizons forecasting on hourly ETT datasets and think there are two reasons: (1) Transformer-based models intrinsically establish long-range dependencies, which plays a crucial role in long sequence forecasting; (2) it employs a simple decomposition by average pooling with a fixed kernel size, which is more suitable for strongly periodic dataset like hourly ETT.
This phenomenon is beneficial to long-term forecasting but limits the sensitivity to local context, and the bonus does not have significant impact on other datasets.
Compared with baselines, LaST extracts the seasonal and trend patterns with disentangled representations adaptively and thus can be applied to intricate time series.
Ablation study.
We investigated the performance benefits brought by each mechanism of LaST on a synthetic dataset (generation details are provided in appendix F.3) and
ETTh1.
The results are shown in Table 3, consisting of two groups: M1 validates the mechanisms of seasonal-trend representations learning framework.
In it, “w/o seasonal” and “w/o trend” denote LaST without the seasonal and trend components respectively; “w/o coe” denotes LaST without autocorrelation and CORT coefficients while estimating the reconstruction loss.
M2 judges the introduction and estimations of MI, where“w/o lower” and “w/o upper” indicate the removal of the lower and upper bounds for
MI in regularization terms respectively; “with MINE” denotes that we replace our lower bound with
MINE.
The results show that all mechanisms improve the performance on the forecasting task.
We notice that the quality drops a lot when removing the trend component.
The reason is the seasonal forecasting derives from the iDFT algorithm, which is essentially a periodical repetition of the history observations.
However, it captures the seasonal patterns and assists the trend component in complete
LaST to bring the superiority, especially in the long-term settings and strongly periodical synthetic dataset.
Besides, we observe that with biased regularization term MINE, the performance becomes unstable and sometimes even worse than LaST without MI lower bound, while our unbiased bound (cf. Eq.(9)) continuously outperforms it.
Representation disentanglement.
We visualize the seasonal-trend representations with the tSNE [58] technique in Figure 2.
We also visualize the embeddings in last layer of Autoformer decoder as a comparison. The points with same color have a clearer and closer clustering in LaST, while they mix together without decomposition mechanisms (“w/o dec” indicates removal of the two decomposition mechanisms (autocorrelation and CORT coefficients, and the upper bound to MI). [Table]
[Table]
Notably, though Autofomer with the simple moving average block achieves satisfying decomposition from the time series perspective [Table] their representations are still prone to entanglement. These results suggest that (1) learning disentangled seasonal-trend representations is not trivial
[Table]
and (2) the proposed decomposition mechanisms successfully disentangle the seasonal-trend representations in latent space, each paying attention to a specific temporal pattern.
Input settings.
We further investigate the influence of hyperparameter input length to validate the sensitivity and Table 4 shows the results.
Long look-back window improves the performance especially in long-term forecasting [Table] while others even have performance degradation.
This verifies that LaST can effectively utilize the past information to understand patterns and make predictions.
[Table]
L a S
T w / o d ec A u t o f o r m e r L a S T
A u t o f o r m e r
L a S
T A u t o f o r m e r L a S T
A u t o f o r m e r
L a S
T A u t o f o r m e r L a S T
A u t o f o r m e r
[Figure]
Observations from a case-study.
We further validate LaST by by visualizing the extracted seasonality and trend in specifica cases.
As shown in Figure 3, LaST can capture the seasonal patterns on realworld datasets.
For example, a strong daily period is indicated on hourly and 15-minutes ETT datasets.
Even the period on Exchange dataset is not obvious, LaST still provides some long-term periods on the daily data.
Besides, trend and seasonal components jointly accurately restore the original sequence with their own perspective, which supports that LaST can produce workable disentangled representations for intricate time series.
Conclusion
We presented LaST, a disentangled variational inference framework with mutual information constraints to disassociate a couple of seasonal-trend representations in latent space, for effective forecasting of time series.
Our extensive experiments demonstrated that LaST successfully disentangles the seasonal-trend representations and achieves state-of-the-art performance.
Our future work will focus on tackling other challenging downstream tasks in the time series domain, e.g., generation and imputation. In addition, we plan to model stochastic factors explicitly in decomposition strategies, which will better understand the real-world time series. Acknowledgments and Disclosure of Funding
This work was supported in part by National Natural Science Foundation of China (Grant
No
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[No] (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Our source code for reproducibility is included in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See “Datasets” and “Evaluation setup” in Sec. 5.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix F.5.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix F.2.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See URLs in Sec.
(b) Did you mention the license of the assets?
[Yes] See folder “license” in supplementary materials.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Our source code for reproducibility is included in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See folder “license” in supplementary materials.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] They do not contain personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Following are the details of the proofs of the claims from the main text – Theorem 1, Theorem 2 and the MI lower bound (cf. Section 4). A.1 Inference of ELBO with decomposition
We follow the decomposition strategy, in the sense that time series and their representations are composed of multiple components such as seasonal Z s and trend Z t .
Sequence X and Y is decomposed as a sum form, i.e., X =
X s + X t , Y = Y s + Y t . Representations Z s and Z t are independent from each other, i.e., P ( Z s , Z t ) =
P ( Z s ) P ( Z t ) and, additionally, Z t and Z s are only associated with their own components.
With the above assumptions, we have:
[Equation]
A
[Equation]
A.2 Proof of the alternative of reconstruction loss
The loss function of log P θ ( X | Z ) can be written with Gaussian distribution as:
[Equation]
A
[Equation]
where µ θ ( Z ) and σ θ ( Z ) are neural networks that reconstruct the X from latent representations.
By the above equation, maximizing the reconstruction loss is regarded as minimizing the euclidean distance between inputs and reconstructions.
Thus, we opt to optimize the reconstruction loss in LaST by finding a way that imposes closer distances between raw input X s , X t , and reconstructed input ˆ X s , ˆ X t .
There are three terms in Eq.
(6) in main text:
[Equation]
The first term reflects the similarity between every instant value in X and its κ -lagged value.
We can regard the function A XX ( κ ) as an unnormalized score that measures the confidence of period κ [1, 2].
A lower distance between the two autocorrelation sequences means more similar periodic characteristics.
With identical autocorrelations, reconstructed ˆ X s will be same as X s in the seasonal view but still holds interference.
The optimized reconstructed ˆ
X s can be formulated as ˆ X s =
X s + ε s , where ε denotes the interference error.
The second normalized term reflects the similarity between the first differences of the input time series, which further measures the simultaneity of two signals’ rising or falling patterns, i.e., the trend.
With this objective, the reconstructed trend input is formulate as ˆ
X t =
X t + ε t similar to ˆ X s .
The third term now naturally becomes ∥
ε s + ε t ∥ 2 aiming to minimize the interference terms.
These three terms are used to reconstruct the pure seasonal and trend components and optimize the above reconstruction loss.
A.3 Biased gradient of MINE
The lower bound of MI between X and Z is estimated by MINE [3], which is defined as
[Equation]
When sampling the time series in a mini-batch B (see details in Algorithm 1)
[Equation]
it becomes
[Equation]
where Z ( i ) and Z ( j ) are derived from parameters-driven encoders
Q ϕ
( Z | X ) with input X .
The gradient of MINE is calculated by general stochastic optimizer such as Adam [4] or others:
[Equation]
where the second term holds bias since the derivative of logarithmic function
[Equation]
B Discrete Fourier transform and autocorrelation
The details of our use of Discrete Fourier Transform and efficient autocorrelation are presented next.
B.1 Discrete Fourier Transform
We apply discrete Fourier Transform (DFT) and inverse Discrete Fourier Transform (iDFT) to construct the mapping between time and frequency domain (cf.
Figure 1 in the main text).
First, given a time domain sequence of seasonality Z s =
{ Z s 1 , Z s 2 , . .
, Z s T } , we consider the first (cid:4)
T + (cid:5) Fourier coefficients according to the Nyquist theorem [5]:
[Equation]
where 1 ≤ k ≤ (cid:4)
T + (cid:5) .
Then time domain seasonal representation is extended and mapped from frequency domain with iDFT:
[Equation]
where 1 ≤ t ≤ T + τ, (B2) where we derive predictable representations
˜
Z s by taking the last τ time steps.
In LaST, we use the fast version of DFT [6] based on halving lemma and divide-and-conquer strategy, based on which the time complexity becomes O
( T log T ) , as opposed to the original O
( T 2 ) .
B.2 Efficient autocorrelation
The autocorrelation of sequence X with κ lagged steps is defined as
[Equation]
To obtain the autocorrelation sequence, it requires T ( T − 1) / 2 times multiplications, each of which holds O ( d 2 ) time complexity, where d is the dimension of X .
To improve the efficiency, we employ the Wiener–Khinchin theorem [7] that allows computing the autocorrelation with fast Fourier transform (FFT) and its inverse transform as:
[Equation]
where the asterisk denotes complex conjugate, and the output A XX (0 : T − 1) denotes sequence {A XX ( κ ) } κ ∈ [0: T − 1] .
In this setting, the number of multiplication calculations drops to T log T (from the original quadratic complexity).
C Discussions of prior distributions
Choosing an appropriate prior for the latent representations in VAE is important for mediating between the encoder and decoder.
The standard Gaussian N (0 , I ) is a commonly used prior assumption because: (1) it is efficient without additional calculation overhead and (2) the input data are preprocessed with standard normalization.
We note that any other advanced approaches on prior learning can be easily incorporated into our model.
In the following, we take VampPrior [8] – which produces prior distribution through the encoder Q ϕ given pseudo inputs – as an example to show the effects of different priors on the forecasting performance of LaST.
The priors P ( Z s ) and P ( Z t ) are obtained by Q ϕ s ( Z s | I ) and Q ϕ t ( Z t | I ) , respectively, with identity matrix as pseudo inputs.
[Table]
Table C1 shows the experimental results.
We can see that VampPrior and standard Gaussian are well matched in LaST on time series forecasting task.
However, none shows clear superiority over the other on four datasets.
Thus, we use the standard Gaussian as the default prior in LaST due to its efficiency.
D Existing upper bounds and the proposed I STUB
We now discuss in detail the portion pertaining to the upper bounds (cf.
Section 4 of the main paper).
D.1 Existing upper bounds
Existing research results have concluded several traceable and appropriate upper bounds to MI between X and Y [9, 10, 11], they include:
[Equation]
where Q ( Y ) is usually a learned marginal density approximation to P ( Y ) and B is the batch size.
These upper bounds are all based on conditional distributions which, defined as Q ( Z s | Z t ) or
Q ( Z t | Z s ) in our LaST
[Equation]
are still untraceable and ineffective.
Though we can replace them with joint distribution via deductions, for example,
[Equation]
it is challenging to establish a joint distribution and estimate its density directly without any additional information [12]
[Equation]
D.2 Derivation of proposed upper bound I STUB
To overcome the limitations of the existing upper bounds
[Equation]
we introduce a normalized energy-based function that uses a critic γ β ( Z s , Z t ) over the variational joint distribution to estimate the probability:
[Equation]
e
[Equation]
Z s
[Equation]
Z t
[Equation]
, where Z β = E Q ( Z s
[Equation]
[Equation]
Applying this formula into Eq.
(D4), we can obtain a traceable upper bound as:
[Equation]
which helps us addressing the joint density calculation challenge.
E Complexity analysis and algorithm
This section presents the complexity analysis and, for completeness, also shows the pseudo-code of the algorithms for training and prediction phases.
E.1 Complexity analysis
The corresponding time complexities of the components in LaST and are shown in Table E1.
The time complexity of seasonal and trend encoders/decoders are linear of fully connected network, i.e.,
O ( Tnd ) , where n is the number of observations and d is the dimension of latent representation.
For the reconstruction phrase, calculations of autocorrelation and CORT incur O ( nT log T ) and O ( nT ) time complexities, respectively.
The MI upper and lower bounds are both based on the energy function implemented by a 2-layer MLP, which has a time complexity of O ( Td 2 ) .
As for the predictor, the time complexity of fast Fourier transform is equal to O ( nT log T ) .
The first FFN in trend forecasting consumes O ( nTτ ) , while the time complexity of other FFNs becomes O ( τnd ) .
We note that only the encoders and predictor are required after training.
Thus, the time complexity of training is
O ( T ( nd + n log T + d 2 ) + τ ( nT + nd )) and that of prediction turns to O ( nTτ + nT log T + τnd ) .
[Table]
E.2 Algorithm of LaST
The pseudo-codes of LaST covering training and prediction phases are summarized in Algorithm E and E2, respectively.
Algorithm E1 Training phase of LaST.
Input: Historical time series X 1: T and future time series X T
+1: T + τ ; Hyperparameters and initialized parameters of LaST.
Output: LaST with optimized parameters.
1: for
Algorithm E2 Prediction phase of LaST.
F Experiment supplementary
In this section, we present more details of the baselines, computing infrastructure, dataset generation process, and full experimental results on ETT benchmark (ETTh1, h2, m1, and m2).
In addition, we provide fluctuation analysis and memory cost comparison.
F.1 Details of baselines
The implementations and settings of the seven baselines are provided below.
Unless otherwise specified, we use the suggested settings described in the respective papers.
CoST [13]: analyses raw signals in time and frequency domain with temporal convolutions and
Fourier transform, and separately produces seasonal and trend representations for time series forecasting task.
We run experiments with their publicly available code: https://github.com/ salesforce/CoST .
TS2Vec [14]: is a universal time series representation learning framework that performs contrastive learning in a hierarchical way over augmented context views, which enables a robust contextual representation for each timestamp.
We run the code from their open source repository: https: //github.com/yuezhihan/ts2vec .
TNC [15]: is a self-supervised framework that introduces the concept of a temporal neighborhood with stationary properties and learns generalizable time series representations.
We use their open-source code: https://github.com/sanatonek/TNC_representation_learning .
As for hyperparameters, we set w = 0 .
005 in the loss function, mc _ sample _ size = 20 , batch size as 8, and learning rate as 10 − 3 with Adam optimizer.
VAE-GRU [16]: combines the strengths of recurrent network and stochastic gradient variational
Bayes, which has been applied for time series forecasting among various domains [17, 18].
We implement it by the GRU [19] variant with initial state sampling from N (0 , I ) and keep the variational settings consistent with our LaST.
Autoformer [20]: designs a novel decomposition architecture with an autocorrelation mechanism to improve Transformer [21].
It exploits average pooling strategy to split time series into seasonal and trend components and regards autocorrelation coefficient as attention score, which boosts the performance of long-term forecasting.
We use their publicly available code: https://github.com/ thuml/Autoformer .
Informer [22]: designs an efficient transformer-based model for time series forecasting with a
ProbSparse self-attention mechanism, highlights distilling module, and generative style decoder.
We use the publicly available code: https://github.com/zhouhaoyi/Informer2020 .
TCN [23]: introduces an architecture that is applied across all tasks for sequence modeling.
It is also popular in time series modeling and forecasting [24, 25].
We use their public repository: https://github.com/locuslab/TCN and set kernel sizes as { 2 1 , 2 2 , · · · , 2 7 } .
F.2 Evaluation environment
LaST is implemented by PyTorch and all experiments are run on a server with a Intel Xeon
Platinum 8124M CPU, a RTX-3090 GPU, and 128GB memory.
F.3 Synthetic dataset generation
Following the method in [13], we synthesize time series with sinusoidal seasonal patterns, as well as linear and non-linear trend patterns.
The seasonal signal consists of three sine waves with the following period, phase, and amplitudes: { (20 , 3 , 0) , (50 , 3 , 0 . 2) , (100 , 3 , 0 . 5) } .
The trend signal is composed of two components: (1) a nonlinear, saturating pattern X t = 1 1+exp β 0 ( t − β 1 ) , where β 0 = 0 .
2 and β 1 = 200 ; (2) an ARMA process whose parameters of AR and MA are { (0 . 9 , − 0 . 1) , (0 . 2 , − 0 . 5) } .
The final time series is generated by the sum of these seasonal and trend signals.
F.4 More results on ETT benchmark
[Table]
[Table]
F.5 Fluctuation analysis
To validate the models’ robustness
[Table]
we conduct fluctuation analysis and have three runs of all the experiments with different seeds for baselines and LaST.
Table F3 reports the performance results in terms of MSE and MAE with standard deviations.
[Table]
[Table]
F.6 Convergence
Epoch
Epoch
[Table]
po
[Table]
h
[Table]
[Figure]
[Figure]
[Table] [Figure] We conduct additional experiments to validate the convergence property of our model. Figure F shows the descending process of training, valid, and test loss of our model as the epochs increase, where we can see that all losses first drop and then goes to level off.
The downward trend is not obvious on ETTm1 dataset.
This is because ETTm1 is a larger-scale dataset and it requires over 1,000 times stochastic gradient descent at every epoch.
We visualize the losses change at the first epoch in figure F2. We can observe that losses drop a lot at the first epoch. These results support that our model converges well on real-world datasets. F.7 Memory cost
We keep the batch size B , input length T and forecasting length τ consistent among baselines and
LaST, and report the memory cost of in Table F4, where we can observe that LaST is ∼ 30% more memory-efficient than CoST and over 40% more memory-efficient than AutoFormer.
CoST contains multiple TCNs with different kernel sizes, which enable it to collect multi-horizon information, but lead the memory consumption to multiply.
Transformer-based Autoformer establishes the correlations between every time step, which captures the long-term dependencies but incurs O
(( T + τ ) 2 ) space complexity.
For example, AutoFormer on Weather dataset with {B = 32 , T = 201 , τ = 720 } allocates 9045MB memory while LaST only needs 1751MB of memory.
Based on variational inference theory, our proposed LaST model regards the time series as a whole and extracts seasonal and trend without additional memory consumption.
[Table]