C O ST: C ONTRASTIVE L EARNING OF D ISENTANGLED S EASONAL -T REND R EPRESENTATIONS FOR T IME S ERIES F ORECASTING
A BSTRACT
Deep learning has been actively studied for time series forecasting, and the mainstream paradigm is based on the end-to-end training of neural network architectures, ranging from classical LSTM/RNNs to more recent TCNs and Transformers.
Motivated by the recent success of representation learning in computer vision and natural language processing, we argue that a more promising paradigm for time series forecasting, is to ﬁrst learn disentangled feature representations, followed by a simple regression ﬁne-tuning step – we justify such a paradigm from a causal perspective.
Following this principle, we propose a new time series representation learning framework for long sequence time series forecasting named CoST, which applies contrastive learning methods to learn disentangled seasonal-trend representations.
CoST comprises both time domain and frequency domain contrastive losses to learn discriminative trend and seasonal representations, respectively.
Extensive experiments on real-world datasets show that CoST consistently outperforms the state-of-the-art methods by a considerable margin, achieving a 21.3% improvement in MSE on multivariate benchmarks.
It is also robust to various choices of backbone encoders, as well as downstream regressors.
Code is available at https://github.com/salesforce/CoST .
I NTRODUCTION
[Equation] Time series forecasting has been widely applied to various domains, such as electricity pricing (Cuaresma et al., 2004), demand forecasting (Carbonneau et al., 2008), capacity planning and management (Kim, 2003), and anomaly detection (Laptev et al., 2017).
Recently, there has been a surge of efforts applying deep learning for forecasting (Wen et al., 2017; Bai et al., 2018;
Zhou et al., 2021), and owing to the increase in data availability and computational resources, these approaches have offered promising performance over conventional methods in forecasting literature.
Compared to conventional approaches, these methods are able to jointly learn feature representations and the prediction function (or forecasting function) by stacking a series of non-linear layers to perform feature extraction, followed by a regression layer focused on forecasting.
However, jointly learning these layers end-to-end from observed data may lead to the model over-ﬁtting and capturing spurious correlations of the unpredictable noise contained in the observed data.
The situation is exacerbated when the learned representations are entangled – when a single dimension of the feature representation encodes information from multiple local independent modules of the data-generating process – and a local independent module experiences a distribution shift.
Figure 1 is an example of such a case, where the observed time series is generated by a seasonal module and nonlinear trend module.
If we know that the seasonal module has experienced a distribution shift, we could still makes a reasonable prediction based on the invariant trend module.
However, if we learn an entangled feature representation from the observed data, it would be challenging for the learned model to handle this distribution shift, even if it only happens in a local component of the data-generating process.
In summary, the learned representations and prediction associations from the end-to-end training approach are unable to transfer nor generalize well when the data is generated from a non-stationary environment, a very common scenario in the time series analysis.
Therefore, in this work, we take a step back and aim to learn disentangled seasonal-trend representations which are more useful for time series forecasting.
To achieve this goal, we leverage the idea of structural time series models (Scott & Varian, 2015;
Qiu et al., 2018), which formulates time series as a sum of trend, seasonal and error variables, and exploit such prior knowledge to learn time series representations.
First, we present the necessity of learning disentangled seasonal-trend representations through a causal lens, and demonstrate that such representations are robust to interventions on the error variable.
Then, inspired by Mitrovic et al.
(2020), we propose to simulate interventions on the error variable via data augmentations and learn the disentangled seasonal-trend representations via contrastive learning.
Based on the above motivations, we propose a novel contrastive learning framework to learn disentangled seasonal-trend representations for the Long Sequence Time-series Forecasting (LSTF) task (Zhou et al., 2021).
Speciﬁcally, CoST leverages inductive biases in the model architecture to learn disentangled seasonal-trend representations.
CoST efﬁciently learns trend representations, mitigating the problem of lookback window selection by introducing a mixture of auto-regressive experts.
It also learns more powerful seasonal representations by leveraging a learnable Fourier layer which enables intra-frequency interactions.
Both trend and seasonal representations are learned via contrastive loss functions.
The trend representations are learned in the time domain, whereas the seasonal representations are learned via a novel frequency domain contrastive loss which encourages discriminative seasonal representations and side steps the issue of determining the period of seasonal patterns present in the data.
The contributions of our work are as follows
We show via a causal perspective, the beneﬁts of learning disentangled seasonal-trend representations for time series forecasting via contrastive learning.
We propose CoST, a time series representation learning approach which leverages inductive biases in the model architecture to learn disentangled seasonal and trend representations, as well as incorporating a novel frequency domain contrastive loss to encourage discriminative seasonal representations.
CoST outperforms existing state-of-the-art approaches by a considerable margin on realworld benchmarks – 21.3% improvement in MSE for the multivariate setting.
We also analyze the beneﬁts of each proposed module, and establish that CoST is robust to various choices of backbone encoders and downstream regressors via extensive ablation studies.
S EASONAL -T REND R EPRESENTATIONS FOR T IME S ERIES
Problem Formulation Let ( x 1 , . . .
x T ) ∈ R T × m be a time series, where m is the dimension of observed signals.
Given lookback window h , our goal is to forecast the next k steps, ˆ X = g ( X ) ,
where X ∈ R h × m , ˆ X ∈ R k × m , and g ( · ) denotes the prediction mapping function, and ˆ X predicts the next k time steps of X .
In this work, instead of jointly learning the representation and prediction association through g ( · ) , we focus on learning feature representations from observed data, with the goal of improving predictive performance.
Formally, we aim to learn a nonlinear feature embedding function V = f
( X ) ,
where X ∈ R h × m and
V ∈ R
h × d , to project m -dimensional raw signals into a d -dimensional latent space for each timestamp.
Subsequently, the learned representation of the ﬁnal timestamp v h is used as inputs for the downstream regressor of the forecasting task.
Disentangled Seasonal-Trend Representation Learning and Its Causal Interpretation As discussed in Bengio et al.
(2013), complex data arise from the rich interaction of multiple sources – a good representation should be able to disentangle the various explanatory sources, making it robust to complex and richly structured variations.
Not doing so may otherwise lead to capturing spurious features that do not transfer well under non i.i.d.
data distribution settings.
To achieve this goal, it is necessary to introduce structural priors for time series.
Here, we borrow ideas from Bayesian Structural Time Series models (Scott & Varian, 2015; Qiu et al., 2018).
As illustrated in the causal graph in Figure 2, we assume that the observed time series data X is generated from the error variable E and the error-free latent variable X (cid:63) .
X (cid:63) in turn, is generated from the trend variable T and seasonal variable S .
As E is not predictable, the optimal prediction can be achieved if we are able to uncover X (cid:63) which only depends on T and S .
Firstly, we highlight that existing work using end-to-end deep forecasting methods to directly model the time-lagged relationship and the multivariate interactions along the observed data X .
Unfortunately, each X includes unpredictable noise E , which might lead to capturing spurious correlations.
Thus, we aim to learn the error-free latent variable X (cid:63) .
Secondly, by the independent mechanisms assumption (Peters et al., 2017; Parascandolo et al., 2018), we can see that the seasonal and trend modules do not inﬂuence or inform each other.
Therefore, even if one mechanism changes due to a distribution shift, the other remains unchanged.
The design of disentangling seasonality and trend leads to better transfer, or generalization in nonstationary environments.
Furthermore, independent seasonal and trend mechanisms can be learned independently and be ﬂexibly re-used and re-purposed.
We can see that interventions on E does not inﬂuence the conditional distribution P ( X (cid:63) | T, S ) , i.e. P do ( E = e i ) ( X (cid:63) | T, S ) =
P do ( E = e j ) ( X (cid:63) | T, S ) , for any e i , e j in the domain of E .
Thus, S and T are invariant under changes in E .
Learning representations for S and T allows us to ﬁnd a stable association with the optimal prediction (of X (cid:63) ) in terms of various types of errors.
Since the targets X (cid:63) are unknown, we construct a proxy contrastive learning task inspired by Mitrovic et al.
Speciﬁcally, we use data augmentations as interventions on the error E and learn invariant representations of T and S via contrastive learning.
Since it is impossible to generate all possible variations of errors, we select three typical augmentations: scale, shift and jitter, which can simulate a large and diverse set of errors, beneﬁcial for learning better representations.
S EASONAL -T REND C ONTRASTIVE L EARNING F RAMEWORK
In this section, we introduce our proposed CoST framework to learn disentangled seasonal-trend representations.
We aim to learn representations such that for each time step, we have the disentangled representations for seasonal and trend components, i.e., V =
[ V ( T ) ; V ( S ) ]
∈ R h × d , where
V ( T ) ∈
R h × d T and V ( S )
∈
R h × d S , such that d = d T + d S .
Figure 3a illustrates our overall framework.
Firstly, we make use of an encoder backbone f b : R h × m
→ R h × d to map the observations to latent space.
Next, we construct both the trend and seasonal representations from these intermediate representations.
Speciﬁcally, the Trend Feature
Disentangler (TFD), f T : R h × d
→ R h × d T , extracts the trend representations via a mixture of autoregressive experts and is learned via a time domain contrastive loss L time .
The Seasonal Feature
Disentangler (SFD), f S : R h ×
d
→ R h × d S , extracts the seasonal representations via a learnable
Fourier layer and is learned by a frequency domain contrastive loss which includes an amplitude component, L amp , and a phase component, L phase .
We give a detailed description of both components in the next section.
The model is learned in an end-to-end fashion, with the overall loss function being
[Equation]
where α is a hyper-parameter which balances the trade-off between trend and seasonal factors.
Finally, we concatenate the outputs of the Trend and Seasonal Feature Disentanglers to obtain our ﬁnal output representations. [Figure]
3. T REND F EATURE R EPRESENTATIONS
Extracting the underlying trend is crucial for modeling time series.
Auto-regressive ﬁltering is one widely used method, as it is able to capture time-lagged causal relationships from past observations.
One challenging problem is to select the appropriate lookback window – a smaller window leads to under-ﬁtting, while a larger model leads to over-ﬁtting and over-parameterization issues.
A straightforward solution is to optimize this hyper-parameter by grid search on the training or validation loss (Hyndman & Khandakar, 2008), but such an approach is too computationally expensive.
Thus, we propose to use a mixture of auto-regressive experts which can adaptively select the appropriate lookback window.
Trend Feature Disentangler (TFD)
As illustrated in Figure 3b, the TFD is a mixture of L +1 autoregressive experts, where L = (cid:98) log 2 ( h/ 2) (cid:99) .
Each expert is implemented as a 1 d causal convolution with d input channels and d T output channels, where the kernel size of the i -th expert is 2 i .
Each expert outputs a matrix ˜ V
( T,i ) =
CausalConv( ˜ V , 2 i ) .
Finally, an average-pooling operation is performed over the outputs to obtain the ﬁnal trend representations,
[Equation]
Time Domain Contrastive Loss
We employ a contrastive loss in the time domain to learn discriminative trend representations.
Speciﬁcally, we apply the MoCo (He et al., 2020) variant of contrastive learning which makes use of a momentum encoder to obtain representations of the positive pair, and a dynamic dictionary with a queue to obtain negative pairs.
We elaborate further on the details of contrastive learning in Appendix A.
Then, given N samples and K negative samples, the time domain contrastive loss is
[Equation]
where given a sample V ( T ) , we ﬁrst select a random time step t for the contrastive loss and apply a projection head, which is a one-layer MLP to obtain q , and k is respectively the augmented version of the corresponding sample from the momentum encoder/dynamic dictionary.
3. S EASONAL F EATURE R EPRESENTATIONS
Spectral analysis in the frequency domain has been widely used in seasonality detection (Shumway et al., 2000).
Thus, we turn to the frequency domain to handle the learning of seasonal representations.
To do so, we aim to address two issues: i) how can we support intra-frequency interactions (between feature dimensions) which allows the representations to encode periodic information more easily, and, ii) what kind of learning signal is required to learn representations which are able to discriminate between different seasonality patterns?
Standard backbone architectures are unable to easily capture intra-frequency level interactions, thus, we introduce the SFD which makes use of a learnable Fourier layer.
Then, in order to learn these seasonal features without prior knowledge of the periodicity, a frequency domain contrastive loss is introduced for each frequency.
Seasonal Feature Disentangler (SFD)
As illustrated in Figure 3c, the SFD is primarily composed of a discrete Fourier transform (DFT) to map the intermediate features to frequency domain, followed by a learnable Fourier layer.
We include further details and deﬁnitions of the DFT in Appendix B.
The DFT is applied along the temporal dimension and maps the time domain representations into the frequency domain, F ( ˜ V ) ∈ C F × d , where F = (cid:98) h/ 2 (cid:99) + 1 is the number of frequencies.
Next, the learnable Fourier layer, which enables frequency domain interactions, is implemented via a perelement linear layer.
It applies an afﬁne transform on each frequency, with a unique set of complexvalued parameters for each frequency, since we do not expect this layer to be translation invariant.
Finally, we transform the representation back to time domain using an inverse DFT operation.
The ﬁnal output matrix of this layer is the seasonal representation, V ( S )
∈
R h × d S .
Formally, we can denote the i, k -th element of the output as
[Equation]
where A ∈ C F × d × d S , B ∈ C F × d S are the parameters of the per-element linear layer.
Frequency Domain Contrastive Loss As illustrated in Figure 3c, the inputs to the frequency domain loss functions are the pre-iFFT representations, denoted by F ∈ C F × d S .
These are complexvalued representations in the frequency domain.
To learn representations which are able to discriminate between different seasonal patterns, we introduce a frequency domain loss function.
As our data augmentations can be interpreted as interventions on the error variable, the seasonal information does not change and thus, a contrastive loss in frequency domain corresponds to discriminating between different periodic patterns given a frequency.
To overcome the issue of constructing a loss function with complex-valued representations, each frequency can be uniquely represented by its amplitude and phase representations, | F i, : | and φ ( F i, : ) .
Then, the loss functions are denoted,
[Equation]
where F ( j ) i, : is the j -th sample in a mini-batch, and ( F ( j ) i, : ) (cid:48) is the augmented version of that sample.
E XPERIMENTS
In this section, we report the results of a detailed empirical analysis of CoST and compare it against a diverse set of time series representation learning approaches, as well as compare against end-to-end supervised forecasting methods.
Appendix F contains further results on runtime analysis.
4. E XPERIMENTAL S ETUP
Datasets We conduct extensive experiments on ﬁve real-world public benchmark datasets.
ETT (Electricity Transformer Temperature)
(Zhou et al., 2021) consists of two hourly-level datasets (ETTh) and one 15-minute-level dataset (ETTm), measuring six power load features and “oil temperature”, the chosen target value for univariate forecasting.
Electricity 2 measures the electricity consumption of 321 clients, and following popular benchmarks, we convert the dataset into hourlylevel measurements and set “MT 320” as the target value for univariate forecasting.
Weather 3 is an hourly-level dataset containing 11 climate features from nearly 1,600 locations in the U.S., and we take “wet bulb” as the target value for univariate forecasting.
Finally, we also include the M5 dataset (Makridakis et al., 2020) in Appendix J.
Evaluation Setup Following prior work, we perform experiments on two settings – multivariate and univariate forecasting.
The multivariate setting involves multivariate inputs and outputs, considering all dimensions of the dataset.
The univariate setting involves univariate inputs and outputs, which are the target values described above.
We use MSE and MAE as evaluation metrics, and perform a 60/20/20 train/validation/test split.
Inputs are zero-mean normalized and evaluated over various prediction lengths.
Following (Yue et al., 2021), self-supervised learning approaches are ﬁrst trained on the train split, and a ridge regression model is trained on top of the learned representations to directly forecast the entire prediction length.
The validation set is used to choose the appropriate ridge regression regularization term α , over a search space of { 0 . 1 , 0 . 2 , 0 . 5 , 1 , 2 , 5 , 10 , 20 , 50 , 100 , 200 , 500 , 1000 } .
Evaluation results are reported on the test set.
Implementation Details For CoST and all other representation learning methods, the backbone encoder used is a Temporal Convolution Network (following similar practice in TS2Vec (Yue et al., 2021))
unless the approach includes or is an architectural modiﬁcation (further details in
Appendix E).
All methods used have a representation dimensionality of 320.
We use a standard hyper-parameter setting on all datasets – a batch size of 256 and learning rate of 1E − 3 , momentum of 0.9 and weight decay of 1E − 4 with SGD optimizer and cosine annealing.
The MoCo implementation for time domain contrastive loss uses a queue size of 256, momentum of 0.999, and temperature of 0.07.
We train for 200 iterations for datasets with less than 100,000 samples, anditerations otherwise.
Details on data augmentations used in CoST can be found in Appendix C.
4. R ESULTS
Among the baselines, we report the performance of representation learning techniques including
TS2Vec, TNC, and a time series adaptation of MoCo in our main results.
A more extensive benchmark of feature-based forecasting approaches can be found in Appendix H due to space limitations.
Further details about the baselines can be found in
Appendix E. We include supervised forecasting approaches - two Transformer based models, Informer (Zhou et al., 2021) and LogTrans(Li et al., 2020), and the backbone TCN trained directly on an end-to-end forecasting loss.
A comparison of end-to-end forecasting approaches can be found in Appendix I.
Table 1 summarizes the results of CoST and top performing baselines for the multivariate setting, and Table 7 (in Appendix G due to space limitations) for the univariate setting.
For end-to-end forecasting approaches, the TCN generally outperforms the Transformer based approaches, Informer and LogTrans.
At the same time, the representation learning methods outperform end-to-end forecasting approaches, but there are indeed cases, such as in certain datasets for the univariate setting, where the end-to-end TCN performs surprisingly well.
While Transformers have been shown to
[Table]
be powerful models in other domains like NLP [Table] this suggests that TCN models are still a powerful baseline which should still be considered for time series.
Overall, our approach achieves state-of-the-art performance, beating the best performing end-toend forecasting approach by 39.3% and 18.22% (MSE) in the multivariate and univariate settings respectively.
CoST also beats next best performing feature-based approach by 21.3% and 4.71% (MSE) in the multivariate and univariate settings respectively.
This indicates that CoST learns more relevant features by learning a composition of trend and seasonal features which are crucial for forecasting tasks.
4. P ARAMETER S ENSITIVITY
[Table]
α controls the weightage of the seasonal components in the overall loss function, L =
L time + α 2 ( L amp + L phase ) .
We perform a sensitivity analysis on this hyper-parameter (Table 2) and show that an optimal value can be chosen and is robust across various settings.
We choose α = 5E −for all other experiments since it performs well on both multivariate and univariate settings.
We note that the small values of α stems from the frequency domain contrastive loss being generally three orders of magnitude larger than the time domain contrastive loss, rather than being an indicator that the seasonal component has lower importance than the trend component.
Further, we highlight that overall, while choosing a smaller value of α leads to better performance on most datasets, there are certain cases for which a larger α might be preferred, as seen in the lower portion of Table 2.
4. A BLATION S TUDY
[Table] [Table]
[Table]
Components of CoST We ﬁrst perform an ablation study to understand the performance beneﬁts brought by each component in CoST.
Table 3 presents the average results over the ETT datasets on all forecast horizon settings (similarly for Tables 4 and 5).
We show that both trend and seasonal components improve performance over the baselines (SimCLR and MoCo), and further, the composition of trend and seasonal components leads to the optimal performance.
We further verify that training our proposed model architecture end-to-end with a supervised forecasting loss leads to worse performance.
Backbones Next, we verify that our proposed trend and seasonal components as well as contrastive loss (both time and frequency domain) are robust to various backbone encoders.
TCN is the default backbone encoder used in all other experiments and we present results on LSTM and Transformer backbone encoders of equivalent parameter size.
While performance using the TCN backbone outperforms LSTM and Transformer, we show that our approach outperforms the competing approach on all three settings.
Regressors Finally, we show that CoST is also robust to various regressors used for forecasting.
Apart from a ridge regression model, we also perform experiments on a linear regression model and a kernel ridge regression model with RBF kernel.
As shown in Table 5, we also demonstrate that
CoST outperforms the competing baseline on all three settings. [Figure] [Figure] ASE
[Figure]
We visualize the learned representations on a simple synthetic time series with both seasonal and trend components, and show that CoST is able to learn representations which are able to discriminate between various seasonal and trend patterns.
The synthetic dataset is generated by deﬁning two trend and three seasonal patterns, and taking the cross product to form six time series (details in Appendix D).
After training the encoders on the synthetic dataset, we can visualize them via the T-SNE algorithm (Van der Maaten & Hinton, 2008).
Figureshows that our approach is able to learn both the trend and seasonal patterns from the dataset and the learned representations has high clusterability, whereas TS2Vec is unable to distinguish between various seasonal patterns.
R ELATED W ORK
Deep forecasting has typically been tackled as an end-to-end supervised learning task, where early work considered using RNN based models (Lai et al., 2018) as a natural approach to modeling time series data.
Recent work have also considered adapting Transformer based models for time series forecasting (Li et al., 2020;
Zhou et al., 2021), speciﬁcally focusing on tackling the quadratic space complexity of Transformer models.
Oreshkin et al.
(2020) proposed a univariate deep forecasting model and showed that deep models outperform classical time series forecasting techniques.
While recent work in time series representation learning focused on various aspects of representation learning such how to sample contrastive pairs (Franceschi et al., 2020; Tonekaboni et al., 2021), taking a Transformer based approach (Zerveas et al., 2021), exploring complex contrastive learning tasks (Eldele et al., 2021), as well as constructing temporally hierarchical representations (Yue et al., 2021), none have touched upon learning representations composed of trend and seasonal features.
Whereas existing work have focused exclusively on time series classiﬁcation tasks, Yue et al.
ﬁrst showed that time series representations learned via contrastive learning establishes a new stateof-the-art performance on deep forecasting benchmarks.
Classical time series decomposition techniques (Hyndman & Athanasopoulos, 2018) have been used to decompose time series into seasonal and trend components to attain interpretability.
There has been recent work on developing more robust and efﬁcient decomposition approaches (Wen et al., 2018; 2020; Yang et al., 2021).
These methods focus decomposing the raw time series into trend and seasonal components which are still interpreted as time series in the original input space rather than learning representations.
Godfrey & Gashler (2017) presents an initial attempt to use neural networks to model periodic and non-periodic components in time series data, leveraging periodic activation functions to model the periodic components.
Different from our work, such a method is only able to model a single time series per model, rather than produce the decomposed seasonaltrend representations given a lookback window.
C ONCLUSION
Our work shows separating the representation learning and downstream forecasting task to be a more promising paradigm than the standard end-to-end supervised training approach for time-series forecasting.
We show this empirically, and also explain it through a causal perspective.
By following this principle, we proposed CoST, a contrastive learning framework that learns disentangled seasonal-trend representations for time series forecasting tasks.
Extensive empirical analysis shows that CoST outperforms the previous state-of-the-art approaches by a considerable margin and is robust to various choices of backbone encoders and regressors.
Future work will extend our framework for other time-series intelligence tasks.
A C ONTRASTIVE L EARNING
Contrastive learning via the instance discrimination task is a powerful approach for self-supervised learning.
Here, we describe the essentials of this method, which underpins our proposed approach.
Firstly, a family of data augmentations, A , is deﬁned.
Given a single sample of data x i ∈ X , two data augmentation operators are sampled, a ∼ A , a (cid:48) ∼ A .
q i = f
( a ( x i )) is referred to as the query representation with encoder f , and k i =
f ( a (cid:48)
( x i )) is the positive key representation.
Finally, the
InfoNCE loss function is
[Equation]
where τ is the temperature hyper-parameter, k j are negative key representations, and K is the total number of negative samples.
Standard approaches use an efﬁcient mechanism to obtain negative samples - by simply treating all other samples in the mini-batch as negative samples, i.e. K = N − 1 .
MoCo (He et al., 2020) introduces the idea of a using a queue of size K (a hyper-parameter) to obtain negative samples.
At each iteration of training, simply pop N samples from the queue, and push the
N representations form the current mini-batch.
B D ISCRETE F OURIER T RANSFORM
The DFT provides a frequency domain view of a given sequence, x = ( x 0 , x 1 , . . .
x N − 1 ) , mapping a time series with regular intervals into the Fourier coefﬁcients, a sequence of complex numbers of equal length.
Due to the conjugate symmetry of the DFT of real-valued signals, we can simply consider the ﬁrst (cid:98) N/ 2 (cid:99) + 1 Fourier coefﬁcients, c = F ( x ) ∈ C (cid:98) N/ 2 (cid:99) +1 . c
[Equation]
Each complex component, c k , can be represented by the amplitude, | c k | , and the phase, φ ( c k ) ,
[Equation]
where R { c k } and I { c k } are the real and imaginary components of c k respectively.
Finally, the inverse DFT maps the frequency domain representation back to the time domain,
[Equation]
C D ATA A UGMENTATIONS
In our experiments, we utilize a composition of three data augmentations, applied in the following order - scaling, shifting, and jittering, activating with a probability of 0.5.
Scaling The time-series is scaled by a single random scalar value, obtained by sampling (cid:15) ∼
N (0 , 0 . 5) , and each time step is ˜ x t = (cid:15)x t .
Shifting The time-series is shifted by a single random scalar value, obtained by sampling (cid:15) ∼
N (0 , 0 . 5) and each time step is ˜ x t = x t + (cid:15) .
Jittering I.I.D. Gaussian noise is added to each time step, from a distribution (cid:15) t ∼ N (0 , 0 . 5) , where each time step is now ˜ x t = x t + (cid:15) t .
D S YNTHETIC D ATA G ENERATION
We ﬁrst construct two trend patterns.
The ﬁrst trend patterns follows a nonlinear, saturating pattern, y t
= 1 1+exp β 0 ( t − β 1 ) + (cid:15) t for β 0 = 0 .
2 , β 1 = 60 , (cid:15) t ∼ N (0 , 0 . 3) .
The second pattern is a mixture of
ARMA process, ARMA(2 , 2) + ARMA(3 , 3) + ARMA(4 , 4) , where the AR and MA parameters are as follows, ( { .9, -.1 } , { .2, -.5 } ), ( { .1, .2, .3 } , { .1, .65, -.45 } ), ( { .3, .5, -.5, -.3 } , { .1, .1, -.2, - .3 } ).
Next, we construct three seasonal patterns, consisting of sine waves with the following period, phase, and amplitudes, { 20, 0, 3 } , { 50, .2, 3 } , { 100, .5, 3 } .
The ﬁnal time series are constructed as follows, generate a trend pattern g ( t ) and seasonal pattern s ( t ) , the ﬁnal time series is y ( t ) =
g ( t ) + s ( t ) , t = 0 , . .
We do this for all pairs of trend and seasonal patterns, constructing a total of 6 time series.
E D ETAILS ON BASELINES
Results for TS2Vec, TNC, MoCo, Triplet, CPC, TST, TCC are based on our reproduction, while results for Informer, LogTrans, and LSTNet (Lai et al., 2018) are directly taken from Yue et al.
(2021) for the ETT and Electricity datasets, and Zhou et al.
(2021) for the Weather dataset.
For all reproduced approaches, we train for 200 iterations for datasets with less than 100,000 samples, otherwise, and 600 iterations otherwise.
The encoders used in all approaches except Triplet and TST is the causal TCN encoder as proposed in TS2Vec (Yue et al., 2021).
A fully connected layer is ﬁrst used to project each time step from the dimensionality of the multivariate time series to the hidden channel (size 64).
Then, there arelayers of convolution blocks.
Each convolution block is a sequence of GELU, DilatedConv, GELU,
DilatedConv, with skip connections across each block.
The DilatedConvs have dilation of 2 i in each layer i of convolution block, all have kernel size of 3 and input/output channel size of 64.
A ﬁnal convolution block is used to map the hidden channels to the output channel (size 320).
For Triplet, the encoder is their proposed causal TCN encoder, while for TST, their proposed Transformer encoder is used.
Further details can be found in their respective papers.
TS2Vec (Yue et al., 2021)
TS2Vec was recently proposed as a universal framework for learning time series representations by performing contrastive learning in a hierarchical manner over augmented context views.
They propose to learn timestamp level representations.
We ran the code from their open source repository 4 as is, hyper-parameters used are all defaults as suggested in their paper.
TNC (Tonekaboni et al., 2021)
TNC proposes a self-supervised framework for learning generalizable representations for non-stationary time series.
They make use of the augmented Dickey-Fuller test for stationarity to ensure positive samples come from the a neighborhood of similar signals.
We use their open source code 5 , setting w = 0 .
005 in the loss function, mc sample size = 20 , batch size of 8, and learning rate of 1E − 3 with Adam optimizer.
MoCo (He et al., 2020) MoCo is a popular self-supervised learning baseline in computer vision, and we implement a time series version of MoCo using their open source code 6 .
We use a batch size of 128, queue of 256, momentum for the momentum encoder is 0.999, temperature of the loss function is 0.07, learning rate of 1E − 3 with SGD optimizer, with cosine annealing.
Data augmentations used are described in Appendix C.
Triplet (Franceschi et al., 2020)
Triplet proposes a time series self-supervised learning approach by taking positive samples to be substrings of the anchor, and negative samples to be randomly sampled from the dataset.
We reproduce their approach by adapting their open source code 7 , making use of their proposed causal TCN model architecture.
We use a batch size of 10, and learning rate 1E − 3 with AdamW optimizer.
CPC (van den Oord et al., 2019) CPC learns representations by predicting the future in latent space using auto-regressive models.
They use a probabilistic contrastive loss which induces latent space to capture information maximally useful to predict future samples.
We reproduce this approach by referencing an unofﬁcial implementation 8 .
We use a GRU for the AR module, use two prediction steps after eight AR steps.
We use a batch size of 8 and learning rate of 1E − 3 with
AdamW optimizer.
TST (Zerveas et al., 2021) TST is a Transformer based approach using a reconstruction loss.
We use the open source implementation 9 as is.
TCC (Eldele et al., 2021)
TCC introduces a temopral contrastive module and a tough cross-view prediction task.
We use their open source implementation 10 .
We use a learning rate of 3E − 4 with
Adam optimizer λ 1 = 1 , λ 2 = 0 .
7 as in their implementation, and jitter scale ratio or 1.1, maximum segment length of 8, and jitter ratio of 0.8, following their HAR setting. F R UNTIME A NALYSIS
[Table]
Table 6 shows the runtime in seconds for each phase for various representation learning methods and end-to-end approaches. All experiments are performed on an NVIDIA A100 GPU. Do note that for Informer, we follow the hyperparameters (including lookback window length) as described by the authors which may vary for different forecasting horizons, thus the runtime may not be strictly increasing as the forecasting horizon increases. We want to highlight that for all representation learning approaches, the ridge regressor portions should be equal since the dimension size used are the same across all methods, any differences are simply due to randomness.
Despite a slightly higher training time compared to some of the baseline approaches, CoST achieves much better results (refer to Table 1 in the main paper).
Furthermore, the extra computation time of CoST as compared to
TS2Vec is due to the sequential computation of each expert in the mixture of AR expert component, it can be further accelerated by parallel methods (He et al., 2021).
G U NIVARIATE FORECASTING BENCHMARK
[Table]
[Figure]
[Table]
H R ESULTS ON FEATURE - BASED FORECASTING BASELINES
[Table] [Table]
[Table] [Table]
We include hand-crafted features (using the same experiment methodology as representation learning approaches) in our benchmark, by using features from the TSFresh package.
We select the same set of features for all datasets and settings, to avoid extensive feature engineering which requires domain expertise.
TSFresh generally under performs in the multivariate benchmark due to the high dimensionality of the generated features, since it extracts univariate features and thus feature size increases linearly with input size.
On the other hand, it performs relatively well on the univariate setting.
I R ESULTS ON END - TO - END FORECASTING BASELINES COMPARED TO C O ST
[Table]
[Table]
ETT
[Table]
[Table] i c it y [Table] [Table]
J R ESULTS ON M5 DATASETS
[Table]
0.953 L 0.
0.990 L 0.
1.332 L 1.
Avg.
K C ASE S TUDY : D ISENTANGLEMENT
[Figure] To exhibit CoST’s ability to disentangle trend and seasonal components, we plot the T-SNE representations of the Trend Feature Disentangler (TFD) and Seasonal Feature Disentangler (SFD) separately.
From Figure 5a, we see that representations from the TFD indeed have better clusterability and the representations from SFD have a degree of overlap between the two trend patterns.
From
Figure 5b, we see that the TFD representations have a higher degree of overlap between the three seasonality patterns than the SFD representations.
Here, to better highlight the stronger capability of seasonality representations in extracting seasonal patterns, we used complex seasonality patterns (described below).
Complex Seasonality Similar to the synthetic data generation process in Appendix D, we generate three different seasonality patterns before combining them with the two trend patterns.
The ﬁrst seasonality pattern is no seasonality.
The second pattern begins with a sine wave of period, phase, and amplitude of { 3 , 0 , 10 } , thereafter, a mask is then applied to the entire pattern, consisting of a repeating pattern of three 1s and seven 0s. The third pattern begins with a sine wave of period, phase, and amplitude of { 10 , 0 . 5 , 15 } , thereafter, a mask is then applied to the entire pattern, consisting of a repeating pattern of two 1s and eight 0s.