T OWARDS U NSUPERVISED T IME S ERIES R EPRESENTATION L EARNING : A D ECOMPOSITION P ERSPECTIVE
Anonymous authors
Paper under double-blind review
A BSTRACT
Existing contrastive methods of universal time series representation learning mainly rely on distilling invariant patterns at varying scales and building contrastive loss with the help of negative sampling.
However, the invariance assumptions may not hold in real-world time-series data, and the infamous negative sampling could bring in new biases for representation learning.
In this work, we propose a novel contrastive learning approach toward time series representation learning on top of trend-seasonality decomposition, namely TS-DC.
TS-DC differentiates itself from prior methods in three folds: 1) a time series decomposition approach is devised to distill different aspects/components of a complex time series; 2) a novel component-wise contrastive loss is proposed in which negative sampling is not necessary; 3) the informative signals of time series can be captured comprehensively by means of adaptive contrasting.
Extensive experiments on different public benchmark datasets validate the superior performance of our proposed representation learning method.
I NTRODUCTION
Representing informative signals from intricate and noisy time series (TS) data is challenging for machine learning models (Gamboa, 2017).
Time series representation learning (TSRL) has been found beneficial for many downstream tasks, including classification, forecasting and anomaly detection (Leng et al., 2009;
Zerveas et al., 2021; Yue et al., 2022).
Though there have been many supervised TS feature extraction methods (Wu et al., 2018; Ruiz et al., 2021; Xiao et al., 2021), they can hardly tackle the high-dimensional and sparsely labeled real-world TS data (Ching et al., 2018).
Unsupervised representation learning aims to extract low-dimensional and unified representation without requiring labels, which sheds light on TS representation learning.
In this work, we study the problem of unsupervised time series representation learning (UTSRL).
Numerous recent works have come up with creative and interesting techniques toward unsupervised representation learning in different domains (Pinheiro et al., 2020; Caron et al., 2018; Ren et al., 2019b; Cheng et al., 2019).
Inspired by this, a few contrastive learning approaches have been proposed to address the UTSRL problem.
The main idea goes like this: there exist key properties in a time series that keep invariant across time and contexts, then one can encourage the model to learn such underlying signals by means of data augmentation at certain or different semantic scales.
For example, rather than focusing on sub-sequence invariant signals (Franceschi et al., 2019), neighborhood smoothness (Tonekaboni et al., 2020), or long-term dependency (Yang & Hong, 2022), Yue et al.
(2022) propose to leverage scale invariant information by means of hierarchical contrasting and data augmentation methods (including series cropping and timestamp masking, etc. ).
Besides, cross-view contrasting is another well applied method to enable the learnt representation to be aware of transformation-/contextual-consistency (Eldele et al., 2021) in time series data.
However, due to the huge diversities among different time series data, it is challenging to construct high-quality negative samples toward universal time series representation.
Furthermore, since multilevel noises exist in real-world time series data (Kostelich & Schreiber, 1993), it might be difficult to tell generated representations from noises.
Though it is useful to leverage large amount of negative samples, increasing the number of negative samples does not necessarily help (Arora et al., 2019), and extra biases could be raised by negative sampling (Chuang et al., 2020).
In addition, distilling the underlying and inherent features encoded in time series data, such as temporal dependencies (Xuan Murphy, 2007), is challenging yet critical for unsupervised TS representation learning.
While the assumptions about TS-property-invariance may lead to spurious correlations, which could harm the generalizability of UTSRL.
To this end, we propose a time series decomposition contrastive learning approach (named TS-DC) for UTSRL.
Specifically, we first propose a deep trend-seasonality decomposition method to examine underlying trend and seasonality in time series data with the help of variational auto-encoder (VAE).
To thoroughly exploit the signals in a time series, the remainder is further decomposed recursively.
Then, a component-wise contrastive loss is proposed to encourage the learnt representation be closer to informative components.
In this way, negative sampling can be avoided since the negative samples are deterministic and conditioned on the decomposition process only.
Afterward, a weight adjustment module is devised to balance the decaying weights of different components, such that the adaptive contrasting can be achieved.
We conduct extensive experiments on various benchmark datasets toward different downstream tasks ( e.g., classification and forecasting), and the rich results demonstrate that TS-DC outperforms existing SOTA unsupervised time series representation learning methods with satisfactory margins.
Main contributions of this work are summarized below:
• We propose a deep decomposition method to yield disentangled yet informative inherent features for time series data without seeking contextual-invariance signals.
• We propose a novel component-wise contrastive learning method devoted to time series data, such that the notorious negative sampling can be avoided.
• To fully capture the multi-faceted features in time series data, an adaptive contrasting mechanism is proposed.
R ELATED W ORK
Unsupervised Time Series Representation Learning.
1) Generative.
Time series representations can be obtained through generative models, like auto-encoders (Yingzhen & Mandt, 2018; Srivastava et al., 2015; Malhotra et al., 2017; Chung et al., 2016).
These methods generally embed the input into a latent space via the encoder and decode the representation back to recover the input signals.
Whereas, the representation distribution of the generative method is modeled at pointwise level, which may result in limited generalizability.
2) Contrastive.
Contrastive learning has achieved remarkable improvements toward unsupervised representation learning and has been applied to many different fields successfully, such as computer vision (Oord et al., 2018; Logeswaran & Lee, 2018;
Tian et al., 2020; Chen et al., 2020; Gao et al., 2021; Kuang et al., 2021) and speech recognition (Schneider et al., 2019; Kharitonov et al., 2021; Cuervo et al., 2022; Wang et al., 2021).
There have been works that successfully apply contrastive learning techniques to time series data (Franceschi et al., 2019;
Tonekaboni et al., 2020; Yue et al., 2022; Eldele et al., 2021; Zerveas et al., 2021).
Franceschi et al.
(2019) proposed an encoder formed by dilated convolutions that admits variable-length inputs, and trained with a triplet loss using time-based negative sampling.
Tonekaboni et al.
(2020) studied the stationary properties of time series and incorporate sample weight adjustment to produce generalizable time series representation.
Eldele et al.
(2021) proposed to encourage the invariance of different data augmentations and put efforts to learn robust representation by means of cross-view prediction and contextual contrasting.
However, the defects of these contrastive methods, such as sampling bias and invariance assumption, could lower the generalizability of learnt time series representation.
To relieve the sampling bias, Yang & Hong (2022) exploit the instance-level augmentation with standard dropout on time-series data and devise a temporalspectral method to enrich the representations.
Instead of negative sampling, we build the contrasting pairs for UTSRL from a decomposition perspective, which is different from existing works.
Time Series Decomposition.
STL (Seasonal-Trend decomposition using Loess) is one of the most classic and widely-used decomposition method (Cleveland et al., 1990; Dokumentov et al., 2015;
Wen et al., 2019).
However, the limited flexibility of these STL-based methods might not be able to handle the time series data with large diversities.
Recently, the effectiveness of applying deep learning algorithms to the time series decomposition has been proved (Asadi & Regan, 2020; Wu et al., 2021; Taylor & Letham, 2018; Oreshkin et al., 2019).
Our decomposition method is similar to N-Beats (Oreshkin et al., 2019), while we aim to maximize the gap between the decomposed trend and seasonality, and re-generate the trend and seasonality, separately.
In addition, we implement a reverting process to assembly all the decomposed artifacts such that the input series can be recovered.
M ETHODOLOGY
Problem Statement.
Given an M -dimension multivariate time series X
=
[ x 1 , x 2 , · · · , x M ] , ∀ m ∈ { 1 , 2 , · · · , M } , x m ∈ R L m where L m denotes the length of m -th series in X .
∃ 1 ≤ m 1 < m 2 ≤ M such that L m 1 ̸
=
L m 2 .
Then, unsupervised time series representation learning (UTSRL) aims to learn a function f θ : R L × M → R L × D ( L = max( L 1 , · · · , L M ) and D > 0 ) that maps X into a representation Z , where Z can describe X as much as possible.
As shown in Figure 1, TS-DC consists of multiple encoders.
The first encoder absorbs the input X ∈ R L × M and generates the initial representation Z ∈ R L × D .
Meanwhile, we decompose X into additive trend, seasonality, and remainder.
Then the remainder is decomposed into pairs of trend and seasonality recursively.
Each pair of trend and seasonality are added back to form a component X n ( n = 1 , 2 , · · · , N ).
After further embedding X n into Z n , a component-wise contrastive loss is applied, separately.
At last, the representations of different components are taken into account comprehensively by means of adaptive contrasting.
We introduce the details of our approach in following subsections separately.
3. T IME S ERIES D ECOMPOSITION
The trend-seasonality decomposition can reveal underlying insights of a complex time series (Laptev et al., 2015; Wen et al., 2019).
Trend usually reflects long-term increase and decrease, seasonality depicts periodic patterns, and spike & dip can be encoded in the remainder.
The detailed operations in a single decomposition can be seen in Figure 2.
Specifically, we construct a time vector T =
[ 1 , t , · · · , t c ] where t = ([1 , 2 , · · · , L − 1] /L )
T and c > 0 is a constant that controls the range of trend.
Then we can obtain the trend for the first decomposition as (cid:102)
X t 1 =
T × FC ( X ) , where FC ( · ) denotes a fully connected network.
Similarly, the seasonality can be extracted with Fourier Transform: (cid:102)
X s 1 = S × FC ( X ) , where S =
[1 , cos (2 π t ) , · · · , cos (2 π ( L 2 − 1) t ) , sin (2 π t ) , · · · , sin (2 π ( L 2 − 1) t )] is the matrix of sinusoidal waveforms for extracting seasonal and periodic patterns.
To ensure the validity of decomposed trend and seasonality, w.l.o.g. , we assume that trend X t 1 and seasonality X s 1 are Gaussian variables, and the distributions of X t 1 and X s 1 can be obtained by a mapping function g ( · ) ,
[Equation]
We then re-sample the trend and seasonality, i.e., X t 1 ∼ p ( X t 1 ) =
N ( X t 1 ; µ t 1 , σ t 1 ) , X s 1 ∼ p
( X s 1 ) =
N ( X s 1 ; µ s 1 , σ s 1 ) .
Since that trend and seasonality should behave differently, the following KLdivergence should be maximized,
[Equation]
As shown in Figure 1, the decomposition operations can be carried out recursively.
The n -th decomposition takes the residual of previous decomposition as input, i.e., X rn =
X rn − 1 − ( X sn + X tn ) .
Then, we aim to maximize the sum of all KL-divergences in different decomposition operations,
[Equation]
To simplify the training procedure, we instead minimize L KL = 1 / (
[Equation]
L KL ) , where α is a hyper-parameter.
In addition, in order to reduce the variance of decomposition operations, the sum of all the decomposed components should be close to the original time series X ,
[Equation]
By minimizing L decomp , we can ensure the quality of decomposition operations w.r.t.
approximating original time series.
3. C OMPONENT -W ISE C ONTRASTING
The above decomposition can be denoted as X ≈ (cid:80) Nn =1 X n
where X n = X sn + X tn .
Basically, component X n implies the trend and seasonality of input X regarding a specific aspect.
To enforce the outcome representation to learn the complex signals encoded in X from different perspectives, for the n -th decomposition, we treat embeddings Z n and Z m ( n ̸ = m ) as positive and negative, respectively.
As a result, the trend and seasonality of X n ( n = 1 , 2 , · · · , N ) can be highlighted by contrasting positives with negatives,
[Equation]
where E ( · ) : R L × M → R L × D denotes an arbitrary encoder.
Essentially, in Equation (5), negative sampling is not required since all the negative samples are deterministic, i.e., the remaining decomposed components { X m | m ̸ = n } .
Negative sampling is required for most contrastive learning methods, which is yet often tricky, biased and time consuming (Liu et al., 2021).
There exist pioneering works to get contrastive learning rid of negative samples in the computer vision field (Grill et al., 2020; Chen & He, 2021).
TS-DC gets the contrastive learning rid of additional biases brought by negative sampling and avoids data augmentation in time series representation learning.
3. A DAPTIVE C ONTRASTING
Both informative signals and noises exist in decomposed components of the original time series.
However, as the decomposition proceeds recursively, it is reasonable to assume that the capability of accounting for the raw time series is decaying to some extent because of the additivity of trendseasonality decomposition, i.e., X rn =
X rn [Equation] −
[Equation]
X tn + X sn ) .
Therefore, we combine the componentwise contrastive losses to form a new loss:
[Equation]
where β > 1 is a constant that controls the decaying speed.
Through minimizing the objective in
Equation (6), different component-wise contrastive losses can be jointly optimized.
Besides, with the monotonically decreasing weights, prior decomposed components will be paid more attention when learning the outcome representation.
However, the above configuration of decaying weights is not flexible enough to handle complex and highly diverse time series data.
Weight Adjustment.
To implement a data-driven weight adjustment, we can inspect the decomposition at a different angle.
The decomposition process can be written as: X 1: N = D
Nθ ( X ) , where
D Nθ ( · ) denotes the decomposition module consisting of N decomposition operations with parameters θ , which means that the decomposition is determined by X and θ : ( X , θ )
→ X 1: N .
On the other hand, according to the Bayes’ theorem (Ghosh et al., 2006), the parameters θ satisfy: p ( θ | X ) = p ( X | θ ) p ( θ ) /p
( X ) .
In other words, θ can be derived from X : X ⇒ θ .
Consequently, the decomposition X 1: N is actually determined by the original time series X only.
Therefore, we adopt a non-linear function h ϕ ( · ) to learn correlations between decomposed components and the time series X directly.
[Equation]
where W ∈ R N × D and W = [ w 1 , w 2 , · · · , w N ] T
We then rewrite the loss in Equation (6) below:
[Equation]
In this way, through contrasting the outcome representation with the signals in different components jointly and adaptively, the outcome encoder will be encouraged to be aware of multi-faceted patterns in the time series data. Optimization. Finally, we construct a composite loss and optimize it jointly.
To be specific, by combining the alternative of Equation (3), plus Equations (4) and (8), we have
[Equation]
where γ , λ , and ζ are trade-off parameters, and A GG ( · ) is a aggregator yielding real-value.
E XPERIMENT
We evaluate the representations learnt by TS-DC on univariate as well as multivariate time series datasets, respectively.
Besides, we use the downstream tasks including classification, forecasting and anomaly detection to report the performance comparisons.
The results of forecasting and anomaly detection are available in Appendices D and E.
4. E XPERIMENT S ETTINGS
Classification Datasets.
The UCR archive (Dau et al., 2019) consisting of 128 datasets is adopted for univariate time series classification, and the UEA archive (Bagnall et al., 2018) (includingdatasets) is employed for multivariate time series classification. We employ zero-padding for missing values to ensure the time series data are with the same length in a dataset. All the datasets in UCR and UEA archives are split into train and test data already, and the time series in 85 UCR datasets are already z-normalized.
We split the train data as 4 : 1 for training and validation for both UCR and UEA archives. For UCR archive, we apply z-normalization of train data to validation and test data accordingly. While for UEA archive, we adopt an instance-wise z-normalization suggested by (Rakthanmanon et al., 2013) for the training, validation, and testing data, separately.
Baselines.
We select six time series representation learning methods as competitors: 1) TLoss (Franceschi et al., 2019) uses time-based negative sampling and a triplet loss to obtain generalpurpose representations; 2) CPC (Oord et al., 2018) learns the representations by predicting the future in latent space with autoregressive model; 3) TNC (Tonekaboni et al., 2020) learns the time series representations by ensuring the distribution of signals within a neighborhood is distinguishable from the distribution of non-neighboring signals; 4)
TS2Vec (Yue et al., 2022) performs contrastive learning over augmented context views to learn the robust contextual representation for each timestamp; 5) TST (Zerveas et al., 2021) presents a Transformer-based framework for time series representation learning; and 6) TS-TCC (Eldele et al., 2021) uses a temporal and contextual contrastive method to learn the time series representations. 7) Supervised. We additionally compare a [Table]
[Table]
supervised method which is implemented with the same encoder adopted by the unsupervised models, but the encoder as well as the downstream model ( e.g., classifier or forecaster) are learned synchronously with the supervision signals (see Appendix A.1). Note that, to fairly compare the effectiveness of different TS representation learners, we employ a Transformer-based encoder (Vaswani et al., 2017) as the backbone encoder for both TS-DC and baselines in the experiments, and more implementation details are available in Appendix A.2.
Implementation Details.
The dimensionality of representation Z is set to 128 . In time series decomposition, a fully connected network ( i.e., FC ( · ) ) consisting of 2 nonlinear layers with the hidden size 256 is adopted.
The trend-seasonality decomposition operation is carried out 3 times in the experiments ( i.e., N = 3 ). The hyperparameter α used in L KL is 0 .
The constant controlling the decaying weights, i.e., β , is 2.
The trade-off parameters in Equation (9) is configured as γ = λ = ζ = 1 . Besides, we construct an RNN-based deep model with the hidden size 128 as the downstream classifier.
The non-linear function h ϕ ( · ) in Equation (7) is a 2-layer fully connected network with the hidden size being 256 . Evaluation Settings.
All the models are trained on an Nvidia P40 GPU with CUDA 10.2.
More parameter settings employed in the evaluation can be found in Appendix A.3. In addition, we use metrics Accuracy and AUPRC ( i.e., area under precision-recall curve) for evaluating classification performance. For both Accuracy and AUPRC, the larger the better.
4. T IME S ERIES C LASSIFICATION
The averaged Accuracy and AUPRC scores of six unsupervised methods and the supervised method on the UCR and UEA archives can be seen in Table 1.
On average, we have achieved 7% accuracy and 6% AUPRC improvements on 128 UCR datasets, and 6% accuracy and 0.9% AUPRC improvements on 30 UEA datasets, respectively.
Furthermore, with comparison to the supervised method, TS-DC also achieves comparable performance.
4.2. U NIVARIATE T IME S ERIES R ESULTS
The full ACC. results of 128 UCR datasets can be found in Appendix F.1.
Here we report the performance of five selected datasets in Table 2 to elaborate the effectiveness of different representation learning methods.
The descriptions about these five datasets [-shortly: 1) Coffee (Briandet et al., 1996) consists of food spectrographs and contains 28 train samples and 28 test samples withtime points; 2) ECG
 (Goldberger et al., 2000) is the ECG recordings with the series length 140 , and there are 500 samples for training and 4500 for testing; 3) FordB has 3636 samples for training and 810 for testing, each time series has 500 time points (the training data were collected-]can be found in Appendix A.4.
TS-DC outperforms most of the UTSRL methods and performs comparably with the supervised method.
Besides, we have following observations in Table 2. 1)
TST achieves better performance when the training data is of large amount, which is also the requisite of Transformer-based model. 2) [Table]
[Table] Both TS-TCC and TS2Vec use the temporal and contextual information to construct the contrastive loss.
TS2Vec utilizes sampled subseries at different scales, while TS-TCC explores the full time series, this difference may account for that TS-TCC performs better than TS2Vec. 3)
TNC highlights the local smoothness in time series data, which might be helpful in handling limited data ( e.g., Coffee dataset) or noisy data ( e.g., FordB dataset).
4) CPC and T-Loss are two distinct contrastive methods, the former adopts a probabilistic contrastive loss and the latter applies instance-wise contrastive loss.
On ECG5000 dataset, CPC achieves better performance than T-Loss, which demonstrates the strength of probabilistic contrastive loss in modeling instant data.
4.2. M ULTIVARIATE T IME S ERIES R ESULTS
The classification performance on 5 UEA datasets are reported in Table 3 (see Appendix F.2 for full ACC. results).
The brief descriptions about these five datasets are available in Appendix A.5.
In general, as depicted in Table 3, TS-DC still performs the best most of the time.
Besides, it can be seen that, TST achieves very competitive performance toward multivariate time series classification, which can validate the superiority of Transformer-based model on high-dimensional data.
T-Loss outperforms CPC w.r.t.
multivariate time series classification, which demonstrates the limitation of the probabilistic contrastive loss for high dimensional data. [Figure]
4.2. V ISUALIZING L EARNT R EPRESENTATIONS
In addition, we employ t-SNE plots to visualize the representations generated by different TS representation methods on ECG5000 dataset in Figure 3.
There exist 5 types of time series in ECG [Table] dataset.
In Figure
[Table] it can be seen that TS-DC achieves competitive performance comparing to the supervised method with respect to clustering samples in the same category.
Besides, for clustering samples of the rare class, i
[Table]
e., class 3, 4 and 5, TS-DC outperforms most of the alternative unsupervised TS representation learning methods. Another showcase can be found in Appendix C.1. [Table]
4. M ODEL A NALYSIS
The superior performance achieved toward downstream tasks validates the effectiveness of TSDC in learning informative TS representations.
Here, we explain underlying mechanisms contributing to the performance achieved by TS-DC (the additional parameter sensitivity analysis can be found in Appendix B). 4.3. A BLATION S TUDY OF L OSS T ERMS IN F INAL O BJECTIVE We implement tailored TS-DC by omitting L KL , L decomp , and L ac in Equation (9), respectively. Table 4 reports the corresponding performance comparisons.
As depicted, the proposed contrasting mechanism benefits the downstream classifier consistently (refer to w/o L ac ). Besides, the performance of univariate TS classification might be lowered if the decomposed components lack expected properties, such as trend-seasonality distinguishability (attributed to L KL ) and TS reconstructability (attributed to L decomp ). However, for multivariate TS classification, the contributions of decomposition are not good enough, which is probably accounted for the limitation of temporal decomposition on high-dimensional data. Basically, the relative contributions of L KL , L decomp , and L ac in Equation (9) are not impressive since the loss terms cannot fully reflect the contributions of decomposition and contrasting mechanism of TS-DC.
L KL and L decomp can enforce the decomposed components behave as expected, and L ac possesses the contrasting power of TS-DC to a certain extent.
In addition, we showcase t-SNE plots of TS-DC and its variants in Appendix C.1.1.
4.3. I NSPECTING T IME S ERIES D ECOMPOSITION
With the time series decomposition, we can dispatch trend and seasonality of a time series into different components, which can describe original time series to varying degrees.
We showcase the univariate time series and the decomposed components on ECG5000, FordB, Ham, and Strawberry datasets in Figure 4.
We can see that more informative signals are captured by the earlier component, and the first component usually acts as the most informative one that accounts for original time series.
As the decomposition proceeds, the latter components contain less informative signals and the level of noise grows as well.
Thus, the prior components should often be paid more attention. In
[Figure] by TS-DC on four datasets. Here, the input time series is decomposed into 3 components, i.e., X = { X 1 , X 2 , X 3 } .
[Table]
addition, it is noteworthy that for some cases [Table] the decomposed components behave similarly ( e
[Table]
g.,
Figure 4c) [Table] which can validate the necessity of adaptive contrasting of TS-DC toward representing complex time series data. More results can be found in Appendix C.2.
[Table]
4.3. A BLATION S TUDY OF A DAPTIVE C ONTRASTING
As discussed in Section 4.3.
[Table]
the amount of insightful signals contained in different components differs.
We combine the monotonically decreasing weights plus a data-driven weight adjustment to realize the adaptive contrasting.
Though we showcased the adaptive contrasting is necessary, we still wonder how the monotonically decreasing weights and the data-driven weight adjustment contribute to the overall performance of downstream tasks.
We separately train the model without weight adjustment ( i.e., TS-DC − W ), without predefined monotonically decreasing weight ( i.e., TS-DC β =0 ) and without adaptive contrasting (TS-DC β
=0 , − W ) to inspect the effects of the decaying weights and the weight adjustment comprehensively.
Here we report the classification performance on the univariate time series and multivariate time series datasets in Table 5, the t-SNE plots of the obtained representations can be found in Appendix C.3.1.
In particular, the predefined decaying weights play an important role in learning useful representations (see TS-DC − W ).
While the weight-adjustmentonly mechanism performs much worse (see TS-DC β =0 ), even harms the learnt representations without any adaptive contrasting mechanisms ( i.e., TS-DC β
=0 , − W ), such as, the performance reported on Ham, FordB, and FaceDetection datasets.
But the weight adjustment mechanism can greatly improve the effectiveness of the representation learning with the help of predefined decaying weights, which further validates the effectiveness of the adaptive contrasting in TS-DC.
D ISCUSSION
The challenges encountered in UTSRL are different from those in other domains because of the intricate characteristics in time series data.
Besides the common issues of contrastive learning, like sampling bias and learning robustness, how to highlight informative time series patterns effectively is of great importance as well.
In this work, we found that through the time series decomposition, the performance of representation learning can be improved with the help of contrastive learning.
Due to the nature of non-stationary, the decomposed components of a time series differ from each other.
To harness the decomposed components comprehensively, we propose an adaptive contrasting mechanism consisting of weight decaying and weight adjustment.
However, it is challenging to learn the best weight configuration for highly diverse time series data.
Therefore, a more robust and automatic weighing mechanism need to be explored.
In addition, we propose a component-wise contrastive learning method toward UTSRL, while the components with high level noise should be resisted, thus, how to pick the high-quality components is another open problem.
CONCLUSION
In this work, we studied the problem of unsupervised time series representation learning which is beneficial for multiple downstream tasks.
To tackle the time series representation learning problem, we proposed a time series decomposition contrastive learning approach, namely TS-DC.
Specifically, TS-DC employs trend-seasonality decomposition to construct a serial of components for a time series.
Then, a component-wise contrastive loss was proposed, such that the negative sampling is avoided and the informative signals of time series data can be fully captured.
Moreover, we devised an adaptive contrasting mechanism to comprehensively leverage the decomposed components in a time series for better representation learning.
Extensive experiments on multiple benchmark datasets toward different downstream tasks validate the effectiveness of TS-DC.
R EPRODUCIBILITY S TATEMENT
The source urls to the baselines and the links to the benchmark datasets used in this work are summarized in ‘src-urls.pdf’ (in the uploaded supplementary materials).
In addition, detailed descriptions about the implementations are listed below.
1) Dataset.
The datasets used for the classification tasks are introduced in Section 4.1, the first paragraph in Section 4.2.1 and Appendix A.5.
The forecasting datasets are introduced in Appendix D, and the datasets used for anomaly detection are introduced in Appendix E. 2) Source Code of TS-DC .
The source code of the proposed algorithm in this paper is included in the supplementary materials, named ‘tsdc.src.zip’.
3) Implementation Details.
The parameter settings of our model can be found in Appendix A. and the implementation details of the baseline methods are described in Appendix A.2.
A E XTRA E XPERIMENT S ETTINGS
In the empirical study, to fairly compare different time series representation learning methods, the same encoder as well as the same downstream algorithm (for classification or forecasting or anomaly detection) is employed to evaluate the effectiveness of learnt representations. A. T RAIN T HE E NCODER : S UPERVISED V . S . U NSUPERVISED
As introduced in the descriptions of baseline settings in Section 4.1, we implement a supervised method as a competitor.
In particular, the training procedure of supervised time series representation learner is different from unsupervised time series representation learning, which is demonstrated in
Figure 5. For supervised method of TS representation learning, the representation learner ( i.e., the Encoder in Figure 5a) is trained together with the downstream model ( e.g., a classification model), guided by the supervision signals. Whereas, as depicted in Figure 5b, for UTSRL, the representation learner is trained with the help of self-supervised loss, such as contrastive loss or reconstruction loss.
Then, the trained Encoder is adopted directly to generate embeddings, and the downstream model is optimized with the supervision signals independently.
[Figure] A. I MPLEMENTATION D ETAILS Implementation Details of Unsupervised Baselines.
1) TNC (Tonekaboni et al., 2020) chooses the adjacent segments as the positive samples and nonneighborhood segments as the negative samples.
In particular, the size of sampling window is set to be ⌊ 1 / 6 · L ⌋ where L is the time series length, and the number of negative samples is set as 5.
Then, we follow the other settings as recommended.
For CPC (Oord et al., 2018) and TLoss (Franceschi et al., 2019), we follow most of the implementation settings introduced in (Tonekaboni et al., 2020).
Besides, the number of time steps for CPC is set as ⌊ 1 / 10 · L ⌋ or 2 if L <( L is the time series length).
The number of negative samples is 3 for T-Loss and the window size is also set as 1 / 10 · L . 3)
For TS2Vec (Yue et al., 2022), the temporal unit is set to 0 and the max train length is 3000. 4)
For TS-TCC (Eldele et al., 2021) and TST (Zerveas et al., 2021), the recommended parameter settings are employed for the evaluation.
Implementation Details of Backbone Encoder and Downstream Classifier.
The encoder used in this paper is implemented following the encoder in Transformer which is built on top of multi-head attention (MHA).
In particular, the number of heads is set to 8, and 3 encoder layers are adopted.
Besides, the embedding dimension in MHA is set to 128.
The encoders in TSDC are with the same architecture and share the same parameters.
For classification, the downstream classifier consists of a 1-layer GRU with hidden size 256 and a 2-layer MLP network with hidden size 256, and the ReLU activation function is adopted. A. M ORE E VALUATION S ETTINGS
The shared experimental settings in the evaluation are as follows. We utilize an Adam optimizer with learning rate 1 e − 3 in the experiments. The batch size is 16 for the datasets that contain more than 100 samples, otherwise, the batch size is set as 2. The training procedure is repeated in 20 epochs at most, and early stopping is adopted where the patience is set to 5.
A. D ESCRIPTIONS OF T HE D ATASETS FOR U NIVARIATE T IME S ERIES C LASSIFICATION
We briefly introduce the selected datasets for elaborating univaraite time series classification results in the main text (refer to Section 4.2.1) as follow.
1) Coffee (Briandet et al., 1996) consists of food spectrographs and contains 28 train samples and 28 test samples with 286 time points; 2) ECG5000 (Goldberger et al., 2000) is the ECG recordings with the series length 140 , and there are 500 samples for training and 4500 for testing; 3) FordB has 3636 samples for training and 810 for testing, each time series has 500 time points (the training data were collected in operating conditions, while the test set were collected under noisy conditions); 4) Ham (Olias et al., 2006) composes of the food spectrographs as well, the train/test size is 109 / 105 ; 5) Strawberry is obtained by Fouriertransform infrared spectroscopy (FTIR) including 613 and 370 samples for training and testing, respectively, with 235 time points being collected.
A. D ESCRIPTION OF THE D ATASETS FOR M ULTIVARIATE T IME S ERIES C LASSIFICATION
The descriptions of the selected datasets for multivariate time series classification (see Section 4.2.2) are briefly introduced below.
1) Epilepsy (Villar et al., 2016) was collected from 6 participants using a tri-axial accelerometer on the dominant wrist whilst conducting 4 different activities.
This dataset contains 137 training samples and 136 testing samples, and the dimension and length of each sample is 3 and 207 , respectively.
2) FaceDetection is an EEG dataset recorded from 16 different subjects, the train/test size is 5890 / 3524 , the length and dimension of each sample is 62 and 114 , respectively.
Heartbeat (Liu et al., 2016) is the heart sound recording dataset.
It includestraining samples and 205 testing samples, and the length of each sample is 405 with dimension 61 .
4) PenDigits (Alimoglu et al., 1996) includes 7494 training samples and 3498 testing samples, and each sample contains 8 points where each one is with 2 dimensions.
RacketSports containstraining instances and 152 testing instances, for each instance, the time series length is 30 and the dimensionality is 6. [Figure] [Table]
B P ARAMETER S ENSITIVITY A NALYSIS
In accordance with the experimental results in the main text [Table] we report the parameter sensitivity analysis on the same datasets adopted in Section 4.
[Table]
B. D IVERGENCE B ETWEEN T REND AND S EASONALITY IN D ECOMPOSITION
The parameter α in L KL (see Section 3.1) affects the divergence between the decomposed trend and seasonality of a time series.
We analyze the classification performance of the learned representation yielded by TS-DC on different datasets with varying α from 0 .
1 to 3 , which is demonstrated in Figure 6.
As depicted (Figures 6a to 6d), univariate time series classification is more sensitive to α , i.e., the KL-divergence between the decomposed trend and seasonality.
The probable reason behind is that the KL-divergence between decomposed trend and seasonality of univariate time series data is relative large, which means that the decomposition of univariate time series data is more easier.
This can be further validated by examining the effect of α on multivariate time series data (see Figures 6e to 6h), which show that multivariate time series is immune to the time series decomposition.
B. T HE C ONSTANT C ONTROLLING D ECAYING W EIGHTS
To evaluate the influence of decaying function in adaptive contrasting, we vary the constant β in Equation (6) (refer to Section 3.3 of the main text) and examine the representation learning performance accordingly.
We report the classification performance of TS-DC with β ranging in { 1 , 2 , 3 , 4 } .
The classification performance on both univariate and multivariate datasets are reported in Table 6, where β = 2 is used in the main experiment.
Generally speaking, the effectiveness of the representation learning function is relatively insensitive to β in terms of classification performance.
Whereas, the most suitable β on different datasets differs.
In particular, when the time series data is of high quality, the decomposed components are distinguishable from each other, a decaying function with conservative β value ( β ≤ 2 ) is preferred.
While if the time series is of high level noise, like FordB, a rapid decaying function is required to distinguish the decomposed components, thus, a large β value is beneficial.
We showcase t-SNE plots of TS-DC and its variants by omitting L KL , L decomp , and L ac in Equation (9), respectively. The clustering results on ECG5000 dataset are demonstrated in Figure 7. As discussed in the main text of Section 4.3.1, the relative contributions of L KL , L decomp , and L ac are not significant, therefore, the tailored variants of TS-DC can achieve competitive clustering results as well.
[Figure] [Figure]
C.1. M ORE R ESULTS OF V ISUALIZING L EARNT R EPRESENTATIONS
In addition to the case study in Section 4.2.3, we showcase more t-SNE plots on the PenDigits dataset (Figure 8).
Frankly speaking, as the time series in PenDigits dataset are not dominated by certain classes, the clustering effect of outcome embeddings yielded by TS-DC is more easy to tell.
Besides, TST, TS-TCC, and TS2Vec perform much better than TNC, CPC, and T-Loss.
C. R EVISIT T IME S ERIES D ECOMPOSITION IN TS-DC
Figure 9 demonstrates the detailed process of time series decomposition in TS-DC framework, which has been described in the main text of Section 3.1. C.2. H OW TO L EARN THE TS D ECOMPOSITION IN TS-DC As introduced in Section 3.1, the decomposition is repeated in a recursive way, which can be formulated as follows:
[Equation]
[Equation]
Calculate
[Equation]
[Equation]
[Equation]
[Equation]
12: θ
N arg until
The decomposition process described above absorbs the remainder X r recursively, such that the complex signals encoded in time series can be fully captured.
Though similar to traditional greedy algorithms in signal recovery, like Orthogonal Matching Pursuit (OMP) (Rezaiifar & Krishnaprasad, 1995), the parameters that govern the decomposition process in TS-DC are determined in an end-to-end fashion.
Specifically, we construct a time vector T to extract time features, a set of trigonometric functions S as the orthogonal functions to utilize the properties of Fourier Transform, and fully connected networks FC ( · ) to implement the trend and seasonality extractions.
Then, let θ n denote the parameter set in the n -th decomposition, and H denote the set of neural networks H
= { FC tn , FC sn , g tn , g sn } , we can rewrite the above decomposition process:
[Equation]
Afterward, with the proposed objectives L KL and L decomp (see Section 3.1), the related parameters θ n can be learned accordingly:
[Equation]
[Equation]
where η is the learning rate and L θ n
=
L KL + L decomp .
For more details of the training procedure, please refer to Algorithm 1.
C.2. M ORE V ISUALIZATION R ESULTS OF TS D ECOMPOSITION
We separately plot the decomposed trend and seasonality in each decomposition step for ECG and Coffee datasets, respectively.
The plots are demonstrated in Figure 10.
We can see that, in the first decomposition step, the combination of trend and seasonality can recover the original series quite well.
As the decomposition proceeds, less informative signals can be observed in the decomposed trend and seasonality.
Besides, the decomposed trend prefer to be conservative in recovering original time series, while seasonality most likely to be more radical.
[Figure]
[Figure]
[Figure]
Moreover, since the information contained in the remainder keeps being lowered as the decomposition proceeds, the trend and seasonality in following decomposition steps become more noisier.
In Figure 11, we plot the original TS as well as the recovered TS, and the recovered TS is realized by fusing the decomposed components.
It can be seen that, as the number of decomposition steps climbs up, the recovered time series fit the original TS better yet more noises come along.
The existence of noise does negatively affect the TS decomposition.
However, this is beneficial for contrastive learning, since the robustness of the TS representation learning can be enhanced.
C.2. T IME S ERIES C LASSIFICATION R ESULTS I N T ERMS O F N
In Table 7, we investigate the effectiveness of outcome representations yielded by TS-DC toward the time series classification task.
In particular, we report classification scores against the number of components to be decomposed ( i.e., N ) in TS-DC framework.
We can see that, with regards to
TS classification, the right number of components differs case by case.
Therefore, it is important to determine the decomposition level for different time series data in a more heuristic and intelligent manner.
[Table]
C. F URTHER A NALYSIS OF A DAPTIVE C ONTRASTING C.3. A BLATION S TUDY OF A DAPTIVE C ONTRASTING : Q UALITATIVE A NALYSIS
In addition to the quantitative ablation study of the adaptive contrasting of TS-DC (see Section 4.3. in the main text)
[Table] we visualize the representations learned by TS-DC and its variants on univariate and multivariate time series datasets, respectively, in Figure 12. In particular
[Table] we visualize the t-SNE plots of the representations learned by TS-DC and its variants, including TS-DC w/o weight adjustment (TS-DC − W ), TS-DC w/o predefined monotonically decreasing weights (TS-DC β =0 ) and
TS-DC w/o adaptive contrasting (TS-DC β
=0 , − W ).
The visualizations are carried out on ECG (univariate TS) and PenDigits (multivariate TS) datasets, respectively.
We can observe that, the representations yielded by TS-DC − W are less distinguishable, while more distinguishable representations can be learned by TS-DC β =0 yet too excessive (even the samples in the same category cannot be grouped together).
This is consistent with the analysis of Section 4.3.3 in the main text.
Moreover, the representations learned by TS-DC β =0 , − W mistakenly group the samples in different classes and separate the samples in the same category.
Therefore, not only weight decaying but also weight adjustment are helpful in contributing to the outcome representation learning. [Figure] [Figure]
C.3. W EIGHT A DJUSTMENT A NALYSIS : I MPACTS ON L EARNING P ROCEDURE
The weight adjustment in TS-DC framework can not only weigh different components in the time series data but also handle the heterogeneity among different types of time series, and thus improve the robustness of representation learning.
Here we analyze the impacts of the weight adjustment on the training procedure of TS-DC by inspecting the learning curves w/ and w/o the weight adjustment.
In Figure 13, we plot the curves of contrastive losses of decomposed components { X 1 , X 2 , · · · , X N } where N = 3 , and compare the loss curves w/ and w/o the weight adjustment, denoted by L nac and L nac, − W ( n = 1 , 2 , 3 ).
Besides, the curves of the original training loss and validation loss are also plotted.
In general, the weight adjustment improves the stability of the overall learning procedure as the validation loss decreases smoothly.
In addition, the curves of different decomposed components behave more discriminatively when the weight adjustment is introduced.
C. C ONTRASTING WITH T REND /S EASONALITY AT D IFFERENT D ECOMPOSITION L EVELS
Rather than contrasting the decomposed component at different levels, in this experiment, we attempt to rebuild a new contrastive loss by contrasting trend/seasonality at different levels.
Specifically, at each decomposition level n ∈ { 1 , 2 , · · · , N } , the decomposed trend/seasonality is treated as the positive and the corresponding seasonality/trend as the negative.
For instance, if we contrast trend X tn (positive) with X sn (negative), then the loss function in Equation (5) can be rewritten as:
[Equation]
then the corresponding objective for adaptive contrasting is:
[Equation]
Vice versa, we can build an alternative of Equation (14) by contrasting seasonality X sn with X tn (denoted by (cid:101)
L s + /t − ).
Afterward, we compare the classification performance of TS-DC on univariate and multivariate time series datasets with different contrastive loss functions and report corresponding experiment results in Table 8.
We can see that, in terms of AUPRC, the classification performance on Ham and Epilepsy datasets can be improved with the new contrasting mechanism, which
[Table]
mainly indicates that a certain temporal pattern (trend or seasonality) in these two datasets should be paid more attentions.
D T IME S ERIES F ORECASTING R ESULTS
In this section [Table] we evaluate the effectiveness of representation learning towards the time series forecasting task.
We conduct the time series forecasting with the learnt TS representation functions on three publicly available datasets: Traffic 1 , Weather 2 and Electricity 3 .
In particular, Traffic dataset includes 862 variables [Table] and it is a collection of 48 months (2015-2016) hourly recorded data from the California Department of Transportation; Electricity dataset collects the electricity consumption of 321 clients from 2012 to 2014.
Weather dataset was recorded between 2020-2021 every 10 minutes including 21 weather indicators.
We split the training/validation/test set as 7 : 1 : 2 for these three datasets [Table] and three prediction lengths ( i.e., 32, 64 and 128) are tested.
To construct the instance set suitable for TS forecasting, an input- L x -predict- L y sliding window is adopted to roll the train, validation and test sets with stride 1 time step, respectively, which is illustrated in Figure 14.
In the experiment, the input length is the same with the prediction length, i.e., L x = L y .
Besides, we employ MSE and MAE as the evaluation metrics.
Moreover, the same settings of the representation learning for the classification task (see Appendices A.1 to A.3) are adopted.
The forecasting results are reported in Table 9, which shows that TS-DC consistently achieves better forecasting results compared to other baseline representation learning methods.
Moreover, TST also performs well toward the time series forecasting task, which can demonstrate the superiority of the Transformer-based model on the high-dimension data among these baseline contrastive learning methods.
TS-TCC
and
TS2Vec both stress the temporal and contextual patterns of the time series data, and the superior forecasting performance than other representation learning methods highlight [Table] [Figure] the important role of temporal pattern modeling in forecasting.
Due to the limited capability of exploring the essential characteristics of time series data, like temporal dependencies, CPC and
TNC perform not satisfactory for the time series forecasting.
Moreover, TNC performs worse than other baselines, which suggests that the signals focusing on local smoothness hinders the model foreseeing the future changes, thus limits the capacity of forecasting.
Showcases of forecasting results.
We additionally showcase the results of different representation learning methods toward time series forecasting on Electricity dataset, which is shown in Figure 15.
Compared to other unsupervised representation learning methods as well as the supervised representation learning method, TS-DC represents the trends and peaks more accurately, which further validate the superiority of TS-DC in capturing the temporal dynamics that are helpful in forecasting the future time series.
E T IME S ERIES A NOMALY D ETECTION R ESULTS
To evaluate the time series representation learning for the time series anomaly detection task, we conduct experiments
[Table]
includes 29 different minutely sampled KPI curves from various Internet companies.
Yahoo dataset is an open anomaly detection dataset released by Yahoo Lab.
This dataset contains 367 hourly sampled time series with anomaly points being labeled.
The settings of constructing training/validation/test sets are as follows.
For the KPI dataset, we follow the training/test split setting of released data, and further split each time series in the training set as 4 : 1 following the time order to form the training and validation sets accordingly.
For each time series in Yahoo dataset, we chronically split a time series as 7 : 1 : 2 and construct the training, validation and test sets.
Subsequently, we employ a streaming evaluation protocol.
To be specific, given a time series in training/validation sets, a length l sliding window is utilized to roll the series with stride d time steps.
In the evaluation, we set l = 128 and d = 12 .
Then, given a sliced time series with fixed length l , for each time point within this sliced series, we predict whether a point is anomaly or not.
Besides, a time series in the test set is evaluated without being sliced.
The representation learning is conducted on training and validation sets, and the dimensionality of the latent representation is 32 and the batch size is 100 .
The other settings of the representation learning for TS-DC and baselines are the same with the descriptions in Appendices A.1 to A.3.
Then, a learned time series representing function takes the time series data in the test set as input and encodes them accordingly.
Afterward, the outcome representations are fed into a downstream anomaly detector.
To fairly compare the effectiveness of different representation learning methods, we employ the same anomaly detection method for different representation learning models.
We take ECOD 4 as the anomaly detector (Zhao et al., 2019; Li et al., 2022) to evaluate the quality of obtained representations yielded by different methods.
The evaluation metrics including accuracy, precision, recall, F1 and AUC are reported.
The results can be seen in Table 10.
We can see that TS-DC consistently outperforms baseline representation learning models, while the margins are not considerable.
In particular, the poor performance of CPC suggests that the forecasting capacity relying on the temporal dependencies does not help in anomaly detection task.
Basically, the overall anomaly detection performance of the representation learning algorithms are not impressive.
The probable reason is that the current time series representation learning models mostly emphasize the macroscopic signals, which is immune to the rare, sparse and disordered outliers.
Therefore, it is still an open question for unsupervised time series representation learning to fuel the downstream anomaly detection task.
F F ULL T IME S ERIES C LASSIFICATION R ESULTS F. F ULL R ESULTS OF U NIVARIATE T IME S ERIES C LASSIFICATION
The full results of univariate time series classification on 128 UCR datasets can be found in Table 11.
In general, the averaged accuracy of TS-DC outperforms other baseline unsupervised time series representation learning methods.
Besides, TS-DC even outperforms the supervised method in terms of averaged accuracy.
[Table]
Computers 0.
CricketX 0.
CricketY 0.
CricketZ 0.
DiatomSizeReduction 0.
DistalPhalanxOutlineAgeGroup 0.
DistalPhalanxOutlineCorrect 0.
DistalPhalanxTW 0.
Earthquakes 0.
ECG 0. ECG 0.
ECGFiveDays 0.
ElectricDevices 0.
[Table] [Table]
Phoneme
[Table] [Table]
Plane
[Table]
ProximalPhalanxTW 0.
RefrigerationDevices 0.
ScreenType 0.
ShapeletSim 0.
ShapesAll 0.
SmallKitchenAppliances 0.
SonyAIBORobotSurface 0. SonyAIBORobotSurface 0.
StarLightCurves 0.
Strawberry 0.
SwedishLeaf 0.
Symbols 0.
[Table] [Table]
EOGVerticalSignal 0.
EthanolLevel 0.
FreezerRegularTrain 0.
FreezerSmallTrain 0.
Fungi 0.
GestureMidAirD 0. GestureMidAirD 0.
GestureMidAirD 0.
GesturePebbleZ 0. GesturePebbleZ 0.
GunPointAgeSpan 0.
GunPointMaleVersusFemale 0.
[Table]
F. F ULL R ESULTS OF M ULTIVARIATE T IME S ERIES C LASSIFICATION
Table 9 shows the full results of TS-DC and other baseline unsupervised representation learning methods as well as the supervised representation learning method on 30 UEA datasets.
As depicted,
TS-DC achieves better classification performance in terms of averaged accuracy.
[Table]