Self-Supervised Time Series Representation Learning with Temporal-Instance Similarity Distillation
Abstract
We propose a self-supervised method for pretraining universal time series representations in which we learn contrastive representations using similarity distillation along the temporal and instance dimensions.
We analyze the effectiveness of both dimensions, and evaluate our pre-trained representations on three downstream tasks: time series classification, anomaly detection, and forecasting.
1. Introduction
Large-scale pre-trained models provide the initial foundation in many real-world machine learning systems, in particular in the domains of computer vision and natural language processing.
Despite the wide range of applications in healthcare, finance, transportation, energy, etc., the development of large-scale pre-trained models for (non-language) time series remains under-explored in the machine learning community.
An important first step towards building such a model is the self-supervised learning of a universal representation for time series.
We present a novel self-supervised representation learning method for time series in which we leverage similarity distillation (Tejankar et al., 2021; Zheng et al., 2021) as an alternative source of self-supervision to traditional negative-positive contrastive pairs.
In particular, we propose to learn contrastive representations using similarity distillation along the temporal and instance dimensions.
We systematically analyze the effectiveness of distillation in both dimensions, as well as the benefit of hierarchical pooling.
We evaluate the performance of our pre-trained time series representations on three downstream tasks: time series classification, anomaly detection, and forecasting.
Our code is available at https://github. com/BorealisAI/ssl-for-timeseries .
2. Related Work
Self-supervised techniques can be used for learning representations without requiring explicit labels.
The pre-trained representations can afterwards be fine-tuned with fewer labelled data for different downstream tasks.
For time series representation learning, pre-training approaches have shifted from simpler approaches such as using a sequenceto-sequence encoder-decoder architecture (Malhotra et al., 2017) to more advanced techniques, such as using either pretext tasks (e.g., learning the masked values in TST (Zerveas et al., 2021)) or contrastive learning on different augmentations of the input series, e.g., in TS-TCC (Eldele et al., 2021), T-Loss (Franceschi et al., 2019), TNC (Tonekaboni et al., 2021), and TS2Vec (Yue et al., 2022).
Contrastive methods have been empirically shown to have a better performance (Yue et al., 2022;
Tonekaboni et al., 2021; Eldele et al., 2021)
and they are trained by augmenting every batch and taking the augmentations of the same input as a positive pair and augmentations of different inputs as negative pairs.
The contrastive loss brings the representations of the elements in positive pairs closer to each other than the elements that form a negative pair.
However, contrastive methods for self-supervised representation learning rely on the assumption that the augmentation of a given sample will generate a negative pair with other samples in the batch.
This assumption is not always valid: there could be samples of the same class in the current batch which would mean that not all the assumed negative samples are truly negative.
To address this issue, instead of using positive and negative pairs with contrastive learning, we can use knowledge distillation based approaches, in which a student network is trained to produce the same similarity PDF as a teacher network (with momentum-updated weights) between the current elements in the batch and a set of anchors.
This is the approach taken in ISD (Tejankar et al., 2021) and ReSSL (Zheng et al., 2021).
While this approach has been used in the computer vision domain, to the best of our knowledge, it has never been used for pre-training time series representations. [Figure] [Figure] sl
[Figure]
l s l
Gradient update d d l queue max sl
[Figure] s l s l d s l s l s l 3. Method 3.1. Problem Definition
We propose to learn a non-linear embedding function for a set of time series X
= { x 1 , x 2 , ..., x N } of size N that maps each time series x i with T i timestamps, to its best describing representation
r
i = { r
i 1 , r i 2 , ..., r iT i } .
Each r ij ∈ R
d is the representation of time series i at timestamp j .
3.2. Model Architecture
Our overall architecture can be viewed in Figure 1.
Inspired by the recent approaches in self-supervised learning (Tejankar et al., 2021; Zheng et al., 2021), we propose a student-teacher framework that uses similarity distillation to learn time series representations in a self-supervised manner.
We incorporate the same augmentation technique as the state-of-the-art TS2Vec method (Yue et al., 2022), i.e., we sample two overlapping subsequences from the same sequence.
The two subsequences are applied to a teacher encoder and a student encoder.
Student and teacher encoders follow TS2Vec, consisting of three components: an input projection layer, a timestamp masking module, and a dilated CNN module.
Gradients of the network are only propagated through the student encoder while the teacher encoder is the moving average of the student encoder to avoid collapse (Tejankar et al., 2021; Zheng et al., 2021;
Chen & He, 2021).
While similarity distillation has been used in computer vision, to the best of our knowledge, we are the first to leverage this framework for time series data.
As shown in Figure 1, applying the student and teacher encoders to the subsequences results in s l × d matrices for the student and teacher representations in the overlapping region, where s l is the length of the overlap (the representations of the nonoverlapping regions are ignored in further processing).
Similar to (Tejankar et al., 2021) and (Zheng et al., 2021), we use a memory buffer, implemented as a queue, to store a set of anchor sequences.
For each input sequence, the teacher representations in the overlapping region are appended to the memory buffer, forming a l × max s l × d matrix of anchor representations, where l is length of the buffer and max s l is the maximum overlap length.
Zero padding is applied where necessary.
To learn an effective representation for time series, we want to capture the relationship between the events at various timestamps within the same sequence (temporal objective) as well as the relationship across different sequences (instance objective).
Let s j denote the student representation of an augmented sequence at temporal position j (illustrated as a single ddimensional slice at the bottom of Figure 1).
The temporal loss is computed as follows.
First, we contrast s j with the other student representations of the same augmented sequence at all other temporal positions (green dotted arrows): given a similarity function sim, such as cosine similarity, we obtain the s l -dimensional probability distribution
[Equation]
where τ is a temperature hyperparameter.
Analogously, with t j denoting the teacher representation of the augmented sequence at temporal position j
(illustrated as a single ddimensional slice at the top of Figure 1), we compute
[Equation]
The temporal loss is obtained by summing the KL divergences KL ( p temp t [Equation] p temp s [Equation]
j ) over all temporal positions:
[Equation]
The instance loss contrasts s j with the representations of buffered sequences at temporal position j (red dotted arrows).
We obtain the l -dimensional student probability distribution
[Equation]
and the teacher probability distribution
[Equation]
where q k denotes the k th anchor sequence in the memory buffer.
The instance loss is then obtained by summing the
KL divergences KL ( p inst t || p inst s ) over all temporal positions:
[Equation]
The overall self-supervised loss is given by
[Equation]
where α is a balancing hyperparameter.
4. Experiments
In this section, we present our experimental results using the pre-trained representations learned by our proposed model.
4.1. Tasks, datasets, and evaluation metrics
We evaluate our model on three different downstream tasks:
Classification: For this task, we use the 125 UCR 1 and
UEA 2 benchmarks, which consist of many small datasets of univariate time series.
To evaluate the performance of our model, we use accuracy.
Anomaly detection: For this task, we use the KPI dataset (Ren et al., 2019), a competition dataset released by the
AIOPS Challenge.
It consists of multiple KPI (key performance indicator) curves from 58 companies with very long sequences varying from 4,000 to 75,000 for the anomaly detection task.
We use precision, recall, and F1 score to evaluate the performance.
Forecasting:
For this task, we use the 3ETT 3 (Zhou et al., 2021) and Electricity 4 (Dua & Graff, 2017) datasets.
We perform both univariate and multivariate forecasting and evaluate the performance by calculating the mean-squarederror (MSE) and mean-absolute-error (MAE).
4.2. Hyperparameter settings
The size of the queue is 128 in all experiments, the temperature τ is 0 . 07 , and the temporal and instance losses have the same weight in optimization, i.e., α = 0 .
For all hyperparameters in common with TS2Vec, we follow the hyperparameter settings reported in Yue et al.
The same hyperparameters are used for all downstream tasks.
4.3. Results
Table 1 shows the overall performance of our full model on the UCR and UEA datasets for classification.
Our model achieves competitive performance: it outperforms several recent self-supervised techniques, including TST (Zerveas et al., 2021)), TS-TCC (Eldele et al., 2021), and TNC (Tonekaboni et al., 2021) on both UCR and UEA; however, TS2Vec outperforms our model on this task.
Interestingly, our results show that adding the hierarchical contrast (as recommended in Yue et al.
(2022)) does not improve our model for classification (details in Section 4.4).
Table 2 shows experimental results on the KPI dataset for anomaly detection.
On this task, our model achieves higher recall and F1 scores, while TS2Vec performs better on precision.
Table 3 shows experimental results on the 3ETT and Elec- [Table] [Table] tricity datasets for univariate and multivariate time series forecasting.
We provide results from Informer (Zhou et al., 2021) as well as TS2Vec (Yue et al., 2022) for comparison.
It is worth noting that Informer is a supervised method. Our model consistently outperforms Informer in both univariate and multivariate forecasting. In univariate forecasting, our model achieves comparable performance to TS2Vec on ETTh1, ETTh2, and Electricity (within a standard deviation), and slightly better performance on ETTm1. In multivariate forecasting, our model achieves comparable performance to TS2Vec on ETTh1 and ETTm1, performs worse on ETTh2, and performs better on Electricity.
4.4. Ablations
We next analyze the impact of similarity distillation in temporal and instance dimensions.
Table 4 presents an ablation study of our approach on all downstream tasks, showing the performance of the temporal loss, instance loss
[Table]
and the full model with both losses.
We also provide ablations on the hierarchical loss as proposed in TS2Vec.
On the classification and anomaly detection tasks, we find that the temporal and instance losses are both important, with the full model obtaining the best results.
On the forecasting task, the combined loss does not improve results.
Hierarchical contrast boosts the performance on the anomaly detection [Table] and forecasting tasks by a small margin.
We include additional sensitivity experiments on the queue size and temperature hyperparameters in the supplementary.
5. Conclusion
We introduced a similarity distillation based self-supervised method for pre-training universal time series representations.
In future work [Table] we plan to broaden our exploration into different self-supervised augmentation options [Table]
Supplementary Material A. Sensitivity Analyses
We performed sensitivity analyses on temperature and queue size to evaluate their impact on different tasks.
Tableshows a set of different temperatures for both student ( s τ ) and teacher ( t τ ) networks on the anomaly detection task.
Using different temperatures for student and teacher networks can improve the final performance for anomaly detection.
However, using the best temperature on anomaly detection does not improve the results on other tasks.
Finding a universal set of best hyperparameters for all tasks remains a challenge.
[Table]
We analyzed the impact of queue size on a subset of datasets.
Table 6 shows that increasing queue size can help with classification and anomaly detection while there seems to be a sweet spot.
On the other hand, it seems that queue size does not have a significant impact on forecasting.
We obtain modestly better results if we use a smaller queue size for forecasting.
[Table] [Table]
B. Detailed forecasting results on each horizon
Table 7 and Table 8 show detailed forecasting results on different horizons.
[Table] [Table]
ETTm  0.
[Table]
[Table]
[Table] [Table]
ETTm  0.
[Table] [Table] [Table]
ETTm  0. [Table] [Table] ETTm  0.
[Table]
[Table]
[Table]
ETTh
[Table]
[Table]
[Table]
[Table]
[Table]
ETTh [Table] [Table] [Table]
[Table]
[Table] [Table] ETTh  0. [Table]
[Table]
ETTh  0. [Table] [Table] ETTh  1.
[Table]
[Table] ETTh  0. [Table] [Table]
ETTh
[Table]
[Table]
[Table]
ETTh  2.293 ± 0.
1.187 ± 0.
ETTh  2.573 ± 0.
1.305 ± 0.
ETTh  2.823 ± 0.
1.439 ± 0. Electricity 0.266 ± 0. 0.359 ± 0. Electricity 0.286 ± 0.
0.375 ± 0.
Electricity 0.315 ± 0.
0.396 ± 0.
Electricity 0.332 ± 0. 0.409 ± 0. Electricity 0.359 ± 0. 0.428 ± 0.