I NFORMATION -A WARE T IME S ERIES M ETA -C ONTRASTIVE L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Various contrastive learning approaches have been proposed in recent years and achieve signiﬁcant empirical success.
While effective and prevalent, contrastive learning has been less explored for time series data.
A key component of contrastive learning is to select appropriate augmentations imposing some priors to construct feasible positive samples, such that an encoder can be trained to learn robust and discriminative representations.
Unlike image and language domains where “desired” augmented samples can be generated with the rule of thumb guided by prefabricated human priors, the ad-hoc manual selection of time series augmentations is hindered by their diverse and human-unrecognizable temporal structures.
How to ﬁnd the desired augmentations of time series data that are meaningful for given contrastive learning tasks and datasets remains an open question.
In this work, we address the problem by encouraging both high ﬁdelity and variety based upon information theory.
A theoretical analysis leads to the criteria for selecting feasible data augmentations.
On top of that, we employ the meta-learning mechanism and propose an information-aware approach, InfoTS, that adaptively selects optimal time series augmentations for contrastive representation learning.
The meta-learner and the encoder are jointly optimized in an endto-end manner to avoid sub-optimal solutions.
Experiments on various datasets show highly competitive performance with up to 11.4% reduction in MSE on forecasting task and up to 2.8% relative improvement in accuracy on classiﬁcation task over the leading baselines.
I NTRODUCTION
Time series data in the real world is high dimensional, unstructured, and complex with unique properties, leading to challenges for data modeling (Yang & Wu, 2006).
In addition, without human recognizable patterns, it is much harder to label time series data than images and languages in realworld applications.
These labeling limitations hinder deep learning methods, which typically require a huge amount of labeled data for training, been applied on time series data (Eldele et al., 2021).
Representation learning learns a ﬁxed-dimension embedding from the original time series that keeps their inherent features.
Comparing to the raw time series data, these representations are with better transferability and generalization capacity.
To deal with labeling limitations, contrastive learning methods have been widely adopted in various domains for their soaring performance on representation learning, including vision, language, and graph-structured data (Chen et al., 2020; Xie et al., 2019; You et al., 2020).
In a nutshell, contrastive learning methods typically train an encoder to map instances to an embedding space where dissimilar (negative) instances are easily distinguishable from similar (positive) ones and model predictions to be invariant to small noise applied to either input examples or hidden states.
Despite being effective and prevalent, contrastive learning has been less explored in the time series domain (Eldele et al., 2021; Franceschi et al., 2019; Fan et al., 2020; Tonekaboni et al., 2021).
Existing contrastive learning approaches often involve a speciﬁc data augmentation strategy that creates novel and realistic-looking training data without changing its label to construct positive alternatives for any input sample.
Their success relies on carefully designed rules of thumb guided by domain expertise.
Routinely used data augmentations for contrastive learning are mainly designed for image and language data, such as color distortion, ﬂip, word replacement, and back-translation (Chen et al., 2020; Luo et al., 2021).
These augmentation techniques generally do not apply to time series data.
Recently, some researchers propose augmentations for time series to enhance the size and quality of the training data (Wen et al., 2021).
For example, Eldele et al.
(2021) and Fan et al.
(2020) propose to adopt jittering, scaling, and permutation strategies to generate augmented instances.
Franceschi et al.
(2019) extracts subsequences for data augmentation.
In spite of the current progress, existing methods have two main limitations.
First, unlike images with human recognizable features, time series data are often associated with inexplicable underlying patterns.
Strong augmentation such as permutation may ruin such patterns and consequently, the model will mistake the negative handcrafts for positive ones.
While weak augmentation methods such as jittering may generate augmented instances that are too similar to the raw inputs to be informative enough for contrastive learning.
On the other hand, time series datasets from different domains may have diverse nature.
Adapting a universal data augmentation method, such as subsequence in Xie et al.
(2019), for all datasets and tasks leads to sub-optimal performances.
Other works follow empirical rules to select suitable augmentations from expensive trial-and-error.
Akin to hand-crafting features, hand-picking choices of data augmentations are undesirable from the learning perspective.
The diversity and heterogeneity of real-life time series data further hinder these methods away from wide applicability.
To address the challenges, we ﬁrst introduce the criteria for selecting good data augmentations in contrastive learning.
Data augmentation beneﬁts generalizable, transferable, and robust representation learning by correctly extrapolating the input training space to a larger region (Wilk et al., 2018).
The positive instances enclose a discriminative zone in which all the data points should be similar to the original instance.
The desired data augmentations for contrastive representation learning should have both high ﬁdelity and high variety .
High ﬁdelity encourages the augmented data to maintain the semantic identity that is invariant to transformations (Wilk et al., 2018).
For example, if the downstream task is classiﬁcation, then the generated augmentations of inputs should be class-preserving.
Meanwhile, generating augmented samples with high variety beneﬁts representation learning by increasing the generalization capacity (Chen et al., 2020).
From the motivation, we theoretically analyze the information ﬂows in data augmentations based upon information theory and derive the criteria for selecting desired time series augmentations.
Due to the inexplicability in practical time series data, we assume that the semantic identity is presented by the target in the downstream task.
Thus, high ﬁdelity can be achieved by maximizing the mutual information between the downstream label and the augmented data.
A one-hot pseudo label is assigned to each instance in the unsupervised setting when downstream labels are unavailable.
These pseudo labels encourage augmentations of different instances to be distinguishable from each other.
We show that data augmentations preserving these pseudo labels can add new information without decreasing the ﬁdelity.
Concurrently, we maximize the entropy of augmented data conditional on the original instances to increase the variety of data augmentations.
Based on the derived criteria, we propose an adaptive data augmentation method, InfoTS (as shown in Figure 1), by employing a meta-learning mechanism to avoid ad-hoc choices or painstakingly trial-and-error tuning.
Speciﬁcally, we utilize a meta-network to learn the augmentation prior in tandem with contrastive learning.
The meta-learner automatically selects optimal augmentations from candidate augmentations to generate feasible positive samples.
Along with random sampled negative instances, augmented instances are then fed into a time series encoder to learn representations in a contrastive manner.
With a reparameterization trick, the meta-network can be efﬁciently optimized with back-propagation based upon the proposed criteria.
Therefore, the meta-network can automatically select data augmentations in a per dataset and per learning task manner without resorting to expert knowledge or tedious downstream validation.
Our main contributions include:
• We propose criteria to guide the selection of data augmentations for contrastive time series representation learning without prefabricated knowledge.
• We propose a meta-learning based method to automatically select feasible data augmentations for different time series datasets, which can be efﬁciently optimized with backpropagation.
• We empirically verify the effectiveness of the proposed criteria to ﬁnd optimal data augmentations.
Extensive experiments demonstrate that InfoTS can achieve highly competitive performance with up to 11.4% reduction in MSE on forecasting task and up to 2.8% relative improvement in accuracy on classiﬁcation task over the leading baselines.
M ETHODOLOGY 2. N OTATIONS AND P ROBLEM D EFINITION
A time series instance x has dimension T × F , where T is the length of sequence and F is the dimension of features.
Given a set of time series instances X , we aim to learn an encoder f θ ( x ) that maps each instance x to a ﬁxed-length vector z ∈ R D , where θ is the learnable parameters of the encoder network and D is the dimension of representation vectors.
In semi-supervised settings, each instance x in the labelled set X L ⊆ X is associated with a label y for the downstream task.
Specially, X L = X holds in the fully supervised setting.
In the work, we use the Sans-serif style lowercase letters, such as x , to denote random time series variables and italic lowercase letters, such as x , for sampled instances.
2. W HAT ARE GOOD AUGMENTATIONS FOR CONTRASTIVE LEARNING ?
The goal of data augmentation for contrastive learning is to create realistically rational instances that maintain semantics through different transformation approaches.
Unlike instances in vision and language domains, the underlying semantics of time series data is not recognizable to human, making it hard, if not impossible, to include human knowledge to data augmentation for time series data.
For example, rotating an image will not change its content or the label.
While permuting a time series instance may ruin its signal patterns and generates a meaningless time series instance.
In addition, the tremendous heterogeneity of real-life time series datasets further makes selections based on trial-and-errors impractical.
Although multiple data augmentation methods have been proposed for time series data, there is less discussion on what is a good augmentation that is meaningful for a given learning task and dataset without prefabricated human priors.
From our perspective, ideal data augmentations for contrastive representation should keep high ﬁdelity, high variety, and adaptive to different datasets.
The illustration and examples are shown in Figure 2.
High Fidelity.
Augmentations with high ﬁdelity maintain the semantic identity that is invariant to transformations.
Considering the inexplicability in practical time series data, it is challenging to visually check the ﬁdelity of augmentations.
Thus, we assume that the semantic identity of a time series instance is presented by its label in the downstream task, which might be either available or unavailable during the training period.
Here, we start our analysis from the supervised case and will extend it to the unsupervised case later.
Based on the information theory (Tishby et al., 2000), we deﬁne the objective that keeps high ﬁdelity as the large mutual information (MI) between augmentation v and the label y , i.e., MI ( v ; y ) .
We consider augmentation v as a probabilistic function of x and a random variable (cid:15) , that v = g
( x ; (cid:15) ) .
From the deﬁnition of mutual information, we have MI ( v ; y )
=
H ( y ) − H
( y | v ) , where H ( y ) is the (Shannon) entropy of y and H ( y | v ) is the entropy of y conditioned on augmentation v .
Since H ( y ) is irrelevant to data augmentations, the objective is equivalent to minimizing the conditional entropy
H ( y | v ) .
Considering the efﬁcient optimization, we follow Ying et al.
and Luo et al.
(2020) to approximate it with cross-entropy between y and ˆ y , where ˆ y is the prediction with augmentation v as the input and calculated via
[Equation]
where z is the representation and h w ( · ) is a prediction projector parameterized by w .
The prediction projector is optimized by the classiﬁcation objective.
Then, the objective of high ﬁdelity for
[Figure]
supervised or semi-supervised cases is to minimize
[Equation]
where C is the number of labels.
In the unsupervised settings where y is unavailable, one-hot encoding y s ∈ R | X | is utilized as the pseudo label to replace y in Eq.
The motivation is that augmented instances are still distinguishable from other instances with the classiﬁer.
We theoretically show that augmentations that preserving pseudo labels have the following properties.
Property 1 (Preserving Fidelity).
If augmentation v preserves the one-hot encoding pseudo label, the mutual information between v and the downstream task label y (although not visible to training) is equivalent to that between raw input x and y , i.e., MI ( v ; y )
=
MI ( x ; y ) .
Property 2 (Adding New Information).
By preserving the one-hot encoding pseudo label, augmentation v contains new information comparing to the raw input x , i.e., H ( v ) ≥ H ( x ) .
Detailed proofs are shown in the Appendix A. These properties show that in the unsupervised setting, preserving the one-hot encoding pseudo label guarantees that the generated augmentations will not decrease the ﬁdelity, regardless of the downstream tasks and variances inherent in the augmentations.
Concurrently, it may introduce new information for contrastive learning.
Since the number of labels is equal to the number of instances in dataset X in an unsupervised case, direct optimization of Eq.
(2) is inefﬁcient and unscalable.
Thus, we further relax it by approximating y with the batch-wise one-hot encoding y B , which decreases the number of labels C from the dataset size to the batch size.
High Variety.
Sufﬁcient variances in augmentations improve the generalization capacity of contrastive learning models.
In the information theory, the uncertainty inherent in the random variable’s possible outcomes is described by its entropy.
Considering that augmented instances are generated based on the raw input x , we maximize the entropy of v conditioned on x , H ( v | x ) , to maintain a high variety of augmentations.
From the deﬁnition of conditional entropy, we have
[Equation]
We dismiss the ﬁrst part since the unconstrained entropy of v can be dominated by meaningless noise.
Considering the continuity of both v and x , we adopt a mutual information neural estimator,
InfoNCE (Hjelm et al., 2018; Oord et al., 2018) to approximately compute the mutual information.
Other MI estimators, such as Jensen-Shannon (JSD) estimator (Nowozin et al., 2016), normalized temperature-scaled cross-entropy (NT-Xent)
(Chen et al., 2020), and Donsker-Varadhan (DV) representation of the KL-divergence (Donsker & Varadhan, 1975) can also conveniently be the plug-andplay component in our framework.
Then, the objective to encourage high variety is to maximize the
InfoNCE between v and x :
[Equation]
where v (cid:48) is an augmented instance of input instance x (cid:48) .
z x , z v , and z v (cid:48) are representations of instance x , v , and v (cid:48) respectively.
sim ( z 1 , z 2 ) =
z T 1 z 2 is the inner product of vectors z 1 and z 2 .
Criteria.
Combining the information aware deﬁnition of both high ﬁdelity and variety, we propose the criteria for selecting good augmentations without prior knowledge,
[Equation]
where β is a hyper-parameter to achieve the trade-off between ﬁdelity and variety.
Note that in the unsupervised settings, y is replaced by one-hot encoding pseudo label..
Relation to Information Bottleneck.
Although the formation is similar to information bottleneck in data compression, min p ( e | x ) MI ( x ; e ) − β MI ( e ; y ) , our criteria are different in the following aspects.
First, e in the information bottleneck is a representation of input x , while v in Eq.(5) represents the augmented instances.
Second, information bottleneck aims to keep minimal and sufﬁcient information for data compression, while our criteria are designed for data augmentations in contrastive learning.
Third, in information bottleneck, the compressed representation e is a deterministic function of input x with no variances.
MI ( e ; y ) and MI ( e ; x ) are constraint by MI ( x ; y ) and H ( x )
that
MI ( e ; y ) ≤ MI ( x ; y ) and MI ( e ; x ) =
H ( e ) , where H ( e ) is the entropy of e .
In our criteria, v is a probabilistic function of input x .
As a result, the variances of v makes the augmentation space much larger than the compression representation space in information bottleneck.
Relation to InfoMin.
InfoMin is designed based on the information bottleneck that good views should keep minimal and sufﬁcient information from the original input (Tian et al., 2020).
Similar to the information bottleneck, InfoMin assumes that augmented views are functions of the input, which heavily constrains the variance of data augmentations.
Besides, high ﬁdelity property is dismissed in the unsupervised setting.
It works for image datasets due to the availability of human knowledge.
However, it may fail to generate reasonable augmentations for time series data.
2. T IME S ERIES M ETA -C ONTRASTIVE L EARNING
We aim to design a learnable augmentation selector that learns to select feasible augmentations in a data-driven manner.
With such adaptive data augmentations, the contrastive loss is then used to train the encoder that learns representations from raw time series.
2.3. A RCHITECTURE
The adopted encoder f θ ( x ) : R T × F → R D consists of two components, a fully connected layer, and a 10-layer dilated CNN module (Franceschi et al., 2019; Yue et al., 2021).
To explore the inherent structure of time series, we include both global-wise (instance-level) and local-wise (subsequencelevel) losses in the contrastive learning framework to train the encoder.
Global-wise contrastive loss is designed to capture the instance level relations in a time series dataset.
Formally, given a batch of time series instances X B ⊆ X , for each instance x ∈ X B , we generate an augmented instance v with an adaptively selected transformation, which will be introduced later.
( x, v ) is regarded as a positive pair and other ( B − 1) combinations { ( x, v (cid:48) ) } , where v (cid:48) is an augmented instance of x (cid:48) and x (cid:48) (cid:54) =
x , are considered as negative pairs.
Following Chen et al.
; You et al.
(2020), we design the global-wise contrastive loss based on InfoNCE (Hjelm et al., 2018).
The batch-wise instance-level contrastive loss is
[Equation]
Local-wise contrastive loss is proposed to explore the intra-temporal relations in time series.
For an augmented instance v of a time series instance x , we ﬁrst split it into a set of subsequences S , each with length L .
For each subsequence s ∈ S , we follow Tonekaboni et al.
(2021) to generate a positive pair ( s, p ) by selecting another subsequence close to it.
Non-neighboring samples, ¯ N s , are adopted to generate negative pairs.
Detailed descriptions can be found in Appendix B.
Then, the local-wise contrastive loss for an instance x is:
[Equation]
Across all instances in a batch, we have L c = 1 | X B | (cid:80) x ∈ X B L c x .
Then, the ﬁnal contrastive objective is:
[Equation]
where α is a hyper-parameter to achieve the trade-off between global and local contrastive losses.
2.3. M ETA - LEARNER N ETWORK
Previous time series contrastive learning methods (Franceschi et al., 2019; Fan et al., 2020; Eldele et al., 2021;
Tonekaboni et al., 2021) generate augmentations with either rule of thumb guided by prefabricated human priors or tedious trial-and-errors, which are designed for speciﬁc datasets and learning tasks.
In this part, we discuss how to adaptively select the optimal augmentations with a meta-learner network based on the proposed information-aware criteria.
We can regard its choice of optimal augmentation as a kind of prior selection.
We ﬁrst choose a set of candidate transformations T , such as jittering and time warping.
Each candidate transformation t i ∈ T is associated with an important weight p i ∈ (0 , 1) , inferring the probability of selecting transformation t i .
For an instance x , the augmented instance v i through transformation t i can be computed by:
[Equation]
Considering multiple transformations, we pad all v i to be with the same length.
Then, the adaptive augmented instance can be achieved by combing candidate ones, v = 1 | T | (cid:80) i v i .
To enable the efﬁcient optimization with gradient-based methods, we approximate discrete Bernoulli processes with binary concrete distributions (Maddison et al., 2016).
Speciﬁcally, we approximate a i in Eq.
(9) with
[Equation]
, (10) where σ ( · ) is the sigmoid function and τ is the temperature controlling the approximation.
The rationality of such approximation is given in Appendix A. Moreover, with temperature τ > 0 , the gradient ∂v∂p i is well-deﬁned.
Therefore, our meta-network is end-to-end differentiable.
Detailed algorithm is shown in Appendix B.
R ELATED W ORK 3. C ONTRASTIVE T IME S ERIES R EPRESENTATION L EARNING
Contrastive learning has been utilized widely in representation learning with superior performances in various domains (Chen et al., 2020; Xie et al., 2019;
You et al., 2020).
Recently, some efforts have been devoted to applying contrastive learning to the time series domain (Oord et al., 2018;
Franceschi et al., 2019; Fan et al., 2020; Eldele et al., 2021;
Tonekaboni et al., 2021;
Yue et al., 2021).
Time Contrastive Learning trains a feature extractor with a multinomial logistic regression classiﬁer to discriminate all segments in a time series (Hyvarinen & Morioka, 2016).
Franceschi et al.
(2019) generates positive and negative pairs based on subsequences.
TNC employs a debiased contrastive objective to ensure that in the representation space, signals in the local neighborhood are distinguishable from non-neighboring signals (Tonekaboni et al., 2021).
SelfTime adopts multiple hand-crafted augmentations for unsupervised time series contrastive learning by exploring both inter-sample and intra-sample relations (Fan et al., 2020).
TS2Vec learns a representation for each time stamp and conducts contrastive learning in a hierarchical way (Yue et al., 2021).
However, data augmentations in these methods are either universal or selected by error-and-trail, hindering them away from been widely applied in complex real-life datasets.
3. T IME S ERIES F ORECASTING
Forecasting is a critical task in time series analysis.
Deep learning architectures used in the literature include Recurrent Neural Networks (RNNs) (Salinas et al., 2020; Oreshkin et al., 2019), Convolutional Neural Networks (CNNs) (Bai et al., 2018), Transformers (Li et al., 2019; Zhou et al., 2021), and Graph Neural Networks (GNNs) (Cao et al., 2021).
N-BEATS deeply stacks fully-connected layers with backward and forward residual links for univariate times series forecasting (Oreshkin et al., 2019).
TCN utilizes a deep CNN architecture with dilated causal convolutions (Bai et al., 2018).
Considering both long-term dependencies and short-term trends in multivariate time series,
LSTnet combines both CNNs and RNNS in a uniﬁed model (Lai et al., 2018).
LogTrans brings the
Transformer model to time series forecasting with causal convolution in its attention mechanism Li et al.
Informer further proposes a sparse self-attention mechanism to reduce the time complexity and memory usage (Zhou et al., 2021).
StemGNN is a GNN based model that considers the intra-temporal and inter-series correlations simultaneously Cao et al.
Unlike these works, we aim to learn general representations for time series data that can not only be used for forecasting but also other tasks, such as classiﬁcation.
Besides, the proposed framework is compatible with various architectures as encoders.
3. A DAPTIVE DATA AUGMENTATION
Data augmentation is an important component in contrastive learning.
Existing researches reveal that the choices of optimal augmentation are dependent on downstream tasks and datasets (Chen et al., 2020; Fan et al., 2020).
Some researchers have explored adaptive selections of optimal augmentations for contrastive learning in the vision ﬁeld.
AutoAugment automatically searches the combination of translation policies via a reinforcement learning method (Cubuk et al., 2019).
Faster-AA improves the searching pipeline for data augmentation using a differentiable policy network (Hataya et al., 2020).
DADA further introduces an unbiased gradient estimator for an efﬁcient one-pass optimization strategy (Li et al., 2020).
Within contrastive learning frameworks, Tian et al.
(2020) applies the Information Bottleneck theory that optimal views should share minimal and sufﬁcient information, to guide the selection of good views for contrastive learning in the vision domain.
Considering the inexplicability of time series data, directly applying the InfoMin framework may keep insufﬁcient information during augmentation.
Different from Tian et al.
(2020), we focus on the time series domain and propose an end-to-end differentiable method to automatically select the optimal augmentations for each dataset.
E XPERIMENTS
In this section, we compare InfoTS with SOTA baselines on time series forecasting and classiﬁcation tasks.
We also conduct case studies to show insights into the proposed criteria and meta-learning framework.
Detailed experimental setups are shown in Appendix C. Full experimental results and extra experiments, such as parameter sensitivity studies, are presented in Appendix D.
T IME S ERIES F ORECASTING
Time series forecasting aims to predict the future L y time stamps, with the last L x observations.
We follow Yue et al.
(2021) to train a linear model regularized with the L2 norm penalty to make predictions.
The output has dimension L y in the univariate case and L y × F for the multivariate case, where F is the feature dimension.
Datasets and Baselines.
Four benchmark datasets for time series forecasting are adopted, including ETTh1, ETTh2, ETTm1 (Zhou et al., 2021), and the Electricity dataset (Dua & Graff, 2017).
These datasets are used in both univariate and multivariate settings.
We compare unsupervised InfoTS to the SOTA baselines, including TS2Vec (Yue et al., 2021), Informer (Zhou et al., 2021),
StemGNN (Cao et al., 2021), TCN (Bai et al., 2018), LogTrans (Li et al., 2019), LSTnet (Lai et al., 2018), and N-BEATS (Oreshkin et al., 2019).
Among these methods, N-BEATS is merely designed for the univariate and StemGNN is for multivariate only.
We refer to Yue et al.
(2021) to set up baselines for a fair comparison.
Standard metrics for a regression problem, Mean Squared Error (MSE), and Mean Absolute Error (MAE) are utilized for evaluation.
Evaluation results of univariate time series forecasting are shown in Table 1, while multivariate forecasting results are reported in the Appendix D.4 due to the space limitation.
Performance.
As shown in Tabel 1 and Tabel 4, comparison in both univariate and multivariate settings indicates that InfoTS consistently matches or outperforms the leading baselines.
Some results of StemGNN are unavailable due to the out-of-memory issue (Yue et al., 2021).
Speciﬁcally, we have the following observations.
TS2Vec, another contrastive learning method with data augmentations, achieves the second-best performance in most cases.
The consistent improvement of TS2Vec
[Table]
[Table]
over other baselines indicates the effectiveness of contrastive learning for time series representations learning.
However, such universal data augmentations may not be the most informative ones to generate positive pairs.
Comparing to TS2Vec, InfoTS decreases the average MSE by 11.4%, and the average MAE by 8.6% in the univariate setting.
In the multivariate setting, the MSE and MAE decrease by 4.6% and 2.3%, respectively.
The reason is that InfoTS can adaptively select the most suitable augmentations in a data-driven manner with high variety and high ﬁdelity.
Encoders trained with such informative augmentations learn representations with higher quality.
4. T IME S ERIES C LASSIFICATION
Following the previous setting, we evaluate the quality of representations on time series classiﬁcation in a standard supervised manner (Franceschi et al., 2019;
Yue et al., 2021).
We train an SVM classiﬁer with a radial basis function kernel on top of representations in the training split and then compare the prediction in the test set.
Datasets and Baselines.
Two types of benchmark datasets are used for evaluations.
The UCR archive (Dau et al., 2019) is a collection of 128 univariate time series datasets and the UEA archive (Bredin, 2017) consists of 30 multivariate datasets.
We compare InfoTS with baselines including TS2Vec (Yue et al., 2021), T-Loss (Franceschi et al., 2019), TS-TCC (Eldele et al., 2021),
TST (Zerveas et al., 2021), and DTW (Franceschi et al., 2019).
For our methods, InfoTS s , training labels are only used to train the meta-network to select suitable augmentations, and InfoTS is with a purely unsupervised setting for representation learning.
Performance.
The results on the UEA datasets are summarized in Table 2.
Full results are provided in Appendix D.4.
With the ground-truth label guiding the meta-network, InfoTS s substantially outperforms other baselines.
On average, it improves the classiﬁcation accuracy by 2.8% over the best baseline, TS2Vec, with an average rank value 2.133 on all 30 UEA datasets.
Under the purely unsupervised setting, InfoTS preserves ﬁdelity by adopting one-hot encoding as the pseudo labels.
InfoTS achieves the second best average performance in Table 2, with an average rank value 2.633.
Performances on the 128 UCR datasets are shown in Table 9 in Appendix D.4.
These datasets are univariate with easily recognized patterns, where data augmentations have marginal or even negative effects (Yue et al., 2021).
However, with adaptively selected augmentations for each dataset based on our criteria, InfoTS s and InfoTS still outperform the state-of-the-arts.
4. E VALUATION OF T HE C RITERIA
In Section 2.2, we propose criteria of data augmentations for time series based on information theory that good augmentations should have high variety and ﬁdelity.
With mutual information neural
[Figure]
[Table]
estimation and cross-entropy as approximations, we get the criteria in Eq.
To empirically verify the effectiveness of the proposed criteria, we adopt the dataset CricketX from the UCR archive (Dau et al., 2019; Fan et al., 2020)
[Table]
and conduct augmentations with different conﬁgurations, such as jittering with noise from N (0 , 0 . 3) and cutout 10% time stamps.
For each conﬁguration, we calculate the criteria score and the corresponding classiﬁcation accuracy within the setting in Section 4.2.
As shown in Figure 3, in general, accuracy performance is positively related to the proposed criteria in both supervised and unsupervised settings, verifying the correctness of using the criteria as the objective in the meta-network training.
4. A BLATION S TUDIES
To present deep insights into the proposed method, we verify key components in InfoTS with multiple ablation studies on the Electricity dataset.
Results are shown in Table 3. 1)
We verify the effectiveness of the adopted contrastive learning objective, Eq.
(8), by comparing InfoTS with its two variants.
“w/o Local” removes the local contrastive loss between subsequences and “w/o
Global” removes the instance-level contrastive loss.
The comparison between these variants and
InfoTS demonstrates that both instance-level and intra-temporal objectives are important in contrastive learning for time series data.
To demonstrate the advantage of adaptive selection of augmentations, we compare InfoTS with variants “Random” and “All”.
“Random” randomly selects an augmentation from candidate transformation functions each time and “All” sequentially applies transformations to generate augmented instances.
Performance decreases are observed in these variances, verifying the key role of adaptive selection in our method.
To show the effects of variety and ﬁdelity objectives in meta-network training, we include two variants, “w/o Fidelity” and “w/o Variety”, which dismiss the ﬁdelity or variety objective, respectively.
The comparison between
InfoTS and the variants empirically conﬁrms both variety and ﬁdelity are important for data augmentation in contrastive learning.
Note that the adopted basic augmentations are manually tuned in a previous paper (Fan et al., 2020) with high qualities for contrastive learning.
Including low quality basic augmentations, we conduct more ablation studies are shown in Appendix D.3.
CONCLUSIONS
We propose criteria of data augmentations for time series data based on the information theory that good augmentations should preserve high variety and high ﬁdelity.
We approximate the criteria with mutual information neural estimation and cross-entropy estimation.
Based on the approximated criteria, we present a meta-learning method to adaptively select optimal augmentations for contrastive representation learning.
Comprehensive experiments show that representations produced by our method are high qualiﬁed and easy to use in various downstream tasks, such as time series forecasting and classiﬁcation, with state-of-the-art performances.
S TATEMENTS
Reproducibility Statement.
The training algorithm of the proposed InfoTS is shown in Appendix B. Detailed experimental settings, including architectures and hyper-parameters settings, are given in Appendix C. Core source codes with an example of usages are attached in the supplementary material.
Full version will be made available upon acceptance.
A D ETAILED P ROOFS Properties of data augmentations that preserve pseudo labels.
We assume that v i and v j are two augmented instances of inputs x i and x j , respectively.
Preserving pseudo labels deﬁned in one-hot encoding requires that the map between variable x
and v is one-tomany.
Formally, x i (cid:54) =
x j → v i (cid:54) = v j .
This can be proved by contradiction.
If we have a pair of ( i, j ) that i (cid:54) =
j and v i = v j , then we have f θ ( v i ) =
f θ ( v j ) , showing that augmentations cannot preserve pseudo labels.
Property 1 (Preserving Fidelity).
If augmentation v preserves the one-hot encoding pseudo label, the mutual information between v and downstream task label y (although not visible to training) is equivalent to that between raw input x and y , i.e., MI ( v ; y )
=
MI ( x ; y ) .
Proof.
From the deﬁnition of mutual information, we have
[Equation]
where V ( x ) is the set of augmented instances of a time series instance x .
In the unsupervised setting where the ground-truth label y is unknown, we assumes that the augmentation v is a (probabilistic) function of x only.
The only qualiﬁer means p ( v | x, y ) =
p ( v | x ) .
Since the mapping from x to v is one-to-many.
For each v ∈ V ( x ) we have p ( v, y ) =
p ( v, x, y ) and p ( v ) = p ( v | x ) p ( x ) .
Thus, we have
[Equation]
Property 2 (Adding New Information).
By preserving the one-hot encoding pseudo label, augmentation v contains new information comparing to the raw input x , i.e., H ( v ) ≥ H ( x ) .
In information theory, entropy describes the amount of information of a random variable.
For simplicity, we assume a ﬁnite number of augmented instances for each input, and each augmented instance is generated independently.
Then, we have p ( x ) =
(cid:80) v ∈ V ( x ) p ( v ) .
Then we have that the entropy of variable x is no larger than the entropy of v .
[Equation]
In the binary concrete distribution, parameter τ controls the temperature that achieves the trade-off between binary output and continuous optimization.
When τ → 0 , we have lim τ → 0 P ( a i = 1) =
p i , which is equivalent to the Bernoulli distribution.
Proof.
[Equation]
Since (cid:15) , and p i are both in (0 , 1) , and function x 1 − x are monotonically increasing in this region.
Thus, we have
[Equation]
B A LGORITHMS B. T RAINING A LGORITHM
The training algorithm of InfoTS under both supervised and unsupervised settings is described in
Algorithm 1.
We ﬁrst randomly initiate parameters in the encoder, meta-network, and classiﬁer (line 2).
Given a batch of training instances X B ⊆ X , for each candidate transformation t i , we utilize binary concrete distribution to get parameters a i (lines 6-7), which indicates whether the transformation should be applied (line 8).
V ( i ) B denotes the batch of augmented instances generated from transformation function t i .
The ﬁnal augmented instances are generated by adaptively considering all candidates transformations (line 10).
Parameters θ in the encoder are updated by minimizing the contrastive objective (lines 11-13).
Meta-network is then optimized with information-aware criteria (lines 14-16).
Then, classiﬁer h w is optimized with the classiﬁcation objective (line 17).
I MPLEMENTATION OF LOCAL - WISE CONTRASTIVE
Local-wise contrastive loss aims to capture the intra-temporal relations in each time series instance.
For an augmented instance v , we ﬁrst split it into multiple subsequences, as shown in Figure 4.
Each subsequence has length L .
For each subsequence s , the neighboring subsequences within window size 1 are considered as positive samples.
If s locates at the end of v , then we choose the subsequence in front of s as the positive pair p .
Otherwise, we choose the subsequence following s instead.
Subsequences out of window size 1 are considered as negative samples.
C E XPERIMENTAL S ETTINGS
C.
D ATA A UGMENTATIONS
We follow Fan et al.
(2020) to set up candidate data augmentations, including jittering, scaling, cutout, time warping, window slicing, window warping and subsequence augmentation (Franceschi et al., 2019).
Detailed descriptions are listed as follows.
• Jittering augmentation adds the random noise sampled from a Gaussian distribution N (0 , 0 . 3) to the input time series.
• Scaling augmentation multiplies the input time series by a scaling factor sampled from a Gaussian distribution N (0 , 0 . 5) .
• Cutout operation replaces features of 10% randomly sampled time stamps of the input with zeros.
• Time warping random changes the speed of the timeline 1 .
The number of speed changes is 100 and the maximal ratio of max/min speed is 10.
If necessary, over-sampling or sampling methods are adopted to ensure the length of the augmented instance is the same as the original one.
• Window slicing randomly crops half the input time series and then linearly interpolates it back to the original length (Le Guennec et al., 2016).
• Window warping ﬁrst randomly selects 30% of the input time series along the timeline and then warps the time dimension by 0.5 or 2.
Finally, we adopt linear interpolation to transform it back to the original length (Le Guennec et al., 2016).
• Subsequence operation random selects a subsequence from the input time series (Yue et al., 2021).
With the ﬁrst 100 time stamps in the univariate Electricity dataset as an example, we visualize the original time series and the augmented ones in Figure 5.
C. H ARDWARE AND IMPLEMENTATIONS
All experiments are conducted on a Linux machine with 4 NVIDIA GeForce RTX 2080 Ti GPUs, each with 11GB memory.
CUDA version is 10.1 and Driver Version is 418.56.
Our method InfoTS is implemented with Python 3.7.7 and Pytorch 1.7.1.
[Figure]
C. H YPERPARAMETERS
We train and evaluate our methods with the following hyperparameters and conﬁgurations.
• Optimizer: Adam optimizer (Kingma & Ba, 2014) with learning rate and decay rates setting to 0.001 and (0.9,0.999), respectively.
• SVM: scikit-learn implementation (Pedregosa et al., 2011) with penalty C ∈ { 10 i | i ∈
[ − 4 , 4] ∪ ∞} Franceschi et al.
• Encoder architecture:
We follow Yue et al.
(2021) to design the encoder.
Speciﬁcally, the output dimension of the linear projection layer is set to 64, the same for the number of channels in the following dilated CNN module.
In the CNN module, GELU (Hendrycks & Gimpel, 2016) is adopted as the activation function, and the kernel size is set to 3.
The dilation is set to 2 i in the i -the block.
• Classiﬁer architecture: a fully connected layer that maps the representations to the label is adopted.
• Trade-off hyperparameters: β in Eq.
(5) and α in Eq.
(8) are searched in [0 . 1 , 0 . 5 , 1 . 0 , 5 , 10] .
Parameter sensitivity studies are shown in Section D.1.
• Temperature in binary concrete distribution: we follow the practice in (Jang et al., 2016) to adopt the strategy by starting the training with a high temperature and anneal to a small value with a guided schedule.
D M ORE E XPERIMENTAL R ESULTS D. P ARAMETER S ENSITIVITY S TUDIES
In this part, we adopt the electricity dataset to analyze the effects of two important hyper-parameters in our method InfoTS.
The hyperparameter α in Eq.
(8) controls the trade-off between local and global contrastive losses when training the encoder.
β in Eq.
(5) achieves the balance between high variety and high ﬁdelity when training the meta-network.
We tune these parameters in range [0 . 1 , 0 . 5 , 1 . 0 , 5 , 10] and show the results in Figure 6.
These ﬁgures show that our method achieves high performance with a wide range of selections, demonstrating the robustness of the proposed method.
In general, setting trade-off parameters to 0.5 or 1 achieves good performance.
D. C ASE STUDY ON SIGNAL DETECTION
In this part, we show the potential usage of InfoTS to detect the informative signals in the time series.
We adopt the CricketX dataset as an example for the case study.
Subsequence augmentations
[Table]
[Figure]
on 0-100, 100-200, and 200-300 periods are adopted as candidate transformations.
We observe that the one operated on 100-200 period has high ﬁdelity and variety, leading to better accuracy performance, which is consistent with the visualization results in Figure 7.
D. M ORE ABLATION STUDIES D.3. U PDATING PROCESS OF I NFO TS
To show that our InfoTS can adaptively detect the most effective augmentation based on the data distribution, we follow the setting in Section 4.4 and conduct more ablation studies to investigate comprehensively into the proposed model.
We compare performances of variants that each applies a single transformation to generate augmented instances in Table 5.
From the table, we know that augmentation with subsequence beneﬁts the most for the Electricity dataset.
We visualize the weight
[Figure]
[Table]
updating process of InfoTS in Figure 8, with each line representing the normalized importance score of the corresponding transformation.
The weight for subsequence increase with the epoch, showing that InfoTS tends to adopt subsequence as the optimal transformation.
Consistency between accuracy performance and weight updating process demonstrates the effectiveness of InfoTS to adaptively select feasible transformations.
Besides, as shown in Table 5, InfoTS outperforms the variant that uses subsequence only.
This comparison shows that the meta-network learns to consider the combinations, which is better than any (single) candidate augmentation.
D.3. I NFO TS WITH C ONTROLLABLE AUGMENTATIONS .
We observe that in Table 3, the improvement of InfoTS over its two variants without Variety and
Fidelity objectives is not signiﬁcant.
The reason is that basic augmentations are adopted from a previous paper (Fan et al., 2020), which have been manually tuned to generate relatively good performances for contrastive learning.
To provide deep insights into the proposed criteria and model, in this part, we consider more augmentations with controllable hyper-parameters for the Electricity dataset.
Two groups of augmentations, subsequence augmentations with different lengths and jitter augmentations with different standard deviations, are considered in this part.
Subsequence augmentations work on the temporal dimension, and jitter augmentations work on the feature dimension.
For the subsequence augmentations, we range the ratio of subsequences r in the range [0 . 01 , 0 . 99] .
The subsequence augmentation with ratio r is denoted by Sub r , such as Sub 0 .
For the jitter augmentations, the standard deviations are chosen from the range [0 . 01 , 3 . 0] .
The jitter augmentation with standard deviation std is denoted by Jitter std , such as Jitter 0 .
Intuitively, with r increasing, Sub r generates augmented instances with lower variety and higher ﬁdelity.
For example, with r = 0 . 01 , Sub r generates subsequences that only keep 1% time stamps from the original input, leading to high variety but extremely low ﬁdelity.
Similarly, for jitter augmentations, with std increasing, Jitter std generates augmented instances with higher variety but lower ﬁdelity.
We show their forecasting performances in Table 6 and Table 7.
We observed that augmentations that only keeps high variety, such as Sub 0 .
01 , Jitter 3 . 0 achieve worse performances in terms of MSE and MAE compared to the ones considering both criteria, such as Sub 0 .
3 and Jitter 0 .
On the other hand, augmentations that only keep high ﬁdelity, such as Sub 0 .
99 , Jitter 0 .
01 , and Jitter
.cannot lead to satisfactory performances either.
Thus, we empirically verify the importance of both variety and ﬁdelity for the selection of optimal data augmentations for contrastive learning.
In Fig-
[Table] [Table]
ure 9, we show the relationship between forecasting performance and our proposed criteria.
The results are consistent with the conclusion drawn from classiﬁcation results that forecasting performance, evaluated with both MSE and MAE, is positively related to the proposed criteria.
[Figure]
With the proposed criteria, we empirically show the advantage of the developed meta-network on learning optimal augmentations.
The overall observations are consistent with ones from Section 4.4.
Here, we provide more details.
1) “Random” is a variant that randomly selects a candidate augmentation method to generate augmented instances.
“All” combines all candidate augmentations evenly to generate augmented instances.
These two variances are routinely utilized in previous works (Chen et al., 2020;
You et al., 2020).
They work well when candidate augmentations are manually veriﬁed.
Otherwise, their performances are heavily affected by the low-quality candidate augmentations, as shown in Tabel 8.
On the other hand, by adaptively selecting optimal augmentations, InfoTS signiﬁcantly improve the forecasting performances even with low-quality candidate augmentations.
By comparing InfoTS to its variant w/o Fidelity and w/o Variety, we further empirically demonstrate that both variety and ﬁdelity make contributions to the superior performances of InfoTS.
D. F ULL RESULTS OF TIME SERIES PREDICTION AND CLASSIFICATION
The full results of multivariate time series forecasting are shown in Tabel 4.
Results of StemGNN with L y = 720 are not available due to the out-of-memory error (Yue et al., 2021).
Full results of univariate time series classiﬁcation on 128 UCR datasets are shown in Tabel 9.
Results of T-Loss,
TS-TCC, and TNC are not reported on several datasets because they are not able to deal with missing observations in time series data.
These unavailable accuracy scores are dismissed when computing average accuracy and considered as 0 when calculating the average rank.
Results of multivariate
[Table]
classiﬁcation on 30 UEA datasets are listed in Tabel 10.
Computations of average accuracy scores and ranks follow the ones in 128 UCR datasets.
[Table]
CricketY
[Table]
DiatomSizeReduction
[Table]
DistalPhalanxOutlineCorrect
[Table]
Earthquakes
[Table]
ElectricDevices
[Table]
FaceAll
[Table]
0.808 FaceFour 0.
0.830 FacesUCR 0.
0.905 FiftyWords 0.
0.690 Fish 0.
0.920 FordA 0.
0.555 FordB 0.
0.620 GunPoint 1.
0.907 Ham 0.
0.467 HandOutlines 0.
0.881 Haptics 0.
0.377 Herring 0.
0.531 InlineSkate 0.
0.384 InsectWingbeatSound 0.
0.355 ItalyPowerDemand 0.
0.950 LargeKitchenAppliances 0.
0.795 Lightning 0.
0.869 Lightning 0.
0.726 Mallat 0.
0.934 Meat 0.
0.933 MedicalImages 0.
0.737 MiddlePhalanxOutlineCorrect 0.
0.698 MiddlePhalanxOutlineAgeGroup 0.
0.500 MiddlePhalanxTW 0.
0.506 MoteStrain 0.
0.835 NonInvasiveFetalECGThorax 0.
0.790 NonInvasiveFetalECGThorax 0.
0.865 OliveOil 0.
0.833 OSULeaf 0.
0.591 PhalangesOutlinesCorrect 0.
0.728 Phoneme 0.
0.228 Plane 1.
1.000 ProximalPhalanxOutlineCorrect 0.
0.784 ProximalPhalanxOutlineAgeGroup 0.
0.805 ProximalPhalanxTW 0.
0.761 RefrigerationDevices 0.
0.464 ScreenType 0.
[Table]
SonyAIBORobotSurface
[Table]
SonyAIBORobotSurface
[Table]
Strawberry
[Table]
SwedishLeaf
[Table]
Symbols
[Table]
SyntheticControl
[Table]
TwoLeadECG
[Table]
UWaveGestureLibraryY
[Table]
[Table]
GestureMidAirD
[Table]
GestureMidAirD
[Table]
GesturePebbleZ
[Table]
GesturePebbleZ
[Table]
MelbournePedestrian
[Table]
MixedShapesSmallTrain
[Table]
PickupGestureWiimoteZ
[Table]
[Table]
HandMovementDirection
[Table]
Handwriting
[Table]
Heartbeat
[Table]
JapaneseVowels
[Table]
MotorImagery
[Table]
PenDigits
[Table]
PhonemeSpectra
[Table]
RacketSports
[Table]
SelfRegulationSCP
[Table]