FEAT: A GENERAL FRAMEWORK FOR F EATURE - AWARE M ULTIVARIATE T IME - SERIES R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Multivariate time-series is complex and uncertain.
The overall temporal patterns change dynamically over time, and each feature is often observed to have a unique pattern.
Therefore, it is challenging to model a framework that can flexibly learn feature-specific unique patterns as well as dynamically changing temporal patterns simultaneously.
We propose a general framework for FEature-Aware multivariate
Time-series representation learning, called FEAT.
Unlike previous methods that only focus on training the overall temporal dependencies, we focus on training feature-specific as well as feature-agnostic representations in a data-driven manner.
Specifically, we introduce a feature-wise encoder to explicitly model the feature-specific information and design an element-wise gating layer that learns the influence of feature-specific patterns per dataset in general.
FEAT outperforms the benchmark models in average accuracy on 29 UEA multivariate time-series classification datasets and in MSE and MAE on four multivariate time-series forecasting datasets.
I NTRODUCTION
Deep learning methods have been applied to time-series data in various fields such as manufacturing, healthcare, finance, and transportation to enhance the performance of classification or forecasting tasks (Guo et al., 2019; Zhang et al., 2021; Essien & Giannetti, 2020).
Approaches related to multivariate time-series data have received considerable attention for leveraging the abundant information of features to improve the performance of downstream tasks (Rasul et al., 2020; Zhang et al., 2020).
However, effectively capturing the intrinsic complexity of multivariate time-series data is difficult (Duan et al., 2022; Chen et al., 2022; Liu et al., 2021; Karim et al., 2019), given the complex and uncertain characteristics of data in which features have varied changing patterns over time (Zhao et al., 2015; Du et al., 2020).
Therefore, modeling a general representation learning framework to extract the significant information for multivariate time-series data is more challenging than for univariate time-series data (Han et al., 2018; Du et al., 2020).
Most time-series representation learning methods have the following two limitations.
First, they mainly focus on training feature-agnostic temporal dependency.
In other words, for multivariate time-series data, the unique patterns of each feature are not given sufficient attention.
Figure 1a shows an example of a multivariate time-series sequence with similar temporal behaviors across the features, which we define as feature-agnostic patterns referring to Cirstea et al.
By contrast,
Figure 1b shows the case where the temporal behavior is different for each individual feature, which we call feature-specific patterns.
Although both types of feature characteristics should be properly captured, current time-series representation learning methods tend to overlook the feature-specific patterns.
Second, few time-series representation learning methods are generally applicable to multivariate time-series data.
We define general as “to be able to perform robustly on various datasets from diverse domains with various tasks” .
However, most previous methods have focused on improving classification task via representation learning (Tonekaboni et al., 2021; Eldele et al., 2021).
Franceschi et al.
(2019) and Zerveas et al.
(2021) conducted classification as well as regression tasks but only a few selected datasets were used for evaluations.
Yue et al.
(2022) proposed a universal
[Figure]
representation learning framework, TS2Vec, which achieved state-of-the-art performance on 29 multivariate time-series classification datasets and four multivariate time-series forecasting benchmarks.
However, TS2Vec is designed to learn only feature-agnostic temporal representation.
Specifically, as the multivariate time-series sequence is embedded along the temporal axis, the original feature dimension cannot be preserved any more, making it difficult to explicitly model feature-specific information (Yang et al., 2015).
To resolve these two limitations, we propose FEAT, a general framework for FEature-Aware multivariate Time-series representation learning, which can be applied to datasets in various domain and tasks.
In contrast to the previous methods that focus on training feature-agnostic temporal patterns, we also focus on training feature-aware temporal representation by leveraging a feature-wise encoder.
Furthermore, we propose an element-wise gating layer that flexibly learns the influence of feature-specific patterns of a given input sequence per dataset.
The main contributions of this paper are summarized as follows:
• We propose FEAT, a general representation learning framework for multivariate time-series data that learns feature-specific patterns along with the feature-agnostic temporal patterns.
• We explicitly model the feature-specific information for the first time by introducing a feature encoder and present an element-wise gating layer that flexibly learns the influence of feature-specific patterns per timestamp in a data-driven manner.
• FEAT outperforms existing state-of-the-art representation learning methods on various benchmark datasets and tasks including classification (improvement of 1.9%p on average in terms of accuracy) and forecasting (improvement of 9.54% on average in terms of MSE).
M ETHOD
The overall architecture of FEAT is shown in Figure 2.
We propose a novel self-supervised learning framework for multivariate time-series that utilizes feature-specific patterns in addition to temporal patterns.
Specifically, FEAT learns representation for the first time in terms of three diversified perspectives: feature-specific patterns, feature-agnostic temporal patterns, and dependency between multiple feature-specific and temporal information.
In the following subsections, we describe the architectural details of the proposed framework.
Notations S =
{ X 1 , X 2 , . . .
, X N } is a set of time-series data with N instances.
X i ∈ R T × F , the i th instance in the set S , is the input of our framework.
T is the sequence length, and F is the number of features of the given input sequence.
d 1 and d 2 are the dimension of timestamp-wise embedding and feature-wise embedding, respectively, and D stands for the final representation dimension.
Note that we mark the notations and the output dimensions of each layer in Figure 2 to help understand the following explanations
. 2. I NPUT E MBEDDING
Timestamp-wise embedding and feature-wise embedding We use two learnable input embedding matrices, W 1 ∈ R F × d 1 and W 2 ∈ R T × d 2 , to learn the timestamp-wise embedding E 1 and the feature-wise embedding E 2 respectively, as shown in Figure 2b.
The timestamp-wise embedding has a limitation in that the embedded vector of size T ×
d 1 cannot preserve the dimension of F as the identity of each feature is entangled after the projection.
This makes it difficult to model the feature-specific patterns in the upper layer.
Therefore, we introduce a feature-wise input embedding matrix W 2 to utilize the feature-specific patterns explicitly.
Without using any meta-information or manual preprocessing, we embed each feature by projecting the transposed raw input sequence X T .
2. T IME - SERIES D ATA A UGMENTATION Masking
Input Embedding We randomly mask the input embedding as a data augmentation for contrastive learning inspired by TS2Vec (Yue et al., 2022).
In our experiment, random binomial masking is applied to the timestamp-wise embedding and no masking is applied to the feature-wise embedding.
The detailed experiment is presented in Appendix A.1.
Note that the masking is only applied during training.
Random Shifting We also introduce random shifting (Figure 2a), which is applied to the raw input time-series to produce randomly augmented contexts.
Yue et al.
(2022) proposed random cropping to generate two overlapping segments in contexts with random lengths for contextual consistency.
However, for the feature-wise embedding, the size of T in the transposed input with the size of
F × T must be fixed.
To mitigate this concern, we propose random shifting, which can maintain the effect of random cropping while keeping the dimension T fixed.
As shown in Figure 2a, we shift the raw input time-series along the horizontal axis (where the shift size is randomly chosen as a hyper-parameter) and pad or trim the sequence to preserve the context length.
Maximum shift size is used to impose constraints to prevent the distribution of each feature from the shifted context from getting distorted from the original distribution.
2. L AYERS
Encoder Layer We design two encoders, temporal encoder (Figure 2c) and feature encoder (Figure 2d), to extract feature-agnostic temporal representation and feature-specific representation, respectively.
We use a dilated convolutional neural network as the temporal encoder to extract the feature-agnostic temporal representation R 1 .
Dilated convolutional neural networks have been actively used for time-series data because of their ability to capture multi-scale temporal dependency (Tonekaboni et al., 2021;
Franceschi et al., 2019;
Yue et al., 2022; Bai et al., 2018;
Oord et al., 2016).
The temporal encoder takes the timestamp-wise embedding as an input, and it focuses on learning dynamic feature-agnostic temporal dependencies rather than the specific patterns of each feature.
To complement the loss of feature-specific information, we design an additional MLP-based feature encoder network, which focuses on learning feature-wise specific representation R 2 by receiving feature-wise embedding.
Element
-wise
Gating Layer
The main goal of time-series representation learning is to represent each timestamp of a given sequence in the hidden dimension space.
This means that the length of input sequence and that of the final output representation should be the same.
Concerns might arise if the output of the feature encoder (Figure 2d) and the output of temporal encoder (Figure 2c) have different lengths; the former is F while the latter is T in the above structure.
To address such concerns, we first align the feature-specific representation R 2 along the temporal axis, learning feature-temporal dependency by a projection layer in Figure 2e.
The aligned feature representation
R ′ 2 is regarded as feature-specific temporal representation while the representation R 1 from the temporal encoder is regarded as feature-agnostic temporal representation.
Once these two types of temporal representations R 1 and R ′ 2 are obtained, we aggregate them using an element-wise gating module in the layer shown in Figure 2e.
We assume that each timestamp can be strongly influenced by inter-dependencies with other timestamps that are inherently shared between features, while some timestamps can be influenced by the representative patterns that uniquely appear in each feature (Li et al., 2021; Cirstea et al., 2022).
Consequently, we adopt a gating mechanism to allow the model to learn the influence of each characteristic flexibly in a data-driven manner.
Through this, the model can more adaptively learn each representation compared to a model with a simple aggregation function such as concatenation or mean.
The final representation is R = { r 1 , r 2 , . . .
, r T } , where r t = a t ∗ R 1( t ) + b t ∗ R ′ 2( t ) ∈
R D .
a t is the weight of t th feature-agnostic temporal representation in R 1 , and b t is the weight of t th feature-specific temporal representation in R ′ 2 .
The sum of scalar pair a t and b t is equal to 1 .
The sets of T gating weights, A
= { a 1 , . .
. , a T } and B = { b 1 , . . .
, b T } , are computed as follows:
[Equation]
We use a linear projection layer for g ( · ) .
Detailed analysis of the gating module is given in Section 3.3.
All components of the encoder layer enable FEAT to model the representations into three different perspectives sophisticatedly – feature-agnostic temporal representation R 1 , feature-specific representation R 2 , feature-specific temporal representation R ′ 2 .
Unlike previous architectures, we first focus on learning the beneficial feature-level representation as well as how the features are dynamically aligned on the temporal axis.
Specifically, through the proposed gating layer, the model can automatically control the usage of feature-specific information at each timestamp.
Decoder Layer We used a reconstruction decoder (Figure 2f) to help train the gating parameters by upstream gradients and learn a richer representation.
An MLP decoder is employed, which is only for the auxiliary loss during training.
2. O BJECTIVE F UNCTION
We design the objective function as follows:
[Equation]
where L contrast t is the hierarchical temporal contrasting loss, L contrast f is the feature contrasting loss, and L recon is the reconstruction loss.
The hierarchical temporal contrasting loss L contrast t , adopted from Yue et al.
(2022), is calculated from the overlapping segments of randomly shifted pairs described in Section 2.2.
The main strategy is to set the same timestamps from the two overlapping representations as positive pairs and set the different timestamps to be negative pairs.
L contrast t can be formulated as follows:
[Equation]
[Equation]
ℓ ( i,t ) temp in Eq.
(3) is the temporal contrasting loss,
where r i,t and r ′ i,t are the i th augmented pair of feature-agnostic temporal representation at timestamp t , Ω is the set of overlapped timestamps from each context pair, and 1 is the indicator function.
It induces the network to represent the same timestamps from different contexts as similar and every different timestamp to be unique.
Specifically, the model can focus more on learning position-agnostic temporal representation under the contextual consistency (Yue et al., 2022).
In addition to temporal contrasting loss, instance-wise contrastive loss ℓ ( i,t ) inst of Eq.
(4) is applied, where the positive pair is the same timestamp from the augmented instance while the negative pair is the same timestamp from different instances in batch
B .
The two losses from Eq.
(3) and Eq.
(4) are computed for every timestamp and instance in a batch as formulated in Eq.
(5), and they are also computed on the max-pooled representations along the temporal axis by hierarchical procedures.
This enables the model to learn the representation that reflects the multi-scale granularities through the hierarchical structure.
In addition to temporal contrasting, we propose the feature contrasting loss L contrast f to force the model to learn the unique representation of each feature.
Assuming that the distribution of each feature sequence from the shifted pairs is similar to each other as described in Section 2.2, we take two augmented feature-specific representations of feature f , the output of the feature encoder in
Figure 2d, as a positive pair.
L contrast f can be formulated as follows:
[Equation]
The main difference of L contrast f from L contrast t is that it is applied to the feature-specific representation R 2 ∈ R F × D and not applied hierarchically, where r i,f and r ′ i,f are the representations of feature f from the i th augmented pair.
The feature contrasting loss helps the model to represent each feature with its own patterns to a vector by referring to different feature patterns in the same instance as well as different feature patterns from other instances.
Additionally, we employed the reconstruction error L recon for the loss function of as an auxiliary task.
An MSE score between the input sequence X and the reconstructed sequence ˆ X is calculated on the overlapped timestamps from the shifted pair.
E XPERIMENTS
We verified the effectiveness of our representation learning framework by evaluating the performance improvements in classification as well as forecasting tasks.
To conduct the downstream tasks, we trained traditional machine learning models on the learned representations following Yue et al.
(2022) and Franceschi et al.
For a fair comparison, we fixed the representation dimension as 320 for all the baseline models for representation learning.
The results of FEAT were calculated by averaging the results of five repeated runs, and we report the performances of the baseline models from their original papers and from TS2Vec.
All the experiments were conducted on a single NVIDIA Titan RTX GPU.
Details of the experimental settings and datasets are described in
Appendix A.1 and Appendix A.2.
[Table]
3. M ULTIVARIATE T IME -S ERIES C LASSIFICATION
For the multivariate time-series classification task
[Table]
experiments were conducted on 30 UEA archivedatasets (Bagnall et al., 2018).
We compared the performance of FEAT against six well-known timeseries representation learning methodologies:
TS2Vec (Yue et al., 2022), T-Loss (Franceschi et al., 2019), TS-TCC (Eldele et al., 2021), TST (Zerveas et al., 2021), TNC (Tonekaboni et al., 2021), and a distance-based algorithm DTW
(Chen et al., 2013).
Following the same evaluation protocol as TS2Vec, we trained an SVM classifier with RBF kernel on a max-pooled representation along the temporal axis.
Table 1 presents the summarized results of the classification task.
We can observe that FEAT resulted in the best average accuracy with 1.9%p improvement on 29 UEA datasets over the second-best model TS2Vec.
We can also observe that the average accuracy is not a biased result on account of a part of datasets, in that the average rank also outperforms the other baselines.
The results in Table 1 imply that feature-aware representation learning benefits the performance compared to methods that only learn feature-agnostic temporal dependencies.
We also analyzed the efficiency of FEAT in terms of total training time in Appendix A.3.
3. M ULTIVARIATE T IME -S ERIES F ORECASTING
For the multivariate time-series forecasting task, experiments were conducted on the four benchmark datasets: ETTh 1 , ETTh 2 , ETTm 1 , and ECL.
We compared the performance of FEAT against the representation learning model
TS2Vec (Yue et al., 2022) and five end-to-end forecasting models (Zhou et al., 2021; Cao et al., 2020; Bai et al., 2018;
Li et al., 2019; Lai et al., 2018).
For a fair comparison, we followed the same protocol as TS2Vec and trained a ridge regression on the final
[Table] [Table]
representation with 320 dimensions.
Specifically, the regression model is trained to predict H future timestamps from the representation of the last timestamp of the given input sequence.
The evaluation results of multivariate time-series forecasting in terms of MSE and MAE are summarized in Table 2.
FEAT generally outperforms both types of baselines models (time-series representation learning-based models and end-to-end forecasting models).
FEAT is placed best or second-best in most dataset-time horizon cases.
The average improvements of FEAT compared to TS2Vec are 9.54% in terms of MSE and 4.56% in terms of MAE.
We provide additional analysis of training time and present forecasting plots in Appendix A.4.
A NALYSIS 3.3. A BLATION STUDY
The following ablation studies were conducted to verify the effectiveness of each component in
FEAT.
First, we investigated the effect of each term in our objective function in Eq.
As summarized in Table 3, the model without the hierarchical temporal contrasting loss degraded the performance in 4.9%p.
Furthermore, the performance without feature contrasting loss shows 1.1%p degradation.
Without using either hierarchical temporal contrasting or feature contrasting loss, there
[Figure]
was 5.5%p performance degradation with larger standard deviations.
It can be concluded that the hierarchical temporal contrasting loss and the feature contrasting loss both significantly contribute to the performance improvement.
We also investigated the effect of the reconstruction loss by removing the reconstruction decoder.
Consequently, substantial performance degradation occurred with larger standard deviations per seed and dataset.
This implies that the decoder helps the model to learn more stable and effective representations by upstream gradient flows, as discussed in Section 2.3.
Second, we investigated the module-level performance changes.
For demonstrating the effectiveness of the MLP-based feature encoder in FEAT (Figure 2d), we evaluated the performance by replacing
MLP network with depth-wise dilated convolution, which can also explicitly model the feature-wise patterns.
Consequently, 3.9%p performance degradation occurred.
Furthermore, we validated the effectiveness of learnable element-wise gating layer in Figure 2e.
When we replaced the gating layer with a simple aggregation module such as concatenation or mean, there were 3.6%p or 2.4%p respective performance degradations.
In addition, FEAT without random masking and random shifting showed lower performances as well.
Among the modules, the MLP-based feature encoder seems to be the most significant module in our feature-aware representation learning framework, followed by the element-wise gating module and the two augmentation techniques.
Further analysis of how the feature encoder works is presented in Appendix A.5.1.
3.3. Q UALITATIVE STUDY
We performed t-SNE visualization to demonstrate that the proposed framework FEAT generally learns the rich representation.
Specifically, we compared FEAT with TS2Vec, which only focuses on learning feature-agnostic temporal dependency.
The t-SNE plots in Figure 3 demonstrate that the instances belonging to the same class are clustered more densely as well as more clearly separated from the other classes compared to TS2Vec.
We also calculated the Silhouette scores in Figure 3 by applying K-means clustering on learned representations with the number of clusters equal to the number of classes.
Along with the t-SNE plots, the Silhouette scores support that the learned representations from FEAT were indeed clustered better than TS2Vec as the scores based on our representations were higher than TS2Vec.
The overall t-SNE plots and Silhouette scores of all
UEA datasets are provided in Appendix A.5.2.
Furthermore, we analyzed the element-wise gating layer in Figure 2e to demonstrate that it can flexibly learn the influence of feature-specific patterns on the various datasets in general.
In Figure 4a, the weights for feature-agnostic representation are distributed above 0.8.
This means that the temporal dependency highly influences the final representation.
Indeed, PhonemeSpectra has similar patterns between features along the temporal axis.
By contrast, in Figure 4b and Figure 4c, plotted from datasets having diverse patterns among the features, the distributions appear to be more spread out than in Figure 4a.
Although the feature-agnostic temporal representation has a more significant influence than the feature-specific temporal representation because of the fundamental characteristic of time-series data, Figure 4 shows that the gating layer can automatically capture the importance of feature-specific patterns in a data-driven manner.
R ELATED W ORK
As large amounts of unlabeled data have been accumulated, extensive self-supervised learning methods have been studied to extract rich representation leveraging these data (Chen et al., 2020;
Gao et al., 2021; Devlin et al., 2018).
In particular, in natural language processing and computer vision, many representation learning methods have been studied by introducing various pretext tasks or techniques like contrasting and augmentation (Chen et al., 2020; Gidaris et al., 2018;
He et al., 2022; Oord et al., 2018).
In contrast, studies on time-series representation learning are limited.
A few methodologies of representation learning on time-series data have been proposed to effectively extract rich representations from complex temporal patterns (Yue et al., 2022;
Tonekaboni et al., 2021; Franceschi et al., 2019; Eldele et al., 2021;
Zerveas et al., 2021;
Woo et al., 2022).
Yue et al.
(2022) proposes a universal framework that learns contextual representations by hierarchical contrastive learning over two overlapping segments from random contexts.
Tonekaboni et al.
(2021) proposes a framework for learning generalizable time-series representations for nonstationary time-series via defining neighborhoods considering stationarity properties to design a debiased contrastive objective.
Franceschi et al.
(2019) employs triplet loss to learn scalable representations for multivariate time-series.
Eldele et al.
(2021) introduces a contrastive learning framework using a tough cross-view prediction task on the augmented sequences.
Zerveas et al.
(2021) employs a transformer-based architecture for multivariate time-series representation learning using a reconstruction loss.
Woo et al.
(2022) proposes contrastive learning methods to learn disentangled seasonal-trend representations for forecasting.
Most of the aforementioned works propose methods based on temporal dependency, and none have focused on learning multivariate time-series representations through utilizing feature-specific information.
We propose a general framework that learns representations with feature-specific information.
Notably, our framework can capture the featurespecific information in a data-driven manner for the various datasets, which does not require any additional data preprocessing or expert knowledges.
C ONCLUSION
This paper presents a general framework for feature-aware multivariate time-series representation learning, called FEAT.
This framework is designed to model the feature-wise specific patterns and to leverage this information for learning rich representation.
The evaluation of our framework was conducted on classification as well as forecasting tasks, and the results demonstrate its effectiveness.
FEAT achieved the best performances on 30 classification datasets and four forecasting datasets on average among the existing time-series representation learning models.
Furthermore, we show that the learnable gating layer in FEAT is flexible enough to learn the feature-specific patterns depending on the datasets.
As our framework explicitly models each feature and aligns it to the timestamps, future work can focus on utilizing external features that can enrich the output representations.
FEAT can also take advantage of its reconstruction structure in anomaly detection tasks.
A A PPENDIX A. R EPRODUCTION D ETAILS
[Table]
We used the default fixed set of hyper-parameters following TS2Vec (Yue et al., 2022) regardless of the dataset or the downstream task.
The only difference is that the number of optimization iterations was increased approximately 1.5 times on average.
The batch size B is set to 8.
The learning rate is 0.001.
The final representation dimension D is set to 320, and the dimensions of timestampwise and feature-wise embedding, d 1 and d 2 , were set to 64.
The hidden dimension for encoders and element-wise gating layer was set to 320, equal to D .
In addition, we fixed the maximum shift size as three for random shifting.
Random binomial masking was applied to the timestampwise embedding but no masking for feature-wise embedding was applied by the ablation results in Table 4.
According to Table 4, no significant improvements in terms of average accuracy or standard deviation were observed at every condition.
The average accuracy decreases when the feature masking ratio increases, showing that each feature embedding contains helpful information.
We used ten blocks consisting of two dilated CNN layers for the temporal encoder with kernel size three and dilation size 2 l for the l th block.
A two-layer MLP network for feature encoder as well as reconstruction decoder was employed.
A single linear layer was used for projecting featurespecific representation to the temporal axis.
These architectural settings were fixed to all datasets and downstream tasks.
[Table]
Additionally, we explored the hyper-parameter sensitivity by evaluating each hyper-parameter’s effectiveness.
In Table 5, we evaluated the model performance in terms of variation in batch size, learning rate, representation dimension, and reconstruction loss function.
The numbers marked with * are the fixed hyper-parameters used for all our experiments.
Considering the overall performance on 30 classification datasets, no harmful degradation occurs around the default settings.
Based on this sensitivity analysis, we found the FEAT is not very sensitive to a slight change of hyper-parameters, which can be another advantage of practical implementation.
A. D ATASET
UEA archive datasets UEA archive provides various types of datasets such as Human Activity,
Audio Spectra, Electrocardiogram (ECG), Electroencephalogram (EEG), and Magnetoencephalography (MEG).
The datasets are labeled per observed sequence.
Electricity Transformer Temperature (ETT) dataset ETT dataset 2 contains data collected by an electricity transformer over two years.
In this study, we use three distinct datasets - ETTh 1 , ETThand ETTm 1 .
ETTh 1 and ETTh 2 are 1-hour-level datasets observed from different counties in China, while ETTm 1 is a 15-minute-level dataset.
As they have different granularities, we can also validate the robustness of our learned representations on the various granularities.
The train/validation/test for the ETT datasets is 12/4/4 months.
Electricity Consuming Load (ECL) dataset ECL 3 is the electricity consumption dataset of 321 clients.
It is preprocessed into 1-hour-level dataset following Zhou et al.
The train/validation/test for the ECL dataset is 60%/20%/20%.
A. S UPPLEMENTARY ANALYSIS FOR MULTIVARIATE TIME - SERIES CLASSIFICATION
[Table]
For further analysis, we compared the total training time in Table 6 on 30 UEA datasets of the top three best models in terms of the average accuracy.
We reimplemented TS-TCC 4 and TS2Vec 5 based on the code from their official repositories.
We followed their default settings, except for unifying the representation dimension and evaluation protocol for a fair comparison.
As FEAT is a more sophisticated model than TS2Vec owing to its feature encoder and reconstruction decoder, TS2Vec records the shortest time on training representation.
However, FEAT achieves the shortest time for training the classifier on the learned representation.
It can be understood that although a longer time is required to learn time-series representation through our proposed framework, sufficiently rich representation is learnt so that the following classifier can converge earlier than the other methods.
In comparison to TS-TCC, which is the transformer-based representation learning framework, the proposed FEAT, the Dilated CNN and MLP-based model, is found to be much lighter.
FEAT is approximately 5.08 times faster on training representation and 6.59 times faster on converging SVM classifier compared to TS-TCC.
A. S UPPLEMENTARY ANALYSIS FOR MULTIVARIATE TIME - SERIES FORECASTING
Table 7 summarizes the time for training and inference of the top three best models on the ETTmdataset in Table 2.
We reimplemented Informer 6 and TS2Vec from the official code from their official repositories.
We followed the default setting for Informer and TS2Vec except for the number of epochs; the number of epochs in TS2Vec was set to that of FEAT.
Like the analysis mentioned in Appendix A.3, TS2Vec resulted in the shortest time in training and inference on the representation learning framework.
In contrast, Informer, a transformer-based end-to-end forecasting model, resulted in the longest time in training as well as inference phases.
Although FEAT resulted in a longer total running time than TS2Vec, the difference is not as significant as that between FEAT and
Informer.
Notably, FEAT is 7.22 times more efficient in total training time and 4.30 times faster in total inference time for all horizons on average compared to Informer.
[Table]
[Figure]
[Table]
Figure 5 shows a slice of the target values and forecasting results of FEAT and TS2Vec on the ETThtest dataset with a length of 336.
It shows that the predictions from our proposed model are better fitted to the ground truths with trends and seasonal patterns rather than the TS2Vec representation learning framework.
In addition
[Table]
Table 8 summarizes ablation study results on multivariate times-series forecasting task.
It was performed on three ETT datasets (ETTh 1 , ETTh 2 , and ETTm 1 )
[Table]
and the results are the averaged MSE and MAE scores of the five repetitions at all forecasting horizons.
In terms of both metrics, all ablations exhibit degradation
[Table]
and the full model achieved the best, which supports that each component of FEAT contributes to performance improvement
[Table]
A.
S UPPLEMENTARY F IGURE
A.5.
F EATURE
[Figure]
SPECIFIC REPRESENTATION
[Figure]
Figure 6 presents the heatmap visualizations of feature-specific representation from FEAT.
For comparison, we selected the sample sequences with different classes having different feature patterns.
Specifically, as shown in Figure 6a, these two sequences have similar temporal patterns for the orange and green line, but opposite patterns for the red and blue line.
We verified that our feature encoder can learn these unique characteristics by visualizing the feature-specific representations in
Figure 6b.
Feature contrasting, as well as feature encoder, helps FEAT to model the feature-specific patterns explicitly.
Specifically, the two opposite patterns of the red and blue line in Figure 6a were represented in opposite manners for each class.