SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction
Abstract
One unique property of time series is that the temporal relations are largely preserved after downsampling into two sub-sequences.
By taking advantage of this property, we propose a novel neural network architecture that conducts sample convolution and interaction for temporal modeling and forecasting, named SCINet .
Specifically, SCINet is a recursive downsample-convolve-interact architecture.
In each layer, we use multiple convolutional filters to extract distinct yet valuable temporal features from the downsampled sub-sequences or features.
By combining these rich features aggregated from multiple resolutions, SCINet effectively models time series with complex temporal dynamics.
Experimental results show that SCINet achieves significant forecasting accuracy improvements over both existing convolutional models and Transformer-based solutions across various real-world time series forecasting datasets.
Our codes and data are available at https://github.com/cure-lab/SCINet .
Introduction
Time series forecasting (TSF) enables decision-making with the estimated future evolution of metrics or events, thereby playing a crucial role in various scientific and engineering fields such as healthcare [1], energy management [42], traffic flow [42] and financial investment [10], to name a few.
There are mainly three kinds of deep neural networks used for sequence modeling and they are all applied for time series forecasting [24]: (i).
recurrent neural networks (RNNs) [13]; (ii).
Transformerbased models [37]; and (iii).
temporal convolutional networks (TCN) [4].
Despite the promising results of TSF methods based on these generic models they do not consider the specialty of time series data during modeling.
For example, one unique property of time series is that the temporal relations (e.g., the trend and the seasonal components of the data) are largely preserved after downsampling into two sub-sequences.
Consequently, by recursively downsampling the time series into sub-sequences, we could obtain a rich set of convolutional filters to extract dynamic temporal features at multiple resolutions.
Motivated by the above, in this paper, we propose a novel neural network architecture for time series modeling and forecasting, named sample convolution and interaction network ( SCINet ).
The main contributions of this paper are as follows:
• We propose SCINet, a hierarchical downsample-convolve-interact TSF framework that effectively models time series with complex temporal dynamics.
By iteratively extracting and exchanging information at multiple temporal resolutions, an effective representation with enhanced predictability can be learned, as verified by its comparatively lower permutation entropy (PE) [16].
[Figure]
• We design the basic building block, SCI-Block , for constructing SCINet, which downsamples the input data/feature into two sub-sequences, and then extracts features of each subsequence using distinct convolutional filters.
To compensate for the information loss during the downsampling procedure, we incorporate interactive learning between the two convolutional features within each SCI-Block.
Extensive experiments on various real-world TSF datasets show that our model consistently outperforms existing TSF approaches by a considerable margin.
Moreover, while SCINet does not explicitly model spatial relations, it achieves competitive forecasting accuracy on spatial-temporal TSF tasks.
Related Work and Motivation
The time series forecasting problem is defined as: Given a long time series X ∗ and a look-back window of fixed length T , at timestamp t , time series forecasting is to predict ˆ X t +1:
t + τ = { x t +1 , ..., x t + τ } based on the past T steps X t − T
+1:
t = { x t − T +1 , ..., x t } .
Here, τ is the length of the forecast horizon, x t ∈ R
d is the value at time step t , and d is the number of variates.
For simplicity, in the following we will omit the subscripts, and use X and ˆ X to represent the historical data and the forecasted data, respectively.
2.1 Related Work
Traditional time series forecasting methods such as the autoregressive integrated moving average (ARIMA) model [8] and Holt-Winters seasonal method [14] have theoretical guarantees.
However, they are mainly applicable for univariate forecasting problems, restricting their applications to complex time series data.
With the increasing data availability and computing power in recent years, it is shown that deep learning-based TSF techniques have the potential to achieve better forecasting accuracy than conventional approaches [24, 29].
Earlier RNN-based TSF methods [31, 32] summarize the past information compactly in the internal memory states that are recursively updated with new inputs at each time step, as shown in Fig. 1(a).
The gradient vanishing/exploding problems and the inefficient training procedure greatly restrict the application of RNN-based models.
In recent years, Transformer-based models [37] have taken the place of RNN models in almost all sequence modeling tasks, thanks to the effectiveness and efficiency of the self-attention mechanisms.
Various Transformer-based TSF methods (see Fig. 1(b)) are proposed in the literature [21, 23, 38, 25].
These works typically focus on the challenging long-term time series forecasting problem, taking advantage of their remarkable long sequence modeling capabilities.
Another popular type of TSF model is the so-called temporal convolutional network [7, 4, 33, 39, 27], wherein convolutional filters are used to capture local temporal features (see Fig. 1(c)).
The proposed
SCINet is also constructed based on temporal convolution.
However, our method has several key differences compared with the TCN model based on dilated causal convolution, as discussed in the following.
2.2 Rethinking Dilated Causal Convolution for Time Series Modeling and Forecasting
The local correlation of time series data is reflected in the continuous changes within a time slot, and convolutional filters can effectively capture such local features.
Consequently, convolutional neural networks are explored in the literature for time series modeling and forecasting.
In particular, dilated causal convolution (DCS) is the current de facto method used in this respect.
DCS was first proposed for generating raw audio waveforms in WaveNet [28].
Later, [4] simplifies the WaveNet architecture to the so-called temporal convolutional networks (see Fig.
1 (c)).
TCN consists of a stack of causal convolutional layers with exponentially enlarged dilation factors, which can achieve a large receptive field with just a few convolutional layers.
Over the years, TCN has been widely used in all kinds of time series forecasting problems and achieve promising results [39, 33].
Moreover, convolutional filters can work seamlessly with graph neural networks (GNNs) to solve various spatial-temporal TSF problems.
With causal convolutions in the TCN architecture, an output i is convolved only with the i th and earlier elements in the previous layer.
While causality should be kept in forecasting tasks, the potential “future information leakage" problem exists only when the output and the input have temporal overlaps.
In other words, causal convolutions should be applied only in autoregressive forecasting, wherein the previous output serves as the input for future prediction.
When the predictions are completely based on the known inputs in the look-back window, there is no need to use causal convolutions.
We can safely apply normal convolutions on the look-back window for forecasting.
More importantly, the dilated architecture in TCN has two inherent limitations:
• A single convolutional filter is shared within each layer.
Such a unified convolutional kernel tends to extract the average temporal features from the data/features in the previous layer.
However, complex time series may contain substantial temporal dynamics.
Hence, it is essential to extract distinct yet valuable features with a rich set of convolutional filters.
• While the final layer of the TCN model has the global view of the entire look-back window, the effective receptive fields of the intermediate layers (especially those close to the inputs) are limited, causing temporal relation loss during feature extraction.
The above limitations of the TCN architecture motivate the proposed SCINet design, as detailed in the following section.
[Figure]
SCI-
[Figure]
SCI-
Block … … … … …
Input: !
𝐗 !"
# ( 𝐗 if 𝑘 = 1 ) #Level= #Level= #Level= L
SCINet
Input: 𝐗 …
SCINet
Output: !
𝐗 $ exp
Split 𝐅 ()) 𝐅 *+*,
Input: 𝐅 𝜙 𝜓 𝜌 𝜂 𝐅 ()).
𝐅 *+*,.
𝐅 ())/
𝐅 *+*,/ exp
InteractiveLearning
Concat
Concat & Realign
Onput: !
𝐗 !
FC
SCINet
Concat #Stack =
#Stack= (c) Stacked SCINet (b)
SCINet (a) SCI-Block ℒ !
#Stack= K … ℒ " ℒ #
SCINet: Sample Convolution and Interaction Network
SCINet adopts an encoder-decoder architecture.
The encoder is a hierarchical convolutional network that captures dynamic temporal dependencies at multiple resolutions with a rich set of convolutional filters.
As shown in Fig.
2(a), the basic building block, SCI-Block (Section 3.1), downsamples the input data or feature into two sub-sequences and then processes each sub-sequence with a set of convolutional filters to extract distinct yet valuable temporal features from each part.
To compensate for the information loss during downsampling, we incorporate interactive learning between the two sub-sequences.
Our SCINet (Section 3.2) is constructed by arranging multiple SCI-Blocks into a binary tree structure (Fig. 2(b)).
A distinctive advantage of such design is that each SCI-Block has both local and global views of the entire time series, thereby facilitating the extraction of useful temporal features.
After all the downsample-convolve-interact operations, we realign the extracted features into a new sequence representation and add it to the original time series for forecasting with a fully-connected network as the decoder.
To facilitate extracting complicated temporal patterns, we could further stack multiple SCINets and apply intermediate supervision to get a Stacked SCINet (Section 3.3), as shown in Fig. 2(c).
3.1 SCI-Block
The SCI-Block (Fig. 2(a)) is the basic module of the SCINet, which decomposes the input feature F into two sub-features F ′ odd and F ′ even through the operations of Spliting and Interactive-learning .
The Splitting procedure downsamples the original sequence F into two sub-sequences F even and
F odd by separating the even and the odd elements, which are of coarser temporal resolution but preserve most information of the original sequence.
Next, we use different convolutional kernels to extract features from F even and F odd .
As the kernels are separate, the extracted features from them would contain distinct yet valuable temporal relations with enhanced representation capabilities.
To compensate for potential information loss with downsampling, we propose a novel interactive-learning strategy to allow information interchange between the two sub-sequences by learning affine transformation parameters from each other.
As shown in Fig.
2 (a), the interactive learning procedure consists of two steps.
First, F even and F odd are projected to hidden states with two different 1D convolutional modules ϕ and ψ , respectively, and transformed to the formats of exp and interact to the F even and F odd with the element-wise product (see Eq.
This can be viewed as performing scaling transformation on
F even and F odd , where the scaling factors are learned from each other using neural network modules.
Here, ⊙ is the Hadamard product or element-wise production.
[Equation]
Second, as shown in Eq.
(11), the two scaled features F seven and F sodd are further projected to another two hidden states with the other two 1D convolutional modules ρ and η , and then added to or subtracted from 1 F seven and F sodd .
The final outputs of the interactive learning module are two updated sub-features F ′ even and F ′ odd .
The default architectures of ϕ , ψ , ρ and η are shown in the
Appendix C.
Compared to the dilated convolutions used in the TCN architecture, the proposed downsampleconvolve-interact architecture achieves an even larger receptive field at each convolutional layer.
More importantly, unlike TCN that employs a single shared convolutional filter at each layer, significantly restricting its feature extraction capabilities, SCI-Block aggregates essential information extracted from the two downsampled sub-sequences that have both local and global views of the entire time series.
3.2 SCINet
With the SCI-Blocks presented above, we construct the SCINet by arranging multiple SCI-Blocks hierarchically and get a tree-structured framework, as shown in Fig.
2 (b).
There are 2 l SCI-Blocks at the l -th level, where l = 1 , . . .
, L is the index of the level, and L is the total number of levels.
Within the k -th SCINet of the stacked SCINet (Section 3.3), the input time series X (for k =1 ) or feature vector ˆ X k − 1 =
{ ˆ x k − 1, ..., ˆ x k − 1 τ } (for k > 1 ) is gradually downsampled and processed by SCI-Blocks through different levels, which allows for effective feature learning of different temporal resolutions.
In particular, the information from previous levels will be gradually accumulated, i.e., the features of the deeper levels would contain extra finer-scale temporal information transmitted from the shallower levels.
In this way, we can capture both short-term and long-term temporal dependencies in the time series.
After going through L levels of SCI-Blocks, we rearrange the elements in all the sub-features by reversing the odd-even splitting operation and concatenate them into a new sequence representation.
It is then added to the original time series through a residual connection [12] to generate a new sequence with enhanced predictability.
Finally, a simple fully-connected network is used to decode the enhanced sequence representation into ˆ X k
=
ˆ x k 1 , ..., ˆ x kτ } .
Note that, to mitigate distribution shift in some TSF tasks, before supplying the data in the look-back window to our model, all the data elements are subtracted with the value of the last element, which is added to all the data elements in the forecasting horizon afterwards.
3.3 Stacked SCINet
When there are sufficient training samples, we could stack K layers of SCINets to achieve even better forecasting accuracy (see Fig.
2 (c)), at the cost of a more complex model structure.
Specifically, we apply intermediate supervision [5] on the output of each SCINet using the groundtruth values, to ease the learning of the intermediate temporal features.
The output of the k -th intermediate SCINet, ˆ X k with length τ , is concatenated with part of the input X t − ( T − τ )+1: t to recover the length to the original input and feeded as input into the ( k + 1) -th
SCINet, where k
= 1 , . .
, K − 1 , and K is the total number of the SCINets in the stacked structure.
The output of the K -th SCINet, ˆ X K , is the final forecasting results.
3.4 Loss Function
To train a stacked SCINet with K ( K ≥ 1 ) SCINets, the loss of the k -th prediction results is calculated as the L1 loss between the output of the k -th SCINet and the ground-truth horizontal window to be predicted:
[Equation]
The total loss of the stacked SCINet can be written as:
[Equation]
3.5 Complexity Analysis
Thanks to the downsampling procedure, the neurons at each convolutional layer of SCINet have a larger receptive field than those of TCN.
More importantly, the set of rich convolutional filters in SCINet enable flexible extraction of temporal features from multiple resolutions.
Consequently,
SCINet usually does not require downsampling the original sequence to the coarsest level for effective forecasting.
Given the look-back window size T , TCN generally requires ⌈ log 2 T
⌉ layers when the dilation factor is 2, while the number of layers L in SCINet could be much smaller than log 2 T .
Our empirical study shows that the best forecasting accuracy is achieved with L ≤ 5 in most cases even with large T (e.g., 168 ).
As for the number of stacks K , our empirical study also shows that K ≤would be sufficient.
Consequently, the computational cost of SCINet is usually on par with that of the TCN architecture.
The worst-case time complexity is O
( T log T ) , much less than that of vanilla Transformer-based solutions: O ( T 2 ) .
Experiments
In this section, we show the quantitative and qualitative comparisons with the state-of-the-art models for time series forecasting.
We also present a comprehensive ablation study to evaluate the effectiveness of different components in SCINet.
More details on datasets , evaluation metrics , data pre-processing , experimental settings , network structures and their hyper-parameters are shown in the Appendix.
4.1 Datasets
We conduct experiments on 11 popular time series datasets: (1) Electricity Transformer Temperature [42] (ETTh) (2) Traffic (3) Solar-Energy (4) Electricity (5) Exchange-Rate (6) PeMS ( PEMS03,
PEMS04, PEMS07 and PEMS08 ).
A brief description of these datasets is listed in Table 1.
All the experiments on these datasets in this section are conducted under multi-variate TSF setting.
To make a fair comparison, we follow existing experimental settings, and use the same evaluation metrics as the original publications [17, 26, 40, 19] in each dataset.
[Table]
4.2 Results and Analyses
Table 2, 3, 4, 5, 6 provide the main experimental results of SCINet.
We observe that SCINet shows superior performance than other TSF models on various tasks, including short-term, long-term and spatial-temporal time series forecasting.
Short-term Time Series Forecasting: we evaluate the performance of the SCINet in short-term TSF tasks with other baseline methods on Traffic , Solar-Energy , Electricity and Exchange-Rate datasets.
The experimental setting is the same as [19], which uses the input length of 168 to forecast different future horizons { 3 , 6 , 12 , 24 } .
As can be seen in Table 2, the proposed SCINet outperforms existing RNN/TCN-based (LSTNet [19],
TPA-LSTM [34], TCN [4], TCN † ) and Transformer-based [38, 42, 37] TSF solutions in most cases, especially for the Solar-Energy and Exchange-Rate datasets.
Note that, TCN † denotes a variant of
TCN wherein causal convolutions are replaced by normal convolutions, and improves the original
TCN across all the datasets, which supports our claim in Sec. 2.2.
Moreover, we can also observe that the Transformer-based methods have poor performance in this task.
For short-term forecasting, the recent data points are typically more important for accurate forecasting.
However, the permutationinvariant self-attention mechanisms used in Transformer-based methods do not pay much attention to such critical information.
In contrast, the general sequential models (RNN/TCN) can formulate it easily, showing quite competitive results in short-term forecasting. [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[40]requirespre-prossesseddatasetsfortraining
[Table]
Long-term Time Series Forecasting: many real-world applications also require to predict long-term events.
Therefore, we conduct the experiments on Exchange Rate, Electricity ,Traffic and ETT datasets to evaluate the performance of SCINet on long-term TSF tasks.
In this experiment, we only compare SCINet with Transformer-based methods [38, 18, 21, 42, 37, 25], since they are more popular in recent long-term TSF research.
As can be seen from Table 3, the SCINet achieves state-of-the-art performances in most benchmarks and prediction length settings.
Overall, SCINet yields 39.89 % average improvements on MSE among the above settings.
In particular, for Exchange-Rate, compared to previous state-of-the-art results,
SCINet gives average 65 % improvements on MSE.
We attribute it to that the proposed SCINet can better capture both short ( local temporal dynamics )- and long ( trend, seasonality )-term temporal dependencies to make an accurate prediction in long-term TSF.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Figure]
[Table]
We conduct both Multivariate Time-series Forecasting and Univariate Time-series Forecasting on ETT datasets [42].
For a fair comparison, we keep all input lengths T the same as those of Informer .
The results are shown in Table 4 and Table 5, respectively.
Multivariate Time-series Forecasting on ETT : as can be seen from Table 4, compared with RNN-based methods such as LSTMa [2] and LSTnet [19], Transformer-based methods [18, 21, 42] produce better forecasting results. One of the primary reasons is that, RNN-based solutions conduct iterative forecasting and it is inevitable to suffer from error accumulation effects.
As another direct forecasting method, TCN further outperforms vanilla Transformer-based methods [18, 21, 42], because the stacked convolutional layers allow for more effective local-to-global temporal relation learning for multivariate time series.
It is worth noting that SCINet outperforms all the above models by a large margin.
Fig.
3 presents the qualitative results on some randomly selected sequences of the ETTh dataset, which clearly demonstrate the capability of SCINet in obtaining the trend and seasonality of time series for TSF.
Univariate Time-series Forecasting on ETT : in this experimental setting, we bring several strong baseline methods for univariate forecasting into comparison, including ARIMA, Prophet [36],
DeepAR [32] and N-Beats [29].
In Table 5, we can observe that N-Beats is superior to other baseline methods in most cases.
In fact, N-Beats also takes the unique properties of time series into consideration and directly learns a trend and a seasonality model using a deep stack of fully-connected layers with residuals, which is a departure from the predominant architectures, such as RNNs, CNNs and Transformers.
Nevertheless, the performance of SCINet is still much better than N-Beats.
The newly-proposed Transformer-based forecasting model, Autoformer [38], achieves the second best performance in all experimental settings and also surpasses SCINet in ETTm1 when the forecasting horizon is large. This is because, on the one hand, Autoformer focuses on modeling seasonal patterns and conducts self-attention at the sub-series level (instead of the raw data), which is much better in extracting long-term temporal patterns than vanilla Transformer-based methods. On the other hand, when forecasting long horizons, it is often the trend/seasonal information instead of the temporal dynamics in the look-back window that play the primary role, wherein the advantages of SCINet are not fully exhibited.
[Table]
MSE 0.
0.207 MAE 0.
0.370 Informer[42]
MSE 0.
0.512 MAE 0.
0.644 MSE 0.
0.110 Autoformer[38] MAE 0.
0.261 MSE 0.
0.165 SCINet
MAE 0.
[Figure]
o [Figure] Spatial-temporal Time Series Forecasting: besides the general TSF tasks, there is also a large category of data related to spatial-temporal forecasting.
For example, traffic datasets PeMS [9] (PEMS03, PEMS04, PEMS07 and PEMS08) are complicated spatial-temporal time series for public traffic network and they have been investigated for decades.
Most recent approaches: DCRNN [22],
STGCN [41], ASTGCN [11], GraphWaveNet [39], STSGCN [35], AGCRN [3], LSGCN [15] and
STFGNN [20] use graph neural networks to capture spatial relations while modeling temporal dependencies via conventional TCN or RNN/LSTM architectures.
We follow the same experimental settings as the above works.
As shown in Table 6, these GNN-based methods generally perform better than pure RNN or TCN-based methods.
However, SCINet still achieves better performance without sophisticated spatial relation modeling, which further proves the superb temporal modeling capabilities of SCINet.
[Table]
Predictability estimation: inspired by [16, 30], we use permutation entropy ( PE ) [6] to measure the predictability of the original input and the enhanced representation learnt by SCINet.
Time series with lower PE values are regarded as less complex, thus theoretically easier to predict 2 .
The PE values of the original time series and the corresponding enhanced representations are shown in Table
[Table]
[Table]
As can be observed, the enhanced representations learnt by SCINet indeed have lower PE values compared with the original inputs, which indicates that it is easier to predict the future from the enhanced representations using the same forecaster.
4.3 Ablation studies
To evaluate the impact of each main component used in SCINet, we experiment on several model variants on two datasets:
ETTh1 and PEMS08 .
SCIBlock : we first set the number of stacks K = 1 and the number of SCINet levels L = 3 .
For the SCI-Block design, to validate the effectiveness of the interactive learning and the distinct convolution weights for processing the sub-sequences, we experiment on two variants, namely w/o.
InterLearn and WeightShare .
The w/o.
InterLearn is obtained by removing the interactive-learning procedure described in Eq.
(1) and (11).
In this case, the two sub-sequences would be updated using
F ′ odd = ρ ( ϕ ( F odd )) and F ′ even =
η ( ψ ( F even )) .
For WeightShare , the modules ϕ , ρ , ψ , and η share the same weight.
The evaluation results in Fig.
4 show that both interactive learning and distinct weights are essential, as they improve the prediction accuracies of both datasets at various prediction horizons.
At the same time, comparing Fig. 4(a) with Fig. 4(b), we can observe that interactive learning is more effective for cases with longer look-back window sizes.
This is because, intuitively, we can extract more effective features by exchanging information between the downsampled sub-sequences when there are longer look-back windows for such interactions.
SCINet : for the design of SCINet with multiple levels of SCI-Blocks, we also experiment on two variants.
The first variant w/o.
ResConn is obtained by removing the residual connection from the complete SCINet.
The other variant w/o.
Linear removes the decoder (i.e., the fully-connected layer) from the complete model.
As can be observed in Fig. 4, removing the residual connection leads to a significant performance drop.
Besides the general benefit in facilitating the model training, more importantly, the predictability of the original time series is enhanced with the help of residuals.
The fully-connected layer is also critical for prediction accuracy, indicating the effectiveness of w/o ResCon w/o Linear w/o InterLearn
WeightShareComplete
M e a n
A b s o l u t e
E rr o r
M e a n
A b s o l u t e
E rr o r
[Figure]
w/o ResCon w/o Linear w/o InterLearn
WeightShareComplete the decoder in extracting and fusing the most relevant temporal information according to the given supervision for prediction
We also conduct comprehensive ablation studies on the impact of K (number of stacks) and L (number of levels), and the selection of operator in the interact learning mechanism.
These results are shown in the Appendix B.2 due to space limitation.
Limitations and Future Work
In this paper, we mainly focus on TSF problem for the regular time series collected at even intervals of time and ordered chronologically.
However, in real-world applications, the time series might contain noisy data, missing data or collected at irregular time intervals, which is referred to as irregular time series .
The proposed SCINet is relatively robust to the noisy data thanks to the progressive downsampling and interactive learning procedure, but it might be affected by the missing data if the ratio exceeds a certain threshold, wherein the downsampling-based multi-resolution sequence representation in SCINet may introduce biases, leading to poor prediction performance.
The proposed downsampling mechanism may also have difficulty handling data collected at irregular intervals.
We plan to take the above issues into consideration in the future development of SCINet.
Moreover, this work focuses on the deterministic time series forecasting problem.
Many application scenarios require probabilistic forecasts, and we plan to revise SCINet to generate such prediction results.
Finally, while SCINet generates promising results for spatial-temporal time series without explicitly modeling spatial relations, the forecasting accuracy could be further improved by incorporating dedicated spatial models.
We plan to investigate such solutions in our future work.
Conclusion
In this paper, we propose a novel neural network architecture, sample convolution and interaction network ( SCINet ) for time series modeling and forecasting, motivated by the unique properties of time series data compared to generic sequence data.
The proposed SCINet is a hierarchical downsample-convolve-interact structure with a rich set of convolutional filters.
It iteratively extracts and exchanges information at different temporal resolutions and learns an effective representation with enhanced predictability.
Extensive experiments on various real world TSF datasets demonstrate the superiority of our model over state-of-the-art methods
Acknowledgments and Disclosure of Funding
This work was supported in part by Alibaba Group Holding Ltd. under Grant No. TA2015393.
We thank the anonymous reviewers for their constructive comments and suggestions.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See section(c)
Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the Appendix and link in abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See the Appendix (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the Appendix 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Detail seen in Appendix.
(b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix
In this appendix, we first introduce the datasets and evaluation metrics used in the experiments in
Section A.
Then, we provide extra experimental results in Section B.
In Section C, we present details of network design, training scheme, and hyper-parameter tuning.
A Datasets and Evaluation Metrics
We conduct experiments on 11 popular time series datasets: (1) Electricity Transformer Temperature [42] (ETTh(1,2),ETTm1) 3 consists of 2 year electric power data collected from two separated counties of China.
Each data point includes an "oil temperature" value and 6 power load features.
(2) Traffic 4 contains the hourly data describing the road occupancy rates (ranging from 0 to 1 ) that are recorded by the sensors on San Francisco Bay area freeways from 2015 to 2016 ( 48 months in total).
(3) Solar-Energy 5 records the solar power production from 137 PV plants in Alabama
State, which are sampled every 10 minutes in 2016 .
(4) Electricity 6 includes the hourly electricity consumption (kWh) records of 321 clients from 2012 to 2014 .
(5) Exchange-Rate 7 collects the daily exchange rates of 8 foreign countries from 1990 to 2016 .
(6) PeMS 8 contains four public traffic network datasets ( PEMS03, PEMS04, PEMS07 and PEMS08 ) which are respectively constructed from Caltrans Performance Measurement System (PeMS) of four districts in California.
The data is aggregated into 5 -minutes windows, resulting in 12 points per hour and 288 points per day.
A.1 Electricity Transformer Temperature (ETT)
For data pre-processing, we perform zero-mean normalization, i.e., X ′ =
( X − mean ( X )) /std
( X ) , where mean ( X ) and std ( X ) are the mean and the standard deviation of historical time series, respectively.
We use Mean Absolute Errors (MAE) [17] and Mean Squared Errors (MSE) [26] for model comparison.
Besides, the train, validation and test sets contain 12 , 4 and 4 months data, respectively.
[Equation]
where ˆ x i is the model’s prediction, and x i is the ground-truth.
τ is the length of the prediction horizon.
A.2 PeMS
Following [17], the data is pre-processed using zero-mean normalization and we use Root Mean
Squared Errors (RMSE) and Mean Absolute Percentage Errors (MAPE) as evaluation metrics on this dataset.
[Equation]
A.3 Traffic, Solar-Energy, Electricity and Exchange-Rate
In our experiments, the length of the look-back window T for these datasets is 168 , and we trained independent models for different length of future horizon (i.e., τ = 3 , 6 , 12 , 24 ).
We use Root Relative
Squared Error (RSE) and Empirical Correlation Coefficient (CORR)
to evaluate the performance of the TSF models on these datasets following [19], which are calculated as follows:
[Equation]
where X and ˆ X are the ground-truth and model’s prediction, respectively.
d is the number of variates.
B Extra Experimental Results
In this section, we first add error bars on different forecasting steps T, and also conduct empirical studies on ETTh1 and PEMS datasets to show the impact of different parameter and operator combinations in SCI-Block .
B.1 Error Bars Evaluation
Since deep models for time series forecasting may be influenced by different random initialization, we report our results with 5 runs on the ETTh1 dataset.
From Table 8, we show the standard deviation (Std.) is basically 2% to 3% of the mean values, indicating SCINet is robust towards different initialization.
[Table]
B.2 Evaluation on the Impact of K and L
We conduct experiments on ETTh1 dataset (with the multivariate experimental setting) to evaluate the impact of K (number of stacks) and L (number of levels)
[Table]
under various look-back window sizes
T .
The prediction horizon is fixed to be 24 .
As can be observed from Table
[Table]
when fixing K =
[Table]
larger L leads to better prediction accuracy for the cases with larger T ( T = 128 or 192 ).
This is because we could further extract essential information from coarser temporal resolutions with deeper levels in the SCINet when T is large.
As for the number of stacks K , when fixing L = 3 , if T is small (e.g. T =24 or 48 ), we find that increasing K would improve prediction accuracy.
This is because, under such circumstances, the information extracted from a single SCINet is insufficient.
By stacking more SCINets, we effectively increase the representation learning capability of the model, which facilitates extracting more robust temporal relations for the forecasting task.
However, when T is large (e.g., 192), a shallow stack can already well capture the temporal dependencies for the time series.
Under such circumstances, using deeper stacks may suffer from overfitting issues with the increase of parameters, which degrades the performance in the inference stage.
From Table 9, we can observe a clear trade-off between L and K .
Moreover, the performance variation under different T also indicates the importance of the look-back window selection for forecasting tasks.
While T is typically pre-determined based on domain knowledge about the time
[Table]
series data, based on our empirical study, L ≤ 5 and K ≤ 3 are usually sufficient and tuning these hyperparameters does not incur much effort.
B.3 Empirical Study on Operator Selection
In interactive-learning equation,
[Equation]
the operators can be either "addition" or "subtraction".
Although the model can learn the operation adaptively during training
[Table]
the parameter initialization would affect the final performance.
As shown in the following table the impact of operator settings is minor.
[Table]
C Reproducibility
Our code is implemented with PyTorch.
All the experiments are conducted on an Nvidia Tesla V
SXM2 GPU (32GB memory), which is sufficient for all our experiments.
Structure of the network modules ϕ , ρ , ψ , and η in SCI-Block: As shown in Fig.
5, ϕ , ρ , ψ , and η use the same network architecture.
First, the replication padding is used to keep the border shrunk caused by the convolution operation.
Then, a 1d convolutional layer with kernel size k is applied to extend the input channel C to h * C and followed with LeakyRelu and Dropout.
h means a scale of the hidden size.
Next, the second 1d convolutional layer with kernel size k is to recover the channel h * C to the input channel C .
The stride of all the convolutions is 1 .
We use a LeakyRelu activation after the first convolutional layer because of its sparsity properties and a reduced likelihood of vanishing gradient.
We apply a Tanh activation after the second convolutional layer since it can keep both positive and negative features into [-1, 1].
Loss Function
To enhance the performance in single-step (short-term time series forecasting Sec. 4.2) forecasting, we revise the loss function of the last SCINet in the stacked SCINet with K ( K ≥ 1 ).
The loss function contains two parts:
[Equation]
ReplicationPad1d
Conv1d (in_channels =
C , out_channels = h*C , Kernel = k )
LeakyRelu (negative_slop = 0.01)
Dropout ( d )
Tanh
Conv1d (in_channels = h*C , out_channels =
C , Kernel = k )
Input : X ( C, L )
Output: 푋 ’ (C, L)
For the last stack K , we introduce a balancing parameter λ ∈ (0 , 1) for the value of the last time-step 9 :
[Equation]
Therefore, the total loss of the stacked SCINet can be written as:
[Equation]
Training details: For all datasets, we fix the random seed to be 4321, and train the model forepochs at most.
The reported results on the test set are based on the model that achieves the best performance on the validation set.
Hyper-parameter tuning: We conduct a grid search over all the essential hyper-parameters on the held-out validation set of the datasets.
The detailed hyper-parameter configurations of ETT are shown in Table 11 10 .
Besides, the parameters of the four datasets in PeMS are presented in Table13.
The
Traffic, Solar-Energy, Electricity and Exchange-rate are shown in Table 12.
Notably, we only apply the weighted loss to the Solar and Exchange-rate data since they show less auto-correlation [19], which indicates the temporal correlation of the distant time-stamp cannot be well modelled by a general L1 loss.
Moreover, to build a non-causal TCN 11 in the paper, we only need to remove the chomps in the code and make the padding equal to the dilation.
all
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]