SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction
Anonymous Author(s)
AffiliationAddress email
Abstract
One unique property of time series is that the temporal relations are largely preserved after downsampling into two sub-sequences.
By taking advantage of thisproperty, we propose a novel neural network architecture that conducts sampleconvolution and interaction for temporal modeling and forecasting, named SCINet .
Specifically, SCINet is a recursive downsample-convolve-interact architecture.
Ineach layer, we use multiple convolutional filters to extract distinct yet valuabletemporal features from the downsampled sub-sequences or features.
By combining these rich features aggregated from multiple resolutions, SCINet effectivelymodels time series with complex temporal dynamics.
Experimental results showthat SCINet achieves significant forecasting accuracy improvements over bothexisting convolutional models and Transformer-based solutions across variousreal-world time series forecasting datasets.
Our codes and data are available athttps://anonymous.4open.science/r/SCINet-2588 .
Introduction
Time series forecasting (TSF) enables decision-making with the estimated future evolution of metrics or events, thereby playing a crucial role in various scientific and engineering fields such ashealthcare [1], energy management [38], traffic flow [38], and financial investment [9], to name a few.
There are mainly three kinds of deep neural networks used for sequence modeling, and they are allapplied for time series forecasting [21]: (i).
recurrent neural networks (RNNs) [11]; (ii).
Transformerbased models [33]; and (iii).
temporal convolutional networks (TCN) [3].
Despite the promising results of TSF methods based on these generic models, they do not consider thespecialty of time series data during modeling.
For example, one unique property of time series is thatthe temporal relations (e.g., the trend and the seasonal components of the data) are largely preservedafter downsampling into two sub-sequences.
Consequently, by recursively downsampling the timeseries into sub-sequences, we could obtain a rich set of convolutional filters to extract dynamictemporal features at multiple resolutions.
Motivated by the above, in this paper, we propose a novel neural network architecture for time seriesmodeling and forecasting, named sample convolution and interaction network ( SCINet ).
The maincontributions of this paper are as follows:
• We propose SCINet, a hierarchical downsample-convolve-interact TSF framework thateffectively models time series with complex temporal dynamics.
By iteratively extracting
and exchanging information at multiple temporal resolutions, an effective representation withenhanced predictability can be learned, as verified by its comparatively lower permutation
entropy (PE) [14].
[Figure]
• We design the basic building block, SCI-Block , for constructing SCINet, which downsamples
the input data/feature into two sub-sequences, and then extracts features of each subsequence using distinct convolutional filters.
To compensate for the information loss
during the downsampling procedure, we incorporate interactive learning between the twoconvolutional features within each SCI-Block.
Extensive experiments on various real-world TSF datasets show that our model consistently outperforms existing TSF approaches by a considerable margin.
Moreover, while SCINet does not explicitlymodel spatial relations, it achieves competitive forecasting accuracy on spatial-temporal TSF tasks.
Related Work and Motivation
The time series forecasting problem is defined as: Given a long time series X ∗ and a look-back windowof fixed length T , at timestamp t , time series forecasting is to predict ˆ X t +1:
t + τ = { x t +1 , ..., x t + τ }based on the past T steps X t − T
+1:
t = { x t − T +1 , ..., x t } .
Here, τ is the length of the forecast horizon,x t ∈ R
d is the value at time step t , and d is the number of variates.
For simplicity, in the followingwe will omit the subscripts, and use X and ˆ X to represent the historical data and the forecasted data,respectively.
2.1 Related Work
Traditional time series forecasting methods such as the autoregressive integrated moving average(ARIMA) model [7] and Holt-Winters seasonal method [12] have theoretical guarantees.
However,they are mainly applicable for univariate forecasting problems, restricting their applications tocomplex time series data.
With the increasing data availability and computing power in recent years,it is shown that deep learning-based TSF techniques have the potential to achieve better forecastingaccuracy than conventional approaches [21, 26].
Earlier RNN-based TSF methods [28, 29] summarize the past information compactly in the internalmemory states that are recursively updated with new inputs at each time step, as shown in Fig. 1(a).
The gradient vanishing/exploding problems and the inefficient training procedure greatly restrict theapplication of RNN-based models.
In recent years, Transformer-based models [33] have taken the place of RNN models in almost allsequence modeling tasks, thanks to the effectiveness and efficiency of the self-attention mechanisms.
Various Transformer-based TSF methods (see Fig. 1(b)) are proposed in the literature [19, 20, 34, 22].
These works typically focus on the challenging long-term time series forecasting problem, takingadvantage of their remarkable long sequence modeling capabilities.
Another popular type of TSF model is the so-called temporal convolutional network [6, 3, 30, 36, 24],wherein convolutional filters are used to capture local temporal features (see Fig. 1(c)).
The proposed
SCINet is also constructed based on temporal convolution.
However, our method has several keydifferences compared with the TCN model based on dilated causal convolution, as discussed in thefollowing.
2.2 Rethinking Dilated Causal Convolution for Time Series Modeling and Forecasting
The local correlation of time series data is reflected in the continuous changes within a time slot, andconvolutional filters can effectively capture such local features.
Consequently, convolutional neuralnetworks are explored in the literature for time series modeling and forecasting.
In particular, dilatedcausal convolution (DCS) is the current de facto method used in this respect.
DCS was first proposed for generating raw audio waveforms in WaveNet [25].
Later, [3] simplifiesthe WaveNet architecture to the so-called temporal convolutional networks (see Fig.
1 (c)).
TCNconsists of a stack of causal convolutional layers with exponentially enlarged dilation factors, whichcan achieve a large receptive field with just a few convolutional layers.
Over the years, TCN has beenwidely used in all kinds of time series forecasting problems and achieve promising results [36, 30].
Moreover, convolutional filters can work seamlessly with graph neural networks (GNNs) to solvevarious spatial-temporal TSF problems.
With causal convolutions in the TCN architecture, an output i is convolved only with the i th andearlier elements in the previous layer.
While causality should be kept in forecasting tasks, the potential“future information leakage" problem exists only when the output and the input have temporal overlaps.
In other words, causal convolutions should be applied only in autoregressive forecasting, wherein theprevious output serves as the input for future prediction.
When the predictions are completely basedon the known inputs in the look-back window, there is no need to use causal convolutions.
We cansafely apply normal convolutions on the look-back window for forecasting.
More importantly, the dilated architecture in TCN has two inherent limitations:
• A single convolutional filter is shared within each layer.
Such a unified convolutional kerneltends to extract the average temporal features from the data/features in the previous layer.
However, complex time series may contain substantial temporal dynamics.
Hence, it is
essential to extract distinct yet valuable features with a rich set of convolutional filters.
• While the final layer of the TCN model has the global view of the entire look-back window,
the effective receptive fields of the intermediate layers (especially those close to the inputs)
are limited, causing temporal relation loss during feature extraction.
The above limitations of the TCN architecture motivate the proposed SCINet design, as detailed inthe following section.
SCI-Block
SCI-Block SCI-Block
SCI-Block SCI-Block
SCI-Block … … … … …
Input: !
𝐗 !"
# ( 𝐗 if 𝑘 = 1 ) #Level= #Level= #Level= L
SCINet
Input: 𝐗 …
SCINet
Output: !
𝐗 $ exp
Split 𝐅 ()) 𝐅 *+*,
Input: 𝐅 𝜙 𝜓 𝜌 𝜂 𝐅 ()).
𝐅 *+*,.
𝐅 ())/
𝐅 *+*,/ exp
InteractiveLearning
Concat
Concat & Realign
Onput: !
𝐗 !
FC
SCINet
Concat #Stack =
#Stack= (c) Stacked SCINet (b)
SCINet (a) SCI-Block ℒ !
#Stack= K … ℒ " ℒ #
SCINet: Sample Convolution and Interaction Network
[Figure]
SCINet adopts an encoder-decoder architecture.
The encoder is a hierarchical convolutional network
[Figure]
that captures dynamic temporal dependencies at multiple resolutions with a rich set of convolutionalfilters.
As shown in Fig.
2(a), the basic building block, SCI-Block (Section 3.1), downsamples theinput data or feature into two sub-sequences and then processes each sub-sequence with a set ofconvolutional filters to extract distinct yet valuable temporal features from each part.
To compensatefor the information loss during downsampling, we incorporate interactive learning between the twosub-sequences.
Our SCINet (Section 3.2) is constructed by arranging multiple SCI-Blocks into abinary tree structure (Fig. 2(b)).
A distinctive advantage of such design is that each SCI-Block hasboth local and global views of the entire time series, thereby facilitating the extraction of usefultemporal features.
After all the downsample-convolve-interact operations, we realign the extractedfeatures into a new sequence representation and add it to the original time series for forecasting witha fully-connected network as the decoder.
To facilitate extracting complicated temporal patterns, wecould further stack multiple SCINets and apply intermediate supervision to get a Stacked SCINet(Section 3.3), as shown in Fig. 2(c).
3.1 SCI-Block
[Figure]
The SCI-Block (Fig. 2(a)) is the basic module of the SCINet, which decomposes the input feature Finto two sub-features F ′ odd and F ′ even through the operations of Spliting and Interactive-learning .
The Splitting procedure downsamples the original sequence F into two sub-sequences F even and
F odd by separating the even and the odd elements, which are of coarser temporal resolution butpreserve most information of the original sequence.
Next, we use different convolutional kernels to extract features from F even and F odd .
As thekernels are separate, the extracted features from them would contain distinct yet valuable temporalrelations with enhanced representation capabilities.
To compensate for potential information loss withdownsampling, we propose a novel interactive-learning strategy to allow information interchangebetween the two sub-sequences by learning affine transformation parameters from each other.
Asshown in Fig.
2 (a), the interactive learning procedure consists of two steps.
First, F even and F odd are projected to hidden states with two different 1D convolutional modules ϕand ψ , respectively, and transformed to the formats of exp and interact to the F even and F odd withthe element-wise product (see Eq.
This can be viewed as performing scaling transformation on
F even and F odd , where the scaling factors are learned from each other using neural network modules.
Here, ⊙ is the Hadamard product or element-wise production.
[Equation] [Equation]
Second, as shown in Eq.
(14), the two scaled features F seven and F sodd are further projected toanother two hidden states with the other two 1D convolutional modules ρ and η , and then added toor subtracted from 1 F seven and F sodd .
The final outputs of the interactive learning module are twoupdated sub-features F ′ even and F ′ odd .
The default architectures of ϕ , ψ , ρ and η are shown in thesupplementary materials.
Compared to the dilated convolutions used in the TCN architecture, the proposed downsampleconvolve-interact architecture achieves an even larger receptive field at each convolutional layer.
Moreimportantly, unlike TCN that employs a single shared convolutional filter at each layer, significantlyrestricting its feature extraction capabilities, SCI-Block aggregates essential information extractedfrom the two downsampled sub-sequences that have both local and global views of the entire timeseries.
3.2 SCINet
With the SCI-Blocks presented above, we construct the SCINet by arranging multiple SCI-Blockshierarchically and get a tree-structured framework, as shown in Fig.
2 (b).
There are 2 l SCI-Blocks at the l -th level, where l = 1 , . . .
, L is the index of the level, and L isthe total number of levels.
Within the k -th SCINet of the stacked SCINet (Section 3.3), the inputtime series X (for k =1 ) or feature vector
ˆ X k − 1 =
{ ˆ x k − 1, ..., ˆ x k − 1 τ } (for k > 1 ) is gradually downsampled and processed by SCI-Blocks through different levels, which allows for effective featurelearning of different temporal resolutions.
In particular, the information from previous levels will begradually accumulated, i.e., the features of the deeper levels would contain extra finer-scale temporalinformation transmitted from the shallower levels.
In this way, we can capture both short-term andlong-term temporal dependencies in the time series.
After going through L levels of SCI-Blocks, we rearrange the elements in all the sub-features byreversing the odd-even splitting operation and concatenate them into a new sequence representation.
It is then added to the original time series through a residual connection [10] to generate a newsequence with enhanced predictability.
Finally, a simple fully-connected network is used to decodethe enhanced sequence representation into ˆ X k
= { ˆ x k 1 , ..., ˆ x kτ } .
3.3 Stacked SCINet
When there are sufficient training samples, we could stack K layers of SCINets to achieve even betterforecasting accuracy (see Fig.
2 (c)), at the cost of a more complex model structure.
Specifically, we apply intermediate supervision [4] on the output of each SCINet using the groundtruth values, to ease the learning of the intermediate temporal features.
The output of the k -thintermediate SCINet, ˆ X k with length τ , is concatenated with part of the input X t − ( T − τ )+1: t torecover the length to the original input and feeded as input into the ( k + 1) -th
SCINet, wherek = 1 , . .
, K − 1 , and K is the total number of the SCINets in the stacked structure.
The output ofthe K -th
SCINet, ˆ X K , is the final forecasting results.
3.4 Loss Function
To train a stacked SCINet with K ( K ≥ 1 ) SCINets, the loss of the k -th prediction results is calculatedas the L1 loss between the output of the k -th SCINet and the ground-truth horizontal window to bepredicted:
[Equation]
The total loss of the stacked SCINet can be written as:
[Equation]
3.5 Complexity Analysis
Thanks to the downsampling procedure, the neurons at each convolutional layer of SCINet havea larger receptive field than those of TCN.
More importantly, the set of rich convolutional filtersin SCINet enable flexible extraction of temporal features from multiple resolutions.
Consequently,
SCINet usually does not require downsampling the original sequence to the coarsest level for effectiveforecasting.
Given the look-back window size T , TCN generally requires ⌈ log 2 T
⌉ layers when thedilation factor is 2, while the number of layers L in SCINet could be much smaller than log 2 T .
Ourempirical study shows that the best forecasting accuracy is achieved with L ≤ 5 in most cases evenwith large T (e.g., 168 ).
As for the number of stacks K , our empirical study also shows that K ≤ 3would be sufficient.
Consequently, the computational cost of SCINet is usually on par with that of the TCN architecture.
The worst-case time complexity is O
( T log T ) , much less than that of vanilla Transformer-basedsolutions: O ( T 2 ) .
Experiments
In this section, we show the quantitative and qualitative comparisons with the state-of-the-art modelsfor time series forecasting.
We also present a comprehensive ablation study to evaluate the effectiveness of different components in SCINet.
More details on datasets , evaluation metrics , datapre-processing , experimental settings , network structures and their hyper-parameters are shown inthe supplementary materials.
4.1 Datasets
We conduct experiments on 9 popular time series datasets: (1) Electricity Transformer Temperature [38] (ETTh1) (2) Traffic (3) Solar-Energy (4) Electricity (5) Exchange-Rate (6) PeMS ( PEMS03,
PEMS04, PEMS07 and PEMS08 ).
A brief description of these datasets is listed in Table 1.
All theexperiments on these datasets in this section are conducted under multi-variate TSF setting.
To make a fair comparison, we follow existing experimental settings, and use the same evaluationmetrics as the original publications [15, 23, 37, 17] in each dataset.
[Table]
4.2 Results and Analyses
Table 2, Table 3 and Table 4 provide the main experimental results of SCINet.
We observe that
SCINet shows superior performance than other TSF models on various tasks, including short-term,long-term and spatial-temporal time series forecasting.
Short-term Time Series Forecasting: we evaluate the performance of the SCINet in short-term TSFtasks with other baseline methods on Traffic , Solar-Energy , Electricity and Exchange-Rate datasets.
The experimental setting is the same as [17], which uses the input length of 168 to forecast differentfuture horizons { 3 , 6 , 12 , 24 } .
As can be seen in Table 2, the proposed SCINet outperforms existing RNN/TCN-based (LSTNet [17],
TPA-LSTM [31], TCN [3], TCN † ) and Transformer-based [34, 38, 33] TSF solutions in most cases,especially for the Solar-Energy and Exchange-Rate datasets.
Note that, TCN † denotes a variant of
TCN wherein causal convolutions are replaced by normal convolutions, and improves the original
TCN across all the datasets, which supports our claim in Sec. 2.2.
Moreover, we can also observe thatthe Transformer-based methods have poor performance in this task.
For short-term forecasting, therecent data points are typically more important for accurate forecasting.
However, the permutationinvariant self-attention mechanisms used in Transformer-based methods do not pay much attention tosuch critical information.
In contrast, the general sequential models (RNN/TCN) can formulate iteasily, showing quite competitive results in short-term forecasting.
[Figure]
[Table]
Forecasting: many real-world applications also require to predict long-termevents.
Therefore, we conduct the experiments on ETT , Exchange Rate, Electricity and Trafficdatasets to evaluate the performance of SCINet on long-term TSF tasks.
In this experiment, we onlycompare SCINet with Transformer-based methods [34, 16, 19, 38, 33], since they are more popularin recent long-term TSF research.
As can be seen from Table 3, the SCINet achieves state-of-the-art performances in most benchmarksand prediction length settings (except ETTh1 (720)).
Overall, SCINet yields 21 % average improvements on MSE among the above settings.
In particular, for Exchange-Rate, compared to previousstate-of-the-art results, SCINet gives average 65 % improvements on MSE.
We attribute it to that theproposed SCINet can better capture both short ( local temporal dynamics )- and long ( trend, seasonality )-term temporal dependencies to make an accurate prediction in long-term TSF.
Besides, comparedwith the vanilla Transformer-based methods [16, 19, 38], the newly-proposed Transformer-basedforecasting model Autoformer [34] achieves the second best performance in all experimental settings and also surpasses the SCINet in ETTh1(720).
This is because, Autoformer incorporates moreprior knowledge about the time series data.
It focuses on modeling seasonal patterns and conductsself-attention at the sub-series level (instead of the raw data), which is much better in extractinglong-term temporal patterns than vanilla Transformer-based methods.
[Table]
Spatial-temporal Time Series Forecasting: besides the general TSF task, there is also a largecategory of data related to spatial-temporal forecasting.
For example, traffic datasets PeMS [8]are complicated spatial-temporal time series and they have been investigated for decades.
Mostrecent approaches [36, 32, 2, 13, 18] use graph neural networks to capture spatial relations whilemodeling temporal dependencies via conventional TCN or LSTM architectures.
We follow thesame experimental settings as the above works.
As shown in Table 4, these GNN-based methodsgenerally perform better than pure RNN or TCN-based methods.
However, SCINet still achievesbetter performance without sophisticated spatial relation modeling, which further proves the superbtemporal modeling capabilities of SCINet.
[Table]
Predictability estimation: inspired by [14, 27], we use permutation entropy ( PE ) [5] to measure thepredictability of the original input and the enhanced representation learnt by SCINet.
Time series withlower PE values are regarded as less complex, thus theoretically easier to predict 2 .
The PE values ofthe original time series and the corresponding enhanced representations are shown in Table
[Table] [Table]
As can be observed
[Table]
the enhanced representations learnt by SCINet indeed have lower PE valuescompared with the original inputs, which indicates that it is easier to predict the future from theenhanced representations using the same forecaster. because the prediction accuracy also depends on many other factors, such as the available data for training, the trend and seasonality elements of the time series data, as well as the predictive model.
Visualization: we present the qualitative results on some randomly selected sequences of the ETTh1dataset in Fig.
3, which clearly shows the superior performance of of SCINet in obtaining the trendand seasonality of time series.
[Figure]
4.3 Ablation studies
[Figure] To evaluate the impact of each main component used in SCINet [Figure] we experiment on several modelvariants on two datasets:
ETTh1 and PEMS08 .
SCIBlock : we first set the number of stacks K = 1 and the number of SCINet levels L = 3 .
For the SCI-Block design, to validate the effectiveness of the interactive learning and the distinctconvolution weights for processing the sub-sequences, we experiment on two variants, namely w/o.
InterLearn and WeightShare .
The w/o.
InterLearn is obtained by removing the interactive-learningprocedure described in Eq.
(1) and (14).
In this case, the two sub-sequences would be updated using
F ′ odd = ρ ( ϕ ( F odd )) and F ′ even =
η ( ψ ( F even )) .
For WeightShare , the modules ϕ , ρ , ψ , and η sharethe same weight.
The evaluation results in Fig.
4 show that both interactive learning and distinct weights are essential,as they improve the prediction accuracies of both datasets at various prediction horizons.
At the sametime, comparing Fig. 4(a) with Fig. 4(b), we can observe that interactive learning is more effective forcases with longer look-back window sizes.
This is because, intuitively, we can extract more effectivefeatures by exchanging information between the downsampled sub-sequences when there are longerlook-back windows for such interactions. SCINet : for the design of SCINet with multiple levels of SCI-Blocks, we also experiment on twovariants.
The first variant w/o.
ResConn is obtained by removing the residual connection from thecomplete SCINet. The other variant w/o. Linear removes the decoder (i.e., the fully-connectedlayer) from the complete model. As can be observed in Fig. 4, removing the residual connection w/o ResCon w/o Linear w/o InterLearn
WeightShareComplete
M e a n
A b s o l u t e
E rr o r
M e a n
A b s o l u t e
E rr o r
[Figure]
w/o ResCon w/o Linear w/o InterLearn
WeightShareComplete leads to a significant performance drop.
Besides the general benefit in facilitating the model training,more importantly, the predictability of the original time series is enhanced with the help of residuals.
The fully-connected layer is also critical for prediction accuracy, indicating the effectiveness ofthe decoder in extracting and fusing the most relevant temporal information according to the givensupervision for prediction.
We also conduct comprehensive ablation studies on the impact of K (number of stacks) and L(number of levels), and the selection of operator in the interact learning mechanism.
These results areshown in the supplementary materials due to space limitation.
Limitations and Future Work
In this paper, we mainly focus on TSF problem for the regular time series collected at even intervals oftime and ordered chronologically.
However, in real-world applications, the time series might containnoisy data, missing data or collected at irregular time intervals, which is referred to as irregulartime series .
The proposed SCINet is relatively robust to the noisy data thanks to the progressivedownsampling and interactive learning procedure, but it might be affected by the missing data ifthe ratio exceeds a certain threshold, wherein the downsampling-based multi-resolution sequencerepresentation in SCINet may introduce biases, leading to poor prediction performance.
The proposeddownsampling mechanism may also have difficulty handling data collected at irregular intervals.
Weplan to take the above issues into consideration in the future development of SCINet.
Moreover, this work focuses on the deterministic time series forecasting problem.
Many applicationscenarios require probabilistic forecasts, and we plan to revise SCINet to generate such predictionresults.
Finally, while SCINet generates promising results for spatial-temporal time series without explicitlymodeling spatial relations, the forecasting accuracy could be further improved by incorporatingdedicated spatial models.
We plan to investigate such solutions in our future work.
Conclusion
In this paper, we propose a novel neural network architecture, sample convolution and interactionnetwork ( SCINet ) for time series modeling and forecasting, motivated by the unique propertiesof time series data compared to generic sequence data.
The proposed SCINet is a hierarchicaldownsample-convolve-interact structure with a rich set of convolutional filters.
It iteratively extractsand exchanges information at different temporal resolutions and learns an effective representationwith enhanced predictability.
Extensive experiments on various real-world TSF datasets demonstratethe superiority of our model over state-of-the-art methods.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See section 5(c)
Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See theAppendix and link in abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See the Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the Appendix4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Detail seen inAppendix.(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix
In this appendix, we first introduce the datasets and evaluation metrics used in the experiments in
Section A.
Then, we provide extra experimental results in Section B.
In Section C, we present detailsof network design, training scheme, and hyper-parameter tuning.
A Datasets and Evaluation Metrics
We conduct experiments on 9 popular time series datasets: (1) Electricity Transformer Temperature [38] (ETTh1) 3 consists of 2 year electric power data collected from two separated counties of
China.
Each data point includes an "oil temperature" value and 6 power load features.
(2) Traffic 4contains the hourly data describing the road occupancy rates (ranging from 0 to 1 ) that are recordedby the sensors on San Francisco Bay area freeways from 2015 to 2016 ( 48 months in total).
(3) SolarEnergy 5 records the solar power production from 137 PV plants in Alabama State, which are sampledevery 10 minutes in 2016 .
(4) Electricity 6 includes the hourly electricity consumption (kWh) recordsof 321 clients from 2012 to 2014 .
(5) Exchange-Rate 7 collects the daily exchange rates of 8 foreigncountries from 1990 to 2016 .
(6) PeMS 8 contains four public traffic network datasets ( PEMS03,
PEMS04, PEMS07 and PEMS08 ) which are respectively constructed from Caltrans Performance
Measurement System (PeMS) of four districts in California.
The data is aggregated into 5 -minuteswindows, resulting in 12 points per hour and 288 points per day.
A.1 Electricity Transformer Temperature (ETT)
For data pre-processing, we perform zero-mean normalization, i.e., X ′ =
( X − mean ( X )) /std
( X ) ,where mean ( X ) and std ( X ) are the mean and the standard deviation of historical time series,respectively.
We use Mean Absolute Errors (MAE) [15] and Mean Squared Errors (MSE) [23] formodel comparison.
Besides, the train, validation and test sets contain 12 , 4 and 4 months data,respectively.
[Equation]
where ˆ x i is the model’s prediction, and x i is the ground-truth.
τ is the length of the predictionhorizon.
A.2 PeMS
Following [15], the data is pre-processed using zero-mean normalization and we use Root Mean
Squared Errors (RMSE) and Mean Absolute Percentage Errors (MAPE) as evaluation metrics on thisdataset.
[Equation]
[Equation]
A.3 Traffic, Solar-Energy, Electricity and Exchange-Rate
Traffic 9 contains the hourly data describing the road occupancy rates (ranging from 0 to 1 ) that arerecorded by the sensors on San Francisco Bay area freeways from 2015 to 2016 ( 48 months in total).
Solar-Energy 10 records the solar power production from 137 PV plants in Alabama State, which aresampled every 10 minutes in 2016 .
Electricity 11 includes the hourly electricity consumption (kWh)records of 321 clients from 2012 to 2014 .
Exchange-Rate 12 collects the daily exchange rates of 8foreign countries from 1990 to 2016 .
In our experiments, the length of the look-back window T for the above datasets is 168 , and wetrained independent models for different length of future horizon (i.e., τ = 3 , 6 , 12 , 24 ).
We use
Root Relative Squared Error (RSE) and Empirical Correlation Coefficient (CORR)
to evaluate theperformance of the TSF models on these datasets following [17], which are calculated as follows:
[Equation]
where X and ˆ X are the ground-truth and model’s prediction, respectively.
d is the number of variates.
B Extra Experimental Results
In this section, we first conduct experiments on 4 extra forecasting datasets to see the performance ofthe SCINet in single-step forecasting task.
Then, we add empirical study on PEMS datasets to showthe impact of different operator combinations in SCI-Block .
B.1 Performance Comparison on Extra Datasets.
To further evaluate the performance of the proposed SCINet , we also conduct the experiments on 4single-step forecasting datasets [17], namely Traffic , Solar-Energy , Electricity and Exchange-Rate .
The task is defined as follows:
Given a long time series X ∗ and a look-back window of fixed length T , at timestamp t , the single-stepforecasting is to predict the future value ˆ X t
+ τ : t
+ τ = { x t + τ } .
Here, τ is the length of the forecasthorizon, x t ∈ R d is the value at time step t , and d is the number of variates.
Loss Function
To enhance the performance in single-step forecasting, we revise the loss function of the last SCINetin the stacked SCINet with K ( K ≥ 1 ).
The loss function contains two parts:
[Equation]
For the last stack K , we introduce a balancing parameter λ ∈ (0 , 1) for the value of the lasttime-step 13 :
[Equation]
Therefore, the total loss of the stacked SCINet can be written as:
[Equation]
Results and Analyses
[Table]
As can be seen in Table ??
, SCINet outperforms existing TSF solutions in most cases, especially forthe Solar-Energy datasets.
At the same time, SCINet is slightly inferior to state-of-the-art spatialtemporal models MTGNN [35] and TPA-LSTM [31] in some cases.
We attribute it to the factthat these datasets have relatively strong spatial relations and hence the dedicated spatial modelingarchitectures in these techniques contribute to their high performance.
[Table]
B.2 Evaluation on the Impact of K and L
We conduct experiments on ETTh1 dataset (with the multivariate experimental setting) to evaluatethe impact of K (number of stacks) and L (number of levels), under various look-back window sizes
T .
The prediction horizon is fixed to be 24 .
As can be observed from Table 6, when fixing K = 1 , larger L leads to better prediction accuracyfor the cases with larger T ( T = 128 or 192 ).
This is because we could further extract essentialinformation from coarser temporal resolutions with deeper levels in the SCINet when T is large.
As for the number of stacks K , when fixing L = 3 , if T is small (e.g. T =24 or 48 ), we find thatincreasing K would improve prediction accuracy.
This is because, under such circumstances, theinformation extracted from a single SCINet is insufficient.
By stacking more SCINets, we effectivelyincrease the representation learning capability of the model, which facilitates extracting more robusttemporal relations for the forecasting task.
However, when T is large (e.g., 192), a shallow stack canalready well capture the temporal dependencies for the time series.
Under such circumstances, usingdeeper stacks may suffer from overfitting issues with the increase of parameters, which degrades theperformance in the inference stage.
[Table]
From Table 6, we can observe a clear trade-off between L and K .
Moreover, the performancevariation under different T also indicates the importance of the look-back window selection forforecasting tasks.
While T is typically pre-determined based on domain knowledge about the timeseries data, based on our empirical study, L ≤ 5 and K ≤ 3 are usually sufficient and tuning thesehyperparameters does not incur much effort.
B.3 Empirical Study on Operator Selection
In interactive-learning equation,
[Equation]
the operators can be either "addition" or "subtraction".
Although the model can learn the operationadaptively during training, the parameter initialization would affect the final performance.
As shownin the following table the impact of operator settings is minor.
C Reproducibility
Our code is implemented with PyTorch.
{+All the experiments are conducted on an Nvidia Tesla V
+}
SXM2 GPU (32GB memory), which is sufficient for all our experiments.
Operators
PEMS
PEMS
PEMS
PEMS08 MAE +, + 15.
+, - 15.
-, + 15.
ReplicationPad1d
[Figure]
[Figure]
[Figure]
Tanh
[Figure]
Input :
[Figure]
[Figure]
Structure of the network modules ϕ , ρ , ψ , and η in SCI-Block: As shown in Fig.
5, ϕ , ρ , ψ , and ηuse the same network architecture.
First, the replication padding is used to keep the border shrunkcaused by the convolution operation.
Then, a 1d convolutional layer with kernel size k is applied toextend the input channel C to h * C and followed with LeakyRelu and Dropout.
h means a scale of thehidden size.
Next, the second 1d convolutional layer with kernel size k is to recover the channel h
* Cto the input channel C .
The stride of all the convolutions is 1 .
We use a LeakyRelu activation afterthe first convolutional layer because of its sparsity properties and a reduced likelihood of vanishinggradient.
We apply a Tanh activation after the second convolutional layer since it can keep bothpositive and negative features into [-1, 1].
Training details: For all datasets, we fix the random seed to be 4321, and train the model for 150epochs at most.
The reported results on the test set are based on the model that achieves the bestperformance on the validation set.
[Table]
Hyper-parameter tuning: We conduct a grid search over all the essential hyper-parameters onthe held-out validation set of the datasets.
The detailed hyper-parameter configurations of ETT areshown in Table 8.
Besides, the parameters of the four datasets in PeMS are presented in Table10.
The
Traffic, Solar-Energy, Electricity and Exchange-rate are shown in Table 9.
Notably, we only apply theweighted loss to the Solar and Exchange-rate data since they show less auto-correlation [17], whichindicates the temporal correlation of the distant time-stamp cannot be well modelled by a general L1loss.
Moreover, to build a non-causal TCN 14 in the paper, we only need to remove the chomps in thecode and make the padding equal to the dilation.
[Table]
14 https://github.com/locuslab/TCN/issues/
[Table]
0.5 SCINet
L (level)              5 Stacked SCINet
K (stack)
[Table]
[Table]
[Table]