MICN: M ULTI - SCALE L OCAL AND G LOBAL CONTEXT MODELING FOR L ONG - TERM S ERIES F ORECASTING
A BSTRACT
Recently, Transformer-based methods have achieved surprising performance in the field of long-term series forecasting, but the attention mechanism for computing global correlations entails high complexity.
And they do not allow for targeted modeling of local features as CNN structures do.
To solve the above problems, we propose to combine local features and global correlations to capture the overall view of time series (e.g., fluctuations, trends).
To fully exploit the underlying information in the time series, a multi-scale branch structure is adopted to model different potential patterns separately.
Each pattern is extracted with down-sampled convolution and isometric convolution for local features and global correlations, respectively.
In addition to being more effective, our proposed method, termed as Multi-scale Isometric Convolution Network ( MICN ), is more efficient with linear complexity about the sequence length with suitable convolution kernels.
Our experiments on six benchmark datasets show that compared with state-of-the-art methods, MICN yields 17.2% and 21.6% relative improvements for multivariate and univariate time series, respectively.
Code is available at https://github. com/wanghq21/MICN .
I NTRODUCTION
Researches related to time series forecasting are widely applied in the real world, such as sensor network monitoring (Papadimitriou & Yu., 2006), weather forecasting, economics and finance (Zhu & Shasha, 2002), and disease propagation analysis (Matsubara et al., 2014) and electricity forecasting.
In particular, long-term time series forecasting is increasingly in demand in reality.
Therefore, this paper focuses on the task of long-term forecasting.
The problem to be solved is to predict values for a future period: X t + 1 , X t + 2 ,..., X t + T − 1 , X t + T , based on observations from a historical period:
X 1 , X 2 ,..., X t − 1 , X t , and T ≫ t .
As a classic CNN-based model, TCN (Bai et al., 2018) uses causal convolution to model the temporal causality and dilated convolution to expand the receptive field.
It can integrate the local information of the sequence better and achieve competitive results in short and medium-term forecasting (Sen et al., 2019)
(Borovykh et al., 2017).
However, limited by the receptive field size, TCN often needs many layers to model the global relationship of time series, which greatly increases the complexity of the network and the training difficulty of the model.
Transformers (Vaswani et al., 2017) based on the attention mechanism shows great power in sequential data, such as natural language processing (Devlin et al., 2019)
(Brown et al., 2020), audio processing (Huang et al., 2019) and even computer vision (Dosovitskiy et al., 2021)
(Liu et al., 2021b).
It has also recently been applied in long-term series forecasting tasks (Li et al., 2019b)
(Wen et al., 2022) and can model the long-term dependence of sequences effectively, allowing leaps and bounds in the accuracy and length of time series forecasts (Zhu & Soricut, 2021) (Wu et al., 2021b)
(Zhou et al., 2022).
The learned attention matrix represents the correlations between different time points of the sequence and can explain relatively well how the model makes future predictions based on past information.
However, it has a quadratic complexity, and many of the computations between token pairs are non-essential, so it is also an interesting research direction to reduce its computational complexity.
Some notable models include: LogTrans (Li et al., 2019b), Informer (Zhou et al., 2021), Reformer (Kitaev et al., 2020), Autoformer Wu et al.
(2021b)
, Pyraformer (Liu et al., 2021a), FEDformer (Zhou et al., 2022).
However, as a special sequence, time series has not led to a unified modeling direction so far.
In this paper, we combine the modeling perspective of CNNs with that of Transformers to build models from the realistic features of the sequences themselves, i.e., local features and global correlations.
Local features represent the characteristics of a sequence over a small period T , and global correlations are the correlations exhibited between many periods T 1 , T 2 ,...
T n − 1 , T n .
For example, the temperature at a moment is not only influenced by the specific change during the day but may also be correlated with the overall trend of a period (e.g., week, month, etc.).
We can identify the value of a time point more accurately by learning the overall characteristics of that period and the correlation among many periods before.
Therefore, a good forecasting method should have the following two properties: (1)
The ability to extract local features to measure short-term changes.
The ability to model the global correlations to measure the long- term
trend. Based on this, we propose Multi-scale Isometric Convolution Network ( MICN ).
We use multiple branches of different convolution kernels to model different potential pattern information of the sequence separately.
For each branch, we extract the local features of the sequence using a local module based on downsampling convolution, and on top of this, we model the global correlation using a global module based on isometric convolution.
Finally, Merge operation is adopted to fuse information about different patterns from several branches.
This design reduces the time and space complexity to linearity, eliminating many unnecessary and redundant calculations.
MICN achieves state-of-the-art accuracy on five real-world benchmarks.
The contributions are summarized as follows:
• We propose MICN based on convolution structure to efficiently replace the self-attention, and it achieves linear computational complexity and memory cost.
• We propose a multiple branches framework to deeply mine the intricate temporal patterns of time series, which validates the need and validity for separate modeling when the input data is complex and variable.
• We propose a local-global structure to implement information aggregation and long-term dependency modeling for time series, which outperforms the self-attention family and Auto-correlation mechanism.
We adopt downsampling one-dimensional convolution for local features extraction and isometric convolution for global correlations discovery.
• Our empirical studies show that the proposed model improves the performance of state-ofthe-art method by 17.2% and 21.6% for multivariate and univariate forecasting, respectively.
R ELATED WORK 2. CNN S AND T RANSFORMERS
Convolutional neural networks (CNN) are widely used in computer vision, natural language processing and speech recognition (Sainath et al., 2013)
(Li et al., 2019a) (Han et al., 2020).
It is widely believed that this success is due to the use of convolution operations, which can introduce certain inductive biases, such as translation invariance, etc.
CNN-based methods are usually modeled from the local perspective, and convolution kernels can be very good at extracting local information from the input.
By continuously stacking convolution layers, the field of perception can be extended to the entire input space, enabling the aggregation of the overall information.
Transformer (Vaswani et al., 2017) has achieved the best performance in many fields since its emergence, which is mainly due to the attention mechanism.
Unlike modeling local information directly from the input, the attention mechanism does not require stacking many layers to extract global information.
Although the complexity is higher and learning is more difficult, it is more capable of learning long-term dependencies (Vaswani et al., 2017).
Although CNNs and Transformers are modeled from different perspectives, they both aim to achieve efficient utilization of the overall information of the input.
In this paper, from the view of combining the modeling principles of CNNs and Transformers, we consider both local and global context, extract local features of data first, and then model global correlation on this basis.
Furthermore, our method achieves lower computational effort and complexity.
2. M ODELING B OTH L OCAL AND G LOBAL C ONTEXT
Both local and global relationships play an important role in sequence modeling.
Some works have been conducted to study how to combine local and global modeling into a unified model to achieve high efficiency and interpretability.
Two well-known architectures are: Conformer (Gulati et al., 2020) and Lite Transformer (Wu et al., 2020).
Conformer is a variant of Transformer and has achieved state-of-the-art performance in many speech applications.
It adopts the attention mechanism to learn the global interaction, the convolution module to capture the relative-offset-based local features, and combines these two modules sequentially.
However, Conformer does not analyze in detail what local and global features are learned and how they affect the final output.
There is also no explanation why the attention module is followed by a convolution module.
Another limitation of Conformer is the quadratic complexity with respect to the sequence length due to self-attention.
Lite Transformer also adopts convolution to extract local information and self-attention to capture long-term correlation, but it separates them into two branches for parallel processing.
A visual analysis of the feature weights extracted from the two branches is also presented in the paper, which can provide a good interpretation of the model results.
However, the parallel structure of the two branches determines that there may be some redundancy in its computation, and it still has the limitation of quadratic complexity.
Whether the convolution and self-attention are serialized to extract local and global relationships step by step or in parallel, it inevitably results in quadratic time and space complexity.
Therefore, in this paper, we propose a new framework for modeling local features and global correlations of time series along with a new module instead of attention mechanism.
We also use the convolution operation to extract its local information and then propose isometric convolution to model the global correlation between each segment of the local features.
This modeling method not only avoids more redundant computations but also reduces the overall time and space complexity to linearity with respect to the sequence length.
M ODEL 3. MICN F RAMEWORK
The overall structure of MICN is shown in Figure 1.
The long time series prediction task is to predict a future series of length O based on a past series of length I , which can be expressed as input − I − predict − O , where O is much larger than I .
Inspired by traditional time series decomposition algorithms (Robert et al., 1990)
(Wu et al., 2021b), we design a multi-scale hybrid decomposition ( MHDecomp ) block to separate complex patterns of input series.
Then we use Seasonal Prediction Block to predict seasonal information and Trend-cyclical Prediction Block to predict trend-cyclical information. Then add the prediction results up to get the final prediction Y pred .
We donate d as the number of variables in multivariate time series and D as the hidden state of the series.
The details will be given in the following sections.
[Figure]
3. M ULTI - SCALE H YBRID D ECOMPOSITION
Previous series decomposition algorithms (Wu et al., 2021b) adopt the moving average to smooth out periodic fluctuations and highlight the long-term trends.
For the input series X ∈ R I × d , the process is:
[Equation]
where: X t , X s ∈ R I × d denote the trend-cyclical and seasonal parts, respectively.
The use of the
Avgpool ( · ) with the padding operation keeps the series length unchanged.
But the parameter kernel of the Avgpool ( · ) is artificially set and there are often large differences in trend-cyclical series and seasonal series obtained from different kernels .
Therefore, we design a multi-scale hybrid decomposition block that uses several different kernels of the Avgpool ( · ) and can separate several different patterns of trend-cyclical and seasonal parts purposefully. Different from the MOEDecomp block of
FEDformer (Zhou et al., 2022), we use simple mean operation to integrate these different patterns because we cannot determine the weight of each pattern before learning its features.
Correspondingly, we put this weighting operation in the Merge part of Seasonal Prediction block after the representation of the features.
Concretely, for the input series X ∈ R I × d , the process is:
[Equation]
where X t , X s ∈ R
I × d denote the trend-cyclical and seasonal part, respectively.
The different kernels are consistent with multi-scale information in Seasonal Prediction block.
The effectiveness is demonstrated experimentally in Appendix B.1.
3. T REND - CYCLICAL P REDICTION B LOCK
Currently, Autoformer (Wu et al., 2021b) concatenates the mean of the original series and then accumulates it with the trend-cyclical part obtained from the inner series decomposition block.
But there is no explanation of this and no proof of its effectiveness.
In this paper, we use a simple linear regression strategy to make a prediction about trend-cyclical, demonstrating that simple modeling of trend-cyclical is also necessary for non-stationary series forecasting tasks (See Section 4.2).
Concretely, for the trend-cyclical series X t ∈ R I × d obtained with MHDecomp block, the process is:
[Equation]
where Y regret ∈ R
O × d denotes the prediction of the trend part using the linear regression strategy.
And we use MICN-regre to represent MICN model with this trend-cyclical prediction method.
For comparison, we use the mean of X t to cope with the series where the trend-cyclical keeps constant:
[Equation]
where Y meant ∈
R
O × d denotes the prediction of the trend part.
And we use MICN-mean to represent
MICN model with this trend-cyclical prediction method.
3. S EASONAL P REDICTION B LOCK
As shown in Figure 2, the Seasonal Prediction Block focuses on the more complex seasonal part modeling.
After embedding the input sequence X s , we adopt multi-scale isometric convolution to capture the local features and global correlations, and branches of different scales model different underlying patterns of the time series.
We then merge the results from different branches to complete comprehensive information utilization of the sequence.
It can be summarised as follows:
[Equation]
where X zero ∈ R
O × d denotes the placeholders filled with zero, and
X embs ∈ R
( I + O ) × D denotes the embedded representation of X s .
Y s , l ∈ R ( I + O )
× D represents the output of l − th multi-scale isometric [Figure] [Figure] convolution (MIC) layer, and Y s ∈ R
O × d represents the final prediction of the seasonal part after a linear function Projection with Y s , N ∈ R
( I + O ) × D and Truncate operation.
The detailed description of
Embedding and MIC will be given as follows.
Embedding The decoder of the latest Transformer-based models such as Informer (Zhou et al., 2021), Autoformer (Wu et al., 2021b) and FEDformer (Zhou et al., 2022) contain the latter half of the encoder’s input with the length I 2 and placeholders with length O filled by scalars, which may lead to redundant calculations.
To avoid this problem and adapt to the prediction length
O , we replace the traditional encoder-decoder style input with a simpler complementary 0 strategy.
Meanwhile, we follow the setting of FEDformer and adopt three parts to embed the input.
The process is:
[Equation]
6) where X embs ∈ R ( I + O ) × D .
TFE represents time features encoding (e.g., MinuteOfHour, HourOfDay,
DayOfWeek, DayOfMonth, and MonthOfYear), PE represents positional encoding and VE represents value embedding.
Multi-scale isometric Convolution(MIC)
Layer MIC layer contains several branches, with different scale sizes used to model potentially different temporal patterns.
In each branch, as shown in Figure 3, the local-global module extracts the local features and the global correlations of the sequence (See Appendix B.7 for more detailed description).
Concretely, after obtaining the corresponding single pattern by avgpool , the local module adopts one-dimensional convolution to implement downsampling.
The process is:
[Equation]
where Y s , l − 1 denotes the output of ( l − 1 )
− th MIC layer and Y s , 0 = X embs .
i ∈ (cid:8)
I[-
 I-] 4 , I 8 ,... (cid:9) denote the different scale sizes corresponding to the different branches in Figure 2.
For Conv1d, we set stride = kernel
= i , which serves as compression of local features.
Y local , i s , l ∈ R
[Equation]
I + O
[Equation]
i × D represents the result obtained by compressing local features, which is a short sequence.
And furthermore, the global module is designed to model the global correlations of the output of the local module.
A commonly used method for modeling global correlations is the self-attention mechanism.
But in this paper, we use a variant of casual convolution, isometric convolution, as an alternative.
As shown in Figure 4, isometric convolution pads the sequence of length S with placeholders zero of length S − 1 , and its kernel is equal to S . It means that we can use a large convolution kernel to measure the global correlation of the whole series.
The current generative prediction approach is to add placeholder to the input sequence, which has no actual sequence information in the second half.
The Isometric Convolution can enable sequential inference of sequences by fusing local features information.
Moreover, the kernel of Isometric convolution is determined by all the training data, which can introduces a global temporal inductive bias (translation equivariance, etc.) and achieve better generalization than self-attention (the correlations are obtained from the product between different elements).
Meanwhile, we demonstrate that for a shorter sequence, isometric convolution is superior to self-attention.
The detailed experiments of the proof are in Appendix B.3.
And to keep the sequence length constant, we upsample the result of the isometric convolution using transposed convolution.
The global module can be formalized as follows:
[Equation]
where Y local , i s ,
l ∈ R
( I + O ) i
× D denote the result after the global correlations [-modeling, i ∈ (cid:8) I 
-]modeling.
Y s , l − 1 is the output of l − 1 MIC layer.
Y global , i s , l
[Equation]
D represents the result of this pattern (i.e., this branch).
[Figure]
Then we propose to use Conv2d to merge the different patterns with different weights instead of the traditional Concat operation.
The validity of this weighting approach is verified in Appendix B.4.
The process is:
[Equation]
where Y s , l ∈ R ( I + O ) × D represents the result of l − th MIC layer.
To get the final prediction of the seasonal part, we use the projection and truncate operations:
[Equation]
10) where Y s , N ∈ R
( I + O )
× D represents the output of N-th MIC layer, and Y s ∈ R
O × d represents the final prediction about the seasonal part.
E XPERIMENTS
Dataset To evaluate the proposed MICN, we conduct extensive experiments on six popular realworld datasets, covering many aspects of life: energy, traffic, economics, and weather.
We follow standard protocol (Zhou et al., 2021) and split all datasets into training, validation and test set in chronological order by the ratio of 6:2:2 for the ETT dataset and 7:1:2 for the other datasets.
More details about the datasets and implementation are described in Appendix A.1 and A.2.
Baselines We include four transformer-based models: FEDformer (Zhou et al., 2022), Autoformer (Wu et al., 2021b), Informer (Zhou et al., 2021), LogTrans (Li et al., 2019b), two RNN-based models:
LSTM (Hochreiter & Schmidhuber, 1997),
LSTNet (Lai et al., 2018b) and CNN-based model TCN (Bai et al., 2018) as baselines.
For the univariate setting, we mainly compare transformer-based models.
For the state-of-the-art model FEDformer, we compare the better one (FEDformer-f).
[Table] [Figure] [Table]
[Table]
I L
I
[Table]
4. M AIN R ESULTS
Multivariate results For multivariate long-term series forecasting, MICN achieves the state-ofthe-art performance in all benchmarks and all prediction length settings (Table 1).
Compared to the previous best model FEDformer, MICN yields an 17.2% averaged MSE reduction.
Especially, under the input-96-predict-96 setting, MICN gives 12% relative MSE reduction in ETTm2, 14% relative MSE reduction in Electricity, 31% relative MSE reduction in Exchange, 12% relative MSE reduction in Traffic, 26% relative MSE reduction in Weather, 17% relative MSE reduction in ILI, and 18.6% average MSE reduction in this setting.
We can also find that MICN makes consistent improvements as the prediction increases, showing its competitiveness in terms of long-term time-series forecasting.
Note that MICN still provides remarkable improvements with a 51% averaged MSE reduction in the Exchange dataset that is without obvious periodicity.
All above shows that MICN can cope well with a variety of time-series forecasting tasks in real-world applications.
More results about other ETT benchmarks are provided in Appendix A.3.
See Appendix C.3 for detailed showcases.
Univariate results We also show the univariate time-series forecasting results in Table 2.
Significantly, MICN achieves a 21.6% averaged MSE reduction compared to FEDformer.
Especially for the Weather dataset, MICN gives 53% relative MSE reduction under the predict-96 setting, 75% relative MSE reduction under the predict-192 setting, 44% relative MSE reduction under the predict-336 setting, and 56% relative MSE reduction under the predict-720 setting.
It again verifies the greater time-series forecasting capacity.
More results about other ETT benchmarks are provided in Appendix A.3.
See Appendix C.2 for detailed showcases.
4. A BLATION S TUDIES
Trend-cyclical Prediction Block
We attempt to verify the necessity of modeling the trend-cyclical part when using a decomposition-based structure.
Like Autoformer (Wu et al., 2021b), previous methods decompose the time series and then take the mean prediction of the trend information, which is then added to the other trend information obtained from the decomposition module in the model.
However, the reasons and rationality are not argued in the relevant papers.
In this paper, we use simple linear regression to predict the trend-cyclical part and we also record the results of the mean prediction for comparison.
Note that with different trend-cyclical prediction blocks, we have different models named MICN-regre and MICN-mean. As shown in Table 3, MICN-regre performs better than MICN-mean overall. Because all the datasets are non-stationary, a simple modeling for trend-cyclical to give model a holistic view of the trend direction is necessary.
See Appendix B.2 for more visualization results and analysis.
[Table]
Local-Global Structure vs. Auto-correlation, self-attention
In this work, we propose the localglobal module to model the underlying pattern of time series, including local features and global correlations, while the previous outstanding model Autoformer uses auto-correlation.
We replace the auto-correlation module in the original Autoformer with our proposed local-global module (we set i ∈ { 12 , 16 } ) for training, and the results are shown in Table 4.
Also, We replace the Local-Global module in MICN-regre with the Auto-Correlation module and self-attention module for training, and the results are shown in Table 5.
They all demonstrate that modeling time series in terms of local features and global correlations is better and more realistic.
[Table]
4. M ODEL A NALYSIS
Impact of input length In time series forecasting tasks, the size of the input length indicates how much historical information the algorithm can utilize.
In general, a model that has a strong ability to model long-term temporal dependency should perform better as the input length increases.
Therefore
[Table]
we conduct experiments with different input lengths and the same prediction length to validate our model.
As shown in Figure [Table] when the input length is relatively long, the performance of Transformer-based models becomes worse because of repeated short-term patterns as stated in (Zhou et al., 2021).
Relatively, the overall performance of MICN prediction gradually gets better as [Table] [Figure] WTH
Robustness analysis We use a simple noise injection to demonstrate the robustness of our model.
Concretely, we randomly select data with proportion ε in the original input sequence and randomly perturb the selected data in the range [ − 2 X i , 2 X i ] , where X i denotes the original data.
The data after noise injection is then trained [Table] and the MSE and MAE metrics are recorded.
The results are shown in
Table 6.
As the proportion of perturbations ε increases, the MSE and MAE metrics of the predictions increase by a small amount.
It indicates that MICN exhibits good robustness in response to less noisy data (up to 10%) and has a great advantage in dealing with many data abnormal fluctuations (e
[Table]
g. abnormal power data caused by equipment damage).
[Table]
C ONCLUSIONS
This paper presents a convolution-based framework MICN, which makes predictions for the trendcyclical part and seasonal part separately.
It achieves O ( LD 2 ) complexity and yields consistent state-of-the-art performance in extensive real-world datasets.
In the Seasonal-Prediction block, we use different scales to mine the sequence for potentially different patterns, each modeled from a local and global perspective, which is implemented by different convolution operations.
The proposed isometric convolution outperforms self-attention in terms of capturing global correlations for a short sequence.
The extensive experiments further demonstrate the effectiveness of our modeling approach for long-term forecasting tasks.
A CKNOWLEDGEMENTS
This work was supported by the Sichuan Science and Technology Program (2023YFG0112) and
the
National Key R&D Program of China (2020YFB0704502) and the Sichuan Science and Technology Program (2022YFG0034) and Postdoctoral Interdisciplinary Innovation Fund(10822041A2137) and Sichuan University and Yibin Cooperation Program (2020CDYB-30) A S UPPLEMENTAL E XPERIMENTS A. D ATASET D ETAILS
In this work, the details of the experiment datasets are summarized as follows: (1) ETT (Zhou et al., 2021)
dataset contains two visions of the sub-dataset: ETTh and ETTm, collected from electricity transformers every 15 minutes and 1 hour between July 2016 and July 2018.
(2) Electricitydataset contains the electricity consumption of 321 customers recorded hourly from 2012 to 2014.
(3) Exchange (Lai et al., 2018a) dataset records daily exchange rates of eight different countries daily ranging from 1990 to 2016.
(4) Traffic 2 contains the data from California Department of
Transportation hourly, which describes the road occupancy rates measured by different sensors on
San Francisco Bay area freeways.
(5) Weather 3 contains 21 meteorological indicators, recorded every 10 minutes for 2020 whole year.
(6) ILI 4 records influenza-like illness (ILI) patients data weekly from
Centers for Disease Control and Prevention of the United States between 2002 and 2021.
Tablesummarizes feature details (Sequence Length: Len, Dimension: Dim, Frequency: Freq) .
[Table]
A. I MPLEMENTATION DETAILS
Our method is trained with the L2 loss, using the ADAM optimizer with an initial learning rate of 10 − 3 .
Batch size is set to 32.
The training process is early stopped after three epochs if there is no loss degradation on the valid set.
The mean square error (MSE) and mean absolute error (MAE) are used as metrics.
All the experiments are repeated 3 times with different seeds, implemented in PyTorch and conducted on NVIDIA RTX A5000 24GB GPU.
The hyper-parameter i is set to { 12 , 16 } , and the hyper-parameter sensitivity analysis can be seen in Appendix A.4 .
For a fairer comparison, we fix the input length to 96 for all datasets (36 for ILI).
MICN contains 1 MIC layer.
We use MICN-regre and MICN-mean to represent the different strategies of trend-cyclical prediction block in the following.
A. F ULL B ENCHMARK ON THE ETT D ATASETS
We build the benchmark on the four ETT datasets in Table 8 and Table 9.
The ETTh1 and ETTh datasets are recorded hourly, while the ETTm1 and ETTm2 datasets are recorded every 15 minutes.
MICN achieves state-of-the-art performance in all benchmarks in general.
Especially for the multivariate ETTm1 dataset, MICN gives 17% relative MSE reduction under the predict-96 setting, gives 15% relative MSE reduction under the predict-192 setting, gives 8% relative MSE reduction under the predict-336 setting, gives 15% relative MSE reduction under the predict-720 setting.
A. H YPER -P ARAMETER S ENSITIVITY
As shown in Table 10, we can verify the model robustness with respect to hyper-parameter i .
Different values of i have slightly different results.
Concretely, when i take one value, MICN performs worse [Table] [Table]
ETT h
[Table] ETT m [Table]
ETT m
[Table]
[Table]
because of the lack of ability to capture complex temporal patterns of the time series.
Meanwhile,
MICN can achieve almost the same better performance when i takes two or three values, indicating that the multi-branch structure is effective.
To be more representative, we set i to { 12 , 16 } in this paper.
A. S ELECTION OF DIFFERENT CONVOLUTION MODES
As shown in Table 11, we also record the performance in different convolution modes: stride = kernel and stride =
kernel 2 .
The second mode makes more comprehensive use of local information, making the convolution more coherent.
MICN achieves similar performance in different convolution modes.
It proves that MICN can make the most of sequence information, and the performance of the model depends on the structure we proposed. [Table] [Table]
B A DDITIONAL M ODEL A NALYSIS B. M ULTI - SCALE H YBRID D ECOMPOSITION
Autoformer harnesses the decomposition as an inner block of deep models and gets good performance.
However, the patterns obtained by its decomposition are simple and cannot effectively deal with the complex and changeable properties of time series.
As shown in Table [Table] we replace the decomposition block in Autoformer with our proposed multi-scale hybrid decomposition block.
For
Exchange, we achieve a similar performance because it has no obvious temporal pattern.
The result verifies that multi-scale hybrid decomposition structure is more in line with the complex temporal patterns in real-time series.
B. V ISUALIZATION OF L EARNED T REND -C YCLICAL P ARTS
As shown in Figure 6 and Figure
[Table]
we plot the results of learned trend-cyclical parts.
The separate modeling of the trend-cyclical part makes better performance and grasp of long-term progression.
We also observe that the mean prediction is slightly better on the ETTm2 dataset.
This is due to the complexity of the trend-cyclical information and the inability of simple linear regression, which may
[Table]
[Figure]
[Figure] [Figure]
prediction-
[Figure]
require a more advanced trend prediction method.
Moreover, the mean value of its trend change is close to constant, so mean prediction is better in this situation.
B. I SOMETRIC CONVOLUTION VS . MASKED SELF - ATTENTION
With the local module in MICN, we get a short sequence characterizing local features.
On this basis, we propose the isometric convolution in global module to model the global correlation of the sequence, while previously the first choice is masked self-attention.
We replace the isometric convolution in the global module of MICN with masked self-attention for training, and the results are shown in Table 13 and Table 14.
It verifies that for a short sequence, isometric convolution outperforms masked self-attention in general [Table] [Table] To compare the isometric convolution and the masked self-attention further
[Table]
we conduct more experiments on full benchmark with different kernel sizes. The results in Table 15 show that the isometric convolution outperforms the masked self-attention in the most cases. And we also note that in some cases the masked self-attention is slightly more effective. We believe that this is related to the corresponding datasets and that we will analyse the datasets in detail in the future.
Moreover, we can conclude that different kernels have a relatively small impact on the final results, which indicates that our advanced structure instead of model parameters plays a major role in the performance. [Table] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
B. C OMPARISON OF M ERGING O PERATIONS
The traditional method of merging branch structures is the concat operation on the hidden state.
In this paper, we propose to adopt 2D convolution to merge multiple branches to better measure the importance of each branch (the kernel represents the weights).
As shown in Table 16, the better performance verifies the effectiveness of our proposed method. [Table]
B. E FFICIENCY A NALYSIS
For MICN, the complexity lies in Downsampling convolution and Isometric convolution in LocalGlobal module. If we set the sequence length to L , the hidden state to D and the multi-scale convolution kernels to i . For downsampling convolution (kernel= stride), the complexity is O ( i ∗ D 2 ∗ Li )
=
O ( LD 2 ) .
For Isometric convolution, the sequence length and kernel are changed to Li , stride=1, padding= Li − 1, so the complexity is O (( Li ) 2 ∗ D 2 ) = O ( L 2 D 2 i 2 ) . In this paper, we set i ∈
{ L 4 , L 8 ... } is factor of L , so the complexity of Isometric convolution is O ( cD 2 ) , where c is a constant.
In summary, the overall complexity is max ( O ( LD 2 ) , O ( cD 2 ))
=
O ( LD 2 ) , which is linear about the sequence length.
The comparisons of the time complexity and memory usage in training and the inference steps in testing are summarized in Table 17.
Furthermore, we compare the running memory and time among Local-Global-based, Auto-correlationbased and self-attention-based models during the training phase.
As shown in Figure 8, the proposed
Local-Global module shows O ( LD 2 ) complexity and achieves better long-term sequences efficiency.
[Table]
As the prediction length increases, our model takes a little more time than Auto-Correlation.
We speculate that this may be due to the use of the convolution operation or the activation function Tanh .
In general, our method is the most portable and valuable in practical applications.
[Figure]
B. M ORE A NALYSIS OF T HE T REND - CYCLICAL P REDICTION B LOCK
To further evaluate the trend-cyclical prediction, we conduct more experiments on full benchmark.
As shown in Table 18, regre is the simple regression prediction without other modules and mean is the simple mean prediction without other modules.
MICN-regre is our proposed method with regression prediction in trend-cyclical block and MICN-mean is the mean prediction in trend-cyclical block.
FEDformer-mean is the original FEDformer using the mean prediction and FEDformer-regre uses regression prediction instead of mean prediction.
We can conclude that simple regression and mean fail to capture the complex temporal correlations.
But for the data Exchange without periodicity, simple regression makes competitive performance.
This result is worth thinking about, and we will conduct more in-depth experiments in the future.
Meanwhile, the results that MICN-regre outperforms FEDformer-regre and MICN-mean outperforms FEDforemr-mean prove the validity of our proposed model.
We also find FEDformer-regre makes worse performance in most cases.
This may be due to the more complex structure of FEDformer, and more complex regression prediction is needed correspondingly.
B. T HE D ETAILED D ESCRIPTION OF T HE L OCAL -G LOBAL M ODULE
We show the detailed description of the Local-global module in Figure 9.
For the input series, we adopt down-sampling convolution with different kernels to extract the local features of different temporal patterns and isometric convolution instead of masked self-attention to capture global correlations.
Then we use up-sampling convolution to recover the length of the series.
Finally, we merge the different branches to complete the modeling of different patterns. [Figure] [Table] [Figure]
B. M ORE D ISCUSSION A BOUT T HE I SOMETRIC C ONVOLUTION
A ND T HE L INEAR
C OMPLEXITY M ODEL :
L INFORMER , F ASTFORMER
[Table]
Linformer Wang et al.
and
Fastformer Wu et al.
(2021a) are other advanced self-attention family models with linear complexity.
We conduct more comparison experiments on full benchmark and show the results in Table 19 and Table 20.
Concretely [Table] we replace the isometric convolution in MICN-regre with the different masked attention mechanisms in Linformer and Fastformer.
We implement the experiments in multivariate and univariate datasets.
We can conclude that Fastformer can make relatively competitive performance but our proposed model MICN performs best in general.
These results provide further evidence of the effectiveness of MICN.
[Table]
[Table]
C S UPPLEMENTARY OF M AIN R ESULTS C. M AIN R ESULTS WITH S TANDARD D EVIATIONS
To get more robust experimental results, we repeat each experiment three times with different random seeds.
For easier comparison, the results are shown in the main text when the seed is set to 2021.
Table 21 shows the standard deviations.
[Table] C. U NIVARIATE SHOWCASES
As shown in Figure 10, Figure 11, Figure 12, Figure 13, Figure [Table]
and Figure [Table] we plot the forecasting results from the test set of univariate dataset Electricity and Traffic for comparison.
Our model gives the best performance among different models.
Moreover, MICN is significantly better at predicting the overall change and peak in the time series than Transformer-based models
[Table]
[Figure]
PredictionGroundTruth prediction-
[Figure] [Figure]
prediction-
[Figure] [Figure] [Figure] prediction-
[Figure]
prediction-
[Figure] [Figure]
prediction
[Figure]
[Figure] [Figure] [Figure] [Figure]
prediction
[Figure] [Figure]
prediction-
[Figure] [Figure]
C. M ULTIVARIATE SHOWCASES
As shown in Figure 16, Figure 17, Figure 18, Figure 19, Figure
[Figure]
and Figure
[Figure]
we also plot the forecasting results from the test set of multivariate datasets ETTm1 and ETTm2 for comparison.
Our model gives the most accurate prediction.
Moreover, MICN is better at predicting rising and falling turning points in time series and closer to ground truth
[Figure]
prediction-
[Figure]
[Figure]
[Figure]
[Figure] [Figure]
[Figure] [Figure]
prediction-
[Figure]
prediction
[Figure] [Figure]
prediction-
[Figure]
prediction-
[Figure]
[Figure]