Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting
Abstract
Transformers have shown great power in time series forecasting due to their global-range modeling ability.
However, their performance can degenerate terribly on non-stationary real-world data in which the joint distribution changes over time.
Previous studies primarily adopt stationarization to attenuate the nonstationarity of original series for better predictability.
But the stationarized series deprived of inherent non-stationarity can be less instructive for real-world bursty events forecasting.
This problem, termed over-stationarization in this paper, leads
Transformers to generate indistinguishable temporal attentions for different series and impedes the predictive capability of deep models.
To tackle the dilemma between series predictability and model capability, we propose Non-stationary
Transformers as a generic framework with two interdependent modules: Series
Stationarization and De-stationary Attention.
Concretely, Series Stationarization unifies the statistics of each input and converts the output with restored statistics for better predictability.
To address the over-stationarization problem, Destationary Attention is devised to recover the intrinsic non-stationary information into temporal dependencies by approximating distinguishable attentions learned from raw series.
Our Non-stationary Transformers framework consistently boosts mainstream Transformers by a large margin, which reduces MSE by 49.43% on
Transformer, 47.34% on Informer, and 46.89% on Reformer, making them the state-of-the-art in time series forecasting.
Code is available at this repository: https://github.com/thuml/Nonstationary_Transformers .
Introduction
Time series forecasting has become increasingly ubiquitous in real-world applications, such as weather forecasting, energy consumption planning, and financial risk assessment.
Recently, Transformers [34] have achieved progressive breakthrough on extensive areas [12, 13, 10, 24].
Especially in time series forecasting, credited to their stacked structure and the capability of attention mechanisms,
Transformers can naturally capture the temporal dependencies from deep multi-level features [39, 19, 22, 37], thereby fitting the series forecasting task perfectly.
Despite the remarkable architectural design, it is still challenging for Transformers to predict realworld time series because of the non-stationarity of data.
Non-stationary time series is characterized by the continuous change of statistical properties and joint distribution over time, which makes the time series less predictable [6, 16].
Besides, it is a fundamental problem to make deep models generalize well on a varying distribution [28, 21, 5].
In previous work, it is generally acknowledged to pre-process the time series by stationarization [26, 29, 17], which can attenuate the non-stationarity of raw time series for better predictability and provide more stable data distribution for deep models.
[Figure]
However, non-stationarity is the inherent property of real-world time series and also good guidance for discovering temporal dependencies for forecasting.
Experimentally, we observe that training on the stationarized series will undermine the distinction of attentions learned by Transformers.
While vanilla
Transformers [34] can capture distinct temporal dependencies from different series in Figure 1(a),
Transformers trained on the stationarized series tend to generate indistinguishable attentions in
Figure 1(b).
This problem, named by the over-stationarization , will bring unexpected side-effect that makes Transformers fail to capture eventful temporal dependencies, limit the model’s predictive ability, and even induce the model to generate outputs with huge non-stationarity deviation from the ground truth.
Thus, how to attenuate time series non-stationarity towards better predictability and mitigate the over-stationarization problem for model capability simultaneously is the key problem to further improve the performance of forecasting.
In this paper, we explore the effect of stationarization in time series forecasting and propose Nonstationary Transformers as a general framework, which empowers Transformer [34] and its efficient variants [19, 39, 37] with great predictive ability for real-world time series.
The proposed framework involves two interdependent modules: Series Stationarization to increase the predictability of nonstationary series and De-stationary Attention to alleviate over-stationarization.
Technically, Series
Stationarization adopts a simple but effective normalization strategy to unify the key statistics of each series without extra parameters.
And De-stationary Attention approximates the attention of unstationarized data and compensates the intrinsic non-stationarity of raw series.
Benefiting from the above designs, Non-stationary Transformers can take advantage of the great predictability of stationarized series and crucial temporal dependencies discovered from original non-stationary data.
Our method achieves state-of-the-art performance on six real-world benchmarks and can generalize to various Transformers for further improvement.
The contributions lie in three folds:
• We refine that the predictive capability of non-stationary series is essential in real-world forecasting.
By detailed analysis, we find out that current stationarization approaches will lead to the over-stationarization problem, limiting the predictive capability of Transformers.
• We propose Non-stationary Transformers as a generic framework, including Series Stationarization to make the series more predictable and De-stationary Attention to avoid the over-stationarization problem by re-incorporating the non-stationarity of original series.
• Non-stationary Transformers consistently boosts four mainstream Transformers by a large margin and achieves state-of-the-art performance on six real-world benchmarks.
Related Work 2.1 Deep Models for Time Series Forecasting
In recent years, deep models with elaboratively designed architectures have achieved great progress in time series forecasting.
RNN-based models [35, 38, 25, 31, 32] are proposed for application in an autoregressive manner for sequence modeling, but the recurrent structure can suffer from modeling long-term dependency.
Soon afterward, Transformer [34] emerges and shows great power in sequence modeling.
To overcome the quadratic computation growth on sequence length, subsequent works aim to reduce Self-Attention’s complexity.
Especially in time series forecasting, Informer [39] extends Self-Attention with KL-divergence criterion to select dominant queries.
Reformer [19] introduces local-sensitive hashing (LSH) to approximate attention by allocated similar queries.
Not only improved by reduced complexity, the following models further develop delicate building blocks for time series forecasting.
Autoformer [37] fuses the decomposition blocks into a canonical structure and develops Auto-Correlation to discover series-wise connections.
Pyraformer [23] designs pyramid attention module (PAM) to capture temporal dependencies with different hierarchies.
Other deep but Transformer-free models also achieve remarkable performance.
N-BEATS [27] proposes the explicit decomposition of trend and seasonal terms with strong interpretability.
N-HiTS [9] introduces hierarchical layout and multi-rate sampling for tackling time series with respective frequency bands.
In this paper, different from previous works focusing on architectural design, we analyze the series forecasting task from the basic view of stationarity, which is an essential property of time series [6, 16].
It is also notable that as a general framework, our proposed Non-stationary Transformers can be easily applied to various Transformer-based models.
2.2 Stationarization for Time Series Forecasting
While stationarity is important to the predictability of time series [6, 16], real-world series always present non-stationarity.
To tackle this problem, the classical statistical method ARIMA [7, 8] stationarizes the time series through differencing.
As for deep models, since the distribution-varying problem accompanied by non-stationarity makes deep forecasting even more intractable, stationarization methods are widely explored and always adopted as the pre-processing for deep model inputs.
Adaptive Norm [26] applies z-score normalization for each series fragment by global statistics of a sampled set.
DAIN [29] employs a nonlinear neural network to adaptively stationarize time series with observed training distribution.
RevIN [17] introduces a two-stage instance normalization [33] that transforms model input and output respectively to reduce the discrepancy of each series.
In contrast, we find out that directly stationarizing time series will damage the model’s capability of modeling specific temporal dependency.
Therefore, unlike previous methods, in addition to the stationarization, Non-stationary Transformers further develops De-stationary Attention to bring the intrinsic non-stationarity of the raw series back to attention.
Non-stationary Transformers
As aforementioned, stationarity is an important element of time series predictability.
Previous “direct stationarization” designs can attenuate non-stationarity of series for better predictability, but they obviously neglect inherent properties of real-world series, which will result in the over-stationarization problem as stated in Figure 1.
To deal with the dilemma, we go beyond previous works and propose
Non-stationary Transformers as a generic framework.
Our model involves two complementary parts: Series Stationarization to attenuate time series non-stationarity and De-stationary Attention to re-incorporate non-stationary information of raw series.
Empowered by these designs, Non-stationary
Transformers can improve data predictability and maintain model capability simultaneously.
3.1 Series Stationarization
Non-stationary time series make the forecasting task intractable for deep models because it is hard for them to generalize well on series with changed statistics during inference, typically varied mean and standard deviation.
The pilot work, RevIN [17] applies instance normalization with learnable affine parameters to each input and restores the statistics to the corresponding output, which makes each series follow a similar distribution.
Experimentally, we find that this design also works well without learnable parameters.
Thus, we propose a more straightforward but effective design to wrap
Transformers as the base model without extra parameters, naming by Series Stationarization.
As is shown in Figure 2, it contains two corresponding operations: Normalization module at first to deal with the non-stationary series caused by varied mean and standard deviation, and De-normalization module at the end to transform the model outputs back with original statistics.
Here are the details.
[Table]
quunQTLIKrkoioy6KbLivYB9Qik+m0DebFZFIopbj1B9zqZ4l/oH/hnTEFtYhOSHLm3HPuzL3XSwI/VY7zWrAWFpeWV4qra+sbm1vbpZ3dZhpnkosGj4NYtj2WisCPREP5KhDtRAoWeoFoeXeXOt4aCZn6cXStxonohmwQ+X2fM0VU5yZkashZMKlNb0tlp+KYZc8DNwdl5Ksel15wgx5icGQIIRBBEQ7AkNLTgQsHCXFdTIiThHwTF5hijbwZqQQpGLF39B3QrpOzEe11ztS4OZ0S0CvJaeOQPDHpJGF9mm3imcms2d9yT0xOfbcx/b08V0iswpDYv3wz5X99uhaFPs5NDT7VlBhGV8fzLJnpir65/aUqRRkS4jTuUVwS5sY567NtPKmpXfeWmfibUWpW73muzfCub0kDdn+Ocx40jyvuacW9OilXL/JRF7GPAxzRPM9QRQ11NEzHH/GEZ6tujaypdf8ptQq5Zw/flvXwAXjZk+U=</latexit> H
Normalization module To attenuate the non-stationarity of each input series, we conduct normalization on the temporal dimension by a sliding window over time.
For each input series x = [ x 1 , x 2 , ..., x S ] ⊤ ∈ R S × C , we transform it by translation and scaling operations and obtain x ′ =
[ x ′ 1 , x ′ 2 , ..., x ′ S ] ⊤ ∈ R S × C , where S and C denote the sequence length and variable number respectively.
The Normalization module can be formulated as follows:
[Equation]
where µ x , σ x ∈ R C × 1 , 1 σ x means the element-wise division and ⊙ is the element-wise product.
Note that Normalization module decreases the distributional discrepancy among each input time series, making the distribution of the model input more stable.
De-normalization module As shown in Figure 2, after the base model H predicting the future value with length- O , we adopt De-normalization to transform the model output y ′
=
[ y ′ 1 , y ′ 2 , ..., y ′ O ]
⊤ ∈ R O × C with σ x and µ x and obtain ˆ y =
[ˆ y 1 , ˆ y 2 , ..., ˆ y O ] ⊤ as the eventual forecasting results.
The
De-normalization module can be formulated as follows:
[Equation]
By means of the two-stage transformation, the base models will receive stationarized inputs, which follow a stable distribution and are easier to generalize.
This design also makes the model equivariant to translational and scaling perturbance of time series, thereby benefiting real-world series forecasting.
3.2 De-stationary Attention
While the statistics of each time series are explicitly restored to the corresponding prediction, the non-stationarity of the original series cannot be fully recovered only by De-normalization.
For instance, Series Stationarization can generate the same stationarized input x ′ from distinct time series x 1 , x 2 (i.e. x 2 =
α x 1 + β ), and the base model will get identical attention that fails to capture crucial temporal dependencies entangled with non-stationarity (Figure 1).
In other words, the undermined effects caused by over-stationarization happen inside the deep model, especially in the calculation of attention.
Furthermore, non-stationary time series are fragmented and normalized into several series chunks with the same mean and variance, which follow more similar distributions than the raw data before stationarization.
Thus, the model is more likely to generate over-stationary and uneventful outputs, which is irreconcilable with the natural non-stationarity of the original series.
To tackle the over-stationarization problem caused by Series Stationarization, we propose a novel
De-stationary Attention mechanism, which can approximate the attention that is obtained without stationarization and discover the particular temporal dependencies from original non-stationary data.
Analysis of the plain model As mentioned above, the over-stationarization problem is caused by the vanishment of inherent non-stationarity information, which will make the base model fail to capture eventful temporal dependencies for forecasting.
Therefore, we try to approximate the attention learned from the original non-stationary series.
We start from the formula of Self-Attention [34]:
[Equation]
where Q , K , V ∈ R S ×
d k are length- S queries, keys and values of d k -dimension respectively, and
Softmax( · ) is conducted row by row.
To simplify the analysis, we assume the embedding and feed-forward layers f to hold the linear properties 2 and f is conducted separately on each time point, that is, each query token in Q =
[ q 1 , q 2 , ..., q S ]
⊤ can be calculated as q i = f
( x i ) with respect to the input series x = [ x 1 , x 2 , · · · , x S ] ⊤ .
Since it is a convention to conduct normalization on each time series variable to avoid certain variable that dominates the scale, we can further assume each variable of series x shares the same variance, and thus original σ x ∈ R C × 1 is reduced to a scalar.
After Normalization module, the model receives the stationarized input x ′ =
( x − 1 µ ⊤ x ) /σ x , where 1 ∈ R S × 1 is an all-ones vector.
Based on the linear property assumption, it can be proved that
the
Attention layer will receive Q ′ =
[ f ( x ′ 1 ) , ..., f ( x ′ S )]
⊤ =
( Q − 1 µ ⊤ Q ) /σ
x , where µ Q ∈ R d k × 1 is the mean of Q along the temporal dimension (See Appendix A for a detailed proof).
And so is the corresponding transformed K ′ , V ′ .
Without Series Stationarization, the input of Softmax( · ) in
Self-Attention should be QK ⊤ / √ d k , while now the attention is calculated based on Q ′ , K ′ :
[Equation]
We find that Q µ K ∈ R S × 1 and µ ⊤ Q µ K ∈ R , and they are repeatedly operated on each column and element of σ 2 x Q ′ K ′⊤ ∈ R S × S respectively.
Since Softmax( · ) is invariant to the same translation on the row dimension of input, we have the following equation:
[Equation]
Equation 5 deduces a direct expression of the attention Softmax (cid:0) QK ⊤ / √ d k (cid:1) learned from raw series x .
Except for the current Q ′ , K ′ from stationarized series x ′ , this expression also requires the non-stationary information σ x , µ Q , K that are eliminated by Series Stationarization.
De-stationary Attention To recover the original attention on non-stationary series, we attempt to bring the vanished non-stationary information back to its calculation.
Based on Equation 5, the key is to approximate the positive scaling scalar τ = σ 2 x ∈ R + and shifting vector ∆ =
K µ Q ∈ R S × 1 , which are defined as de-stationary factors .
Since the strict linear property hardly holds for a deep model, other than estimating and utilizing real factors with great effort, we try to learn de-stationary factors directly from the statistics of unstationarized x , Q and K by a simple but effective multilayer perceptron layer.
As we can only discover limited non-stationary information from current
Q ′ , K ′ , the unique and reasonable source to compensate non-stationarity is the original x without being normalized.
Thus, as a direct deep learning implementation of Equation 5, we apply a multilayer perceptron as the projector to learn de-stationary factors τ, ∆ from the statistics µ x , σ x of unstationarized x individually.
And the De-stationary Attention is calculated as follows:
[Equation]
where the de-stationary factors τ and ∆ are shared by De-stationary Attention of all layers (Figure 2).
De-stationary Attention mechanism learns the temporal dependencies from both stationarized series
Q ′ , K ′ and non-stationary series x , µ x , σ x , and multiplies by the stationarized values V ′ .
Therefore, it can benefit from the predictability of stationarized series and maintain the inherent temporal dependencies of raw series simultaneously.
Overall architecture Following the prior use of Transformers [39, 37] in time series forecasting, we adopt the standard Encoder-Decoder structure (Figure 2), where the encoder is to extract information from past observations, and the decoder is to aggregate past information and refine the prediction from simple initialization.
The canonical Non-stationary Transformer is wrapped by Series Stationarization to both the input and output of vanilla Transformer [34], and replacing the Self-Attention by our proposed De-stationary Attention, which can boost the non-stationary series predictive capability of the base model.
For the Transformer variants [19, 39, 37], we transform the terms inside Softmax( · ) with the de-stationary factors τ , ∆ to re-integrate the non-stationary information (See Appendix E. for the implementation details).
Experiments
We conduct extensive experiments to evaluate the performance of Non-stationary Transformers on six real-world time series forecasting benchmarks and further validate the generality of the proposed framework on various mainstream Transformer variants.
Datasets Here are the descriptions of the datasets: (1) Electricity [1] records the hourly electricity consumption of 321 clients from 2012 to 2014.
(2) ETT [39] contains the time series of oil destationary factors and power load collected by electricity transformers from July 2016 to July 2018.
ETTm1 /ETTm2 are recorded every 15 minutes, and ETTh1/ETTh2 are recorded every hour.
Exchange [20] collects the panel data of daily exchange rates from 8 countries from 1990 to 2016.
(4) ILI [2] collects the ratio of influenza-like illness patients versus the total patients in one week, which is reported weekly by Centers for Disease Control and Prevention of the United States from 2002 and 2021.
(5) Traffic [3] contains hourly road occupancy rates measured by 862 sensors on
San Francisco Bay area freeways from January 2015 to December 2016.
(6) Weather [4] includes meteorological time series with 21 weather indicators collected every 10 minutes from the Weather
Station of the Max Planck Biogeochemistry Institute in 2020.
Especially, in this paper, we adopt the Augmented Dick-Fuller (ADF) test statistic [14] as the metric to quantitatively measure the degree of stationarity .
A smaller ADF test statistic indicates a higher degree of stationarity, which means the distribution is more stable.
Table 1 summarizes the overall statistics of the datasets and lists them in ascending order by degree of stationarity.
We follow the standard protocol that divides each dataset into the training, validation, and testing subsets according to the chronological order.
The split ratio is 6:2:2 for the ETT dataset and 7:1:2 for others.
[Table]
Baselines We evaluate the vanilla Transformer [34] equipped by the Non-stationary Transformers framework in both multivariate and univariate settings to demonstrate its effectiveness.
For multivariate forecasting, we include six state-of-the-art deep forecasting models: Autoformer [37],
Pyraformer [23], Informer [39], LogTrans [22], Reformer [19] and LSTNet [20].
For univariate forecasting, we include seven competitive baselines: N-HiTS [9], N-BEATS [27], Autoformer [37],
Pyraformer [23], Informer [39], Reformer [19] and ARIMA [7].
In addition, we adopt the proposed framework on both the canonical and efficient variants of Transformers: Transformer [34],
Informer [39], Reformer [19] and Autoformer [37] to validate the generality of our framework.
Implementation details All the experiments are implemented with PyTorch [30] and conducted on a single NVIDIA TITAN V 12GB GPU.
Each model is trained by ADAM [18] using L2 loss with the initial learning rate of 10 − 4 and batch size of 32.
Each Transformer-based model contains two encoder layers and one decoder layer.
Considering the efficiency of hyperparameters search, we use two-layer perceptron projector with the hidden dimension varying in { 64 , 128 , 256 } in De-stationary
Attention.
We repeat each experiment three times with different random seeds and report the test
MSE/MAE under different prediction lengths, and the standard deviations are also provided in the
Appendix C.2.
A lower MSE/MAE indicates better performance.
4.1 Main Results
Forecasting results As for multivariate forecasting results, the vanilla Transformer equipped with our framework consistently achieves state-of-the-art performance in all benchmarks and prediction lengths (Table 2).
Notably, Non-stationary Transformer outperforms other deep models impressively on datasets characterized by high non-stationarity: under the prediction length of 336, we achieve 17% MSE reduction ( 0 . 509 → 0 . 421 ) on Exchange and 25% ( 2 .
669 → 2 . 010 ) on ILI compared to previous state-of-the-art results, which indicates that the potential of deep model is still constrained on non-stationary data.
We also list the univariate results of two typical datasets with different stationarity in Table 3.
Non-stationary Transformer still realizes remarkable forecasting performance.
sequence is
[Table]
[Table]
W t h
[Table]
Framework generality We apply our framework to four mainstream Transformers and report the performance promotion of each model (Table 4).
Our method consistently improves the forecasting ability of different models.
Overall, it achieves averaged 49.43% promotion on Transformer, 47.34% on Informer, 46.89% on Reformer and 10.57% on Autoformer, making each of them surpass previous state-of-the-art.
Compared to native blocks of the models, there is hardly any parameter and computation increase by applying our framework (See Appendix C.5 for details) [Table] and thereby their computational complexities can be preserved.
It validates that Non-stationary Transformer is an effective and lightweight framework that can be widely applied to Transformer-based models and enhances their non-stationary predictability to achieve state-of-the-art performance. [Table]
We report the averaged MSE/MAE of all prediction lengths (stated in Table 2) and the relative
MSE reduction ratios (Promotion) by our framework.
Full results (under all prediction lengths and promotion on ETSformer [36], FEDformer [40]) can be found in Appendix C
[Table] [Table] 4.2 Ablation Study
Quality evaluation To explore the role of each module in our proposed framework, we compare the prediction results on ETTm2 obtained by three models: vanilla Transformer, Transformer with only
Series Stationarization, and our Non-stationary Transformer.
In Figure 3, we find out that the two modules strengthen the non-stationary forecasting ability of Transformer from different perspectives.
Series Stationarization focuses on the alignment of statistical properties among each series input that benefits Transformer a lot to generalize on out-of-distribution data.
However, as is shown in
Figure 3(b), the over-stationarized circumstance for training makes the deep model more likely to output uneventful series with significant high stationarity and neglect the nature of non-stationary real-world data.
With the aid of De-stationary Attention, the model gives concern back to the inherent non-stationarity of real-world time series.
It is beneficial for an accurate prediction of the detailed series variation, which is vital in real-world time series forecasting.
[Figure]
[Table]
Quantitative performance
In addition to the above case study, we also provide quantitative forecasting performance comparison with stationarization methods: a deep method RevIN [17] and
Series Stationarization (Section 3.1).
As is shown in Table
[Table]
the forecasting results assisted by
RevIN and Series Stationarization are basically the same, which indicates that the parameter-free version of normalization in our framework performs sufficiently to stationarize time series.
Besides, the proposed De-stationary Attention in Non-stationary Transformers further boosts the performance and achieves the best in all six benchmarks.
The MSE reduction brought by De-stationary Attention becomes significant, especially when the dataset is highly non-stationary (Exchange: 0 .
569 → 0 . 461 ,
ETTm2: 0 .
461 → 0 . 306 ).
The comparison reveals that simply stationarizing time series still limits the predictive capability of Transformers, and the complementary mechanisms in Non-stationary
Transformers can properly release the models’ potential for non-stationary series forecasting.
4.3 Model Analysis
Over-stationarization problem To verify the over-stationarization problem from a statistical view
, we train Transformers with the aforementioned methods respectively, arrange all predicted time series in chronological order and compare the degree of stationarity with the ground truth (Figure 4).
While models solely equipped with stationarization methods tend to output series with unexpected high degree of stationarity, the results assisted by De-stationary Attention are close to the actual value (relative stationarity ∈ [97% , 103%] ).
Besides, as the degree of series stationarity increases, the overstationarization problem becomes more significant.
The huge discrepancy of the degree of stationarity can account for the inferior performance of Transformer with only stationarization.
And it also demonstrates that De-stationary Attention as an internal renovation alleviates over-stationarization.
[Figure]
Exploring of Non-stationary Information Re-incorporation It is notable that by specifying over-stationarization as less distinguishable attention, we narrow down our design space into the attention calculation mechanism.
To explore other approaches to retrieve non-stationary information, we conduct experiments by re-incorporating the µ and σ into feed-forward layers (DeFF), which is the left part of the Transformer architecture.
In detail, we feed learned µ and σ into each feed-forward layer iteratively.
As is shown in Table 6, re-incorporating non-stationarity is necessary only when the inputs are stationarized (Stationary), which is beneficial for forecasting but will lead to stationarity discrepancy of model outputs.
And our proposed design (Stat + DeAttn) makes further promotion and achieves the best in most cases (77%).
In addition to the theoretical analysis, experimental results further validate the effectiveness of our design in re-incorporating non-stationarity on attention [Table] [Table]  Conclusion
This paper addresses time series forecasting from the view of stationarity.
Unlike previous studies that simply attenuate non-stationarity leading to over-stationarization, we propose an efficient way to increase series stationarity and renovate the internal mechanism to re-incorporate non-stationary information, thus boosting data predictability and model predictive capability simultaneously.
Experimentally, our method shows great generality and performance on six real-world benchmarks.
And detailed derivations and ablations are provided to testify the effectiveness of each component in our proposed Non-stationary Transformers framework.
In the future, we will explore a more model-agnostic solution for the over-stationarization problem.
Acknowledgments
This work was supported by the National Key Research and Development Plan (2021YFC3000905),
National Natural Science Foundation of China (62022050 and 62021002), Beijing Nova Program (Z201100006820041), and BNRist Innovation Fund (BNR2021RC01002).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] See Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Appendix G. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix F. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix A. (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide the data link and code in https://github.com/thuml/Nonstationary_Transformers .
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix E.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix C.2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] The data source is described in Section 4.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We provide the code in https://github.com/thuml/Nonstationary_Transformers .
(d) Did you discuss whether and how consent was obtained from people whose data you’re us- ing/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Proof of De-stationary Attention
Definition.
Self-Attention [34] is defined as:
[Equation]
where Q , K and V ∈ R S ×
d k are length- S query, key and value, where S is the length of input sequence and d k is the feature dimension, and Softmax( · ) is conducted on each row.
Assumption 1.
The embedding layer and feed forward layer are functions conducted separately at each time point of the input and hold the linear property.
For example, the query Q as the input of the first Attn( · ) layer is obtained by feeding the input x =
[ x 1 , x 2 , · · · , x S ] ⊤ ∈ R S × C into the embedding layer f : R C × 1 → R d k × 1 , where C is the number of series variables.
And each of the query token in Q = [ q 1 , q 2 , ..., q S ] ⊤ can be calculated as q i = f
( x i ) w.r.t.
each time point in x = [ x 1 , x 2 , · · · , x S ] ⊤ .
Function f holds the linear property means that f ( ax + by ) =
af ( x )+ bf ( y ) , where a, b are scalars and x, y are vectors.
Assumption 2.
Each variable of the input series has the same variance.
For each input time series x , we calculate its mean and variance as follows:
[Equation]
where µ x , σ x ∈ R C × 1 is the mean and standard deviation of all x i s. Since it is a convention to conduct normalization on each series variable to avoid certain variable that dominates the scale, we can assume that each variable shares the same variance, and thus σ x is reduced to a scalar.
Theorem.
[Equation]
Equation 8 means the Softmax (cid:0) QK ⊤ / √ d k (cid:1) learned from raw series x can be calculated by current Q ′ , K ′ learned from stationarized series x ′ , and the calculation also requires the non-stationary information σ x , µ Q , K that are eliminated during stationarization.
Proof 1.
(First layer analysis) After our stationarization, the model receives the normalized input x ′ =
[ x ′ 1 , x ′ 2 , ..., x ′ S ]
⊤ and each x ′ i
=
(1 /σ x ) ⊙ ( x i − µ x ) .
Based on Assumption 2, σ x is reduced to a scalar and we can simplify the normalized input of each time point to x ′ i =
( x i − µ x ) /σ x .
Then x ′ is fed into the embedding layer f .
Based on Assumption 1, we get current query Q ′ =
[ q ′ 1 , ..., q ′ S ]
⊤ of the first Attn( · ) layer:
[Equation]
where µ Q = 1 S (cid:80) Si =1 q i ∈ R d k × 1 .
Then Q ′ =
[ q ′ 1 , ..., q ′ S ] ⊤ can be written as ( Q − 1 µ ⊤ Q ) /σ
x and 1 ∈ R S ×is an all-ones vector.
And so is the corresponding transformed K ′ .
Without the stationarization, the input of Softmax( · ) in Self-Attention should be ( QK ⊤ / √ d k ) , while now the attention is calculated based on Q ′ , K ′ .
And we have the following equations:
[Equation]
We find that Q µ K ∈ R S × 1 and µ ⊤ Q µ K ∈ R , and they are repeatedly operated on each column and element of σ 2 x Q ′ K ′⊤ ∈ R S × S .
Since Softmax( · ) is invariant to the same translation on the row dimension of input, we have the following equation:
[Equation]
Proof 2.
(Multiple layers analysis)
We have deduced an equivalent expression of the output of the first Softmax( · ) .
If we can successfully approximate the attention map that related to Q and K , we only need to consider Attn( · ) with respect to the change of V .
Fortunately, Attn( · ) as the function of V gives each time point of the output E =
[ e 1 , ..., e S ] ⊤ ∈ R S
×
d k as a simplex: e j
[Equation]
which also holds the linear property f ( a V 1 + b V 2 )
=
af ( V 1 )
+ bf ( V 2 ) .
Therefore, the Attn( · ) layer is also a function that satisfies our Assumption 1.
We will have each time point of the output E varies linearly with each time point of the input x , and then E will become the next block’s input.
As the feed forward layer, residual adding and Attn( · ) layer are the repeating building blocks of Transformer, they also compose a function with linear property as stated in Assumption 1.
By the first layer analysis and induction on each layer, Equation 8 will holds for Softmax( · ) of all layers under our assumptions.
Attention design Based on the analysis, we develop De-stationary Attention as:
[Equation]
where τ ∈ R + and ∆ ∈ R S × 1 is defined as the scaling and shifting de-stationary factors respectively to approximate σ 2 x and K µ Q under the real scenario.
Since the key to making Equation 8 established is to approximate the attention map successfully, we apply a direct deep learning implementation.
To be concisely, we use a multi-layer perceptron as the projector to learn de-stationary factors τ, ∆ from the statistics µ x , σ x and unstationarized x .
De-stationary Attention learns the temporal dependencies from both stationarized series Q ′ ,
K ′ and non-stationary series x , µ x , σ x , and multiplies by the stationarized values V ′ to keep the linear property.
It can benefit from the predictability of stationarized series and re-incorporate the inherent non-stationarity of raw series simultaneously.
B Hyperparameter Sensitivity
We verify the robustness of the proposed Non-stationary Transformers framework with respect to hyper-parameter dim , which is the hidden layer dimension of the MLP projector that learns de-stationary factors.
Considering the efficiency of hyperparameters search, we fix the number of hidden layers, and the hidden layer dimension varies in { 64 , 128 , 256 } .
The results are shown in Table 7.
For datasets with relatively high non-stationarity (Exchange and ILI), large dim would be a better choice, which indicates that non-stationary information entangled with unstationarized input should be learned by a projector with big capacity.
Besides, as the dataset presents higher non-stationarity, the influence of de-stationary project design becomes more significant.
[Table]
C Supplementary of Main Results C.1 Multivariable Forecasting Results
As shown in Table 8, we list additional benchmark on the ETT datasets [39], which includes the hourly recorded ETTh1/ETTh2 and 15-minutely recorded ETTm1.
Non-stationary Transformer also achieves remarkable improvement over the state-of-the-art on various forecasting horizons.
For the input-96-predict-336 long-term setting, Non-stationary Transformer surpasses previous best results by 4.4% ( 0 . 615 → 0 . 588 ) in ETTh1, 3.5% ( 0 .
572 → 0 . 552 )
in ETTh2 and 26.7% ( 0 . 675 → 0 . 495 ) MSE reduction in ETTm1.
The overall results show averaged 11.5% MSE reduction over previous state-of-the-art deep forecasting models.
We also list additional model comparison in Table 9, including the concurrent work FEDformer [40], and non-Transformer models LSSL [15] and GRU [11].
Our method still outperforms these models in most cases (83%).
Notably, LSSL [15] achieves good performance on Weather [4] dataset with the highest stationarity but poorly performs on others, especially non-stationary datasets.
[Table]
ETT
[Table] [Table]
I L I 2.
ETT m 0.
E l ec t r i c it y 0.
T r a f fi c 0.
W ea t h e r 0.
C.2 Performance of Non-stationary Transformer and Variants
We apply our proposed Non-stationary Transformers framework to six Transformer variants: Transformer [34], Informer [39], Reformer [19], Autoformer [37], ETSformer [36] and FEDformer [40].
The averaged results are shown in Table 4 due to the limited pages.
We provide supplementary forecasting results in Table 10 and [Table]
ETT
[Table]
E
[Table] ec [Table] i c it
[Table]
C.3 Comparison with Stationarization Methods
We provide full comparison among Non-stationary Transformers and two stationarization methods: Revin[17] and Series Stationarization.
The averaged results are shown in Table 5 due to the limited pages.
As is listed in Table 12, our framework achieves the state-of-the-art performance especially on datasets with high non- stationarity.
For Transformer, the proposed method achieves 25.6% ( 1 . 467 → 1 . 092 ) MSE reduction on
Exchange under the predict-720 settings, 10.8% ( 2 . 572 → 2 . 294 ) on ILI under the predict-24 settings, and 30.3% ( 0 .
→ 0 . 417 ) on ETTm2 under the predict-720 settings.
As for Reformer, since De-stationary
Attention is not directly deduced from the LSH attention [19], current approximation as stated in Equation 8 may not be the optimal solution
[Table]
but the introducing of De
[Table] [Table]
0 . 559 ) promotion on ETTm2 and 4.4% ( 2 .
→ 2 . 770 ) on ILI under the predict-336 setting.
The comparison demonstrates De-stationary Attention mechanism can further benefit the predictive ability of Transformers.
[Table]
C.4 Prediction Showcases
We provide supplementary showcases of predictions given by three models: vanilla Transformer, Transformer with Series Stationarization
[Table]
and Non-stationary Transformer.
We plot the last dimension of forecasting results that comes from the test set of ETTm1 for qualitative comparison
[Table]
As is shown in Figures 5, 6,
[Table]
and
[Table]
we find that vanilla Transformer is inclined to output predictions with scale and level far from the ground truth
[Table]
but its ability to capture local series variation remains strong.
While Series
Stationarization benefits Transformer by aligning the statistics among each series
[Table]
the base model neglects the intrinsic non-stationarity of time series and becomes more likely to output stationary but uneventful series.
With the help of our framework
[Table]
the equipped model will be free from the disturbance caused by data non-stationarity and fulfill the potential to capture local variations.
[Table]
C.5 Efficiency of Non-stationary Transformers
As is shown in Table 13, we list the parameters increment and the performance gain (see Promotion in Table 4) of our proposed method.
It is obvious that Non-stationary Transformers significantly boosts the forecasting performance by a large margin with hardly any additional parameters.
[Figure] [Figure]
[Figure] [Figure]
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
D Ablations D.1 Effects of Series Stationarization
We propose Series Stationarization, which has no additional learnable parameters, to increase the degree of stationarity and make the time series distribution more stable.
As is shown in Table 14, after our normalization module processing, the ADF test statistic of the time series gets obviously smaller, which verifies normalization as an effective design to attenuate the non-stationarity of real world time series.
D.2 Ablation of De-stationary Factors
To explore the influence of de-stationary factors
[Table]
we compare the forecasting results obtained by three variants: only using τ , only using ∆ , and using both.
We conduct experiments on two typical datasets: Exchange ( variables) and Electricity (321 variables).
As is shown in Table 15, the forecasting performance will degrade in all cases if we only employ single one of τ and ∆ , especially without τ ( 0 .
→ 0 . 212 , 0 .
441 → 0 .
550 under the predict-336 setting), which validates the complete form as stated in Equation 9 is a better choice.
[Table]
E Non-stationary Transformers: Experimental Details E.1 Detailed Experiment Configurations
We compare each Transformers with and without our framework using the same training strategy.
The only hyperparameters for our framework come from the projector design which learns de-stationary factors.
We search the hyperparameters as stated in Appendix B.
The best hyperparameter is selected on the validation set .
As for other forecast models for the baseline comparison, most of the results are from Autoformer [37].
By contacting the authors of Autoformer, we obtain the hyper-parameter selection strategy as follows: for NBEATS [27], we conduct a grid search for hidden channel in { 256 , 512 , 768 } , number of layers in { 2 , 3 , 4 , 5 } , learning rate in { 5 × 10 − 5 , 1 × 10 − 4 , 5 × 10 − 4 , 1 × 10 − 3 } .
For LSTNet [20], since the paper also experiments on the Traffic [3], Electricity [1] and Exchange [20] datasets, the hyper-parameter setting is following the experimental details of the original paper.
For N-HiTs [9], ETSformer [36], and FEDformer [40], as these methods share the same benchmark, we use their official code with three random seeds.
E.2 Implementation Details of Non-stationary Transformer and Variants
We provide the pseudo-code of Series Stationarization, De-stationary Attention and Non-stationary Transformers in Algorithms 1, 2, 3 and 4.
All Transformers have two-layer encoder and one-layer decoder with the feature dimension d k =512, including Transformer [34], Informer [39], Reformer [19], Autoformer [37], ETSformer [36] and FEDformer [40].
Besides, we adopt embedding method and one-step generation strategy of Informer [39].
It is worth noting that for the row length of attention map differs from S × S , where S is the initial input sequence length, we omit the shifting de-stationary factor ∆ in Equation 9 (i.e., the Self-Attention layer of
Transformer decoder, and the Self-Attention layer of the Informer encoder where the shape of attention map is changed over layers), since the performance of only use τ will not degenerate a lot as shown in Table 15.
For the cross attention, we first conduct the rescaling operation with de-stationary factors and then multiply by the corresponding mask.
For Transformer variants, we conduct the rescaling operation on the pre- Softmax scores.
Algorithm
Require:
Input past time series x ∈ R S × C ; Input Length S ; Variables number C
▷ µ x ∈
C
▷ σ x ∈ R
C
Algorithm
Require:
Predicted time series y ′ ∈
R O × C by the base model; original statistics input µ x x
Output
Variables number
Algorithm 3 De-stationary Attention.
Require:
Queries Q ′ ∈ R S
×
d k ; Keys
R
Input
Feature dimension d k .
Algorithm
Transformers -
Architecture.
Require:
Input past time series x ∈ R S × C ; Input Length S ; Predict length O ; Variables number C ;
Feature dimension d k ; Encoder layers number N ; Decoder layers number M .
Technically, we set d k as 512, N as 2, M as 1.
1: x ′ , µ x , σ x = Normaliztion ( x ) ▷
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
l − 1 ′ dec + FFN ( x l − 1 ′ dec ) (cid:1)
▷ x l ′ dec ∈ R ( S 2 + O ) ×
d k 14: End for 15: y ′ =
MLP ( x M ′ dec ) S 2 : S 2 + O ▷ y ′ ∈
R
O × d k 16:
y =
De normaliztion ( y ′ , µ x , σ x )
▷ y ∈ R
O × d k 17: Return y ▷ Return the prediction results
F Broader Impact F.1 Impact on Real-world Applications
We focus on real-world time series forecasting, which is challenging for Transformers because of data non- stationarity.
Our method goes beyond previous studies that only stationarize the time series.
We fully utilize the predictive capability of attention mechanism that captures essential temporal dependencies associated with inherent non-stationarity. Our proposed method achieves state-of-the-art performance in five real-world applications, which makes it more promising for Transformers to tackle real-world forecasting applications, and helps our society make better decisions and prevent risks in advance for various fields. And our paper mainly focuses on scientific research and has no obvious negative social impact. F.2 Impact on Future Research
In this paper, we analyze the generalization difficulty of Transformers in distribution-varying time series forecasting.
We propose a general framework to fulfill the potential of Transforms constrained by data nonstationary.
Our work introduces an essential and promising direction to improve forecasting performance: to increase the stationarity of time series towards better predictability and mitigate the over-stationarization problem for the predictive capability of deep models simultaneously.
The remarkable generality and effectiveness of the proposed framework can be instructive for future research.
G Limitation
Our De-stationary Attention is deduced by analyzing the vanilla Self-Attention, which may not be the optimal solution for advanced attention mechanisms.
There also remains room for re-incorporating non-stationarity on other classical stationarization methods, like differencing and quantile.
Besides, the proposed framework is currently limited to the Transformer-based models, while the over-stationarization problem can appear on any deep time forecasting models if using stationarization methods inappropriately.
Therefore, a more model-agnostic solution for the over-stationarization problem will be our exploring direction.