Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting
Anonymous Author(s)
AffiliationAddress email
Abstract
Transformers have shown great power in time series forecasting due to their globalrange modeling ability.
However, their performance can degenerate terribly onnon-stationary real-world data in which the joint distribution changes over time.
Previous studies primarily adopt stationarization to attenuate the non-stationarity oforiginal series for better predictability.
But the stationarized series deprived of inherent non-stationarity can be less instructive for real-world bursty events forecasting.
This problem, termed over-stationarization in this paper, leads Transformers togenerate indistinguishable temporal attentions for different series and impedesthe predictive capability of deep models.
To tackle the dilemma between seriespredictability and model capability, we propose Non-stationary Transformers as ageneric framework with two interdependent modules: Series Stationarization and
De-stationary Attention.
Concretely, Series Stationarization unifies the statistics ofeach input and converts the output with restored statistics for better predictability.
To address the over-stationarization problem, De-stationary Attention is devised torecover the intrinsic non-stationary information into temporal dependencies by approximating distinguishable attentions learned from raw series.
Our Non-stationary
Transformers framework consistently boosts mainstream Transformers by a largemargin, which reduces 49.43% MSE on Transformer, 47.34% on Informer, and46.89% on Reformer, making them the state-of-the-art in time series forecasting.
Introduction
Time series forecasting has become increasingly ubiquitous in real-world applications, such asweather forecasting, energy consumption planning, estimation of future illness cases and financialrisk assessment.
Recently, Transformers [32] have achieved progressive breakthrough on extensiveareas [11, 12, 10, 22].
Especially in time series forecasting, credited to their stacked structure and thecapability of attention mechanisms, Transformers can naturally capture the temporal dependenciesfrom deep multi-level features [36, 17, 20, 34], thereby fitting the series forecasting task perfectly.
Despite the remarkable architectural design, it is still challenging for Transformers to predict realworld time series because of the non-stationarity of data.
Non-stationary time series are characterizedby the continuous change of statistical properties and joint distribution over time [6, 14].
Onthe one hand, non-stationarity makes the time series less predictable [6, 14].
On the other hand,it is a fundamental but challenging problem to make deep models generalize well on a varyingdistribution [26, 19, 5].
To further specify the analysis of stationarity, we introduce the degreeof stationarity to represent the distribution change degree of time series, which can be measuredquantitatively by widely-used statistics [13].
In previous work, it is generally acknowledged to preprocess the time series by stationarization [24, 27, 15], which can increase the degree of stationarityof raw time series for better predictability and provide more stable data distribution for deep models.
[Figure]
However, non-stationarity is the inherent property of real-world time series and provides good guidance for discovering temporal dependencies for forecasting.
Experimentally, we observe that trainingon the stationarized series will undermine the distinction of attentions learned by Transformers.
Asshown in Figure 1(a), vanilla Transformers [32] can capture distinct temporal dependencies fromdifferent series. However, in Figure 1(b), Transformers trained on the stationarized series tends togenerate indistinguishable attentions, naming by the over-stationarization problem.
It will bring unexpected side-effect that make Transformers fail to capture eventful temporal dependencies, limit themodel’s predictive ability, and even induce the model to generate outputs with huge non-stationaritydeviation from the ground truth.
Thus, how to increase the degree of series stationarity towards betterpredictability and mitigate the over-stationarization problem for model capability simultaneously isthe key problem to further improve the performance of forecasting.
In this paper, we explore the effect of stationarization in time series forecasting and propose Nonstationary Transformers as a general framework, which empowers Transformer [32] and its efficientvariants [17, 36, 34] with great predictive ability for real-world time series.
The proposed frameworkinvolves two interdependent modules: Series Stationarization to increase the predictability of nonstationary series and De-stationary Attention to alleviate over-stationarization.
Technically, Series
Stationarization adopts a simple but effective normalization strategy to unify the key statistics ofeach series without extra parameters.
And De-stationary Attention approximates the attention ofunstationarized data and compensates the intrinsic non-stationarity of raw series.
Benefiting fromthe above designs, Non-stationary Transformers can take advantage of the great predictability ofstationarized series and crucial temporal dependencies discovered from original non-stationary data.
Our method achieves state-of-the-art performance on six real-world benchmarks and can generalizeto various Transformers for further improvement on base models.
The contributions lie in three folds:
• We refine that the predictive capability of non-stationary series is essential in real-world
forecasting.
By detailed analysis, we find out that current stationarization approaches willlead to the over-stationarization problem, limiting the predictive capability of Transformers.•
We propose Non-stationary Transformers as a generic framework, including Series Stationarization to make the series more predictable and De-stationary Attention to avoid the
over-stationarization problem by re-incorporating the non-stationarity of original series.• Non-stationary Transformers consistently boosts four mainstream Transformers by a large margin and achieves state-of-the-art performance on six real-world benchmarks.
Related Work
2.1 Deep Models for Time Series Forecasting
In recent years, deep models with elaboratively designed architectures have achieved great progressin time series forecasting.
RNN-based models [33, 35, 23, 29, 30] are proposed for application in anautoregressive manner for sequence modeling, but the recurrent structure can suffer from modeling long-term dependency.
Soon afterward, Transformer [32] emerges and shows great power in sequencemodeling.
To overcome the quadratic computation growth on sequence length, subsequent worksaim to reduce Self-Attention’s complexity.
Especially in time series forecasting, Informer [36]extends Self-Attention with KL-divergence criterion to select dominant queries.
Reformer [17]introduces local-sensitive hashing (LSH) to approximate attention by allocated similar queries.
Notonly improved by reduced complexity, the following models further develop delicate building blocksfor time series forecasting.
Autoformer [34] fuses the decomposition blocks into a canonical structureand develops Auto-Correlation to discover series-wise connections.
Pyraformer [21] designs pyramidattention module (PAM) to capture temporal dependencies with different hierarchies.
Other deepbut Transformer-free models also achieve remarkable performance.
N-BEATS [25] proposes theexplicit decomposition of trend and seasonal terms with strong interpretability.
N-HiTS [9] introduceshierarchical layout and multi-rate sampling for tackling time series with respective frequency bands.
In this paper, different from previous works focusing on architectural design, we analyze the seriesforecasting task from the basic view of stationarity, which is an essential property of time series [6, 14].
It is also notable that as a general framework, our proposed Non-stationary Transformers can beeasily applied to various Transformer-based models.
2.2 Stationarization for Time Series Forecasting
While stationarity is important to the predictability of time series [6, 14], real-world series alwayspresent non-stationarity.
To tackle this problem, the classical statistical method ARIMA [7, 8]stationarizes the time series through differencing.
As for deep models, since the distribution-varyingproblem accompanied by non-stationarity makes deep forecasting even more intractable, stationarization methods are widely explored and always adopted as the pre-processing for deep model inputs.
Adaptive Norm [24] applies z-score normalization for each series fragment by global statistics of asampled set.
DAIN [27] employs a nonlinear neural network to adaptively stationarize time serieswith observed training distribution.
RevIN [15] introduces a two-stage instance normalization [31]that transforms model input and output respectively to reduce the discrepancy of each series.
Incontrast, we find out that directly stationarizing time series will damage the model’s capability ofmodeling specific temporal dependency.
Therefore, unlike previous methods, in addition to thestationarization, Non-stationary Transformers further develops De-stationary Attention to bring theintrinsic non-stationarity of the raw series back to attention.
Non-stationary Transformers
As aforementioned, stationarity is an important element of time series predictability.
Previous “directstationarization” designs can increase the degree of series stationarity for better predictability, but theyobviously neglect inherent properties of real-world series, which will result in the over-stationarizationproblem as stated in Figure 1.
To deal with the dilemma, we go beyond previous works and propose Non-stationary Transformers as a generic framework.
Our model involves two complementary parts: Series Stationarization to increase the degree of series stationarity and De-stationary Attention tore-incorporate non-stationary information of raw series.
Empowered by these designs, Non-stationary
Transformers can improve data predictability and maintain model capability simultaneously.
3.1 Series Stationarization
Non-stationary time series make the forecasting task intractable for deep models because it is hardfor them to generalize well on series with changed statistics during inference, typically varied meanand standard deviation.
The pilot work, RevIN [15] applies instance normalization with learnableaffine parameters to each input and restores the statistics to the corresponding output, which makeseach series follow a similar distribution.
Experimentally, we find that this design also works wellwithout learnable parameters.
Thus, we propose a more straightforward but effective design to wrap
Transformers as the base model without extra parameters, naming by Series Stationarization.
As isshown in Figure 2, it contains two corresponding operations: Normalization module at first to dealwith the non-stationary series caused by varied mean and standard deviation, and De-normalizationmodule at the end to transform the model outputs back with original statistics.
Here are the details.
[Table]
quunQTLIKrkoioy6KbLivYB9Qik+m0DebFZFIopbj1B9zqZ4l/oH/hnTEFtYhOSHLm3HPuzL3XSwI/VY7zWrAWFpeWV4qra+sbm1vbpZ3dZhpnkosGj4NYtj2WisCPREP5KhDtRAoWeoFoeXeXOt4aCZn6cXStxonohmwQ+X2fM0VU5yZkashZMKlNb0tlp+KYZc8DNwdl5Ksel15wgx5icGQIIRBBEQ7AkNLTgQsHCXFdTIiThHwTF5hijbwZqQQpGLF39B3QrpOzEe11ztS4OZ0S0CvJaeOQPDHpJGF9mm3imcms2d9yT0xOfbcx/b08V0iswpDYv3wz5X99uhaFPs5NDT7VlBhGV8fzLJnpir65/aUqRRkS4jTuUVwS5sY567NtPKmpXfeWmfibUWpW73muzfCub0kDdn+Ocx40jyvuacW9OilXL/JRF7GPAxzRPM9QRQ11NEzHH/GEZ6tujaypdf8ptQq5Zw/flvXwAXjZk+U=</latexit>
H
Normalization module To increase the degree of stationarity for input series, we conduct normalization on the temporal dimension by a sliding window over time.
For each input seriesx = [ x 1 , x 2 , ..., x S ] ⊤ ∈ R S × C , we transform it by translation and scaling operations and obtain x ′ =
[ x ′ 1 , x ′ 2 , ..., x ′ S ] ⊤ ∈ R S × C , where S and C denote the sequence length and variable numberrespectively.
The Normalization module can be formulated as follows:
[Equation]
where µ x , σ x ∈ R C × 1 , 1 σ x means the element-wise division and ⊙ is the element-wise product.
Notethat Normalization module decreases the distributional discrepancy among each input time series,making the distribution of the model input more stable.
De-normalization module As shown in Figure 2, after the base model H predicting the future valuewith length-
O , we adopt De-normalization to transform the model output y ′ =
[ y ′ 1 , y ′ 2 , ..., y ′
O ] ⊤ ∈
R O × C with σ x and µ x and obtain ˆ y =
[ˆ y 1 , ˆ y 2 , ..., ˆ y O ] ⊤ as the eventual forecasting results.
The
De-normalization module can be formulated as follows:
[Equation]
By means of the two-stage transformation, the base models will receive inputs with higher degree ofstationarity, that is following a more stable distribution. This design also makes the model equivariantto translational and scaling perturbance of time series, thereby benefiting real-world series forecasting.
3.2 De-stationary Attention
While the statistics of each time series are explicitly restored to the corresponding prediction, thenon-stationarity of the original series can not be fully recovered only by De-normalization.
Forinstance, Series Stationarization can generate the same stationarized input x ′ from distinct time seriesx 1 , x 2 (i.e. x 2 =
α x 1 + β ), and the base model will get identical attention that fail to capture crucialtemporal dependencies entangled with non-stationarity (Figure 1).
In other words, the underminedeffects caused by over-stationarization happen inside the deep model, especially in the calculation ofattention.
Furthermore, non-stationary time series are fragmented and normalized into several serieschunks with the same mean and variance, which follow more similar distributions than the raw databefore stationarization. Thus, the model is more likely to generate over-stationary and uneventfuloutputs, which is irreconcilable with the natural non-stationarity of the original series.
To tackle the over-stationarization problem caused by Series Stationarization, we propose a novel
De-stationary Attention mechanism, which can approximate the attention that is obtained withoutstationarization and discover the particular temporal dependencies from original non-stationary data.
Analysis of the plain model As mentioned above, the over-stationarization problem is causedby the vanishment of inherent non-stationarity information, which will make the base model fail to capture eventful temporal dependencies for forecasting.
Therefore, we try to approximate the attentionlearned from the original non-stationary series.
We start from the formula of Self-Attention [32]:
[Equation]
where Q , K , V ∈ R S ×
d k are length- S queries, keys and values of d k -dimension respectively, and
Softmax( · ) is conducted row by row.
To simplify the analysis, we assume the embedding andfeed-forward layers f to hold the linear properties 1 and f is conducted separately on each time point,that is, each query token in Q =
[ q 1 , q 2 , ..., q S ] ⊤ can be calculated as q i = f
( x i ) with respect tothe input series x =
[ x 1 , x 2 , · · · , x S ] ⊤ .
Since it is a convention to conduct normalization on each 160 time series variable to avoid certain variable that dominates the scale, we can further assume eachvariable of series x shares the same variance, and thus original σ x ∈ R C × 1 is reduced to a scalar.
After Normalization module, the model receives the stationarized input x ′ =
( x − 1 µ ⊤ x ) /σ x , where1 ∈ R S × 1 is an all-ones vector.
Based on the linear property assumption, it can be proved that
the
Attention layer will receive Q ′ =
[ f ( x ′ 1 ) , ..., f ( x ′ S )]
⊤ = ( Q − 1 µ ⊤ Q ) /σ x , where µ Q ∈ R d k × 1 is themean of Q along the temporal dimension (See supplementary materials for a detailed proof).
And sois the corresponding transformed K ′ , V ′ .
Without Series Stationarization, the input of Softmax( · ) in
Self-Attention should be QK ⊤ / √ d k , while now the attention is calculated based on Q ′ , K ′ :
[Equation]
We find that Q µ K ∈ R S × 1 and µ ⊤ Q µ K ∈ R , and they are repeatedly operated on each column andelement of σ 2 x Q ′ K ′⊤ ∈ R S × S respectively.
Since Softmax( · ) is invariant to the same translation onthe row dimension of input, we have the following equation:
[Equation]
The Equation 5 deduces a direct expression of the attention Softmax (cid:0) QK ⊤ / √ d k (cid:1) learned from rawseries x .
Except for the current Q ′ , K ′ from stationarized series x ′ , this expression also requires thenon-stationary information σ x , µ Q , K that are eliminated by Series Stationarization.
De-stationary Attention To recover the original attention on non-stationary series , we attempt tobring the vanished non-stationary information back to its calculation.
Based on the Equation 5, the keyis to approximate the positive scaling scalar τ = σ 2 x ∈ R + and shifting vector ∆ =
K µ Q ∈ R S × 1 ,which are defined as de-stationary factors .
Since the strict linear property hardly holds for a deepmodel, other than estimating and utilizing real factors with great effort, we try to learn de-stationaryfactors directly from the statistics of unstationarized x , Q and K by a simple but effective multilayer perceptron layer.
As we can only discover limited non-stationary information from current Q ′ , K ′ , the unique and reasonable source to compensate non-stationarity is the original x withoutbeing normalized.
Thus, as a direct deep learning implementation of Equation 5, we apply a multilayer perceptron as the projector to learn de-stationary factors τ, ∆ from the statistics µ x , σ x ofunstationarized x individually.
And the De-stationary Attention is calculated as follows:
[Equation]
where the de-stationary factors τ and ∆ are shared by De-stationary Attention of all layers (Figure 2).
De-stationary Attention mechanism learns the temporal dependencies from both stationarized series
Q ′ , K ′ and non-stationary series x , µ x , σ x , and multiplies by the stationarized values V ′ .
Therefore,it can benefit from the predictability of stationarized series and maintain the inherent temporaldependencies of raw series simultaneously.
Overall architecture Following the prior use of Transformers [36, 34] in time series forecasting, we adopt the standard Encoder-Decoder structure (Figure 2), where the encoder is to extractinformation from past observations, and the decoder is to aggregate past information and refinethe prediction from simple initialization.
The canonical Non-stationary Transformer is wrapped by
Series Stationarization to both the input and output of vanilla Transformer [32], and replacing the
Self-Attention by our proposed De-stationary Attention, which can boost the non-stationary seriespredictive capability of the base model.
For the Transformer variants [17, 36, 34], we transformthe terms inside Softmax( · ) with the de-stationary factors τ , ∆ to re-integrate the non-stationaryinformation (See supplementary materials for the implementation details).
Experiments
We conduct extensive experiments to evaluate the performance of Non-stationary Transformers onsix real-world time series forecasting benchmarks and further validate the generality of the proposedframework on various mainstream Transformer variants.
See supplementary materials for the code.
Datasets Here are the descriptions of the datasets: (1) Electricity [1] records the hourly electricityconsumption of 321 clients from 2012 to 2014.
(2) ETT [36] contains the time series of oil destationary factors and power load collected by electricity transformers from July 2016 to July 2018.
ETTm1 /ETTm2 are recorded every 15 minutes, and ETTh1/ETTh2 are recorded every hour.
Exchange [18] collects the panel data of daily exchange rates from 8 countries from 1990 to 2016.(4) ILI [2] collects the ratio of influenza-like illness patients versus the total patients in one week,which is reported weekly by Centers for Disease Control and Prevention of the United States from2002 and 2021.
(5) Traffic [3] contains hourly road occupancy rates measured by 862 sensors on
San Francisco Bay area freeways from January 2015 to December 2016.
(6) Weather [4] includesmeteorological time series with 21 weather indicators collected every 10 minutes from the Weather
Station of the Max Planck Biogeochemistry Institute in 2020.
Especially, in this paper, we adopt the
Augmented Dick-Fuller (ADF) test statistic [13] as the metric to measure the degree of stationarity. A smaller ADF test statistic indicates a higher degree of stationarity, which means the distributionis more stable. Table 1 summarizes the overall statistics of the datasets and lists them in ascendingorder by degree of stationarity.
We follow the standard protocol that divides each dataset into thetraining, validation, and testing subsets according to the chronological order.
The split ratio is 6:2:2for the ETT dataset and 7:1:2 for others.
[Table]
Baselines We evaluate the vanilla Transformer [32] equipped by the Non-stationary Tranformers framework in both multivariate and univariate settings to demonstrate its effectiveness.
Formultivariate forecasting, we include six state-of-the-art deep forecasting models: Autoformer [34],
Pyraformer [21], Informer [36], LogTrans [20], Reformer [17] and LSTNet [18].
For univariateforecasting, we include seven competitive baselines: N-HiTS [9], N-BEATS [25], Autoformer [34],
Pyraformer [21], Informer [36], Reformer [17] and ARIMA [7].
In addition, we adopt the proposed framework on both the canonical and efficient variants of Transformers: Transformer [32],
Informer [36], Reformer [17] and Autoformer [34] to validate the generality of our framework.
Implementation details All the experiments are implemented in PyTorch [28] and conducted forthree runs on a single NVIDIA TITAN V 12GB GPU.
Each model is trained by ADAM [16] using
L2 loss with the initial learning rate of 10 − 4 and batch size of 32.
Each Transformer-based modelcontains two encoder layers and one decoder layer.
Considering the efficiency of hyperparameterssearch, we use two-layer perceptron projector with the hidden dimension varying in { 64 , 128 , 256 }in De-stationary Attention.
with different random seeds. We repeat each experiment three times andreport the test MSE/MAE under different prediction lengths, and the standard deviations are alsoprovided in the supplementary materials.
A lower MSE/MAE indicates better performance.
4.1 Main Results
Forecasting results As for multivariate forecasting results, the vanilla Transformer equipped withour framework consistently achieves state-of-the-art performance in all benchmarks and predictionlengths (Table 2).
Notably, Non-stationary Transformer outperforms other deep models impressivelyon datasets characterized by high non-stationarity.
With the prediction length of 336, we achieve17% MSE reduction ( 0 . 509 → 0 . 421 ) on Exchange and 25% ( 2 . 669 → 2 . 010 ) on ILI compared toprevious state-of-the-art results, which indicates that the potential of deep model is still constrainedon non-stationary data.
We also list the univariate results of two typical datasets with differentstationarity in Table 3.
Non-stationary Transformer still realizes remarkable forecasting performance.
[Table] [Table]
[Table] [Table]
ETT
[Table]
[Table]
Framework generality We apply our framework to four mainstream Transformers and report theperformance promotion of each model (Table 4).
Our method consistently improves the forecastingability of different models.
Overall, it achieves averaged 49.43% promotion on Transformer, 47.34%on Informer, 46.89% on Reformer and 10.57% on Autoformer, making each of them surpassprevious state-of-the-art.
Compared to native blocks of the models, there is hardly any parameterand computation increase by applying our framework (See supplementary materials for details),and thereby their computational complexities can be preserved.
It validates that Non-stationary
Transformers is an effective and lightweight framework that can be widely applied to Transformerbased models and enhances their non-stationary predictability to achieve state-of-the-art performance.
4.2 Model Analysis
Quality evaluation To explore the role of each module in our proposed framework, we comparethe prediction results on ETTm2 obtained by three models: vanilla Transformer, Transformer withsolely Series Stationarization, and Non-stationary Transformer.
In Figure 3, we find out that the twomodules strengthen the non-stationary forecasting ability of Transformer from different perspectives.
Series Stationarization focuses on the alignment of statistical properties among each series inputthat benefits Transformer a lot to generalize on out-of-distribution data.
However, as is shown in
Figure 3(b), the over-stationarized circumstance for training makes the deep model more likely tooutput uneventful series with significant stronger stationarity than raw data and neglect the nature ofnon-stationary real-world data.
With the aid of De-stationary Attention, the model gives concern backto the inherent non-stationarity of real-world time series.
It is beneficial for an accurate prediction ofthe detailed series variation, which is vital in real-world time series forecasting.
[Figure]
Quantitative performance
In addition to the above case study, we also provide quantitativeforecasting performance comparison with stationarization methods: a deep method RevIN [15] and
Series Stationarization (Section 3.1).
As is shown in Table 5, the forecasting results assisted by
RevIN and Series Stationarization are basically the same, which indicates that the parameter-free version of normalization in our framework performs sufficiently to stationarize time series.
Besides,the proposed De-stationary Attention in Non-stationary Transformers further boosts the performanceand achieves the best in all six benchmarks.
The MSE reduction brought by De-stationary Attentionbecomes significant, especially when the dataset is highly non-stationary (Exchange: 0 .
569 → 0 . 461 ,
ETTm2: 0 .
461 → 0 . 306 ).
The comparison reveals that solely stationarizing time series still limitsthe predictive capability of Transformers, and the complementary mechanisms in Non-stationary
Transformers can properly release the models’ potential for non-stationary series forecasting.
[Table]
Over-stationarization problem To verify the over-stationarization problem from a statistical view
,we train Transformers with the aforementioned methods respectively, arrange all predicted time seriesin chronological order and compare the degree of stationarity with the ground truth (Figure 4).
Whilemodels solely equipped with stationarization methods tend to output series with unexpected highdegree of stationarity, the results assisted by De-stationary Attention are close to the actual value(relative stationarity ∈ [97% , 103%] ).
Besides, as the degree of series stationarity increases, the overstationarization problem becomes more significant.
The huge discrepancy of the degree of stationaritycan account for the inferior performance of Transformer with stationarization only.
And it alsodemonstrates that De-stationary Attention as an internal renovation can alleviate over-stationarization.
[Figure]
Conclusion
This paper addresses time series forecasting from the perspective of stationarity.
Unlike previousstudies that solely increase the degree of stationarity of original series, we propose Non-stationary Transformers to increase the predictability of time series and fully exploit the predictive capability ofthe model simultaneously.
Our proposed framework consists of two complementary mechanisms,where Series Stationarization is an efficient implementation to increase the degree of stationarity oftime series and De-stationary Attention renovates the internal mechanism to recover the inherentnon-stationarity of real-world series.
The introduction of our framework boosts the forecastingperformance of mainstream Transformers to the state-of-the-art on six real-world benchmarks.
Wewill