Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting
Abstract
Transformers have shown great power in time series forecasting due to their global-range modeling ability.
However, their performance can degenerate terribly on non-stationary real-world data in which the joint distribution changes over time.
Previous studies primarily adopt stationarization to attenuate the nonstationarity of original series for better predictability.
But the stationarized series deprived of inherent non-stationarity can be less instructive for real-world bursty events forecasting.
This problem, termed over-stationarization in this paper, leads
Transformers to generate indistinguishable temporal attentions for different series and impedes the predictive capability of deep models.
To tackle the dilemma between series predictability and model capability, we propose Non-stationary
Transformers as a generic framework with two interdependent modules: Series
Stationarization and De-stationary Attention.
Concretely, Series Stationarization unifies the statistics of each input and converts the output with restored statistics for better predictability.
To address the over-stationarization problem, Destationary Attention is devised to recover the intrinsic non-stationary information into temporal dependencies by approximating distinguishable attentions learned from raw series.
Our Non-stationary Transformers framework consistently boosts mainstream Transformers by a large margin, which reduces MSE by 49.43% on
Transformer, 47.34% on Informer, and 46.89% on Reformer, making them the state-of-the-art in time series forecasting.
Code is available at this repository: https://github.com/thuml/Nonstationary_Transformers .
Introduction
Time series forecasting has become increasingly ubiquitous in real-world applications, such as weather forecasting, energy consumption planning, and financial risk assessment.
Recently, Transformers [32] have achieved progressive breakthrough on extensive areas [11, 12, 10, 22].
Especially in time series forecasting, credited to their stacked structure and the capability of attention mechanisms,
Transformers can naturally capture the temporal dependencies from deep multi-level features [37, 17, 20, 35], thereby fitting the series forecasting task perfectly.
Despite the remarkable architectural design, it is still challenging for Transformers to predict realworld time series because of the non-stationarity of data.
Non-stationary time series is characterized by the continuous change of statistical properties and joint distribution over time, which makes the time series less predictable [6, 14].
Besides, it is a fundamental problem to make deep models generalize well on a varying distribution [26, 19, 5].
In previous work, it is generally acknowledged to pre-process the time series by stationarization [24, 27, 15], which can attenuate the non-stationarity of raw time series for better predictability and provide more stable data distribution for deep models.
[Figure]
However, non-stationarity is the inherent property of real-world time series and also good guidance for discovering temporal dependencies for forecasting.
Experimentally, we observe that training on the stationarized series will undermine the distinction of attentions learned by Transformers.
While vanilla
Transformers [32] can capture distinct temporal dependencies from different series in Figure 1(a),
Transformers trained on the stationarized series tend to generate indistinguishable attentions in
Figure 1(b).
This problem, named by the over-stationarization , will bring unexpected side-effect that makes Transformers fail to capture eventful temporal dependencies, limit the model’s predictive ability, and even induce the model to generate outputs with huge non-stationarity deviation from the ground truth.
Thus, how to attenuate time series non-stationarity towards better predictability and mitigate the over-stationarization problem for model capability simultaneously is the key problem to further improve the performance of forecasting.
In this paper, we explore the effect of stationarization in time series forecasting and propose Nonstationary Transformers as a general framework, which empowers Transformer [32] and its efficient variants [17, 37, 35] with great predictive ability for real-world time series.
The proposed framework involves two interdependent modules: Series Stationarization to increase the predictability of nonstationary series and De-stationary Attention to alleviate over-stationarization.
Technically, Series
Stationarization adopts a simple but effective normalization strategy to unify the key statistics of each series without extra parameters.
And De-stationary Attention approximates the attention of unstationarized data and compensates the intrinsic non-stationarity of raw series.
Benefiting from the above designs, Non-stationary Transformers can take advantage of the great predictability of stationarized series and crucial temporal dependencies discovered from original non-stationary data.
Our method achieves state-of-the-art performance on six real-world benchmarks and can generalize to various Transformers for further improvement.
The contributions lie in three folds:
• We refine that the predictive capability of non-stationary series is essential in real-world forecasting.
By detailed analysis, we find out that current stationarization approaches will lead to the over-stationarization problem, limiting the predictive capability of Transformers.
• We propose Non-stationary Transformers as a generic framework, including Series Stationarization to make the series more predictable and De-stationary Attention to avoid the over-stationarization problem by re-incorporating the non-stationarity of original series.
• Non-stationary Transformers consistently boosts four mainstream Transformers by a large margin and achieves state-of-the-art performance on six real-world benchmarks.
Related Work 2.1 Deep Models for Time Series Forecasting
In recent years, deep models with elaboratively designed architectures have achieved great progress in time series forecasting.
RNN-based models [33, 36, 23, 29, 30] are proposed for application in an autoregressive manner for sequence modeling, but the recurrent structure can suffer from modeling long-term dependency.
Soon afterward, Transformer [32] emerges and shows great power in sequence modeling.
To overcome the quadratic computation growth on sequence length, subsequent works aim to reduce Self-Attention’s complexity.
Especially in time series forecasting, Informer [37] extends Self-Attention with KL-divergence criterion to select dominant queries.
Reformer [17] introduces local-sensitive hashing (LSH) to approximate attention by allocated similar queries.
Not only improved by reduced complexity, the following models further develop delicate building blocks for time series forecasting.
Autoformer [35] fuses the decomposition blocks into a canonical structure and develops Auto-Correlation to discover series-wise connections.
Pyraformer [21] designs pyramid attention module (PAM) to capture temporal dependencies with different hierarchies.
Other deep but Transformer-free models also achieve remarkable performance.
N-BEATS [25] proposes the explicit decomposition of trend and seasonal terms with strong interpretability.
N-HiTS [9] introduces hierarchical layout and multi-rate sampling for tackling time series with respective frequency bands.
In this paper, different from previous works focusing on architectural design, we analyze the series forecasting task from the basic view of stationarity, which is an essential property of time series [6, 14].
It is also notable that as a general framework, our proposed Non-stationary Transformers can be easily applied to various Transformer-based models.
2.2 Stationarization for Time Series Forecasting
While stationarity is important to the predictability of time series [6, 14], real-world series always present non-stationarity.
To tackle this problem, the classical statistical method ARIMA [7, 8] stationarizes the time series through differencing.
As for deep models, since the distribution-varying problem accompanied by non-stationarity makes deep forecasting even more intractable, stationarization methods are widely explored and always adopted as the pre-processing for deep model inputs.
Adaptive Norm [24] applies z-score normalization for each series fragment by global statistics of a sampled set.
DAIN [27] employs a nonlinear neural network to adaptively stationarize time series with observed training distribution.
RevIN [15] introduces a two-stage instance normalization [31] that transforms model input and output respectively to reduce the discrepancy of each series.
In contrast, we find out that directly stationarizing time series will damage the model’s capability of modeling specific temporal dependency.
Therefore, unlike previous methods, in addition to the stationarization, Non-stationary Transformers further develops De-stationary Attention to bring the intrinsic non-stationarity of the raw series back to attention.
Non-stationary Transformers
As aforementioned, stationarity is an important element of time series predictability.
Previous “direct stationarization” designs can attenuate non-stationarity of series for better predictability, but they obviously neglect inherent properties of real-world series, which will result in the over-stationarization problem as stated in Figure 1.
To deal with the dilemma, we go beyond previous works and propose
Non-stationary Transformers as a generic framework.
Our model involves two complementary parts: Series Stationarization to attenuate time series non-stationarity and De-stationary Attention to re-incorporate non-stationary information of raw series.
Empowered by these designs, Non-stationary
Transformers can improve data predictability and maintain model capability simultaneously.
3.1 Series Stationarization
Non-stationary time series make the forecasting task intractable for deep models because it is hard for them to generalize well on series with changed statistics during inference, typically varied mean and standard deviation.
The pilot work, RevIN [15] applies instance normalization with learnable affine parameters to each input and restores the statistics to the corresponding output, which makes each series follow a similar distribution.
Experimentally, we find that this design also works well without learnable parameters.
Thus, we propose a more straightforward but effective design to wrap
Transformers as the base model without extra parameters, naming by Series Stationarization.
As is shown in Figure 2, it contains two corresponding operations: Normalization module at first to deal with the non-stationary series caused by varied mean and standard deviation, and De-normalization module at the end to transform the model outputs back with original statistics.
Here are the details.
[Table]
quunQTLIKrkoioy6KbLivYB9Qik+m0DebFZFIopbj1B9zqZ4l/oH/hnTEFtYhOSHLm3HPuzL3XSwI/VY7zWrAWFpeWV4qra+sbm1vbpZ3dZhpnkosGj4NYtj2WisCPREP5KhDtRAoWeoFoeXeXOt4aCZn6cXStxonohmwQ+X2fM0VU5yZkashZMKlNb0tlp+KYZc8DNwdl5Ksel15wgx5icGQIIRBBEQ7AkNLTgQsHCXFdTIiThHwTF5hijbwZqQQpGLF39B3QrpOzEe11ztS4OZ0S0CvJaeOQPDHpJGF9mm3imcms2d9yT0xOfbcx/b08V0iswpDYv3wz5X99uhaFPs5NDT7VlBhGV8fzLJnpir65/aUqRRkS4jTuUVwS5sY567NtPKmpXfeWmfibUWpW73muzfCub0kDdn+Ocx40jyvuacW9OilXL/JRF7GPAxzRPM9QRQ11NEzHH/GEZ6tujaypdf8ptQq5Zw/flvXwAXjZk+U=</latexit>
H
Normalization module To attenuate the non-stationarity of each input series, we conduct normalization on the temporal dimension by a sliding window over time.
For each input series x = [ x 1 , x 2 , ..., x S ] ⊤ ∈ R S × C , we transform it by translation and scaling operations and obtain x ′ =
[ x ′ 1 , x ′ 2 , ..., x ′ S ] ⊤ ∈ R S × C , where S and C denote the sequence length and variable number respectively.
The Normalization module can be formulated as follows:
[Equation]
where µ x , σ x ∈ R C × 1 , 1 σ x means the element-wise division and ⊙ is the element-wise product.
Note that Normalization module decreases the distributional discrepancy among each input time series, making the distribution of the model input more stable.
De-normalization module As shown in Figure 2, after the base model H predicting the future value with length- O , we adopt De-normalization to transform the model output y ′
=
[ y ′ 1 , y ′ 2 , ..., y ′ O ]
⊤ ∈ R O × C with σ x and µ x and obtain ˆ y =
[ˆ y 1 , ˆ y 2 , ..., ˆ y O ] ⊤ as the eventual forecasting results.
The
De-normalization module can be formulated as follows:
[Equation]
By means of the two-stage transformation, the base models will receive stationarized inputs, which follow a stable distribution and are easier to generalize.
This design also makes the model equivariant to translational and scaling perturbance of time series, thereby benefiting real-world series forecasting.
3.2 De-stationary Attention
While the statistics of each time series are explicitly restored to the corresponding prediction, the non-stationarity of the original series cannot be fully recovered only by De-normalization.
For instance, Series Stationarization can generate the same stationarized input x ′ from distinct time series x 1 , x 2 (i.e. x 2 =
α x 1 + β ), and the base model will get identical attention that fails to capture crucial temporal dependencies entangled with non-stationarity (Figure 1).
In other words, the undermined effects caused by over-stationarization happen inside the deep model, especially in the calculation of attention.
Furthermore, non-stationary time series are fragmented and normalized into several series chunks with the same mean and variance, which follow more similar distributions than the raw data before stationarization.
Thus, the model is more likely to generate over-stationary and uneventful outputs, which is irreconcilable with the natural non-stationarity of the original series.
To tackle the over-stationarization problem caused by Series Stationarization, we propose a novel
De-stationary Attention mechanism, which can approximate the attention that is obtained without stationarization and discover the particular temporal dependencies from original non-stationary data.
Analysis of the plain model As mentioned above, the over-stationarization problem is caused by the vanishment of inherent non-stationarity information, which will make the base model fail to capture eventful temporal dependencies for forecasting.
Therefore, we try to approximate the attention learned from the original non-stationary series.
We start from the formula of Self-Attention [32]:
[Equation]
where Q , K , V ∈ R S ×
d k are length- S queries, keys and values of d k -dimension respectively, and
Softmax( · ) is conducted row by row.
To simplify the analysis, we assume the embedding and feed-forward layers f to hold the linear properties 2 and f is conducted separately on each time point, that is, each query token in Q =
[ q 1 , q 2 , ..., q S ]
⊤ can be calculated as q i = f
( x i ) with respect to the input series x = [ x 1 , x 2 , · · · , x S ] ⊤ .
Since it is a convention to conduct normalization on each time series variable to avoid certain variable that dominates the scale, we can further assume each variable of series x shares the same variance, and thus original σ x ∈ R C × 1 is reduced to a scalar.
After Normalization module, the model receives the stationarized input x ′ =
( x − 1 µ ⊤ x ) /σ x , where 1 ∈ R S × 1 is an all-ones vector.
Based on the linear property assumption, it can be proved that the Attention layer will receive Q ′ =
[ f ( x ′ 1 ) , ..., f ( x ′ S )]
⊤ =
( Q − 1 µ ⊤ Q ) /σ
x , where µ Q ∈ R d k ×is the mean of Q along the temporal dimension (See Appendix for a detailed proof).
And so is the corresponding transformed K ′ , V ′ .
Without Series Stationarization, the input of Softmax( · ) in
Self-Attention should be QK ⊤ / √ d k , while now the attention is calculated based on Q ′ , K ′ :
[Equation]
We find that Q µ K ∈ R S × 1 and µ ⊤ Q µ K ∈ R , and they are repeatedly operated on each column and element of σ 2 x Q ′ K ′⊤ ∈ R S × S respectively.
Since Softmax( · ) is invariant to the same translation on the row dimension of input, we have the following equation:
[Equation]
Equation 5 deduces a direct expression of the attention Softmax (cid:0) QK ⊤ / √ d k (cid:1) learned from raw series x .
Except for the current Q ′ , K ′ from stationarized series x ′ , this expression also requires the non-stationary information σ x , µ Q , K that are eliminated by Series Stationarization.
De-stationary Attention To recover the original attention on non-stationary series, we attempt to bring the vanished non-stationary information back to its calculation.
Based on Equation 5, the key is to approximate the positive scaling scalar τ = σ 2 x ∈ R + and shifting vector ∆ =
K µ Q ∈ R S × 1 , which are defined as de-stationary factors .
Since the strict linear property hardly holds for a deep model, other than estimating and utilizing real factors with great effort, we try to learn de-stationary factors directly from the statistics of unstationarized x , Q and K by a simple but effective multilayer perceptron layer.
As we can only discover limited non-stationary information from current
Q ′ , K ′ , the unique and reasonable source to compensate non-stationarity is the original x without being normalized.
Thus, as a direct deep learning implementation of Equation 5, we apply a multilayer perceptron as the projector to learn de-stationary factors τ, ∆ from the statistics µ x , σ x of unstationarized x individually.
And the De-stationary Attention is calculated as follows:
[Equation]
where the de-stationary factors τ and ∆ are shared by De-stationary Attention of all layers (Figure 2).
De-stationary Attention mechanism learns the temporal dependencies from both stationarized series
Q ′ , K ′ and non-stationary series x , µ x , σ x , and multiplies by the stationarized values V ′ .
Therefore, it can benefit from the predictability of stationarized series and maintain the inherent temporal dependencies of raw series simultaneously.
Overall architecture Following the prior use of Transformers [37, 35] in time series forecasting, we adopt the standard Encoder-Decoder structure (Figure 2), where the encoder is to extract information from past observations, and the decoder is to aggregate past information and refine the prediction from simple initialization.
The canonical Non-stationary Transformer is wrapped by Series Stationarization to both the input and output of vanilla Transformer [32], and replacing the Self-Attention by our proposed De-stationary Attention, which can boost the non-stationary series predictive capability of the base model.
For the Transformer variants [17, 37, 35], we transform the terms inside Softmax( · ) with the de-stationary factors τ , ∆ to re-integrate the non-stationary information (See Appendix for the implementation details).
Experiments
We conduct extensive experiments to evaluate the performance of Non-stationary Transformers on six real-world time series forecasting benchmarks and further validate the generality of the proposed framework on various mainstream Transformer variants.
Datasets Here are the descriptions of the datasets: (1) Electricity [3] records the hourly electricity consumption of 321 clients from 2012 to 2014.
(2) ETT [37] contains the time series of oil destationary factors and power load collected by electricity transformers from July 2016 to July 2018.
ETTm1 /ETTm2 are recorded every 15 minutes, and ETTh1/ETTh2 are recorded every hour.
Exchange [18] collects the panel data of daily exchange rates from 8 countries from 1990 to 2016.
(4) ILI [1] collects the ratio of influenza-like illness patients versus the total patients in one week, which is reported weekly by Centers for Disease Control and Prevention of the United States from 2002 and 2021.
(5) Traffic [2] contains hourly road occupancy rates measured by 862 sensors on
San Francisco Bay area freeways from January 2015 to December 2016.
(6) Weather [4] includes meteorological time series with 21 weather indicators collected every 10 minutes from the Weather
Station of the Max Planck Biogeochemistry Institute in 2020.
Especially, in this paper, we adopt the Augmented Dick-Fuller (ADF) test statistic [13] as the metric to quantitatively measure the degree of stationarity .
A smaller ADF test statistic indicates a higher degree of stationarity, which means the distribution is more stable.
Table 1 summarizes the overall statistics of the datasets and lists them in ascending order by degree of stationarity.
We follow the standard protocol that divides each dataset into the training, validation, and testing subsets according to the chronological order.
The split ratio is 6:2:2 for the ETT dataset and 7:1:2 for others.
[Table]
Baselines We evaluate the vanilla Transformer [32] equipped by the Non-stationary Transformers framework in both multivariate and univariate settings to demonstrate its effectiveness.
For multivariate forecasting, we include six state-of-the-art deep forecasting models: Autoformer [35],
Pyraformer [21], Informer [37], LogTrans [20], Reformer [17] and LSTNet [18].
For univariate forecasting, we include seven competitive baselines: N-HiTS [9], N-BEATS [25], Autoformer [35],
Pyraformer [21], Informer [37], Reformer [17] and ARIMA [7].
In addition, we adopt the proposed framework on both the canonical and efficient variants of Transformers: Transformer [32],
Informer [37], Reformer [17] and Autoformer [35] to validate the generality of our framework.
Implementation details All the experiments are implemented with PyTorch [28] and conducted on a single NVIDIA TITAN V 12GB GPU.
Each model is trained by ADAM [16] using L2 loss with the initial learning rate of 10 − 4 and batch size of 32.
Each Transformer-based model contains two encoder layers and one decoder layer.
Considering the efficiency of hyperparameters search, we use two-layer perceptron projector with the hidden dimension varying in { 64 , 128 , 256 } in De-stationary
Attention.
We repeat each experiment three times with different random seeds and report the test
MSE/MAE under different prediction lengths, and the standard deviations are also provided in
Appendix.
A lower MSE/MAE indicates better performance.
4.1 Main Results
Forecasting results As for multivariate forecasting results, the vanilla Transformer equipped with our framework consistently achieves state-of-the-art performance in all benchmarks and prediction lengths (Table 2).
Notably, Non-stationary Transformer outperforms other deep models impressively on datasets characterized by high non-stationarity: under the prediction length of 336, we achieve 17% MSE reduction ( 0 . 509 → 0 . 421 ) on Exchange and 25% ( 2 .
669 → 2 . 010 ) on ILI compared to previous state-of-the-art results, which indicates that the potential of deep model is still constrained on non-stationary data.
We also list the univariate results of two typical datasets with different stationarity in Table 3.
Non-stationary Transformer still realizes remarkable forecasting performance.
[Table]
[Table]
Framework generality We apply our framework to four mainstream Transformers and report the performance promotion of each model (Table 4).
Our method consistently improves the forecasting ability of different models.
Overall, it achieves averaged 49.43% promotion on Transformer, 47.34% on Informer, 46.89% on Reformer and 10.57% on Autoformer, making each of them surpass previous state-of-the-art.
Compared to native blocks of the models, there is hardly any parameter and computation increase by applying our framework (See Appendix for details), and thereby their computational complexities can be preserved.
It validates that Non-stationary Transformer is an effective and lightweight framework that can be widely applied to Transformer-based models and enhances their non-stationary predictability to achieve state-of-the-art performance.
[Table]
We report the averaged MSE/MAE of all prediction lengths (stated in Table 2) and the relative
MSE reduction ratios (Promotion) by our framework.
Full results (under all prediction lengths and promotion on ETSformer [34], FEDformer [38]) can be found in Appendix.
[Table]
4.2 Ablation Study
Quality evaluation To explore the role of each module in our proposed framework, we compare the prediction results on ETTm2 obtained by three models: vanilla Transformer, Transformer with only
Series Stationarization, and our Non-stationary Transformer.
In Figure 3, we find out that the two modules strengthen the non-stationary forecasting ability of Transformer from different perspectives.
Series Stationarization focuses on the alignment of statistical properties among each series input that benefits Transformer a lot to generalize on out-of-distribution data.
However, as is shown in
Figure 3(b), the over-stationarized circumstance for training makes the deep model more likely to output uneventful series with significant high stationarity and neglect the nature of non-stationary real-world data.
With the aid of De-stationary Attention, the model gives concern back to the inherent non-stationarity of real-world time series.
It is beneficial for an accurate prediction of the detailed series variation, which is vital in real-world time series forecasting.
[Figure]
[Table]
Quantitative performance
In addition to the above case study, we also provide quantitative forecasting performance comparison with stationarization methods: a deep method RevIN [15] and
Series Stationarization (Section 3.1).
As is shown in Table
[Table]
the forecasting results assisted by
RevIN and Series Stationarization are basically the same, which indicates that the parameter-free version of normalization in our framework performs sufficiently to stationarize time series.
Besides, the proposed De-stationary Attention in Non-stationary Transformers further boosts the performance and achieves the best in all six benchmarks.
The MSE reduction brought by De-stationary Attention becomes significant, especially when the dataset is highly non-stationary (Exchange: 0 .
569 → 0 . 461 ,
ETTm2: 0 .
461 → 0 . 306 ).
The comparison reveals that simply stationarizing time series still limits the predictive capability of Transformers, and the complementary mechanisms in Non-stationary
Transformers can properly release the models’ potential for non-stationary series forecasting.
4.3 Model Analysis
Over-stationarization problem To verify the over-stationarization problem from a statistical view
, we train Transformers with the aforementioned methods respectively, arrange all predicted time series in chronological order and compare the degree of stationarity with the ground truth (Figure 4).
While models solely equipped with stationarization methods tend to output series with unexpected high degree of stationarity, the results assisted by De-stationary Attention are close to the actual value (relative stationarity ∈ [97% , 103%] ).
Besides, as the degree of series stationarity increases, the overstationarization problem becomes more significant.
The huge discrepancy of the degree of stationarity can account for the inferior performance of Transformer with only stationarization.
And it also demonstrates that De-stationary Attention as an internal renovation alleviates over-stationarization.
[Figure]
Exploring of Non-stationary Information Re-incorporation It is notable that by specifying over-stationarization as less distinguishable attention, we narrow down our design space into the attention calculation mechanism.
To explore other approaches to retrieve non-stationary information, we conduct experiments by re-incorporating the µ and σ into feed-forward layers (DeFF), which is the left part of the Transformer architecture.
In detail, we feed learned µ and σ into each feed-forward layer iteratively.
As is shown in Table 6, re-incorporating non-stationarity is necessary only when the inputs are stationarized (Stationary), which is beneficial for forecasting but will lead to stationarity discrepancy of model outputs.
And our proposed design (Stat + DeAttn) makes further promotion and achieves the best in most cases (77%).
In addition to the theoretical analysis, experimental results further validate the effectiveness of our design in re-incorporating non-stationarity on attention
[Table] [Table]
Conclusion
This paper addresses time series forecasting from the view of stationarity.
Unlike previous studies that simply attenuate non-stationarity leading to over-stationarization, we propose an efficient way to increase series stationarity and renovate the internal mechanism to re-incorporate non-stationary information, thus boosting data predictability and model predictive capability simultaneously.
Experimentally, our method shows great generality and performance on six real-world benchmarks.
And detailed derivations and ablations are provided to testify the effectiveness of each component in our proposed Non-stationary Transformers framework.
In the future, we will explore a more model-agnostic solution for the over-stationarization problem.
Acknowledgments
This work was supported by the National Key Research and Development Plan (2021YFC3000905),
National Natural Science Foundation of China (62022050 and 62021002), Beijing Nova Program (Z201100006820041), and BNRist Innovation Fund (BNR2021RC01002).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] See Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Section 7 of the Appendix. (c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6 of the Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 1 of the Appendix. (b) Did you include complete proofs of all theoretical results?
[Yes] See Section 1 of the Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide the data link and code in https://github.com/thuml/Nonstationary_Transformers .
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 5.1 of the Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Table 4 of the Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] The data source is described in Section 4.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We provide the code in https://github.com/thuml/Nonstationary_Transformers .
(d) Did you discuss whether and how consent was obtained from people whose data you’re us- ing/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[-Exchange under the predict-720 settings, 10.8% ( 2 . 572 → 2 . 294 ) on ILI under the predict-24 settings, and 30.3% ( 0 . 
-]
[-0 . 559 ) promotion on ETTm2 and 4.4% ( 2 . 
-]
[N/A]
[-As is shown in Table 15, the forecasting performance will degrade in all cases if we only employ single one of τ and ∆ , especially without τ ( 0 . 
-]