First De-Trend then Attend: Rethinking Attention for Time-Series Forecasting
Abstract
Transformer-based models have gained large popularity and demonstrated promising results in long-term time-series forecasting in recent years.
In addition to learning attention in time domain, recent works also explore learning attention in frequency domains (e.g., Fourier domain, wavelet domain), given that seasonal patterns can be better captured in these domains.
In this work, we seek to understand the relationships between attention models in different time and frequency domains.
Theoretically, we show that attention models in different domains are equivalent under linear conditions (i.e., linear kernel to attention scores).
Empirically, we analyze how attention models of different domains show different behaviors through various synthetic experiments with seasonality, trend and noise, with emphasis on the role of softmax operation therein.
Both these theoretical and empirical analyses motivate us to propose a new method: TDformer (Trend Decomposition
Transformer), that first applies seasonal-trend decomposition, and then additively combines an MLP which predicts the trend component with Fourier attention which predicts the seasonal component to obtain the final prediction.
Extensive experiments on benchmark time-series forecasting datasets demonstrate that TDformer achieves state-of-the-art performance against existing attention-based models.
Introduction
Transformer [18] recently gains wide popularity in time-series forecasting, inspired by its success in natural language processing and its ability to capture long-range dependencies [19].
Apart from the vanilla Transformer that calculates attention in time domain, recently variants of Transformer which calculate attention in frequency domains (e.g., Fourier domain or wavelet domain) (Figure 2) [22, 21, 24, 20, 14] have also been proposed to better model global characteristics of time series.
Despite the progress made by Transformer-based methods for time series forecasting, there lacks a rule of thumb to select the domain in which attention is best learned.
Our work is driven by better understanding the following research question: Does learning attention in one domain offer better representation ability than the other?
If so, how?
We show mathematically that under linear conditions, learning attention in time or frequency domains leads to equivalent representation power.
We then show that due to the softmax non-linearity used for normalization, this theoretical linear equivalence does not hold empirically.
In particular, attention models in different domains demonstrate different empirical advantages.
This finding sheds light on how to best apply attention
GroundTruthPrediction (a) ECL TDformer    400 1.5 1.
GroundTruthPrediction (b) ECL FEDformer    400 0.
GroundTruthPrediction (c)
Weather TDformer    400 0.
GroundTruthPrediction
[Figure]
models under different practical scenarios.
We propose TDformer based on these insights and demonstrate that we achieve state-of-the-art performance against current attention-based models.
More specifically, we find that (1) for data with strong seasonality , frequency-domain attention models are more sample-efficient compared with time-domain attention models, as softmax with exponential terms correctly amplify the dominant frequency modes in Fourier space.
(2) For data with trend , attention models generally show inferior generalizability , as attention models by nature interpolate rather than extrapolate the context.
This finding of difference in performances of attention models on various types of time series data emphasizes the importance of seasonal-trend decomposition module in the attention model framework.
(3) For data with noisy spikes , frequency-domain attention models are more robust to such spiky data, as large-value spikes in the time domain correspond to small-amplitude high-frequency modes, whose attention would be filtered out by softmax operations.
Due to the different performances of the various attention modules on data with seasonality and trend, we propose TDformer that first decomposes the context time series into trend and seasonal components.
We use a MLP for predicting the future trend, Fourier attention to predict the future seasonal part, and add these two components to obtain the final prediction.
Extensive experiments on benchmark forecasting datasets demonstrate the effectiveness of our proposed approach.
As a motivating example, we visualize predictions of TDformer and one of the best performing baselines
FEDformer in Figure 1.
On data with strong seasonality (Figure 1a and Figure 1b) TDformer preserves both the seasonality and trend of the original data, while FEDformer [24] deviates from the trend of the ground truth.
On data with strong trend (Figure 1c and Figure 1d), TDformer generates predictions that better follow the trend of the original data.
In summary, our contributions are:
• We theoretically show that under linear conditions, attention models in time domain, Fourier domain and wavelet domain have the same representation power; • We empirically analyze attention models in different domains with synthetic data of different characteristics, given the non-linearity of softmax.
We show that frequency-domain attention performs the best on data with seasonality, and attention models in general have inferior generalizability on trend data, which motivates the design of a hybrid model based on seasonal trend decomposition; • We propose TDformer that separately models the trend with MLP and seasonality with Fourier attention, and shows state-of-the-art performance against current attention models on time-series forecasting benchmarks.
Related Work
Time-Domain Attention Forecasting Models .
Informer [22] proposes efficient ProbSparse selfattention mechanism.
Autoformer [21] renovates time-series decomposition as a basic inner block and designs Auto-Correlation mechanism for dependencies discovery.
Non-stationary Transformer [14] proposes Series Stationarization and De-stationary Attention to address over-stationarization.
Frequency-Domain Attention Forecasting Models .
FEDformer [24] proposes Fourier and wavelet enhanced blocks based on Multiwavelet-based Neural Operator Learning [4] to capture important structures in time series through frequency domain mapping.
ETSformer [20] selects top-K largest [Table] [Table]
/gRUERr/4hb/03ZtsetPXBwOO9GWbmBRFn2rju1MmtrW9sbuW3Czu7e/sHxcOjpg5jRWiDhDxU7QBrypmkDcMMp+1IUSwCTlvB+C7zW09UaRbKR5NE1Bd4KNmAEWwyqULOCr1i2a26M6BV4i1IuVbqnr9Oa0m9V/zu9kMSCyoN4VjrjudGxk+xMoxwOil0Y00jTMZ4SDuWSiyo9tPZrRN0apU+GoTKljRopv6eSLHQOhGB7RTYjPSyl4n/eZ3YDG78lMkoNlSS+aJBzJEJUfY46jNFieGJJZgoZm9FZIQVJsbGk4XgLb+8SpoXVe+qevlg07iFOfJwAiWogAfXUIN7qEMDCIzgGd7g3RHOi/PhfM5bc85i5hj+wPn6Ad4QkHQ=</latexit [Figure] analysis follows for multiple layers. See precise notations in Section 3.
amplitude modes as frequency attention and combines with exponential smoothing attention.
Adaptive
Fourier Neural Operator (AFNO)
[3] builds upon FNO [13] and proposes an efficient token mixer that learns to mix in the Fourier domain.
FNet [11] replaces the self-attention with Fourier Transform and promotes efficiency without much loss of accuracy on NLP benchmarks.
T-WaveNet [15] constructs a tree-structured network with each node built with invertible neural network (INN) based wavelet transform unit for iterative decomposition.
Adaptive Wavelet Transformer Network (AWT-Net) [5] generates wavelet coefficients to classify each point into high or low sub-bands components and exploits Transformer to enhance the original shape features.
Decomposition-Based Forecasting Models decompose time series into trend and seasonality (with i.e., STL decomposition [2]).
Apart from attention-based Autoformer and FEDformer, N-BEATS [16] models trend with small-degree polynomials and seasonality with Fourier series.
N-HiTS [1] redefines
N-BEATS
by enhancing its input decomposition via multi-rate data sampling and its output synthesizer via multi-scale interpolation.
FreDo [17] incorporates frequency-domain features into AverageTile model that averages history sub-series.
FiLM [23] applies Legendre Polynomials projections to approximate historical information and Fourier projection to remove noise.
DeepFS [6] encodes temporal patterns with self-attention and predicts Fourier series parameters and trend with MLP.
Despite the success of attention models in time, Fourier, and wavelet domains, there is still a lack of notion for understanding their relationships and respective advantages.
Decomposition-based methods also adopt decomposition layers without giving strong reasoning for their necessity.
We propose to fill this gap from both theoretical and empirical perspectives, and based on these analysis build a new framework that shows better forecasting performance.
Linear Equivalence of Attention in Various Domains 3.1 Formulation of Attention Models
We first briefly introduce the canonical Transformers.
Denote input queries, keys and values as q ∈ R L × D , k ∈ R L × D , v ∈ R L × D , which are transformed from input x through linear embeddings.
Denote output of attention module as o ( q , k , v ) ∈
R L × D .
As shown in Figure 2 (a), The attention operation in canonical attention is formulated as
[Equation]
where d q is the dimension for queries that serves as normalization term in attention operation, and σ ( · ) represents activation function.
When σ ( · ) = softmax( · ) 3 , we have softmax attention : o ( q , k , v ) = softmax (cid:0) qk T / (cid:112)
d q (cid:1) v .
When σ ( · ) = Id( · ) (identity mapping), we have linear attention : o ( q , k , v )
=
qk T v (we ignore the normalization term (cid:112)
d q for simplicity).
Definition 3.1 (Time Attention) .
Equation 1 refers to time domain attention where q , k , v are all in original time domain, shown in Figure 2 (a).
Definition 3.2 (Fourier Attention) .
Fourier attention first converts queries, keys, and values with
Fourier Transform, performs a similar attention mechanism in the frequency domain, and finally converts the results back to the time domain using inverse Fourier transform, shown in Figure 2 (b).
Let F ( · ) , F − 1 ( · ) denote Fourier transform and inverse Fourier transform, then Fourier attention is
[Equation]
Definition 3.3 (Wavelet Attention) .
Wavelet transform applies wavelet decomposition and reconstruction to obtain signals of different scales.
Wavelet attention performs attention calculation to decomposed queries, keys, and values in each scale, and reconstructs the output from attention results in each scale, illustrated in Figure 2 (c).
Let W ( · ) , W − 1 ( · ) denote wavelet decomposition and wavelet reconstruction, then wavelet attention is o ( q , k , v ) =
W − 1 (cid:16) σ (cid:0) W ( q ) W ( k T ) / (cid:112) d q (cid:1) W ( v ) (cid:17) .
3.2 Linear Equivalence of Time, Fourier and Wavelet Attention
In this section we formally prove that time, Fourier and wavelet attention models are equivalent under linear attention case.
Lemma 3.1.
When σ ( · ) = Id( · ) (linear attention), time, Fourier and wavelet attention are equivalent.
Proof.
Let W = ( ω jk √ L )
∈
C L × L , ω =
e − 2 πjL denote the Fourier matrix, then Fourier transform to signal x ∈ R L × D can be expressed as X
= Wx , X ∈ C L × D , and inverse Fourier transform can be expressed as x = W H X , where W H is the Hermitian (conjugate transpose) of W .
Given properties of Fourier matrix, we could easily show that
[Equation]
Following this expression, Fourier domain linear attention can be written as
[Equation]
Therefore, calculating attention in Fourier domain is equivalent to time-domain attention.
For wavelet attention, we take single-scale wavelet decomposition and reconstruction as an example, and multi-scale wavelet transform follows the same analysis.
Using the same notation, let W ∈
R L × L 2 , W − 1 ∈ R L 2 × L denote the wavelet decomposition and reconstruction matrix, then wavelet decomposition to signal x ∈ R L × D can be expressed as X
=
Wx , X ∈ R L 2 × D , and wavelet reconstruction can be expressed as x = W − 1 X .
Since wavelet matrix is orthogonal, we have the property that W T W
=
I .
Wavelet linear attention is
[Equation]
which is again equivalent to time-domain attention.
Therefore, we show that mathematically, time,
Fourier and wavelet attention models are equivalent given linear assumptions.
Investigation on the Role of Softmax
Although these attention models are equivalent given linear assumptions, in practice we apply softmax as normalization, which changes the behavior of different attention models.
In this section, we empirically analyze how softmax causes such performance gaps on datasets with three different representative properties: seasonality, trend and noise.
For all experiments in this section, the task is to predict the next 96 time steps given history 96 time steps.
We implement the wavelet-domain attention model based on multiwavelet transform model [4].
4.1 Data with Seasonality
For data with fixed seasonality, Fourier attention is the most sample-efficient.
We use sin(x) as an example of seasonal data (visualized in Figure 3a and Figure 3b).
There exist dominant frequency modes for data with seasonality.
We visualize linear attention (Figure 3c) and softmax attention (Figure 3d) in Fourier space.
Attention scores are concentrated on the dominant frequency mode.
As softmax with exponential terms has the “polarization” effect (increasing the gap between large and small values), softmax attention further concentrates the scores on the dominant frequency, helping the model to better capture seasonal information.
Therefore, we find that frequency-domain attention
[Figure]
Fourier softmax attention amplifies the dominant frequency modes, but also neglects the smallamplitude modes that embed the localized frequency information.
(i)-(l): Data with linear trend.
Fourier softmax attention incorrectly amplifies the low-frequency modes compared with Fourier linear attention.
(m)-(p): Data with spikes as noise.
Fourier softmax attention filters out the noisy components and emphasizes the correct frequency modes compared with Fourier linear attention.
[Table]
models are capable of quickly recognizing the dominant frequency modes (more sample efficient) compared with time-domain models (Figure 4a). To further illustrate such polarization effect, we also compare softmax attention with polynomial kernels σ ( x ) = x di / (cid:80) i x di , where d is the degree of polynomials (without loss of generality we assume x i > 0 , ∀ i ). Polarization effect increases with respect to polynomial degrees. As shown in Figure 4c, the performance also increases as we increase the polarization effect and approaches the performance of softmax operations.
We also notice that apart from the polarization effect from exponential terms, normalization itself also introduces performance gaps between different attention models.
The possible reason is that it’s easier to optimize in the sparse Fourier domain compared with time domain.
We leave this as our future explorations.
For data with varying seasonality, wavelet attention is the most effective.
We use alternating sin(x) and sin(2x) as an example of varying seasonal data (visualized in Figure 3e and Figure 3f). [Figure] [Table]
The Fourier representation has both dominant modes as well as small-amplitude modes, where the latter embeds the varying-seasonality information.
The Fourier softmax attention correctly amplifies the dominant frequency modes, but at the same time neglects the small-amplitude modes that convey the information of varying seasonality.
By contrast, wavelet attention combines multi-scale timefrequency representation, and provides better localized frequency information.
As shown in Figure 4b, wavelet attention is the most effective for varying-seasonality data.
4.2 Data with Trend
For data with trend, all attention models show inferior generalizability, especially Fourier attention.
We take linear trend data as an example (Figure 3i and Figure 3j) and evaluate different attention models.
The first several frequency modes in Fourier space carry large values; the attention scores hence mostly focus on the first few frequency modes (top-left corner of Figure 3k).
With the polarization effect of softmax, attention scores emphasize even more on these low-frequency components (Figure 3l) and generate misleading reconstruction results.
We evaluate different attention models in Table 1.
Fourier attention, with inappropriate polarization, leads to the largest errors.
Moreover, all these attention models fail to extrapolate linear trend well and suffer from large errors, since attention mechanism by nature works through interpolating the context history.
By contrast,
MLP perfectly predicts such trend signals, as shown in Table 1.
This motivates us to decompose the time series into trend and seasonality [21, 24], apply attention mechanism only for seasonality, and use MLP for modeling trend.
4.3 Data with Spikes
For data carrying noise, Fourier attention is the most robust.
We randomly inject large-value spikes into the training set of sin(x) as a motivating example (Figure 3m and Figure 3n).
Spikes which have large values in time domain result in small-amplitude frequency components after
Fourier transforms.
With the polarization effect of softmax, time-domain softmax attention focuses incorrectly on large-value spikes while Fourier-domain softmax attention correctly filters out the noisy components and attends to the dominant frequency modes induced by sin(x) .
Comparing Figure 3o and Figure 3p, linear attention still distributes attention to the noisy frequency modes, while softmax attention mostly focuses correctly on the dominant frequency modes.
Therefore, frequency-domain attention models are more robust to spikes, as shown in Table 2.
All these analysis on datasets with different characteristics help guide our model design in the next section. [Table] [Figure] [Table]
+0eJxyhBuopjGrO1BjimJcFMQQXE7YRiGHsW2N7jKffseM07i6FYME+yEsBeRgCAolOTqu90Qij6CVLYzVxYfFkofZ9mdpEcnmatXzZpZwJgm1phU6wdbdvZkbzdc/avrxygNcSQQhZx3LDMRjoRMEERxVummHCcQDWAPdxSNYIi5I4tDMmNfKb4RxEy9SBiF+rtDwpDzYeipynxTPunl4n9eJxXBhSNJlKQCR2g0KEipIWIjT8XwCcNI0KEiEDGidjVQHzKIhMquokKwJk+eJq3jmnVWO71RaVyCEcpgB+yBQ2CBc1AH16ABmgCBB/AMXsGb9qi9aO/ax6i0pI17NsEfaJ8/Bnyc2Q==</latexit> X l, 3de <latexit sha1_base64="i6gc/
Ku7FB69EXXaM+4YXENS45s=">AAACCHicbVC7TsMwFHV4lvIKMDIQtULqgKqkQpSxgoWxSPQhtSFyXLe1ajuR7SBVUUYWfoFPYGEAIVY2VjYEH4OTdoCWI1k6Pude3XuPH1IilW1/GguLS8srq7m1/PrG5ta2ubPblEEkEG6ggAai7UOJKeG4oYiiuB0KDJlPccsfnad+6wYLSQJ+pcYhdhkccNInCCoteeZBl0E1RJDG7cSLs49gMeZJch3To0rimUW7bGew5okzJcVaofT9VX2/r3vmR7cXoIhhrhCFUnYcO1RuDIUiiOIk340kDiEawQHuaMohw9KNs0MS61ArPasfCP24sjL1d0cMmZRj5uvKdFM566Xif14nUv1TNyY8jBTmaDKoH1FLBVaaitUjAiNFx5pAJIje1UJDKCBSOru8DsGZPXmeNCtl56R8fKnTOAMT5MA+KIAScEAV1MAFqIMGQOAWPIAn8GzcGY/Gi/E6KV0wpj174A+Mtx8oj55v</latexit>
[Figure]  Our Method: TDformer
The performance difference in data with various characteristics motivates our model design.
For data with seasonality, Fourier softmax attention amplifies dominant frequency modes and demonstrates the best performance.
For data with trend, Fourier softmax incorrectly attends to only the low-frequency modes and produces large errors.
Meanwhile, all attention models which work through interpolating the historical context, do not generalize well on trend data compared with MLP.
These analyses motivate us to decompose time series into trend and seasonality, use Fourier attention to predict the seasonal part and MLP to predict the trend part.
Figure 5 overviews our proposed model architecture.
We first decompose the time series into trend parts and seasonal parts following FEDformer [24].
More specifically, we apply multiple average filters with different sizes to extract different trend patterns, and apply adaptive weights to combine these patterns into the final trend component.
The seasonal component is acquired by subtracting trend from the original time series:
[Equation]
where σ, w ( x ) , f ( x ) denote the softmax operation, data-dependent weights and average filters.
For the trend component, we use a three-layer MLP to predict the future trend.
As reversible instance normalization (RevIN) proves to be effective to remove and restore the non-stationary information [7, 23] which mainly resides in trend, we also add RevIN layers before and after MLP:
X trend = RevIN(MLP(RevIN( x trend ))) .
For the seasonal component, we adopt Transformer architecture but replace time-domain attention with Fourier-domain attention.
More specifically, we first feed the seasonal part to N layers of encoder:
[Equation]
[Equation] where X 0en = x seasonal , FA and FF are short for Fourier Attention and Feed Forward network.
Fourier Attention computes the attention in Fourier space and converts the output to time domain at the end (Definition 3.2) with σ ( · ) = softmax( · ) :
[Equation]
The seasonal part is also zero-padded for the future part and fed into M layers of decoder to obtain the final seasonal output:
[Equation]
where X 0de =
Padding( x seasonal ) .
We add the trend prediction from MLP and seasonal prediction from Transformer to obtain the final output prediction, i.e., X final =
X trend + X M de .
Optimization is based on a reconstruction MSE loss between predicted and ground truth future time series.
Remark.
While FEDformer [24] and Autoformer [21] also have seasonal-trend decomposition, their trend and seasonal components are not disentangled; the trend prediction still comes from the attention module, which is sub-optimal based on our analysis in Section 4 and our empirical results in Section 6.
By contrast, we apply seasonal-trend decomposition in the beginning, and apply Fourier attention only on seasonality components.
This seemingly simple different way of decomposition brings significant performance gains to see in the experiment section, with even less model complexity.
Non-stationary Transformer also computes attention for trend data.
Moreover, with RevIN, TDformer has the similar effect of stationarization.
[Table]
Experiments 6.1 Dataset and Baselines
We conduct experiments on benchmark time-series forecasting datasets: ETTm2 [22], electricity 4 , exchange [10], traffic 5 , weather 6 .
We quantify the strength of seasonality for each dataset (details in Appendix).
Electricity, traffic and ETTm2 are strongly seasonal data [Table] while exchange rate and weather demonstrate less seasonality and more trend.
We compare TDformer with state-of-the-art attention models: Non-stationary Transformer [14], FEDformer [24], Autoformer [21], Informer [22], LogTrans [12], Reformer [9].
As classical models (e.g., ARIMA), RNN-based models and CNNbased models generate large errors as shown in previous papers [22, 21], here we do not include their performance in the comparison.
We use Adam [8] optimizer with a learning rate of 1 e − 4 and batch size of 32 .
We split the dataset with 7 : 2 : 1 into training, validation and test set, use validation set for hyperparameter tuning and report the results on the test set.
For all real-world experiments, we feed the past 96 timesteps as context to predict the next 96 , 192 , 336 , 720 timesteps following previous works [24, 21].
All experiments are repeated 5 times and we report the mean MSE and
MAE.
We implement in Pytorch on NVIDIA V100 16GB GPUs.
6.2 Comparing Attention Models on Real-World Datasets
As an extension to experiments on synthetic data (Section 4), we also compare attention models on real-world datasets, and observe consistent results as on synthetic datasets.
Note that for a fair comparison, we directly compare the attention models without additional components like decomposition blocks or additional learnable transformation kernels [21, 24].
We choose traffic dataset as data with seasonality and weather dataset as data with trend.
As shown in Table 3, frequency-domain attention models demonstrate better performance with seasonal data, which aligns with our observations on synthetic datasets.
For trend data, Fourier-attention models show larger errors compared with time and wavelet attention models, which is also consistent with our observations on synthetic datasets.
Compared with the reported performance after seasonal-trend decomposition as in FEDformer [24] and Autoformer [21], the errors on seasonal data remain similar, while errors increase significantly on trend data.
This emphasizes the importance of de-trending.
We also replace
Fourier attention with time or wavelet attention in TDformer in Section 6.4.
6.3 Main Results
We compare TDformer with the state-of-the-art baselines and report on MSE and MAE in Table 4.
TDformer consistently demonstrates better performance across different datasets and forecasting horizons.
On average, TDformer reduces the MSE by 9 .
14% compared with Non-stationary Transformer and by 14 .
69% compared with FEDformer,
and we attribute such improvement to our separate modeling of trend and seasonality with MLP and Fourier attention.
As we mention in Remark 5, trend prediction of FEDformer and Non-stationary Transformer still come from attention modules, while TDformer decouples the modeling of trend and seasonality, and demonstrates better forecasting results.
See Figure 1 for qualitative comparison.
6.4 Ablation Study
To separately understand the effect of trend and seasonal modules, we conducted ablation studies.
TDformer-MLP-TA(WA) replaces Fourier attention with time (wavelet) attention for seasonality,
[Table] [Table]
and shows larger errors especially on seasonal data (traffic), as Fourier attention is more capable of capturing seasonality.
Exchange data is mainly composed of trend, so different attention variants demonstrate similar performance with time attention being slightly better.
We also replace MLP with time attention for trend (TDformer-TA-FA) and observe large errors, as attention models show inferior generalization ability on trend data.
TDformer w/o RevIN removes RevIN normalization and displays larger errors, which shows the importance of normalization for non-stationary data.
Conclusion
In this work we are driven by better understanding the relationships and separate benefits of attention models in time, Fourier and wavelet domains.
We show that theoretically these three attention models are equivalent given linear assumptions.
However, empirically due to the role of softmax, these models have respective benefits when applied to datasets with specific properties.
Moreover, all attention models show inferior generalizability on data with trend.
Based on these analyses of performance differences, we propose TDformer which separately models trend and seasonality with MLP and Fourier attention models after seasonal trend decomposition.
TDformer achieves state-of-the-art performance against current attention models on time-series forecasting benchmarks.
In the future, we plan to explore more complicated models to predict trend (e.g., autoregressive models) and explore other seasonal-trend decomposition methods.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A]
We focus on promoting understanding of existing models, and propose method that improves on forecasting benchmarks.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section(b) Did you include complete proofs of all theoretical results?
[Yes] See Section3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Section(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section(b)
Did you mention the license of the assets?
[Yes] See Section(c)
Did you include any new assets either in the supplemental material or as a URL?
[N/A]
We didn’t introduce new datasets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We use public benchmark forecasting datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] We use public benchmark forecasting datasets.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] We didn’t use crowdsourcing or involve human subjects in this paper.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
We didn’t use crowdsourcing or involve human subjects in this paper.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] We didn’t use crowdsourcing or involve human subjects in this paper.
A Appendix
We first apply STL decomposition [2] for each dataset
[Equation]
where T t , S t , R t respectively represent the trend, seasonal and remainder component.
For data with strong seasonality, the seasonal component would have much larger variation than the remainder component; while for data with little seasonality, the two variances should be similar.
Therefore, we can quantify the strength of seasonality as
[Equation]
Following this equation, we summarize the seasonality strength of each dataset in Table 6.
Electricity, traffic and ETTm2 are strongly seasonal data, while exchange rate and weather demonstrate less seasonality and more trend.
[Table]