S CALEFORMER : I TERATIVE M ULTI - SCALE R EFINING T RANSFORMERS FOR T IME S ERIES F ORECASTING
Anonymous authors
Paper under double-blind review
A BSTRACT
The performance of time series forecasting has recently been greatly improved by the introduction of transformers.
In this paper, we propose a general multi-scale framework that can be applied to the state-of-the-art transformer-based time series forecasting models (FEDformer, Autoformer, etc.).
By iteratively reﬁning a forecasted time series at multiple scales with shared weights, introducing architecture adaptations, and a specially-designed normalization scheme, we are able to achieve signiﬁcant performance improvements, from 5 .
5% to 38 .
5% across datasets and transformer architectures, with minimal additional computational overhead.
Via detailed ablation studies, we demonstrate the effectiveness of each of our contributions across the architecture and methodology.
Furthermore, our experiments on various public datasets demonstrate that the proposed improvements outperform their corresponding baseline counterparts.
Our code is publicly available in https://github.com/Scaleformer/Scaleformer.
I NTRODUCTION
C oa r s e
F o rec a s t i n g F i n e F o rec a s t i n g
Time
V a l u e V a l u e
V a l u e V a l u e
[Figure]
at different time scales.
Iterative reﬁnement of a time series forecast is a strong structural prior that beneﬁts time series forecasting.
Integrating information at different time scales is essential to accurately model and forecast time series (Mozer, 1991; Ferreira et al., 2006).
From weather patterns that ﬂuctuate both locally and globally, as well as throughout the day and across seasons and years, to radio carrier waves which contain relevant signals at different frequencies, time series forecasting models need to encourage scale awareness in learnt representations.
While transformer-based architectures have become the mainstream and stateof-the-art for time series forecasting in recent years, advances have focused mainly on mitigating the standard quadratic complexity in time and space, e.g., attention (Li et al., 2019;
Zhou et al., 2021) or structural changes (Xu et al., 2021;
Zhou et al., 2022b), rather than explicit scale-awareness.
The essential cross-scale feature relationships are often learnt implicitly, and are not encouraged by architectural priors of any kind beyond the stacked attention blocks that characterize the transformer models.
Autoformer (Xu et al., 2021) and Fedformer (Zhou et al., 2022b) introduced some emphasis on scale-awareness by enforcing different computational paths for the trend and seasonal components of the input time series; however, this structural prior only focused on two scales: low- and high-frequency components.
Given their importance to forecasting, can we make transformers more scale-aware?
We enable this scale-awareness with Scaleformer .
In our proposed approach, showcased in Figure 1, time series forecasts are iteratively reﬁned at successive time-steps, allowing the model to better capture the inter-dependencies and speciﬁcities of each scale.
However, scale itself is not sufﬁcient.
Iterative reﬁnement at different scales can cause signiﬁcant distribution shifts between intermediate forecasts which can lead to runaway error propagation.
To mitigate this issue, we introduce cross-scale
Our approach re-orders model capacity to shift the focus on scale awareness, but does not fundamentally alter the attention-driven paradigm of transformers.
As a result, it can be readily adapted to work jointly with multiple recent time series transformer architectures, acting broadly orthogonally to their own contributions.
Leveraging this, we chose to operate with various transformer-based backbones (e.g. Fedformer, Autoformer, Informer, Reformer, Performer) to further probe the effect of our multi-scale method on a variety of experimental setups.
Our contributions are as follows: (1) we introduce a novel iterative scale-reﬁnement paradigm that can be readily adapted to a variety of transformer-based time series forecasting architectures.
(2) To minimize distribution shifts between scales and windows, we introduce cross-scale normalization on outputs of the Transformer.
(3) Using Informer and AutoFormer, two state-of-the-art architectures, as backbones, we demonstrate empirically the effectiveness of our approach on a variety of datasets.
Depending on the choice of transformer architecture, our mutli-scale framework results in mean squared error reductions ranging from 5 .
5% to 38 .
Via a detailed ablation study of our ﬁndings, we demonstrate the validity of our architectural and methodological choices.
R ELATED WORKS
Time-series forecasting : Time-series forecasting plays an important role in many domains, including: weather forecasting (Murphy, 1993), inventory planning (Syntetos et al., 2009), astronomy (Scargle, 1981), economic and ﬁnancial forecasting (Krollner et al., 2010).
One of the speciﬁcities of time series data is the need to capture seasonal trends (Brockwell & Davis, 2009).
There exits a vast variety of time-series forecasting models (Box & Jenkins, 1968; Hyndman et al., 2008; Salinas et al., 2020;
Rangapuram et al., 2018; Bai et al., 2018; Wu et al., 2020).
Early approaches such as ARIMA (Box & Jenkins, 1968) and exponential smoothing models (Hyndman et al., 2008) were followed by the introduction of neural network based approaches involving either Recurrent Neural Netowkrs (RNNs) and their variants (Salinas et al., 2020; Rangapuram et al., 2018; Salinas et al., 2020) or Temporal
Convolutional Networks (TCNs) (Bai et al., 2018).
More recently, time-series Transformers (Wu et al., 2020; Zerveas et al., 2021; Tang & Matteson, 2021) were introduced for the forecasting task by leveraging self-attention mechanisms to learn complex patterns and dynamics from time series data.
Informer (Zhou et al., 2021) reduced quadratic complexity in time and memory to O
( L log L ) by enforcing sparsity in the attention mechanism with the ProbSparse attention.
Yformer (Madhusudhanan et al., 2021) proposed a Y-shaped encoderdecoder architecture to take advantage of the multi-resolution embeddings.
Autoformer (Xu et al., 2021) used a cross-correlation-based attention mechanism to operate at the level of subsequences.
FEDformer (Zhou et al., 2022b) employs frequency transform to decompose the sequence into multiple frequency domain modes to extract the feature, further improving the performance of
Autoformer.
Multi-scale neural architectures : Multi-scale and hierarchical processing is useful in many domains, such as computer vision (Fan et al., 2021; Zhang et al., 2021;
Liu et al., 2018), natural language processing (Nawrot et al., 2021; Subramanian et al., 2020;
Zhao et al., 2021) and time series forecasting (Chen et al., 2022; Ding et al., 2020).
Multiscale Vision Transformers (Fan et al., 2021) is proposed for video and image recognition, by connecting the seminal idea of multiscale feature hierarchies with transformer models, however, it focuses on the spatial domain, specially designed for computer vision tasks.
Cui et al.
(2016) proposed to use different transformations of a time series such as downsampling and smoothing in parallel to the original signal to better capture temporal patterns and reduce the effect of random noise.
Many different architectures have been proposed recently (Chung et al., 2016;
Che et al., 2018; Shen et al., 2020; Chen et al., 2021) to improve RNNs in tasks such as language processing, computer vision, time-series analysis, and speech recognition.
However, these methods are mainly focused on proposing a new RNN-based module which is not applicable to transformers directly.
The same direction has been also investigated in Transformers,
TCN, and MLP models.
Recent work Du et al.
(2022) proposed multi-scale segment-wise correlations as a multi-scale version of the self-attention mechanism.
Our work is orthogonal to the above methods as a model-agnostic framework to utilize multi-scale time-series in transformers while keeping the number of parameters and time complexity roughly the same.
Loss function
Upsample
[Figure]
Pooling
X" !"
#$
Loss function
Upsample
Cross-scale Normalization
Pooling
ForecastingModule
Loss function
Cross-scale Normalization
Pooling
ForecastingModule
Loss function Upsample
Cross-scale Normalization
ForecastingModule
X !" X !"
#$%&
Step
Step i
Step m Step i
X" !
$
X !&’(
X !)
*&’( X "#$
X !"
#$
X !%"$ X !$%&
X ’$%&
X ($%&
X ()*+ X *,
[Figure]
M ETHOD
In this section, we ﬁrst introduce the problem setting in Sec.
3.1, then describe the proposed framework in Sec. 3.2, and the normalization scheme in Sec. 3.3.
We provide details on the input’s representation in Sec. 3.4, and the loss function in Sec. 3.5.
3. P ROBLEM S ETTING
We denote X ( L ) and X ( H ) the look-back and horizon windows for the forecast, respectively, of corresponding lengths (cid:96) L , (cid:96) H .
Given a starting time t 0 we can express these time-series of dimension d x , as follows: X ( L ) =
{ x t | x t ∈ R
d
x , t ∈ [ t 0 , t 0 + (cid:96) L ] } and X ( H ) =
{ x t | x t ∈ R
d
x , t ∈ [ t 0 + (cid:96) L + 1 , t
+ (cid:96) L + (cid:96) H ] } .
The goal of the forecasting task is to predict the horizon window X ( H ) given the look-back window X ( L ) .
3. M ULTI - SCALE F RAMEWORK
Our proposed framework applies successive transformer modules to iteratively reﬁne a time-series forecast, at different temporal scales.
The proposed framework is shown in Figure 2.
Given an input time-series X ( L ) , we iteratively apply the same neural module mutliple times at different temporal scales.
Concretely, we consider a set of scales S =
{ s m , ..., s 2 , s 1 , 1 } (i.e. for the default scale of s = 2 , S is a set of consecutive powers of 2 ), where m = (cid:98) log s (cid:96) L (cid:99) − 1 and s is a downscaling factor.
The input to the encoder at the i -th step ( 0 ≤ i ≤ m ) is the original look-back window X ( L ) , downsampled by a scale factor of s i ≡ s m − i via an average pooling operation.
The input to the decoder, on the other hand, is X out i − 1 upsampled by a factor of s via a linear interpolation.
Finally, X dec 0 is initialized to an array of 0s.
The model performs the following operations:
[Equation]
where X ( L ) i and X ( H ) i are the look-back and horizon windows at the i th step at time t with the scale factor of s m − i and with the lengths of (cid:96) L,i and (cid:96) H,i , respectively.
Assuming x (cid:48) t,i − 1 is the output of the forecasting module at step i − 1 and time t , we can deﬁne X enc i and X dec i as the inputs to the normalization:
[Equation]
Finally, we calculate the error between X ( H ) i and X out i as the loss function to train the model.
Please refer to Algorithm 1 for details on the sequence of operations performed during the forward pass.
Require: input lookback window X ( L )
∈ R (cid:96)
L × d x , scale factor s , a set of scales S = { s m , ..., s 2 , s 1 , 1 } , Horizon length (cid:96) H , and Transformer module F .
for i ← 0 to m do
X enc i ← AvgPool (cid:0) X
( L ) , window size= s m − i (cid:1) (cid:46) Equation (1) and (2) of the paper if i = 0 then
X dec i ← (cid:104)
[Equation]
0 (cid:96)
[Equation]
else
X dec i ← Upsample (cid:0) X out i − 1 , scale= s (cid:1) (cid:46) Equation (5) and (6) of the paper end if ¯ µ X i ←
[Equation]
X
[Equation]
end for
Ensure: X out : (cid:46) return the prediction at all scales
3. C ROSS - SCALE N ORMALIZATION
Given a set of input series ( X enc i , X dec i ), with dimensions (cid:96) L i × d x and (cid:96) H i × d x , respectively for the encoder and the decoder of the transformer in i th step, we normalize each series based on the temporal average of X enc i and X dec i .
More formally:
[Equation]
where ¯ µ X i ∈ R
d x is the average over the temporal dimension of the concatenation of both look-back window and the horizon.
Here, ˆ X enc i and ˆ X dec i are the inputs of the i th step to the forecasting module.
[Figure]
Distribution shift is when the distribution of input to a model or its sub-components changes across training to deployment (Shimodaira, 2000; Ioffe & Szegedy, 2015).
In our context, two distinct distribution shifts are common.
First, there is a natural distribution shift between the look-back window and the forecast window (the covariate shift).
Additionally, there is a distribution shift between the predicted forecast windows at two consecutive scales which is a result of the upsampling operation alongside the error accumulation during the intermediate computations.
As a result, normalizing the output at a given step by either the look-back window statistics or the previously predicted forecast window statistics result in an accumulation of errors across steps.
We mitigate this by considering a moving average of forecast and look-back statistics as the basis for the output normalization.
While this change might appear relatively minor, it has a signiﬁcant impact on the resulting distribution of outputs.
The improvement is more evident when compared to the alternative approaches, namely, normalizing by either look-back or previous forecast window statistics.
3. I NPUT E MBEDDING
Following the previous works, we embed our input to have the same number of features as the hidden dimension of the model.
The embedding consists of three parts: (1) Value embedding which uses a linear layer to map the input observations of each step x t to the same dimension as the model.
We further concatenate an additional value 0 , 0 . 5 , or 1 respectively showing if each observation is coming from the look-back window, zero initialization, or the prediction of the previous steps.
(2) Temporal
Embedding which again uses a linear layer to embed the time stamp related to each observation to the hidden dimension of the model.
Here we concatenate an additional value 1 /s i − 0 .
5 as the current scale for the network before passing to the linear layer.
(3) We also use a ﬁxed positional embedding which is adapted to the different scales s i as follows:
[Equation]
3. L OSS F UNCTION
Using the standard MSE objective to train time-series forecasting models leaves them sensitive to outliers.
One possible solution is to use objectives more robust to outliers, such as the Huber loss (Huber, 1964).
However, when there are no major outliers, such objectives tend to underperform.
Given the heterogeneous nature of the data, we instead utilize the adaptive loss (Barron, 2019):
[Equation]
[Table]
with ξ =
( X out i − X
( H ) i )
in step i .
The parameters α and c , which modulate the loss sensitivity to outliers, are learnt in an end-to-end fashion during training.
To the best of our knowledge, this is the ﬁrst time this objective has been adapted to the context of time-series forecasting.
E XPERIMENTS
In this section, we ﬁrst showcase the main results of our proposed approach on a variety of forecasting dataset in Sec. 4.1.
Then, we provide an ablation study of the different components of our model in
Sec. 4.2, and also present qualitative results in Sec. 4.3.
Moreover, we discuss a series of additional extensions of our method in Sec. 4.4, shedding light on promising future directions.
4. M AIN RESULTS
Baselines : To measure the effectiveness of the proposed framework, we mainly use state-of-the-art transformer-based models FedFormer (Zhou et al., 2022b)
Reformer (Kitaev et al., 2020), Performer (Choromanski et al., 2020), Informer (Zhou et al., 2021) and Autoformer (Xu et al., 2021) which are proven to have beaten other transformer-based (e.g. LogTrans (Li et al., 2019), Reformer (Kitaev et al., 2020)), RNN-based (e.g. LSTMNet (Lai et al., 2018), LSTM) and TCN (Bai et al., 2018) models.
For brevity, we only keep comparisons with the transformer models in the tables.
Datasets : We consider four public datasets with different characteristics to evaluate our proposed framework.
Electricity Consuming Load (ECL) 1 corresponds to the electricity consumption (Kwh) of 321 clients.
Trafﬁc 2 aggregates the hourly occupancy rate of 963 car lanes of San Francisco bay area freeways.
Weather 3 contains 21 meteorological indicators, such as air temperature, humidity, etc, recorded every 10 minutes for the entirety of 2020.
Exchange-Rate (Lai et al., 2018) collects the daily exchange rates of 8 countries (Australia, British, Canada, Switzerland, China, Japan, New
Zealand and Singapore)
from 1990 to 2016.
National Illness (ILI) 4 corresponds to the weekly
[Table]
recorded inﬂuenza-like illness patients from the US Center for Disease Control and Prevention.
We consider horizon lengths of 24, 32, 48, and 64 with an input length of 32.
Implementation details : Following previous work (Xu et al., 2021; Zhou et al., 2021), we pass
X enc =
X ( L ) as the input to the encoder.
While an array of zero-values would be the default to pass to the decoder, the decoder instead takes as input the second half of the look-back window padded with zeros X dec
= { x t 0 + (cid:96) L / 2 , ..., x (cid:96) L , 0 , 0 , ..., 0 } with length (cid:96) L / 2 +
(cid:96) H .
The hidden dimension of models is 512 with a batch size of 32.
We use the Adam optimizer with a learning rate of 1e-4.
The look-back window size is ﬁxed to 96, and the horizon is varied from 96 to 720.
We repeat each experiment 5 times and report average values to reduce randomness.
For additional implementation details on our model and baselines please refer to Appendix A.
Main results and comparison with baselines :
Table 1 shows the results of the proposed framework compared to the baselines.
Our proposed multi-scale framework with the adaptive loss outperforms the baselines in almost all of the experiments with an average improvement of 5 .
6% over FEDFormer, 13% over Autoformer and 38% over Informer which are the three most recent transformer-based architectures on MSE.
We also achieved signiﬁcant error reduction on MAE.
The improvement is statistically signiﬁcant in all cases, and in certain cases quite substantial.
In particular for the exchange-rate dataset, with Informer and Reformer base models, our approach improves upon the respective baselines by over 50% averaged over the different horizon lengths.
Time and memory complexity : The proposed framework uses the same number of parameters for the model as the baselines (except two parameters α and c of the Adaptive loss).
Our framework sacriﬁces a small amount of computation efﬁciency for the sake of a signiﬁcant performance improvement.
We expand our analysis in Appendix C. As shown in Table 4 in Appendix, if we replace the operation at the ﬁnal scale by an interpolation of the prior output, we can achieve improved performance over the baselines, at no computational overhead. [Figure] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
4. A BLATION STUDY
We present main ablation studies in this section, more ablation results are shown in Appendix G.
Impact of each component : Two important components of our approach are the multi-scale framework and the use of the adaptive loss.
We conduct multiple experiments (1) removing the multi-scale framework and/or (2) replacing the Adaptive loss by the MSE for training, in order to demonstrate the beneﬁts of these two components.
Figure 4 shows the effect of multi-scale and the loss function with different base models.
Considering the impact of ablating the adaptive loss, we can see that for both the multi-scale and base models, training with the adaptive loss improves performance.
Similarly, adding the multi-scale framework improves performance, both with and without the adaptive loss.
Overall, combining the adaptive loss with the multi-scale framework results in the best performance.
Cross-scale normalization : As we discussed in Section 3.3, having the cross-scale normalization is crucial to avoid the distribution shifts.
To conﬁrm that, we conduct two experiments.
Firstly, we use the multi-scale framework without the cross-scale normalization to argue that the error accumulation and covariate shift between the scales leads higher error compared to only a single scale.
As shown in
Table 2, while the multi-scale framework can get better results in a few cases, it mostly have higher errors than baselines.
On the other hand, adding the normalization with only a single scale can still help to achieve better performance by reducing the effect of covariate shift between the training and the test series.
As shown in Table 3, the normalization improves the results of Informer and FEDformer consistently.
The decomposition layer of the Autoformer solves a similar problem and replacing that with our normalization harms the capacity of the model.
4. Q UALITATIVE R ESULTS
We have also shown the qualitative comparisons between the vanilla Informer and FEDformer versus the results of our framework in Figure 5.
Most notably, in both cases our approach appears signiﬁcantly better at forecasting the statistical properties (such as local variance) of the signal.
Our scaleformer-based models capture trends (and other human-relevant) information better than their baselines.
Despite these interesting ﬁndings, we would like to emphasize that these are randomly selected qualitative examples that may have their own limitations.
For more qualitative results, please refer to Section I in the Appendix.
4. E XTENSIONS AND D ISCUSSION
The Scaleformer structural prior has been shown to be beneﬁcial when applied to transformer-based, deterministic time series forecasting.
It is not however limited to those settings.
In this section, we show it can be extended to probabilisitc forecasting and non transformer-based encoders, both of which are closely coupled with our primary application.
We also aim to highlight potential promising future directions.
[Figure]
E l ec t r i c it y
W ea t h e r
E x c h a ng e -r a t e
Time
Time
V a l u e V a l u e
V a l u e
FEDformer
FEDformer-MSA
Time Time
We show that our Scaleformer can improve performance in a probabilistic forecasting setting (please refer to Table 9 in Appendix for more details).
We adopt the probabilistic output of DeepAR (Salinas et al., 2020), which is the most common probabilistic forecasting treatment.
In this setting, instead of a point estimate, we have two prediction heads, predicting the mean µ and standard deviation σ , trained with a negative log likelihood loss (NLL).
NLL and continuous ranked probability score (CRPS) are used as evaluation metrics.
All other hyperparameters remain unchanged.
Here, again, scaleformers continue to outperform the probabilistic Informer.
While we have mainly focused on improving transformer-based models, they are not the only encoders.
Recent models such as NHits (Challu et al., 2022) and FiLM (Zhou et al., 2022a) attain competitive performance, while assuming a ﬁxed length univariate input/output.
They are less ﬂexible compared with variable length of multi-variate input/output, but result in strong performance and faster inference than transformers, making them interesting to consider.
The Scaleformer prior demonstrates a statistically signiﬁcant improvement, on average, when adapted by NHits and FiLM to iteratively reﬁne predictions.
For more details please refer to Appendix K.
The results mentioned above demonstrate the fact that ScaleFormer can adapt to settings distinct from point-wise time-series forecasts with transformers (the primary scope of our paper), such as probabilistic forecasts and non-transformer models.
We consider such directions to therefore be promising for future work.
C ONCLUSION
Noting that introducing structural priors that account for multi-scale information is essential for accurate time-series forecastings, this paper proposes a novel multi-scale framework on top of recent state-the-art methods for time-series forecasting using Transformers.
Our framework iteratively reﬁnes a forecasted time-series at increasingly ﬁne-grained scales, and introduces a normalization scheme that minimizes distribution shifts between scales.
These contributions result in vastly improved performance over baseline transformer architectures, across a variety of settings, and qualitatively result in forecasts that better capture the trend and local variations of the target signal.
Additionally, our detailed ablation study shows that the different components synergetically work together to deliver this outcome.
For future work, it is promising to extend the preliminary work that has been done applying ScaleFormer architectures to both probabilistic forecasting and non-transformer models.
A I MPLEMENTATION D ETAILS
Our implementation is based on the Pytorch (Paszke et al., 2019) implementation 5 of the Autoformer and Informer (Xu et al., 2021).
The hidden dimension of models are ﬁxed to 512 with a batch size of and we train each model for for 10 epochs with early stop enabled.
To optimize the models, an
Adam optimizer has been used with a learning rate of 1e-4 for the forecasting model and 1e-3 for optimizing the adaptive loss.
The forecasting module is ﬁxed to 2 encoder layers and 1 decoder layer.
The look-back window size is ﬁxed to 96, and the horizon is varied from 96 to 720.
We repeat each experiment 5 times to reduce the effect of randomness in the reported values.
In all experiments, the temporal scale factor s is ﬁxed to
For Informer, we train the model without any changes as the core of our framework.
However,
Autoformer uses a decomposition layer at the input of the decoder and does not pass the trend series to the network, which makes the model unaware of the previous predictions.
To resolve this, we pass zeros as the trend and the series without decomposition as the input to the decoder.
For
Reformer, we used the available implementation 6 of the model from Xu et al.
In addition, we used the pytorch library 7 of Performer (Choromanski et al., 2020) for our performer baseline using the same parameters as our Reformer model, and ﬁnally, we use the ofﬁcial implementationof FEDformer (Zhou et al., 2022b) for the FEDformer model.
We ﬁxed the number of modes to 64 following the original paper and Wavelet Enhanced Structure for the core modules.
To make it consistent with our other experiments, we use the moving average with the kernel size 25 as in Xu et al.
(2021), however, FEDformer (Zhou et al., 2022b) is using moving average with the kernel size of 24.
Our random seed is ﬁxed to 2022 in all experiments.
B M ORE MOTIVATIONS
This section aims to provide more motivations for the use of a multi-scale architecture.
Let us ﬁrst consider the following classical example, highlighted in section 2 of Ferreira et al.
(2006), corresponding to the monthly ﬂows of the Fraser River from January of 1913 to December of 1990.
As shown in the their corresponding plot, the annual averages are strongly inter-related, pointing to the fact that seasonality alone will not sufﬁce to model the variations.
In the context of the paper, this showcases a failure mode of an ARMA model, but this failing is more general: models that do not explicitly account for inter-scale dependencies will perform poorly on similar datasets.
Different approaches have attempted to introduce multi-scale processing (Ferreira et al., 2006; Mozer, 1991) in ways that differ from our own approach.
The multi-scale temporal structure for music composition is introduced in (Mozer, 1991).
Ferreira et al.
(2006) proposed a time series model with rich autocorrelation structures by coupling processes evolving at different levels of resolution through time.
However, their base models are constrained to simple statistical models, e.g. Autoregressive models.
To conclude, we note the following: (1) the approaches mentioned above have applied multi-scale modeling with success, and (2) we are the ﬁrst work to explicitly consider a multi-scale prior by construction for transformers.
C R EDUCING COMPUTATIONAL COST
To obtain an estimation of the total running time of Scaleformer, the running time of each scale as the terms of a geometric progression based on the scale factor s which results the total time of 1 − s m 1 − s multiplied by the running time of the baseline method.
In this regard, Table 4 shows the running times of different experiments where s = 2 with the batch size is 32 on a GeForce GTX 1080 Ti GPU with 64 cores.
While our current code is not optimised, still the scaleformer of each method takes roughly twice of the baselines which is consistent with the mentioned formula.
Note that this is the smallest scale which means that our method is bounded to twice of the baseline and considering a larger scale factor can reduce the time overhead.
Table 5 shows the impact of replacing the Scaleformer operation at the real scale by an interpolation of values of the previous scale, i.e., at scale s 0 we do not apply the transformer but rather compute the results by linear interpolation of X out m − 1 , thereby reducing the compute cost.
This lower cost alternative results in better performance than the baselines, yet worse results than the full Scaleformer.
This shows that adding scale is indeed more efﬁcient, and that there is the possibility of a trade-off between further improved performance (full Scaleformer) or improved computational efﬁciency (interpolated Scaleformer).
[Table]
[Table]
D J USTIFICATION FOR THE ADAPTIVE LOSS
Mathematically, the justiﬁcation for the adaptive loss is as follows.
Considering the ξ term in equation 10, the function is asymptotically close (but not equivalent due to the denominator) to ξ α .
As a result, for outliers (for which ξ will be large), the loss term will function as a L α penalty on ξ , which will penalize outliers more for large α .
The converse of this is that we would expect a model trained with such a loss to learn lower values of α for settings with fewer outliers.
E I NTERPLAY BETWEEN ADAPTIVE LOSS AND MULTI - SCALE ARCHITECTURE
The main reason for combining the adaptive loss and multi-scale architecture under a uniﬁed framework is: they are synergetic.
How do we explain this synergy?
Compared to other transformer architectures (notably the baselines used), ScaleFormer has more iterative steps: the sequential multi-scale operations.
Iterative computation tends to accumulate more errors, which will behave like outliers for the purpose of this loss.
As a result, the process that leads to the need for the two components can be expressed as: (1) The multi-scale architecture is beneﬁcial for performance as a useful structural prior for time series data.
The multi-scale architecture however relies on sequential computation that increases the likelihood of explosive error accumulation.
The adaptive loss serves to mitigate this issue, leading to more stable learning and better performance.
F S INGLE - SCALE MODEL WITH MEAN - NORMALIZATION
A single-scale model with mean normalization performs better than the multi-scale version without normalization.
The reason for this is that normalization as our proposed scheme is targeting at all forms of internal distribution shift, not only those induced by the multi-scale architecture.
In our submission, we make the case for the multi-scale prior as a natural prior to add to transformers.
From empirical observations, we found that such a prior requires adapting normalization.
When investigating means of normalizing, we observed additional beneﬁts to non-multiscale architectures as well.
This means that they also suffer from other forms of distribution shift: we attribute it in the paper to e.g. shifts between lookback and forecast distributions.
G M ORE ABLATION STUDIES AND PARAMETER ANALYSIS G. M ORE RESULTS ON DIFFERENT COMPONENTS
Table 6 extends the results of Figure 4 of the paper to all four datasets for both Autoformer and
Informer, in the multivariate setting and for the two backbones, Autoformer and Informer.
[Table]
G. I TERATIVE REFINEMENT USING THE SAME SCALE
To further conﬁrm the effect of the multi-scale framework [Table] we also compare our proposed framework with another baseline by keeping the original scale in each iteration.
As Table 7 shows [Table] using multiscale framework outperforms keeping the original scale while having signiﬁcantly lower memory and time complexity overhead.
Following the main experiments, we use the scale factor of 2 which results S = { 16 , 8 , 4 , 2 , 1 } for Scaleformer (showed by -MSA ) and a scale factor of 1 with similarly 5 iterations for iterative reﬁnement (showed by -IA ).
G. R ESULTS ON DIFFERENT SCALES
Table 8 showcases the results of our model using different values of the scale parameter s .
It shows that a scale of 2 results in better performance. [Table]
G. E FFECT OF α ON THE LOSS FUNCTION
Figure 6 shows the impact of α on the shape of the loss function on the left, and example ground truth time-series corresponding to the horizon window on the right.
As noted in the main paper, lower values of α tend to result in better robustness to outliers.
This is indeed conﬁrmed empirically in the case of the weather dataset, which corresponds to the lowest learnt α value, and has the outliers with the largest relative scales.
For simplicity, we have excluded c on the analysis as it does not impact the robustness with regards to the outliers, as shown in Barron (2019).
[Figure]
[Table]
A u t o f o r m e r- M S
A
E l ec t r i c it y
I n f o r m e r- M S
A
W ea t h e r
Time Time
Time Time
Time
F E D f
o r m e r- M S
A
[Table]
e
Scale Scale
[Figure]
H D ISCUSSION ON NORMALIZATION FOR A UTOFORMER
There is limited performance gain for Autoformer from the normalization alone.
The reason is that
Autoformer beneﬁts from an inner series decomposition module which acts as the normalization by nature.
Indeed one beneﬁt of our proposed framework is a simple solution to bring the beneﬁts of these speciﬁed designs to other baselines which signiﬁcantly reduces the gap between for example
Informer and Autoformer.
AutoFormer already beneﬁts from an internal component that reduces internal distribution shift: the series decomposition module.
I A DDITIONAL QUALITATIVE RESULTS
Figure 7, Figure 8, and Figure 9 provide additional qualitative results respectively showing the intermediate results of each scale and the comparison between the baselines and our approach with the horizon length of 720 and 192.
As we can see, Scaleformer is able to better learn local and global time-series variations.
[Figure]
E l ec t r i c it y
W ea t h e r
T r a ff i c
E x c h a ng e - r a t e
Time
[Figure]
V a l u e
[Figure]
V a l u e V a l u e
V a l u e
[Figure]
MSA
E l ec t r i c it y
W ea t h e r
E x c h a ng e -r a t e
Time
Time
V a l u e V a l u e
V a l u e
Autoformer
Autoformer-MSA
Time Time
T r a ff i c
V a l u e
J P ROBABILISTIC F ORECASTING
Table 9 shows the comparison of probabilistic methods for Informer by following the probabilistic output of DeepAR (Salinas et al., 2020), which is the most common probabilistic forecasting treatment.
On implementation details, instead of point estimate, we have two prediction heads (one predict µ and one predict σ ) with negative log likelihood loss.
The other hyper parameters is the same as before.
We use NLL loss and continuous ranked probability score (CRPS) which are commonly used in probabilistic forecasting as evaluation metrics.
Dataset
[Figure]
[Table]
K E XTENSION TO O THER M ETHODS
Table 10 shows the comparison results of NHiTs (Challu et al., 2022) and FiLM (Zhou et al., 2022a) as two baselines.
For each method, we copy original model to have model for different scales and we concatenate the input with the output of previous scale for the new scale.
The training hyperparameters such as optimizer and learning rate is the same as the previous baselines.
[Table]
L C ROSS - SCALE N ORMALIZATION
The choice of normalization approach is essential to our method, due to the aforementioned distribution shifts.
In this section, we showcase multiple experiments that were done to study the impact of different ways of normalizing inputs.
In Table 11, we compare using two forms of normalization with
Informer-MSA as the baseline: mean-only versus normalization using both mean and standard deviation.
We consider that adding normalization to our method is a trade-off between two issues faced during training.
One one hand, (internal) distribution shifts hinder performance, as demonstrated in the experiments (See Table 2).
On the other hand, normalizing the internal representations results in a form of information loss (during the processing of the input): we lose information about mean and standard deviation of the input data.
We believe that the reason mean-normalization outperforms mean and standard deviation normalization is because it results in a better trade-off, losing less information while still managing to address the internal distribution shift.
Fig.
10 shows the different dataset distributions.
As we can see, mean normalization works better for datasets such as electricity and trafﬁc with almost unimodal distributions.
For datasets such as exchange-rate and to some extent weather, which have multi-modal distributions, we ﬁnd that mean and standandard deviation works better.
Our hypothesis is that mean-normalization is too simple an approach for more complex input distributions.
We note that, in all cases, either form of normalization results in strong improvements compared to the absence of normalization .
M S YNTHETIC D ATASET
To further evaluate our proposed framework and adaptive loss function, we generate a new dataset based on the Mackey–Glass equations Mackey & Glass (1977).
Concretely, we use the following
[Table]
equation:
[Equation]
where we follow L ´ opez-Caraballo et al.
; Farmer et al.
and we also consider x (0)
= 1 . 2 .
We create three series of length 10k.
First we create a series by only using the above equation and considering τ = 18 where the series has a chaotic behaviour when τ ≥ 17 Farmer et al.
We also provide two additional series with τ = 12 and τ = 9 , and also with added seasonal and trend components.
Our synthetic dataset is a combination of these 3 series (See Figure 11 for an example).
We compare our method using the new dataset on both Informer Zhou et al.
(2021) and
Autoformer Xu et al.
(2021) in Table 12, showing our proposed framework signiﬁcantly improves the performance over the baselines.
[Table] [Figure] [Table] [Figure]
In addition, to further analyze the effect of the adaptive loss on noisy datasets with outliers, we randomly replace a percentage of the training data with outliers by deﬁning an outlier as a sample with a distance of larger than 50 times of standard deviation of the series from the median.
Figureshows the improvement of using adaptive loss on both Informer baseline and our multi-scale version of Informer.
Adaptive loss increases the performance up to roughly 70% when there is a noisy input with extreme outlieers ( 5% of data) while it gets comparable results for a clean dataset, demonstrating it as a strong candidate to replace MSE loss in the time series forecasting tasks.