Original: ETS FORMER : E XPONENTIAL S MOOTHING T RANSFORMERS FOR T IME - SERIES F ORECASTING
Revised: ETS FORMER : E XPONENTIAL S MOOTHING T RANSFORMERS FOR T IME - SERIES F ORECASTING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: ABSTRACT
Revised: A

Revised: BSTRACT

Original: Transformers have recently been actively studied for time-series forecasting.
Revised: Transformers have recently been actively studied for time-series forecasting.

Original: While often showing promising results in various scenarios, traditional Transformers are not designed to fully exploit the characteristics of time-series data and thus suffer some fundamental limitations e.g. they are generally not decomposable or interpretable and are neither effective nor efficient for long -term forecasting.
Revised: While often showing promising results in various scenarios, traditional Transformers are not designed to fully exploit the characteristics of time-series data and thus suffer some fundamental limitations, e.g., they are generally not decomposable or interpretable, and are neither effective nor efficient for long-term forecasting.

Original: In this paper, we propose ETSformer a novel time series Transformer architecture which exploits the principle of exponential smoothing methods in improving
Revised: In this paper, we propose ETSformer, a novel time-series Transformer architecture, which exploits the principle of exponential smoothing methods in improving

Original: Transformers for time-series forecasting.
Revised: Transformers for time-series forecasting.

Original: Specifically ETSformer leverages a novel level-growth-seasonality decomposed Transformer architecture which leads to more interpretable and disentangled decomposed forecasts
Revised: Specifically, ETSformer leverages a novel level-growth-seasonality decomposed Transformer architecture which leads to more interpretable and disentangled decomposed forecasts.

Original: We further propose two novel attention mechanisms  the exponential smoothing attention and frequency attention which are specially designed to overcome the limitations of the vanilla attention mechanism for time-series data
Revised: We further propose two novel attention mechanisms – the exponential smoothing attention and frequency attention, which are specially designed to overcome the limitations of the vanilla attention mechanism for time-series data.

Original: Extensive experiments on various time-series benchmarks validate the efficacy and advantages of the proposed method
Revised: Extensive experiments on the long sequence time-series forecasting (LSTF) benchmark validates the efficacy and advantages of the proposed method.

Original: Code is attached in the supplementary material and will be made publicly available
Revised: Code is attached in the supplementary material, and will be made publicly available.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transformer models have achieved great success in the fields of natural language processing (Vaswani et al., 2017; Devlin et al., 2019), computer vision (Carion et al., 2020;
Revised: Transformer models have achieved great success in the fields of natural language processing (Vaswani et al., 2017; Devlin et al., 2019), computer vision (Carion et al., 2020;

Original: Dosovitskiy et al., 2021), and even more recently, time-series (Li et al., 2019; Wu et al., 2021;
Revised: Dosovitskiy et al., 2021), and even more recently, time-series (Li et al., 2019; Wu et al., 2021;

Original: Zhou et al., 2021; Zerveas et al., 2021;
Revised: Zhou et al., 2021; Zerveas et al., 2021;

Original: Zhou et al., 2022).
Revised: Zhou et al., 2022).

Original: While the success of Transformer models have been widely attributed to the self-attention mechanism, alternative forms of attention, infused with the appropriate inductive biases, have been introduced to tackle the unique properties of their underlying task or data (You et al., 2020; Raganato et al., 2020).
Revised: While the success of Transformer models have been widely attributed to the self-attention mechanism, alternative forms of attention, infused with the appropriate inductive biases, have been introduced to tackle the unique properties of their underlying task or data (You et al., 2020; Raganato et al., 2020).

Original: In time-series forecasting, decomposition-based architectures such as
Revised: In time-series forecasting, decomposition-based architectures such as

Original: Autoformer and FEDformer models (Wu et al., 2021;
Revised: Autoformer and FEDformer models (Wu et al., 2021;

Original: Zhou et al., 2022) have incorporated time-series specific inductive biases, leading to increased accuracy, and more interpretable forecasts (by decomposing forecasts into seasonal and trend components).
Revised: Zhou et al., 2022) have incorporated time-series specific inductive biases, leading to increased accuracy, and more interpretable forecasts (by decomposing forecasts into seasonal and trend components).

Original: Their success has been motivated by: (i) disentangling seasonal and trend representations via seasonal-trend decomposition (Cleveland & Tiao, 1976; Cleveland et al., 1990;
Revised: Their success has been motivated by: (i) disentangling seasonal and trend representations via seasonal-trend decomposition (Cleveland & Tiao, 1976; Cleveland et al., 1990;

Original: Woo et al., 2022), and (ii) replacing the vanilla pointwise dot-product attention which handle time-series patterns such as seasonality and trend inefficiently, with time-series specific attention mechanisms such as the Auto-Correlation mechanism and Frequency-Enhanced Attention.
Revised: Woo et al., 2022), and (ii) replacing the vanilla pointwise dot-product attention which handle time-series patterns such as seasonality and trend inefficiently, with time-series specific attention mechanisms such as the Auto-Correlation mechanism and Frequency-Enhanced Attention.

Original: While these existing work introduce the promising direction of interpretable and decomposed time-series forecasting for Transformer-based architectures, they suffer from two drawbacks.
Revised: While these existing work introduce the promising direction of interpretable and decomposed time-series forecasting for Transformer-based architectures, they suffer from two drawbacks.

Original: Firstly, they suffer from entangled seasonal-trend representations, evidenced in Figure 1, where the trend forecasts exhibit periodical patterns which should only be present in the seasonal component, and the seasonal component does not accurately track the (multiple) periodicities present in the ground truth seasonal component.
Revised: Firstly, they suffer from entangled seasonal-trend representations, evidenced in Figure 1, where the trend forecasts exhibit periodical patterns which should only be present in the seasonal component, and the seasonal component does not accurately track the (multiple) periodicities present in the ground truth seasonal component.

Original: This arises due to their decomposition mechanism which detects trend via a simple moving average over the input signal and detrends the signal by removing the detected trend component – an arguably naive approach.
Revised: This arises due to their decomposition mechanism which detects trend via a simple moving average over the input signal and detrends the signal by removing the detected trend component – an arguably naive approach.

Original: This method has many known pitfalls (Hyndman & Athanasopoulos, 2018), such as the trend-cycle component not being available for the first and last few observations, and over-smoothing rapid rises and falls.
Revised: This method has many known pitfalls (Hyndman & Athanasopoulos, 2018), such as the trend-cycle component not being available for the first and last few observations, and over-smoothing rapid rises and falls.

Original: Secondly, their proposed replacements for the vanilla attention mechanism are not human interpretable – demonstrated in Section 3.3.
Revised: Secondly, their proposed replacements for the vanilla attention mechanism are not human interpretable – demonstrated in Section 3.3.

Original: Model inspection and diagnosis allows us to better understand the fore-
Revised: Model inspection and diagnosis allows us to better understand the fore-

Original: [Figure]
Revised: [Figure]

Original: casts generated by our models, attributing predictions to each component to make better downstream decisions.
Revised: casts generated by our models, attributing predictions to each component to make better downstream decisions.

Original: For an attention mechanism focusing on seasonality, we would expect the cross-attention map visualization to produce clear periodic patterns which shift smoothly across decoder time steps.
Revised: For an attention mechanism focusing on seasonality, we would expect the cross-attention map visualization to produce clear periodic patterns which shift smoothly across decoder time steps.

Original: Yet, the Auto-Correlation mechanism from Autoformer does not exhibit this property, yielding similar attention weights across decoder time steps, while the Frequency-Enhanced Attention from
Revised: Yet, the Auto-Correlation mechanism from Autoformer does not exhibit this property, yielding similar attention weights across decoder time steps, while the Frequency-Enhanced Attention from

Original: FEDformer does not have such model interpretability capabilities due to its complicated frequency domain attention.
Revised: FEDformer does not have such model interpretability capabilities due to its complicated frequency domain attention.

Original: To address these limitations, we look towards the more principled approach of level-growth-season decomposition from ETS methods (Hyndman et al., 2008) (further introduced in Appendix A).
Revised: To address these limitations, we look towards the more principled approach of level-growth-season decomposition from ETS methods (Hyndman et al., 2008) (further introduced in Appendix A).

Original: This principle further deconstructs trend into level and growth components.
Revised: This principle further deconstructs trend into level and growth components.

Original: To extract the level and growth components, we also look at the idea of exponential smoothing, where more recent data gets weighted more highly than older data, reflecting the view that the more recent past should be considered more relevant for making new predictions or identifying current trends, to replace the naive moving average.
Revised: To extract the level and growth components, we also look at the idea of exponential smoothing, where more recent data gets weighted more highly than older data, reflecting the view that the more recent past should be considered more relevant for making new predictions or identifying current trends, to replace the naive moving average.

Original: At the same time, we leverage the idea of extracting the most salient periodic components in the frequency domain via the Fourier transform, to extract the global seasonal patterns present in the signal.
Revised: At the same time, we leverage the idea of extracting the most salient periodic components in the frequency domain via the Fourier transform, to extract the global seasonal patterns present in the signal.

Original: These principles help yield a stronger decomposition strategy by first extracting global periodic patterns as seasonality, and subsequently extracting growth as the change in level in an exponentially smoothed manner.
Revised: These principles help yield a stronger decomposition strategy by first extracting global periodic patterns as seasonality, and subsequently extracting growth as the change in level in an exponentially smoothed manner.

Original: Motivated by the above, we propose ETSformer, an interpretable and efficient Transformer architecture for time-series forecasting which yields disentangled seasonal-trend forecasts.
Revised: Motivated by the above, we propose ETSformer, an interpretable and efficient Transformer architecture for time-series forecasting which yields disentangled seasonal-trend forecasts.

Original: Instead of reusing the moving average operation for detrending, ETSformer overhauls the existing decomposition architecture by leveraging the level-growth-season principle, embedding it into a novel Transformer framework in a non-trivial manner.
Revised: Instead of reusing the moving average operation for detrending, ETSformer overhauls the existing decomposition architecture by leveraging the level-growth-season principle, embedding it into a novel Transformer framework in a non-trivial manner.

Original: Next, we introduce interpretable and efficient attention mechanisms – Exponential Smoothing Attention (ESA) for trend, and Frequency Attention (FA) for seasonality.
Revised: Next, we introduce interpretable and efficient attention mechanisms – Exponential Smoothing Attention (ESA) for trend, and Frequency Attention (FA) for seasonality.

Original: ESA assigns attention weights in an exponentially decreasing manner, with high values to nearby time steps and low values to far away time steps, thus specialising in extracting growth representations.
Revised: ESA assigns attention weights in an exponentially decreasing manner, with high values to nearby time steps and low values to far away time steps, thus specialising in extracting growth representations.

Original: FA leverages frequency domain representations to extract dominating seasonal patterns by selecting the Fourier bases with the K largest amplitudes.
Revised: FA leverages frequency domain representations to extract dominating seasonal patterns by selecting the Fourier bases with the K largest amplitudes.

Original: Both mechanisms have efficient implementations with O ( L log L ) complexity.
Revised: Both mechanisms have efficient implementations with O ( L log L ) complexity.

Original: Furthermore, we demonstrate human interpretable visualizations of both mechanisms in Section 3.3.
Revised: Furthermore, we demonstrate human interpretable visualizations of both mechanisms in Section 3.3.

Original: To summarize, our key contributions are as follows:
Revised: To summarize, our key contributions are as follows:

Original: • We introduce a novel decomposition Transformer architecture, incorporating the timetested level-growth-season principle for more disentangled, human-interpretable time-series forecasts.
Revised: • We introduce a novel decomposition Transformer architecture, incorporating the timetested level-growth-season principle for more disentangled, human-interpretable time-series forecasts.

Original: • We introduce two new attention mechanisms, ESA and FA, which incorporate stronger time-series specific inductive biases.
Revised: • We introduce two new attention mechanisms, ESA and FA, which incorporate stronger time-series specific inductive biases.

Original: They achieve better efficiency than vanilla attention, and yield interpretable attention weights upon model inspection.
Revised: They achieve better efficiency than vanilla attention, and yield interpretable attention weights upon model inspection.

Original: • The resulting method is a highly effective, efficient, and interpretable deep forecasting model.
Revised: • The resulting method is a highly effective, efficient, and interpretable deep forecasting model.

Original: We show this via extensive empirical analysis, that ETSformer achieves performance competitive with state-of-the-art methods over 6 real world datasets on both multivariate and univariate settings, and is highly efficient compared to competing methods.
Revised: We show this via extensive empirical analysis, that ETSformer achieves performance competitive with state-of-the-art methods over 6 real world datasets on both multivariate and univariate settings, and is highly efficient compared to competing methods.

Original: Input Embedding
Revised: Input Embedding

Original: Lookback Window: 𝑿 !"
Revised: Lookback Window: 𝑿 !"

Original: GrowthDamping
Revised: GrowthDamping

Original: FrequencyAttention + 𝑩 !
Revised: FrequencyAttention + 𝑩 !

Original: % 𝑺 !"
Revised: % 𝑺 !"

Original: % 𝑩 !:!
Revised: % 𝑩 !:!

Original: +𝑺 !:!
Revised: +𝑺 !:!

Original: Forecast Horizon: 𝑿E !:!
Revised: Forecast Horizon: 𝑿E !:!

Original: Layer
Revised: Layer

Original: Layer𝑬 !"
Revised: Layer𝑬 !"

Original: Encoder
Revised: Encoder

Original: G+S Stack
Revised: G+S Stack

Original: G+S
Revised: G+S

Original: Stack N
Revised: Stack N

Original: G+S Stack+
Revised: G+S Stack+

Original: Level Stack +
Revised: Level Stack +

Original: Linear
Revised: Linear

Original: Decoder
Revised: Decoder

Original: Layer N
Revised: Layer N

Original: Linear
Revised: Linear

Original: Concat
Revised: Concat

Original: Difference
Revised: Difference

Original: Linear
Revised: Linear

Original: Exponential Smoothing Attention 𝒁 !"
Revised: Exponential Smoothing Attention 𝒁 !"

Original: %"( 𝑩 !"
Revised: %"( 𝑩 !"

Original: % (Growth)
Revised: % (Growth)

Original: DFT
Revised: DFT

Original: Top-K Amplitude iDFT
Revised: Top-K Amplitude iDFT

Original: 𝒁 !"
Revised: 𝒁 !"

Original: %"( 𝑺 !"
Revised: %"( 𝑺 !"

Original: % (Season) + Feedforward
Revised: % (Season) + Feedforward

Original: LayerNorm
Revised: LayerNorm

Original: Multi-
Revised: Multi-

Original: [Figure]
Revised: [Figure]

Original: Attention
Revised: Attention

Original: LayerNormFrequencyAttention -
Revised: LayerNormFrequencyAttention -

Original: L e v e l 𝒁 !"
Revised: L e v e l 𝒁 !"

Original: % 𝒁 !"
Revised: % 𝒁 !"

Original: %"( 𝑬 !" %"( 𝑬 !"
Revised: %"( 𝑬 !" %"( 𝑬 !"

Original: % (Level) 𝑩 !"
Revised: % (Level) 𝑩 !"

Original: % 𝑺 !"
Revised: % 𝑺 !"

Original:  ETS FORMER
Revised:  ETS FORMER

Original: Problem Formulation Let x t ∈ R m denote an observation of a multivariate time-series at time step t .
Revised: Problem Formulation Let x t ∈ R m denote an observation of a multivariate time-series at time step t .

Original: Given a lookback window X t − L : t =
Revised: Given a lookback window X t − L : t =

Original: [ x t − L , . . .
Revised: [ x t − L , . . .

Original: , x t − 1 ] , we consider the task of predicting future values over a horizon, X t : t
Revised: , x t − 1 ] , we consider the task of predicting future values over a horizon, X t : t

Original: + H =
Revised: + H =

Original: [ x t , . . .
Revised: [ x t , . . .

Original: , x t + H − 1 ] .
Revised: , x t + H − 1 ] .

Original: We denote ˆ X t : t + H as the point forecast of X t : t + H .
Revised: We denote ˆ X t : t + H as the point forecast of X t : t + H .

Original: Thus, the goal is to learn a forecasting function ˆ X t : t
Revised: Thus, the goal is to learn a forecasting function ˆ X t : t

Original: + H = f
Revised: + H = f

Original: ( X t − L : t ) by minimizing some loss function L : R H × m × R H × m → R .
Revised: ( X t − L : t ) by minimizing some loss function L : R H × m × R H × m → R .

Original: In the following, we explain how ETSformer infuses level-growth-seasonal decomposition the the classical encoder-decoder Transformer architecture, specializing for interpretable time-series forecasting.
Revised: In the following, we explain how ETSformer infuses level-growth-seasonal decomposition the the classical encoder-decoder Transformer architecture, specializing for interpretable time-series forecasting.

Original: Our architecture design methodology relies on three key principles: (1) the architecture leverages the stacking of multiple layers to progressively extract a series of level, growth, and seasonal representations from the intermediate latent residual; (2) performs level-growth-seasonal decomposition of latent representations, by extracting salient seasonal patterns while modeling level and growth components following an exponential smoothing formulation; (3) the final forecast is a composition of level, growth, and seasonal components making it human interpretable.
Revised: Our architecture design methodology relies on three key principles: (1) the architecture leverages the stacking of multiple layers to progressively extract a series of level, growth, and seasonal representations from the intermediate latent residual; (2) performs level-growth-seasonal decomposition of latent representations, by extracting salient seasonal patterns while modeling level and growth components following an exponential smoothing formulation; (3) the final forecast is a composition of level, growth, and seasonal components making it human interpretable.

Original: 2. O VERALL A RCHITECTURE
Revised: 2. O VERALL A RCHITECTURE

Original: Figure 2 illustrates the overall encoder-decoder architecture of ETSformer.
Revised: Figure 2 illustrates the overall encoder-decoder architecture of ETSformer.

Original: At each layer, the encoder is designed to iteratively extract growth and seasonal latent components from the lookback window.
Revised: At each layer, the encoder is designed to iteratively extract growth and seasonal latent components from the lookback window.

Original: The level is then extracted in a similar fashion to classical level smoothing in Equation (3).
Revised: The level is then extracted in a similar fashion to classical level smoothing in Equation (3).

Original: These extracted components are then fed to the decoder to further generate the final H -step ahead forecast via a composition of level, growth, and seasonal forecasts, which is defined:
Revised: These extracted components are then fed to the decoder to further generate the final H -step ahead forecast via a composition of level, growth, and seasonal forecasts, which is defined:

Original: [Equation]
Revised: [Equation]

Original: where E t : t + H ∈ R
Revised: where E t : t + H ∈ R

Original: H
Revised: H

Original: × m , and B ( n ) t : t
Revised: × m , and B ( n ) t : t

Original: + H , S ( n ) t : t
Revised: + H , S ( n ) t : t

Original: + H ∈ R H
Revised: + H ∈ R H

Original: × d represent the level forecasts, and the growth and seasonal latent representations of each time step in the forecast horizon, respectively.
Revised: × d represent the level forecasts, and the growth and seasonal latent representations of each time step in the forecast horizon, respectively.

Original: The superscript represents the stack index, for a total of N encoder stacks.
Revised: The superscript represents the stack index, for a total of N encoder stacks.

Original: Note that Linear( · ) : R d → R m operates element-wise along each time step, projecting the extracted growth and seasonal representations from latent to observation space.
Revised: Note that Linear( · ) : R d → R m operates element-wise along each time step, projecting the extracted growth and seasonal representations from latent to observation space.

Original: 2.1. I NPUT E MBEDDING
Revised: 2.1. I NPUT E MBEDDING

Original: Raw signals from the lookback window are mapped to latent space via the input embedding module, defined by Z (0) t − L : t = E (0) t − L : t =
Revised: Raw signals from the lookback window are mapped to latent space via the input embedding module, defined by Z (0) t − L : t = E (0) t − L : t =

Original: Conv( X t − L : t ) , where Conv is a temporal convolutional filter with kernel size 3, input channel m and output channel d .
Revised: Conv( X t − L : t ) , where Conv is a temporal convolutional filter with kernel size 3, input channel m and output channel d .

Original: In contrast to prior work (Li et al., 2019;
Revised: In contrast to prior work (Li et al., 2019;

Original: Wu et al., 2020; 2021;
Revised: Wu et al., 2020; 2021;

Original: Zhou et al., 2021), the inputs of ETSformer do not rely on any other manually designed dynamic time-dependent covariates (e.g. month-of-year, day-of-week) for both the lookback window and forecast horizon.
Revised: Zhou et al., 2021), the inputs of ETSformer do not rely on any other manually designed dynamic time-dependent covariates (e.g. month-of-year, day-of-week) for both the lookback window and forecast horizon.

Original: This is because the proposed Frequency Attention module (details in Section 2.2.2) is able to automatically uncover these seasonal patterns, which renders it more applicable for challenging scenarios without these discriminative covariates and reduces the need for feature engineering.
Revised: This is because the proposed Frequency Attention module (details in Section 2.2.2) is able to automatically uncover these seasonal patterns, which renders it more applicable for challenging scenarios without these discriminative covariates and reduces the need for feature engineering.

Original: 2.1. E NCODER
Revised: 2.1. E NCODER

Original: The encoder focuses on extracting a series of latent growth and seasonality representations in a cascaded manner from the lookback window.
Revised: The encoder focuses on extracting a series of latent growth and seasonality representations in a cascaded manner from the lookback window.

Original: To achieve this goal, traditional methods rely on the assumption of additive or multiplicative seasonality which has limited capability to express complex patterns beyond these assumptions.
Revised: To achieve this goal, traditional methods rely on the assumption of additive or multiplicative seasonality which has limited capability to express complex patterns beyond these assumptions.

Original: Inspired by (Oreshkin et al., 2019; He et al., 2016), we leverage residual learning to build an expressive, deep architecture to characterize the complex intrinsic patterns.
Revised: Inspired by (Oreshkin et al., 2019; He et al., 2016), we leverage residual learning to build an expressive, deep architecture to characterize the complex intrinsic patterns.

Original: Each layer can be interpreted as sequentially analyzing the input signals.
Revised: Each layer can be interpreted as sequentially analyzing the input signals.

Original: The extracted growth and seasonal signals are then removed from the residual and undergo a nonlinear transformation before moving to the next layer.
Revised: The extracted growth and seasonal signals are then removed from the residual and undergo a nonlinear transformation before moving to the next layer.

Original: Each encoder layer takes as input the residual from the previous encoder layer Z ( n − 1) t − L : t and emits Z ( n ) t − L : t , B ( n ) t − L : t , S ( n ) t − L : t , the residual, latent growth, and seasonal representations for the lookback window via the Multi-Head Exponential Smoothing
Revised: Each encoder layer takes as input the residual from the previous encoder layer Z ( n − 1) t − L : t and emits Z ( n ) t − L : t , B ( n ) t − L : t , S ( n ) t − L : t , the residual, latent growth, and seasonal representations for the lookback window via the Multi-Head Exponential Smoothing

Original: Attention (MH-ESA) and Frequency Attention (FA) modules (detailed description in Section 2.2).
Revised: Attention (MH-ESA) and Frequency Attention (FA) modules (detailed description in Section 2.2).

Original: The following equations formalizes the overall pipeline in each encoder layer, and for ease of exposition, we use the notation :
Revised: The following equations formalizes the overall pipeline in each encoder layer, and for ease of exposition, we use the notation :

Original: = for a variable update.
Revised: = for a variable update.

Original: [Equation]
Revised: [Equation]

Original: L : t :=
Revised: L : t :=

Original: LN( Z ( n − 1) t − L : t − B ( n ) t − L : t
Revised: LN( Z ( n − 1) t − L : t − B ( n ) t − L : t

Original: [Equation]
Revised: [Equation]

Original: Z ( n ) t − L : t = LN
Revised: Z ( n ) t − L : t = LN

Original: ( Z ( n − 1) t − L : t + FF
Revised: ( Z ( n − 1) t − L : t + FF

Original: ( Z ( n − 1) t − L : t ))
Revised: ( Z ( n − 1) t − L : t ))

Original: LN is layer normalization (Ba et al., 2016), FF( x )
Revised: LN is layer normalization (Ba et al., 2016), FF( x )

Original: =
Revised: =

Original: Linear( σ (Linear( x ))) is a position-wise feedforward network (Vaswani et al., 2017) and σ ( · ) is the sigmoid function.
Revised: Linear( σ (Linear( x ))) is a position-wise feedforward network (Vaswani et al., 2017) and σ ( · ) is the sigmoid function.

Original: Level Module Given the latent growth and seasonal representations from each layer, we extract the level at each time step t in the lookback window in a similar way as the level smoothing equation in
Revised: Level Module Given the latent growth and seasonal representations from each layer, we extract the level at each time step t in the lookback window in a similar way as the level smoothing equation in

Original: Equation (3).
Revised: Equation (3).

Original: Formally, the adjusted level is a weighted average of the current (de-seasonalized) level and the level-growth forecast from the previous time step t − 1 .
Revised: Formally, the adjusted level is a weighted average of the current (de-seasonalized) level and the level-growth forecast from the previous time step t − 1 .

Original: It can be formulated as:
Revised: It can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where α ∈ R m is a learnable smoothing parameter, ∗ is an element-wise multiplication term, and
Revised: where α ∈ R m is a learnable smoothing parameter, ∗ is an element-wise multiplication term, and

Original: Linear( · ) : R d → R m maps representations to observation space.
Revised: Linear( · ) : R d → R m maps representations to observation space.

Original: Finally, the extracted level in the last layer E ( N ) t − L : t can be regarded as the corresponding level for the lookback window.
Revised: Finally, the extracted level in the last layer E ( N ) t − L : t can be regarded as the corresponding level for the lookback window.

Original: We show in
Revised: We show in

Original: Appendix B.3 that this recurrent exponential smoothing equation can also be efficiently evaluated using the efficient A ES algorithm (Algorithm 1) with an auxiliary term.
Revised: Appendix B.3 that this recurrent exponential smoothing equation can also be efficiently evaluated using the efficient A ES algorithm (Algorithm 1) with an auxiliary term.

Original: 2.1. D ECODER
Revised: 2.1. D ECODER

Original: The decoder is tasked with generating the H -step ahead forecasts.
Revised: The decoder is tasked with generating the H -step ahead forecasts.

Original: As shown in Equation (1), the final forecast is a composition of level forecasts E t : t + H , growth representations B ( n ) t : t + H and seasonal representations S ( n ) t : t + H in the forecast horizon.
Revised: As shown in Equation (1), the final forecast is a composition of level forecasts E t : t + H , growth representations B ( n ) t : t + H and seasonal representations S ( n ) t : t + H in the forecast horizon.

Original: It comprises N Growth + Seasonal (G+S) Stacks, and a Level Stack.
Revised: It comprises N Growth + Seasonal (G+S) Stacks, and a Level Stack.

Original: The G+S Stack consists of the Growth Damping (GD) and FA blocks, which leverage
Revised: The G+S Stack consists of the Growth Damping (GD) and FA blocks, which leverage

Original: B ( n ) t , S ( n ) t − L : t to predict B ( n ) t : t + H , S ( n ) t : t + H , respectively.
Revised: B ( n ) t , S ( n ) t − L : t to predict B ( n ) t : t + H , S ( n ) t : t + H , respectively.

Original: Growth: B ( n
Revised: Growth: B ( n

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: Seasonal
Revised: Seasonal

Original: [Equation]
Revised: [Equation]

Original: To obtain the level in the forecast horizon, the Level Stack repeats the level in the last time step t along the forecast horizon.
Revised: To obtain the level in the forecast horizon, the Level Stack repeats the level in the last time step t along the forecast horizon.

Original: It can be defined as E t : t + H =
Revised: It can be defined as E t : t + H =

Original: Repeat H ( E ( N ) t ) =
Revised: Repeat H ( E ( N ) t ) =

Original: [ E ( N ) t , . .
Revised: [ E ( N ) t , . .

Original: , E ( N ) t ] , with Repeat H ( · ) : R 1 × m → R H × m .
Revised: , E ( N ) t ] , with Repeat H ( · ) : R 1 × m → R H × m .

Original: [Figure]
Revised: [Figure]

Original: Growth Damping To obtain the growth representation in the forecast horizon, we follow the idea of trend damping in Equation (4) to make robust multi-step forecast.
Revised: Growth Damping To obtain the growth representation in the forecast horizon, we follow the idea of trend damping in Equation (4) to make robust multi-step forecast.

Original: Thus, the trend representations can be formulated as:
Revised: Thus, the trend representations can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where 0 < γ < 1 is the damping parameter which is learnable, and in practice, we apply a multi-head version of trend damping by making use of n h damping parameters.
Revised: where 0 < γ < 1 is the damping parameter which is learnable, and in practice, we apply a multi-head version of trend damping by making use of n h damping parameters.

Original: Similar to the implementation for level forecast in the Level Stack, we only use the last trend representation in the lookback window
Revised: Similar to the implementation for level forecast in the Level Stack, we only use the last trend representation in the lookback window

Original: B ( n ) t to forecast the trend representation in the forecast horizon.
Revised: B ( n ) t to forecast the trend representation in the forecast horizon.

Original: 2. E XPONENTIAL S MOOTHING A TTENTION AND F REQUENCY A TTENTION M ECHANISM
Revised: 2. E XPONENTIAL S MOOTHING A TTENTION AND F REQUENCY A TTENTION M ECHANISM

Original: Considering the ineffectiveness of existing attention mechanisms in handling time-series data, we develop the Exponential Smoothing Attention (ESA) and Frequency Attention (FA) mechanisms to extract latent growth and seasonal representations.
Revised: Considering the ineffectiveness of existing attention mechanisms in handling time-series data, we develop the Exponential Smoothing Attention (ESA) and Frequency Attention (FA) mechanisms to extract latent growth and seasonal representations.

Original: ESA is a non-adaptive, learnable attention scheme with an inductive bias to attend more strongly to recent observations by following an exponential decay, while FA is a non-learnable attention scheme, that leverages Fourier transformation to select dominating seasonal patterns.
Revised: ESA is a non-adaptive, learnable attention scheme with an inductive bias to attend more strongly to recent observations by following an exponential decay, while FA is a non-learnable attention scheme, that leverages Fourier transformation to select dominating seasonal patterns.

Original: A comparison between existing work and our proposed ESA and FA is illustrated in Figure 3.
Revised: A comparison between existing work and our proposed ESA and FA is illustrated in Figure 3.

Original: 2.2. E XPONENTIAL S MOOTHING A TTENTION
Revised: 2.2. E XPONENTIAL S MOOTHING A TTENTION

Original: Vanilla self-attention can be regarded as a weighted combination of an input sequence, where the weights are normalized alignment scores measuring the similarity between input contents (Tsai et al., 2019).
Revised: Vanilla self-attention can be regarded as a weighted combination of an input sequence, where the weights are normalized alignment scores measuring the similarity between input contents (Tsai et al., 2019).

Original: Inspired by the exponential smoothing in Equation (3), we aim to assign a higher weight to recent observations.
Revised: Inspired by the exponential smoothing in Equation (3), we aim to assign a higher weight to recent observations.

Original: It can be regarded as a novel form of attention whose weights are computed by the relative time lag, rather than input content.
Revised: It can be regarded as a novel form of attention whose weights are computed by the relative time lag, rather than input content.

Original: Thus, the ESA mechanism can be defined as A ES : R L × d → R L × d , where A ES ( V ) t ∈ R d denotes the t -th row of the output matrix, representing the token corresponding to the t -th time step.
Revised: Thus, the ESA mechanism can be defined as A ES : R L × d → R L × d , where A ES ( V ) t ∈ R d denotes the t -th row of the output matrix, representing the token corresponding to the t -th time step.

Original: Its exponential smoothing formula can be further written as:
Revised: Its exponential smoothing formula can be further written as:

Original: [Equation]
Revised: [Equation]

Original: where 0 < α < 1 and v 0 are learnable parameters known as the smoothing parameter and initial state respectively.
Revised: where 0 < α < 1 and v 0 are learnable parameters known as the smoothing parameter and initial state respectively.

Original: Efficient A ES algorithm
Revised: Efficient A ES algorithm

Original: The straightforward implementation of the ESA mechanism by constructing the attention matrix, A ES and performing a matrix multiplication with the input sequence (detailed algorithm in Appendix B.4) results in an O ( L 2 ) computational complexity.
Revised: The straightforward implementation of the ESA mechanism by constructing the attention matrix, A ES and performing a matrix multiplication with the input sequence (detailed algorithm in Appendix B.4) results in an O ( L 2 ) computational complexity.

Original: [Equation]
Revised: [Equation]

Original: Yet, we are able to achieve an efficient algorithm by exploiting the unique structure of the exponential smoothing attention matrix, A ES , which is illustrated in Appendix B.1.
Revised: Yet, we are able to achieve an efficient algorithm by exploiting the unique structure of the exponential smoothing attention matrix, A ES , which is illustrated in Appendix B.1.

Original: Each row of the attention matrix can be regarded as iteratively right shifting with padding (ignoring the first column).
Revised: Each row of the attention matrix can be regarded as iteratively right shifting with padding (ignoring the first column).

Original: Thus, a matrix-vector multiplication can be computed with a cross-correlation operation, which in turn has an efficient fast Fourier transform implementation (Mathieu et al., 2014).
Revised: Thus, a matrix-vector multiplication can be computed with a cross-correlation operation, which in turn has an efficient fast Fourier transform implementation (Mathieu et al., 2014).

Original: The full algorithm is described in Algorithm 1, Appendix B.2, achieving an O ( L log L ) complexity.
Revised: The full algorithm is described in Algorithm 1, Appendix B.2, achieving an O ( L log L ) complexity.

Original: Multi-Head Exponential Smoothing Attention (MH-ESA)
Revised: Multi-Head Exponential Smoothing Attention (MH-ESA)

Original: We use A ES as a basic building block, and develop the Multi-Head Exponential Smoothing Attention to extract latent growth representations.
Revised: We use A ES as a basic building block, and develop the Multi-Head Exponential Smoothing Attention to extract latent growth representations.

Original: Formally, we obtain the growth representations by taking the successive difference of the residuals.
Revised: Formally, we obtain the growth representations by taking the successive difference of the residuals.

Original: [Equation]
Revised: [Equation]

Original: where MH- A ES is a multi-head version of A ES and v ( n )is the initial state from the ESA mechanism.
Revised: where MH- A ES is a multi-head version of A ES and v ( n )is the initial state from the ESA mechanism.

Original: 2.2. F REQUENCY A TTENTION
Revised: 2.2. F REQUENCY A TTENTION

Original: The goal of identifying and extracting seasonal patterns from the lookback window is twofold.
Revised: The goal of identifying and extracting seasonal patterns from the lookback window is twofold.

Original: Firstly, it can be used to perform de-seasonalization on the input signals such that downstream components are able to focus on modeling the level and growth information.
Revised: Firstly, it can be used to perform de-seasonalization on the input signals such that downstream components are able to focus on modeling the level and growth information.

Original: Secondly, we are able to extrapolate the seasonal patterns to build representations for the forecast horizon.
Revised: Secondly, we are able to extrapolate the seasonal patterns to build representations for the forecast horizon.

Original: The main challenge is to automatically identify seasonal patterns.
Revised: The main challenge is to automatically identify seasonal patterns.

Original: Fortunately, the use of power spectral density estimation for periodicity detection has been well studied (Vlachos et al., 2005).
Revised: Fortunately, the use of power spectral density estimation for periodicity detection has been well studied (Vlachos et al., 2005).

Original: Inspired by these methods, we leverage the discrete Fourier transform (DFT, details in Appendix C) to develop the FA mechanism to extract dominant seasonal patterns.
Revised: Inspired by these methods, we leverage the discrete Fourier transform (DFT, details in Appendix C) to develop the FA mechanism to extract dominant seasonal patterns.

Original: Specifically, FA first decomposes input signals into their Fourier bases via a DFT along the temporal dimension, F ( Z ( n − 1) t − L : t )
Revised: Specifically, FA first decomposes input signals into their Fourier bases via a DFT along the temporal dimension, F ( Z ( n − 1) t − L : t )

Original: ∈ C F × d where F = ⌊ L/ 2 ⌋ + 1 , and selects bases with the K largest amplitudes.
Revised: ∈ C F × d where F = ⌊ L/ 2 ⌋ + 1 , and selects bases with the K largest amplitudes.

Original: An inverse DFT is then applied to obtain the seasonality pattern in time domain.
Revised: An inverse DFT is then applied to obtain the seasonality pattern in time domain.

Original: Formally, this is given by the following equations:
Revised: Formally, this is given by the following equations:

Original: [Equation]
Revised: [Equation]

Original: where Φ k,i , A k,i are the phase/amplitude of the k -th frequency for the i -th dimension, arg Top-K returns the arguments of the top K amplitudes, K is a hyperparameter, f k is the Fourier frequency of the corresponding index, and ¯ f k , ¯ Φ k,i are the Fourier frequency/amplitude of the corresponding conjugates.
Revised: where Φ k,i , A k,i are the phase/amplitude of the k -th frequency for the i -th dimension, arg Top-K returns the arguments of the top K amplitudes, K is a hyperparameter, f k is the Fourier frequency of the corresponding index, and ¯ f k , ¯ Φ k,i are the Fourier frequency/amplitude of the corresponding conjugates.

Original: Finally, the latent seasonal representation of the i -th dimension for the lookback window is formulated as S ( n ) t − L : t,i =
Revised: Finally, the latent seasonal representation of the i -th dimension for the lookback window is formulated as S ( n ) t − L : t,i =

Original: [ S ( n ) t − L,i , . . .
Revised: [ S ( n ) t − L,i , . . .

Original: , S ( n ) t − 1 ,i ] .
Revised: , S ( n ) t − 1 ,i ] .

Original: For the forecast horizon, the FA module extrapolates beyond the lookback window via, S ( n ) t : t + H,i =
Revised: For the forecast horizon, the FA module extrapolates beyond the lookback window via, S ( n ) t : t + H,i =

Original: [ S ( n ) t,i , . .
Revised: [ S ( n ) t,i , . .

Original: , S ( n ) t + H − 1 ,i ] .
Revised: , S ( n ) t + H − 1 ,i ] .

Original: Since K is a hyperparameter typically chosen for small values, the complexity for the FA mechanism is similarly O
Revised: Since K is a hyperparameter typically chosen for small values, the complexity for the FA mechanism is similarly O

Original: ( L log L ) .
Revised: ( L log L ) .

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section presents extensive empirical evaluations on the LSTF task over 6 real world multivariate datasets, ETT, ECL, Exchange, Traffic, Weather, and ILI, coming from a variety of application areas (details in Appendix E).
Revised: This section presents extensive empirical evaluations on the LSTF task over 6 real world multivariate datasets, ETT, ECL, Exchange, Traffic, Weather, and ILI, coming from a variety of application areas (details in Appendix E).

Original: Performance is evaluated via the mean squared error (MSE) and mean absolute error (MAE) metrics.
Revised: Performance is evaluated via the mean squared error (MSE) and mean absolute error (MAE) metrics.

Original: For the main benchmark, datasets are split into train, validation, and test sets chronologically, following a 60/20/20 split for the ETT datasets and 70/10/20 split for other datasets.
Revised: For the main benchmark, datasets are split into train, validation, and test sets chronologically, following a 60/20/20 split for the ETT datasets and 70/10/20 split for other datasets.

Original: The multivariate benchmark makes use of all dimensions, while univariate benchmark selects the last dimension of the datasets as the target variable, following previous work (Zhou et al., 2021; Wu et al., 2021).
Revised: The multivariate benchmark makes use of all dimensions, while univariate benchmark selects the last dimension of the datasets as the target variable, following previous work (Zhou et al., 2021; Wu et al., 2021).

Original: Data is pre-processed by performing standardization based on train set statistics.
Revised: Data is pre-processed by performing standardization based on train set statistics.

Original: Further details on implementation and hyperparameters can be found in Appendix D.
Revised: Further details on implementation and hyperparameters can be found in Appendix D.

Original: This is followed by an ablation study of the various contributing components, and interpretability experiments of our proposed model, and finally an analysis on computational efficiency.
Revised: This is followed by an ablation study of the various contributing components, and interpretability experiments of our proposed model, and finally an analysis on computational efficiency.

Original: 3. R ESULTS
Revised: 3. R ESULTS

Original: For the multivariate benchmark, baselines include recently proposed time-series/efficient Transformers – FEDformer, Autoformer, Informer, LogTrans (Li et al., 2019), and Reformer (Kitaev et al., 2020), and RNN variants – LSTnet (Lai et al., 2018), and LSTM (Hochreiter & Schmidhuber, 1997).
Revised: For the multivariate benchmark, baselines include recently proposed time-series/efficient Transformers – FEDformer, Autoformer, Informer, LogTrans (Li et al., 2019), and Reformer (Kitaev et al., 2020), and RNN variants – LSTnet (Lai et al., 2018), and ES-RNN (Smyl, 2020).

Original: Univariate baselines further include N-BEATS (Oreshkin et al., 2019), DeepAR (Salinas et al., 2020), ARIMA, Prophet (Taylor & Letham, 2018), and AutoETS (Bhatnagar et al., 2021).
Revised: Univariate baselines further include N-BEATS (Oreshkin et al., 2019), DeepAR (Salinas et al., 2020), ARIMA, Prophet (Taylor & Letham, 2018), and AutoETS (Bhatnagar et al., 2021).

Original: We obtain baseline results from the following papers: (Wu et al., 2021;
Revised: We obtain baseline results from the following papers: (Wu et al., 2021;

Original: Zhou et al., 2021), and further run AutoETS from the Merlion library (Bhatnagar et al., 2021).
Revised: Zhou et al., 2021), and further run AutoETS from the Merlion library (Bhatnagar et al., 2021).

Original: Table 1 summarize the results of ETSformer against top performing baselines on a selection of datasets, for the multivariate setting, and Table 6 in Appendix G for space.
Revised: Table 1 summarize the results of ETSformer against top performing baselines on a selection of datasets, for the multivariate setting, and Table 6 in Appendix G for space.

Original: Results for ETSformer are averaged over three runs (standard deviation in Appendix H).
Revised: Results for ETSformer are averaged over three runs (standard deviation in Appendix H).

Original: Overall, ETSformer achieves competitive performance, achieving best performance on 14 out of 24 datasets/settings on MSE for the multivariate case, and within topperformance across all 24 datasets/settings.
Revised: Overall, ETSformer achieves competitive performance, achieving best performance on 14 out of 24 datasets/settings on MSE for the multivariate case, and within top 2 performance across alldatasets/settings.

Original: 3. A BLATION S TUDY
Revised: 3. A BLATION S TUDY

Original: We study the contribution of each major component which the final forecast is composed of level, growth, and seasonality.
Revised: We study the contribution of each major component which the final forecast is composed of level, growth, and seasonality.

Original: Table 2 first presents the performance of the full model, and subsequently, the performance of the resulting model by removing each component.
Revised: Table 2 first presents the performance of the full model, and subsequently, the performance of the resulting model by removing each component.

Original: We observe that the composition of level, growth, and season provides the most accurate forecasts across a variety of application areas, and removing any one component results in a deterioration.
Revised: We observe that the composition of level, growth, and season provides the most accurate forecasts across a variety of application areas, and removing any one component results in a deterioration.

Original: In particular, estimation of the level of the time-series is critical.
Revised: In particular, estimation of the level of the time-series is critical.

Original: We also analyse the design of the MH-ESA in Section 3.2, replacing it with a vanilla multi-head attention and an FC layer performing token mixing – we observe that our trend attention formulation indeed is more effective.
Revised: We also analyse the design of the MH-ESA in Section 3.2, replacing it with a vanilla multi-head attention and an FC layer performing token mixing – we observe that our trend attention formulation indeed is more effective.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 3. I NTERPRETABILITY
Revised: 3. I NTERPRETABILITY

Original: ETSformer generates interpretable forecasts which can be decomposed into disentangled level, growth, and seasonal components.
Revised: ETSformer generates interpretable forecasts which can be decomposed into disentangled level, growth, and seasonal components.

Original: We showcased this ability compared to baselines in Figure 1 on synthetic data containing (nonlinear) trend and seasonality patterns (details in Appendix F) , since we are not able to obtain ground truth decomposition from real-world data.
Revised: We showcased this ability compared to baselines in Figure 1 on synthetic data containing (nonlinear) trend and seasonality patterns (details in Appendix F) , since we are not able to obtain ground truth decomposition from real-world data.

Original: Forecast decompositions (without component ground truth) can be found in Appendix I. Furthermore, we report quantitative results over the test set in Table 4.
Revised: Forecast decompositions (without component ground truth) can be found in Appendix J. Furthermore, we report quantitative results over the test set in Table 4.

Original: ETSformer successfully forecasts interpretable level, trend (level + growth), and seasonal components, as observed in the trend and seasonality components closely tracking the ground truth patterns.
Revised: ETSformer successfully forecasts interpretable level, trend (level + growth), and seasonal components, as observed in the trend and seasonality components closely tracking the ground truth patterns.

Original: Despite obtaining a good combined forecast, competing decomposition based approaches, struggles to disambiguate between trend and seasonality.
Revised: Despite obtaining a good combined forecast, competing decomposition based approaches, struggles to disambiguate between trend and seasonality.

Original: Furthermore, ETSformer produces human interpretable attention weights for both the FA and ESA mechanisms, visualized in Figure 4.
Revised: Furthermore, ETSformer produces human interpretable attention weights for both the FA and ESA mechanisms, visualized in Figure 4.

Original: The FA weights visualized exhibit clear periodicity which can be used to identify the dominating seasonal patterns, while ESA weights exhibit exponentially decaying property as per the inductive biases.
Revised: The FA weights visualized exhibit clear periodicity which can be used to identify the dominating seasonal patterns, while ESA weights exhibit exponentially decaying property as per the inductive biases.

Original: This is contrasted to Autoformer’s Auto-Correlation visualization in Figure 5 which does not follow periodicity properties despite being specialized to handle seasonality.
Revised: This is contrasted to Autoformer’s Auto-Correlation visualization in Figure 5 which does not follow periodicity properties despite being specialized to handle seasonality.

Original: 3. C OMPUTATIONAL
Revised: 3. C OMPUTATIONAL

Original: [Figure]
Revised: [Figure]

Original: Figure 6 charts ETSformer’s empirical efficiency with that of competing Transformer-based approaches.
Revised: Figure 6 charts ETSformer’s empirical efficiency with that of competing Transformer-based approaches.

Original: ETSformer maintains competitive efficiency with competing quasilinear and linear complexity Transformers.
Revised: ETSformer maintains competitive efficiency with competing quasilinear and linear complexity Transformers.

Original: This is especially so when forecast horizon increasese, due to ETSformer’s unique decoder architecture which relies on its Trend Damping and Frequency Attention modules rather than relying on a cross attention mechanism.
Revised: This is especially so when forecast horizon increasese, due to ETSformer’s unique decoder architecture which relies on its Trend Damping and Frequency Attention modules rather than relying on a cross attention mechanism.

Original: Of note, while FEDformer claims linear complexity, our empirical results show that it incurs significant overhead especially in terms of runtime efficiency.
Revised: Of note, while FEDformer claims linear complexity, our empirical results show that it incurs significant overhead especially in terms of runtime efficiency.

Original: This slowdown arises from their (official) implementation still relying on the straightfor-
Revised: This slowdown arises from their (official) implementation still relying on the straightfor-

Original: [Figure]
Revised: [Figure]

Original: ward FFT operation, incurring O
Revised: ward FFT operation, incurring O

Original: ( L log L ) complexity, as well as their Frequency Enhanced Modules requiring a large number of trainable parameters.
Revised: ( L log L ) complexity, as well as their Frequency Enhanced Modules requiring a large number of trainable parameters.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Transformer-based ForecastingTransformer-based time-series forecasting models have been actively studied recently. LogTrans (Li et al., 2019)introduces local context to Transformer models via causal convolutions in the query-key projection layer, and propose the LogSparse attention to reduce  complexity to O ( L log L ) .
Revised: Deep Forecasting LogTrans (Li et al., 2019) and AST (Wu et al., 2020) first introduced Transformer based methods to reduce computational complexity of attention.

Original:  Informer (Zhou et al., 2021) extends the Transformer  by proposing the ProbSparse attention and distillation operation to achieve O ( L log L ) complexity.AST (Wu et al., 2020) leverages a sparse normalization transform, α -entmax , to implement a sparse attention layer.
Revised: The LSTF benchmark was first introduced by Informer (Zhou et al., 2021), extending the Transformer architecture by proposing the ProbSparse attention and distillation operation to achieve O ( L log L ) complexity.

Original: It further incorporates an adversarial loss to mitigate the adverse effect of error accumulation in inference.

Original: Similar to our work that incorporates prior knowledge of time-series structure, Autoformer (Wu et al., 2021) introduces the Auto-Correlation attention mechanism which focuses on sub-series based similarity and is able to extract periodic patterns.
Revised: Similar to our work that incorporates prior knowledge of time-series structure, Autoformer (Wu et al., 2021) introduces the Auto-Correlation attention mechanism which focuses on sub-series based similarity and is able to extract periodic patterns.

Original: FEDformer (Zhou et al., 2022) extends this line of work by incorporating Frequency Enhanced structures.
Revised: FEDformer (Zhou et al., 2022) extends this line of work by incorporating Frequency Enhanced structures.

Revised: N-HiTS (Challu et al., 2022) introduced hierarchical interpolation and multi-rate data sampling by building on top of N-BEATS (Oreshkin et al., 2019) for the LSTF task.

Revised: ES-RNN (Smyl, 2020) has explored combining ETS methods with neural networks.

Revised: However, they treat ETS as a pre and post processing step, rather than baking it into the model architecture.

Revised: Furthermore, their method requires prior knowledge on seasonality patterns, and they were not proposed for LSTF, leading to high computation costs over long horizons.

Original: Attention Mechanisms
Revised: Attention Mechanisms

Original: The self-attention mechanism in Transformer models has recently received much attention, its necessity has been greatly investigated in attempts to introduce more flexibility and reduce computational cost.
Revised: The self-attention mechanism in Transformer models has recently received much attention, its necessity has been greatly investigated in attempts to introduce more flexibility and reduce computational cost.

Original: Synthesizer (Tay et al., 2021) empirically studies the importance of dot-product interactions, and show that a randomly initialized, learnable attention mechanisms with or without token-token dependencies can achieve competitive performance with vanilla self-attention on various NLP tasks.
Revised: Synthesizer (Tay et al., 2021) empirically studies the importance of dot-product interactions, and show that a randomly initialized, learnable attention mechanisms with or without token-token dependencies can achieve competitive performance with vanilla self-attention on various NLP tasks.

Original: You et al.
Revised: You et al.

Original: (2020) utilizes an unparameterized Gaussian distribution to replace the original attention scores, concluding that the attention distribution should focus on a certain local window and can achieve comparable performance.
Revised: (2020) utilizes an unparameterized Gaussian distribution to replace the original attention scores, concluding that the attention distribution should focus on a certain local window and can achieve comparable performance.

Original: Raganato et al.
Revised: Raganato et al.

Original: (2020) replaces attention with fixed, non-learnable positional patterns, obtaining competitive performance on NMT tasks.
Revised: (2020) replaces attention with fixed, non-learnable positional patterns, obtaining competitive performance on NMT tasks.

Original: Lee-Thorp et al.
Revised: Lee-Thorp et al.

Original: (2021) replaces self-attention with a non-learnable Fourier Transform and verifies it to be an effective mixing mechanism.
Revised: (2021) replaces self-attention with a non-learnable Fourier Transform and verifies it to be an effective mixing mechanism.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Inspired by the classical exponential smoothing methods and emerging Transformer approaches for time-series forecasting, we propose ETSformer, a novel level-growth-season decomposition Transfomer.
Revised: Inspired by the classical exponential smoothing methods and emerging Transformer approaches for time-series forecasting, we propose ETSformer, a novel level-growth-season decomposition Transfomer.

Original: ETSformer leverages the novel Exponential Smoothing Attention and Frequency Attention mechanisms which are more effective at modeling time-series than vanilla self-attention, and at the same time achieves O
Revised: ETSformer leverages the novel Exponential Smoothing Attention and Frequency Attention mechanisms which are more effective at modeling time-series than vanilla self-attention, and at the same time achieves O

Original: ( L log L ) complexity, where L is the length of lookback window.
Revised: ( L log L ) complexity, where L is the length of lookback window.

Original: We performed extensive empirical evaluation, showing that ETSformer has extremely competitive accuracy and efficiency, while being highly interpretable.
Revised: We performed extensive empirical evaluation, showing that ETSformer has extremely competitive accuracy and efficiency, while being highly interpretable.

Original: Limitations & Future Work ETSformer currently only produces point forecasts.
Revised: Limitations & Future Work ETSformer currently only produces point forecasts.

Original: Probabilistic forecasting would be a valuable extension of our current work due to it’s importance in practical applications.
Revised: Probabilistic forecasting would be a valuable extension of our current work due to it’s importance in practical applications.

Original: Other future directions which ETSformer does not currently consider but would be useful are additional covariates such as holiday indicators and other dummy variables to consider holiday effects which cannot be captured by the FA mechanism.
Revised: Other future directions which ETSformer does not currently consider but would be useful are additional covariates such as holiday indicators and other dummy variables to consider holiday effects which cannot be captured by the FA mechanism.

Original: A C LASSICAL E XPONENTIAL S MOOTHING
Revised: A C LASSICAL E XPONENTIAL S MOOTHING

Original: We instantiate exponential smoothing methods (Hyndman et al., 2008) in the univariate forecasting setting.
Revised: We instantiate exponential smoothing methods (Hyndman et al., 2008) in the univariate forecasting setting.

Original: They assume that time-series can be decomposed into seasonal and trend components, and trend can be further decomposed into level and growth components.
Revised: They assume that time-series can be decomposed into seasonal and trend components, and trend can be further decomposed into level and growth components.

Original: Specifically, a commonly used model is the additive Holt-Winters’ method (Holt, 2004; Winters, 1960), which can be formulated as:
Revised: Specifically, a commonly used model is the additive Holt-Winters’ method (Holt, 2004; Winters, 1960), which can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where p is the period of seasonality, and ˆ x t + h | t is the h -steps ahead forecast.
Revised: where p is the period of seasonality, and ˆ x t + h | t is the h -steps ahead forecast.

Original: The above equations state that the h -steps ahead forecast is composed of the last estimated level e t , incrementing it by h times the last growth factor, b t , and adding the last available seasonal factor s t + h − p .
Revised: The above equations state that the h -steps ahead forecast is composed of the last estimated level e t , incrementing it by h times the last growth factor, b t , and adding the last available seasonal factor s t + h − p .

Original: Specifically, the level smoothing equation is formulated as a weighted average of the seasonally adjusted observation ( x t − s t − p ) and the non-seasonal forecast, obtained by summing the previous level and growth ( e t − 1 + b t − 1 ) .
Revised: Specifically, the level smoothing equation is formulated as a weighted average of the seasonally adjusted observation ( x t − s t − p ) and the non-seasonal forecast, obtained by summing the previous level and growth ( e t − 1 + b t − 1 ) .

Original: The growth smoothing equation is implemented by a weighted average between the successive difference of the (de-seasonalized) level, ( e t − e t − 1 ) , and the previous growth, b t − 1 .
Revised: The growth smoothing equation is implemented by a weighted average between the successive difference of the (de-seasonalized) level, ( e t − e t − 1 ) , and the previous growth, b t − 1 .

Original: Finally, the seasonal smoothing equation is a weighted average between the difference of observation and (de-seasonalized) level, ( x t − e t ) , and the previous seasonal index s t − p .
Revised: Finally, the seasonal smoothing equation is a weighted average between the difference of observation and (de-seasonalized) level, ( x t − e t ) , and the previous seasonal index s t − p .

Original: The weighted average of these three equations are controlled by the smoothing parameters α , β and γ , respectively.
Revised: The weighted average of these three equations are controlled by the smoothing parameters α , β and γ , respectively.

Original: A widely used modification of the additive Holt-Winters’ method is to allow the damping of trends, which has been proved to produce robust multi-step forecasts (Svetunkov, 2016; McKenzie &
Revised: A widely used modification of the additive Holt-Winters’ method is to allow the damping of trends, which has been proved to produce robust multi-step forecasts (Svetunkov, 2016; McKenzie &

Original: Gardner Jr, 2010).
Revised: Gardner Jr, 2010).

Original: The forecast with damping trend can be rewritten as:
Revised: The forecast with damping trend can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: where the growth is damped by a factor of ϕ .
Revised: where the growth is damped by a factor of ϕ .

Original: If ϕ = 1 , it degenerates to the vanilla forecast.
Revised: If ϕ = 1 , it degenerates to the vanilla forecast.

Original: For 0 < ϕ < 1 , as h → ∞ this growth component approaches an asymptote given by ϕb t / (1 − ϕ ) .
Revised: For 0 < ϕ < 1 , as h → ∞ this growth component approaches an asymptote given by ϕb t / (1 − ϕ ) .

Original: B E XPONENTIAL S MOOTHING A TTENTION B. E XPONENTIAL S MOOTHING A TTENTION M ATRIX
Revised: B E XPONENTIAL S MOOTHING A TTENTION B. E XPONENTIAL S MOOTHING A TTENTION M ATRIX

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: j . . .
Revised: j . . .

Original: α  
Revised: α  

Original: B. E FFICIENT E XPONENTIAL S MOOTHING A TTENTION A LGORITHM
Revised: B. E FFICIENT E XPONENTIAL S MOOTHING A TTENTION A LGORITHM

Original: [Table]
Revised: [Table]

Original: B. L EVEL S MOOTHING VIA E XPONENTIAL S MOOTHING A TTENTION
Revised: B. L EVEL S MOOTHING VIA E XPONENTIAL S MOOTHING A TTENTION

Original: [Equation]
Revised: [Equation]

Original: Based on the above expansion of the level equation
Revised: Based on the above expansion of the level equation

Original: [Equation]
Revised: [Equation]

Original: we observe that E ( t ) n can be computed by a sum of two terms, the first of which is given by an A ES term
Revised: we observe that E ( t ) n can be computed by a sum of two terms, the first of which is given by an A ES term

Original: [Equation]
Revised: [Equation]

Original: and we finally, we note that the second term can also be calculated using the conv1d fft algorithm, resulting in a fast implementation of level smoothing.
Revised: and we finally, we note that the second term can also be calculated using the conv1d fft algorithm, resulting in a fast implementation of level smoothing.

Original: B. F URTHER D ETAILS ON ESA I MPLEMENTATION Algorithm
Revised: B. F URTHER D ETAILS ON ESA I MPLEMENTATION Algorithm

Original: A ES
Revised: A ES

Original: mm : matrix multiplication, outer : outer product repeat : einops style tensor operations, gather : gathers values along an axis specified by dim # V: value matrix, shape: L x d # v0: initial state, shape: d # alpha: smoothing parameter, shape:
Revised: mm : matrix multiplication, outer : outer product repeat : einops style tensor operations, gather : gathers values along an axis specified by dim # V: value matrix, shape: L x d # v0: initial state, shape: d # alpha: smoothing parameter, shape:

Original: L, d = V.shape # obtain exponentially decaying weights powers = arange(L) # L weight =
Revised: L, d = V.shape # obtain exponentially decaying weights powers = arange(L) # L weight =

Original: alpha ∗ (1 − alpha).pow(flip(powers))
Revised: alpha ∗ (1 − alpha).pow(flip(powers))

Original: # L # perform a strided roll operation # rolls a matrix along the columns in a strided manner # i.e. first row is shifted right by L − 1 positions, # second row is shifted L − 2, ..., last row is shifted by 0.
Revised: # L # perform a strided roll operation # rolls a matrix along the columns in a strided manner # i.e. first row is shifted right by L − 1 positions, # second row is shifted L − 2, ..., last row is shifted by 0.

Original: weight =
Revised: weight =

Original: repeat(weight, ' L − > T L ' , T=L) # L x L indices = repeat(arange(L), ' L − > T L ' , T=L) indices =
Revised: repeat(weight, ' L − > T L ' , T=L) # L x L indices = repeat(arange(L), ' L − > T L ' , T=L) indices =

Original: (indices − (arange(L) + 1).unsqueeze(1) )
Revised: (indices − (arange(L) + 1).unsqueeze(1) )

Original: % L weight =
Revised: % L weight =

Original: gather(weight, dim=−1, index=indices) # triangle masking to achieve the exponential smoothing attention matrix weight = triangle causal mask(weight) output = mm(weight, V) init weight =
Revised: gather(weight, dim=−1, index=indices) # triangle masking to achieve the exponential smoothing attention matrix weight = triangle causal mask(weight) output = mm(weight, V) init weight =

Original: (1 − alpha)
Revised: (1 − alpha)

Original: ∗∗ (powers + 1) init output = outer(init weight, v0) return init output + output
Revised: ∗∗ (powers + 1) init output = outer(init weight, v0) return init output + output

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Algorithm 2 describes the naive implementation for ESA by first constructing the exponential smoothing attention matrix, A ES , and performing the full matrix-vector multiplication.
Revised: Algorithm 2 describes the naive implementation for ESA by first constructing the exponential smoothing attention matrix, A ES , and performing the full matrix-vector multiplication.

Original: Efficient
Revised: Efficient

Original: A ES relies on Algorithm 3, to achieve an O ( L log L ) complexity, by speeding up the matrix-vector multiplication.
Revised: A ES relies on Algorithm 3, to achieve an O ( L log L ) complexity, by speeding up the matrix-vector multiplication.

Original: Due to the structure lower triangular structure of A ES (ignoring the first column), we note that performing a matrix-vector multiplication with it is equivalent to performing a convolution with the last row.
Revised: Due to the structure lower triangular structure of A ES (ignoring the first column), we note that performing a matrix-vector multiplication with it is equivalent to performing a convolution with the last row.

Original: Algorithm 3 describes the pseudocode for fast convolutions using fast Fourier transforms.
Revised: Algorithm 3 describes the pseudocode for fast convolutions using fast Fourier transforms.

Original: C D ISCRETE F OURIER T RANSFORM
Revised: C D ISCRETE F OURIER T RANSFORM

Original: The DFT of a sequence with regular intervals, x = ( x 0 , x 1 , . . .
Revised: The DFT of a sequence with regular intervals, x = ( x 0 , x 1 , . . .

Original: , x N − 1 ) is a sequence of complex numbers,
Revised: , x N − 1 ) is a sequence of complex numbers,

Original: [Equation]
Revised: [Equation]

Original: for k = 0 , 1 , . . .
Revised: for k = 0 , 1 , . . .

Original: , N − 1 , where c k are known as the Fourier coefficients of their respective Fourier frequencies.
Revised: , N − 1 , where c k are known as the Fourier coefficients of their respective Fourier frequencies.

Original: Due to the conjugate symmetry of DFT for real-valued signals, we simply consider the first ⌊ N/ 2 ⌋ + 1 Fourier coefficients and thus we denote the DFT as F :
Revised: Due to the conjugate symmetry of DFT for real-valued signals, we simply consider the first ⌊ N/ 2 ⌋ + 1 Fourier coefficients and thus we denote the DFT as F :

Original: R N → C ⌊ N/ 2 ⌋ +1 .
Revised: R N → C ⌊ N/ 2 ⌋ +1 .

Original: The DFT maps a signal to the frequency domain, where each Fourier coefficient can be uniquely represented by the amplitude, | c k | , and the phase, ϕ ( c k ) ,
Revised: The DFT maps a signal to the frequency domain, where each Fourier coefficient can be uniquely represented by the amplitude, | c k | , and the phase, ϕ ( c k ) ,

Original: [Equation]
Revised: [Equation]

Original: where R { c k } and I { c k } are the real and imaginary components of c k respectively.
Revised: where R { c k } and I { c k } are the real and imaginary components of c k respectively.

Original: Finally, the inverse DFT maps the frequency domain representation back to the time domain,
Revised: Finally, the inverse DFT maps the frequency domain representation back to the time domain,

Original: [Equation]
Revised: [Equation]

Original: D I MPLEMENTATION D ETAILS D. H YPERPARAMETERS
Revised: D I MPLEMENTATION D ETAILS D. H YPERPARAMETERS

Original: For all experiments, we use the same hyperparameters for the encoder layers, decoder stacks, model dimensions, feedforward layer dimensions, number of heads in multi-head exponential smoothing attention, and kernel size for input embedding as listed in Table 5.
Revised: For all experiments, we use the same hyperparameters for the encoder layers, decoder stacks, model dimensions, feedforward layer dimensions, number of heads in multi-head exponential smoothing attention, and kernel size for input embedding as listed in Table 5.

Original: We perform hyperparameter tuning via a grid search over the number of frequencies K , lookback window size, and learning rate, selecting the settings which perform the best on the validation set based on MSE (on results averaged over three runs).
Revised: We perform hyperparameter tuning via a grid search over the number of frequencies K , lookback window size, and learning rate, selecting the settings which perform the best on the validation set based on MSE (on results averaged over three runs).

Original: The search range is reported in Table 5, where the lookback window size search range was decided to be set as the values for the horizon sizes for the respective datasets.
Revised: The search range is reported in Table 5, where the lookback window size search range was decided to be set as the values for the horizon sizes for the respective datasets.

Original: [Table]
Revised: [Table]

Original: Learning rate
Revised: Learning rate

Original: [Table]
Revised: [Table]

Original: D. O PTIMIZATION
Revised: D. O PTIMIZATION

Original: We use the Adam optimizer (Kingma & Ba, 2015) with β 1 = 0 .
Revised: We use the Adam optimizer (Kingma & Ba, 2015) with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , and ϵ = 1 e − 08 , and a batch size of 32.
Revised: 999 , and ϵ = 1 e − 08 , and a batch size of 32.

Original: We schedule the learning rate with linear warmup over 3 epochs, and cosine annealing thereafter for a total of 15 training epochs for all datasets.
Revised: We schedule the learning rate with linear warmup over 3 epochs, and cosine annealing thereafter for a total of 15 training epochs for all datasets.

Original: The minimum learning rate is set to 1e-30.
Revised: The minimum learning rate is set to 1e-30.

Original: For smoothing and damping parameters, we set the learning rate to be 100 times larger and do not use learning rate scheduling.
Revised: For smoothing and damping parameters, we set the learning rate to be 100 times larger and do not use learning rate scheduling.

Original: Training was done on an Nvidia A100 GPU.
Revised: Training was done on an Nvidia A100 GPU.

Original: D. R EGULARIZATION
Revised: D. R EGULARIZATION

Original: We apply two forms of regularization during the training phase.
Revised: We apply two forms of regularization during the training phase.

Original: Dropout We apply dropout (Srivastava et al., 2014) with a rate of p = 0 .
Revised: Dropout We apply dropout (Srivastava et al., 2014) with a rate of p = 0 .

Original: 2 across the model.
Revised: 2 across the model.

Original: Dropout is applied on the outputs of the Input Embedding, Frequency Self-Attention and Multi-Head
Revised: Dropout is applied on the outputs of the Input Embedding, Frequency Self-Attention and Multi-Head

Original: ES Attention blocks, in the Feedforward block (after activation and before normalization), on the attention weights, as well as damping weights.
Revised: ES Attention blocks, in the Feedforward block (after activation and before normalization), on the attention weights, as well as damping weights.

Original: Noise Injection We utilize a composition of three noise distributions, applied in the following order - scale, shift, and jitter, activating with a probability of 0.5.
Revised: Noise Injection We utilize a composition of three noise distributions, applied in the following order - scale, shift, and jitter, activating with a probability of 0.5.

Original: Scale – The time-series is scaled by a single random scalar value, obtained by sampling ϵ ∼ N (0 , 0 . 2) , and each time step is ˜ x t = ϵx t . 2.
Revised: Scale – The time-series is scaled by a single random scalar value, obtained by sampling ϵ ∼ N (0 , 0 . 2) , and each time step is ˜ x t = ϵx t . 2.

Original: Shift – The time-series is shifted by a single random scalar value, obtained by sampling ϵ ∼ N (0 , 0 . 2) and each time step is ˜ x t = x t + ϵ . 3.
Revised: Shift – The time-series is shifted by a single random scalar value, obtained by sampling ϵ ∼ N (0 , 0 . 2) and each time step is ˜ x t = x t + ϵ . 3.

Original: Jitter – I.I.D. Gaussian noise is added to each time step, from a distribution ϵ t ∼ N (0 , 0 . 2) , where each time step is now ˜ x t = x t + ϵ t .
Revised: Jitter – I.I.D. Gaussian noise is added to each time step, from a distribution ϵ t ∼ N (0 , 0 . 2) , where each time step is now ˜ x t = x t + ϵ t .

Original: E D ATASETS
Revised: E D ATASETS

Original: ETT 1 Electricity Transformer Temperature (Zhou et al., 2021) is a multivariate time-series dataset, comprising of load and oil temperature data recorded every 15 minutes from electricity transformers.
Revised: ETT 1 Electricity Transformer Temperature (Zhou et al., 2021) is a multivariate time-series dataset, comprising of load and oil temperature data recorded every 15 minutes from electricity transformers.

Original: ETT consists of two variants, ETTm and ETTh, whereby ETTh is the hourly-aggregated version of
Revised: ETT consists of two variants, ETTm and ETTh, whereby ETTh is the hourly-aggregated version of

Original: ETTm, the original 15 minute level dataset.
Revised: ETTm, the original 15 minute level dataset.

Original: ECL 2 Electricity Consuming Load measures the electricity consumption of 321 households clients over two years, the original dataset was collected at the 15 minute level, but is pre-processed into an hourly level dataset.
Revised: ECL 2 Electricity Consuming Load measures the electricity consumption of 321 households clients over two years, the original dataset was collected at the 15 minute level, but is pre-processed into an hourly level dataset.

Original: Exchange 3 Exchange (Lai et al., 2018) tracks the daily exchange rates of eight countries (Australia,
Revised: Exchange 3 Exchange (Lai et al., 2018) tracks the daily exchange rates of eight countries (Australia,

Original: United Kingdom, Canada, Switzerland, China, Japan, New Zealand, and Singapore) from 1990 to 2016.
Revised: United Kingdom, Canada, Switzerland, China, Japan, New Zealand, and Singapore) from 1990 to 2016.

Original: Traffic 4 Traffic is an hourly dataset from the California Department of Transportation describing road occupancy rates in San Francisco Bay area freeways.
Revised: Traffic 4 Traffic is an hourly dataset from the California Department of Transportation describing road occupancy rates in San Francisco Bay area freeways.

Original: Weather 

Revised: Weather

Original: Weather measures 21 meteorological indicators like air temperature, humidity, etc., every 10 minutes for the year of 2020.
Revised: Weather measures 21 meteorological indicators like air temperature, humidity, etc., every 10 minutes for the year of 2020.

Original: ILI 6 Influenza-like Illness records the ratio of patients seen with ILI and the total number of patients on a weekly basis, obtained by the Centers for Disease Control and Prevention of the United States between 2002 and 2021.
Revised: ILI 6 Influenza-like Illness records the ratio of patients seen with ILI and the total number of patients on a weekly basis, obtained by the Centers for Disease Control and Prevention of the United States between 2002 and 2021.

Original: F S YNTHETIC D ATASET
Revised: F S YNTHETIC D ATASET

Original: The synthetic dataset is constructed by a combination of trend and seasonal component.
Revised: The synthetic dataset is constructed by a combination of trend and seasonal component.

Original: Each instance in the dataset has a lookack window length of 720 and forecast horizon length of 192.
Revised: Each instance in the dataset has a lookack window length of 720 and forecast horizon length of 192.

Original: The trend pattern follows a nonlinear, saturating pattern, b ( t )
Revised: The trend pattern follows a nonlinear, saturating pattern, b ( t )

Original: =
Revised: =

Original: 1 1+exp β 0 ( t − β 1 ) , where β 0 = − 0 .
Revised: 1 1+exp β 0 ( t − β 1 ) , where β 0 = − 0 .

Original: 2 , β 1 = 720 .
Revised: 2 , β 1 = 720 .

Original: The seasonal pattern follows a complex periodic pattern formed by a sum of sinusoids.
Revised: The seasonal pattern follows a complex periodic pattern formed by a sum of sinusoids.

Original: Concretely, s ( t ) =
Revised: Concretely, s ( t ) =

Original: A 1 cos(2 πf 1 t )
Revised: A 1 cos(2 πf 1 t )

Original: + A 2 cos(2 πf 2 t , where f 1 = 1 / 10 , f 2 = 1 / 13 are the frequencies, A 1 =
Revised: + A 2 cos(2 πf 2 t , where f 1 = 1 / 10 , f 2 = 1 / 13 are the frequencies, A 1 =

Original: A 2 = 0 .
Revised: A 2 = 0 .

Original: 15 are the amplitudes.
Revised: 15 are the amplitudes.

Original: During training phase, we use an additional noise component by adding i.i.d.
Revised: During training phase, we use an additional noise component by adding i.i.d.

Original: gaussian noise with 0.05 standard deviation.
Revised: gaussian noise with 0.05 standard deviation.

Original: Finally, the i -th instance of the dataset is x i =
Revised: Finally, the i -th instance of the dataset is x i =

Original: [ x i (1) , x i (2) , . . .
Revised: [ x i (1) , x i (2) , . . .

Original: , x i (720 + 192)] , where x i ( t ) =
Revised: , x i (720 + 192)] , where x i ( t ) =

Original: b ( t ) + s ( t + i ) .
Revised: b ( t ) + s ( t + i ) .

Original: G U NIVARIATE F ORECASTING B ENCHMARK
Revised: G U NIVARIATE F ORECASTING B ENCHMARK

Original: [Table]
Revised: [Table]

Original: H ETS FORMER S TANDARD D EVIATION
Revised: H ETS FORMER S TANDARD D EVIATION

Original: [Table]
Revised: [Table]

Original: ETT m 0.189 (0.002)
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: E C L 0.187 (0.001)

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: E x c h a ng e 0.085 (0.000) 0.204 (

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: T r a f fi c 0.607 (0.005) 0.392 (0.

Original: [Table]
Revised: [Table]

Original: W ea t h e r 0.197 (0.007) 0.281 (0.008) 0.237 (0.005) 0.312 (0.004) 0.298 (0.003) 0.353 (0.003) 0.352 (0.007) 0.388 (0.002)
Revised: W ea t h e r 0.197 (0.007) 0.281 (0.008) 0.237 (0.005) 0.312 (0.004) 0.298 (0.003) 0.353 (0.003) 0.352 (0.007) 0.388 (0.002)

Original: I L I 2.527 (0.061) 1.020 (0.021) 2.615 (0.103) 1.007 (0.013) 2.359 (0.056) 0.972 (0.011) 2.487 (0.006) 1.016 (0.007)
Revised: I L I 2.527 (0.061) 1.020 (0.021) 2.615 (0.103) 1.007 (0.013) 2.359 (0.056) 0.972 (0.011) 2.487 (0.006) 1.016 (0.007)

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: MSE (SD
Revised: MSE (SD

Original: [Table]
Revised: [Table]

Original: MAE (SD
Revised: MAE (SD)

Original: [Table]

Original: ETT m 0.080 (0.001) 0.212 (0.001) 0.150 (0.024) 0.302 (0.026) 0.175 (0.012) 0.334 (0.014) 0.224 (0.008) 0.379 (0.006)
Revised: ETT m 0.080 (0.001) 0.212 (0.001) 0.150 (0.024) 0.302 (0.026) 0.175 (0.012) 0.334 (0.014) 0.224 (0.008) 0.379 (0.006)

Original: E x c h a ng e 0.099 (0.003) 0.230 (0.003) 0.223 (0.015) 0.353 (0.009) 0.421 (0.002) 0.497 (0.000) 1.114 (0.049) 0.807 (0.016)
Revised: E x c h a ng e 0.099 (0.003) 0.230 (0.003) 0.223 (0.015) 0.353 (0.009) 0.421 (0.002) 0.497 (0.000) 1.114 (0.049) 0.807 (0.016)

Original: I R EAL - WORLD D ECOMPOSED F ORECASTS
Revised: I L

Original: ETTh1Ground Truth
Revised: AYER A NALYSIS

Original: ForecastTrendSeason    80 1.5 1.
Revised: [Table] We provide additional analysis on the number of layers, and also ablations on the observation space (meaning that there is no projection into representation space by removing the embedding layer). We observe that learning deep representations lead to a significant increase in performance, and the optimal number of layers is around 2 3, before overfitting occurs.

Original: ECLGround Truth ForecastTrendSeason    80 1.0 0.
Revised: J R EAL - WORLD D ECOMPOSED F ORECASTS

Original: [Figure]
Revised: [Figure]

Revised: [Figure]