R EVERSIBLE I NSTANCE N ORMALIZATION FOR A CCURATE T IME -S ERIES F ORECASTING AGAINST D ISTRIBUTION S HIFT
Anonymous authors
Paper under double-blind review
[Figure]
day
A BSTRACT
The statistical properties in a time-series often change, i.e., the data distributions differ over time.
This temporal distribution change is one of the main challenges that prevent accurate time-series forecasting.
To address this issue, we propose a simple but effective normalization method called reversible instance normalization (RevIN).
To be speciﬁc, RevIN consists of two different steps, normalization and denormalization.
The former normalizes the input to ﬁx its distribution in terms of the mean and variance, while the latter returns the output to the original distribution.
In addition, RevIN is model-agnostic, generally applicable to various time-series forecasting models with signiﬁcant improvements in forecasting performance.
As shown in Fig.
1, RevIN effectively enhances the performance of the baselines.
Our extensive experimental results verify the general applicability and performance improvements on various real-world datasets.
I NTRODUCTION
Time-series forecasting plays a signiﬁcant role in various daily problems such as health-care, economics, and trafﬁc engineering (Che et al., 2018; Bauer et al., 2016; Zhang et al., 2017).
Recently, time-series forecasting models have achieved comparative performance on these problems by overcoming several challenges, e.g., long-term forecasting (Zhou et al., 2021) and missing value im- putation
(Tang et al., 2020).
However, these time-series forecasting models often severely suffer from statistical properties that change over time, which is also widely known as a distribution shift problem; for example, the mean and variance of the time-series data distribution vary over time.
Accordingly, the input sequences to the forecasting models have different underlying distributions from each other.
In the case of the train and test data, they are usually divided into data before and after a speciﬁc point in time.
Thus, the distributions of the two data often hardly overlap, which is commonly known as the reason for model performance degradation.
Thus, we assume that if we normalize every sequence in the data to have identical mean and variance and provide the transformed data to the model, the discrepancy in the distribution will be reduced, thereby improving the model performance.
However, if the input to the model is shifted and scaled, the prediction of the model can be shifted and scaled as much as the input; since this will differ from the groundtruth distribution, the accuracy of the model will decrease.
Also, we further hypothesize that if the model output, shifted and scaled, can be returned to its original distribution explicitly, we can alleviate the discrepancy in the distribution of the data during the forward pass of the model as well as allow its prediction to obey the original distribution.
Here, reconstructing the original distribution can be achieved by applying denormalization on the output of the model as a reversed function of the normalization applied on the input data.
Inspired by these, we propose a simple yet effective normalization method, reversible instance normalization (RevIN) , which ﬁrst normalizes the input sequences and then denormalizes the model output sequences to solve the time-series forecasting task against the distribution shift problem.
RevIN is symmetrically structured to return the model output to the original distribution in the denormalization layer by scaling and shifting the model output as much as the input data is shifted and scaled in the normalization layer.
Adding only the normalization and denormalization layers on the input and output of the model, respectively, RevIN can be applied to any model without modiﬁcation to the model architecture.
We empirically found that RevIN effectively alleviates the distribution discrepancy within the data, as well as signiﬁcantly improving the model performance when applied to various state-of-the-art time-series forecasting methods.
To verify the effectiveness of RevIN, we conduct extensive quantitative evaluations with various baselines:
Informer (Zhou et al., 2021), N-BEATS (Oreshkin et al., 2020), and SCINet (Liu et al., 2021).
We also present an in-depth analysis of the behavior of the model, including the veriﬁcation of the assumptions on reversible instance normalization (See Section 3.2).
RevIN is a ﬂexible, end-to-end trainable layer that can be applied to any arbitrarily chosen layers, effectively suppressing non-stationary information (mean and standard deviation of the instance) from one layer and restoring it on the other layer at a virtually symmetric position.
Despite its remarkable performance, there has been no work on generalizing and expanding the instance-wise normalization-and-denormalization as a ﬂexibly applicable, trainable layer in the time-series domain.
Recently, deep learning-based time-series forecasting approaches, such as Informer (Zhou et al., 2021) and N-BEATS (Oreshkin et al., 2020), have shown outstanding performance in timeseries forecasting.
However, they overlooked the importance of normalization, merely using simple global preprocessing to the model input without further exploration, and expecting their endto-end deep learning model to replace the role.
Despite the simplicity of our method, there have been no cases of using such techniques in modern deep-learning-based time-series forecasting approaches (Zhou et al., 2021; Liu et al., 2021; Oreshkin et al., 2020).
In this sense, we enlight the importance of the appropriate normalization method in deep-learning-based time-series approaches; we propose a carefully designed, deep-learning-friendly module, for time-series forecasting, by combining the learnable afﬁne transformation with the method, which has been widely accepted in recent deep-learning-based normalization work (Ulyanov et al., 2016).
In summary, our contributions are as follows:
• We propose a simple and effective normalization-and-denormalization method for timeseries, called RevIN, symmetrically structured to remove and restore the statistical information of a time-series instance; it is a generally-applicable layer to arbitrary deep neural networks with negligible cost and signiﬁcantly improves the performance by reducing the distribution discrepancy within the data.
• By adding RevIN to the baseline, we achieve the state-of-the-art performance on four largescale real-world datasets with a signiﬁcant margin.
• We conduct extensive evaluations on RevIN with quantitative analysis and qualitative visualizations to verify its effectiveness, addressing the distribution shift problem.
R ELATED W ORK
Time-series forecasting.
Time-series forecasting methods are mainly categorized into three distinct approaches: (1) statistical methods, (2) hybrid methods, and (3) deep learning-based methods.
As an example of the statistical models, exponential smoothing forecasting (Holt, 2004; Winters, 1960) is a well-established benchmark to predict future values.
Statistical models have several advantages, e.g., interpretability and theoretically well guaranteed.
In order to boost the performance, recent work proposed a hybrid model (Smyl, 2020) which incorporates a deep learning module with a statistical model.
It achieved better performance over the statistical methods in the M4 timeseries forecasting competition.
The deep learning-based method basically follows the sequenceto-sequence framework to model the time-series forecasting.
Initially, deep learning-based models utilized variations of recurrent neural networks (RNNs).
However, to overcome the limitation of the limited receptive ﬁeld, several studies utilize advanced techniques, such as the dilatation and attention module.
For instance, SCINet (Liu et al., 2021) and Informer (Zhou et al., 2021) modiﬁed the sequence-to-sequence-based model to improve the performance for long sequences.
However, most previous deep learning-based models are hard to interpret compared to statistical models.
Thus, inspired by statistical models, N-BEATS (Oreshkin et al., 2020) designed an interpretable layer for time-series forecasting by encouraging the model to explicitly learn trend, seasonality, and residual components.
This model shows superior performance on M4 competition datasets.
Distribution shift.
Although there are various models for time-series forecasting, they often suffer from non-stationary time-series, where the data distribution changes over time.
Domain adaptation (DA) (Tzeng et al., 2017; Ganin et al., 2016; Wang et al., 2018) and domain generalization (DG) (Wang et al., 2021; Li et al., 2018; Muandet et al., 2013) are common solutions for alleviating the distribution shift.
A domain adaptation algorithm attempts to reduce the distribution gap between source and target domains.
A domain generalization algorithm only relies on the source domain and hopes to generalize on the target domain.
Both DA and DG have a common objective that bridges the gap between source and target distributions.
However, in non-stationary time-series, deﬁning a domain is not as straightforward since the data distribution shifts over time.
Recently, Du et al.
(Du et al., 2021) proposed Adaptive RNNs (AdaRNNs) to handle the distribution shift problems for non-stationary time-series data.
It ﬁrst characterizes the distribution information by splitting the training data into periods.
Then, it matches distributions of the discovered periods to generalize the model.
However, unlike AdaRNNs that are costly expensive, RevIN is simple yet effective and model-agnostic since the method can be adopted easily to any deep-learning model.
P ROPOSED M ETHOD
In this section, we propose reversible instance normalization (RevIN) for alleviating the distribution shift problem in time-series, which causes the discrepancy among the input sequences of the training data, as well as the discrepancy between train and test data distributions.
Section 3.1 describes RevIN in detail and Section 3.2 discusses how RevIN alleviates the distribution discrepancy in time-series throughout qualitative visualization.
3. R EVERSIBLE I NSTANCE N ORMALIZATION
We consider the multivariate time-series forecasting task in discrete time, with a sliding window size of T x for input data.
Here, N, K, T x , and T y indicate the number of the sequences, the number of variables, the input sequence length, and the prediction length.
Let X
= { x ( i ) } Ni =1 and
Y =
{ y ( i ) } Ni =1 denote the input data and corresponding target, respectively.
In RevIN, the input sequence length T x and the prediction length T y can be different since the observations are normalized and denormalized across the temporal dimension, as will be explained below.
Given an input sequence x ( i ) ∈ R K × T x , we aim to solve the time-series forecasting problem, which is to predict the subsequent values y ( i
) ∈ R K × T y .
Our proposed method, RevIN, symmetrically transforms the input data x
( i ) and the prediction output y
( i ) of the network, as illustrated in Fig.
First, we normalize the input data x ( i ) with the instance-speciﬁc mean and standard deviation, which is widely accepted as instance normalization (Ulyanov et al., 2016).
The mean and standard deviation are computed for
0(b - 2) (b - 1) (a - 3)  0(b - 4) (b - 3)
[Figure]
RevIN 𝝁𝝁 , 𝝈𝝈 , 𝜷𝜷 , 𝜸𝜸
RevIN (a - 2) 𝜃𝜃
[Figure]
every instance x ( i ) k · of the input data (Fig. 2(a-3)) as
[Equation]
and
[Equation]
Using these statistics, we normalize the input data x ( i ) as (Fig. 2(a-1))
[Equation]
where γ, β ∈ R K are learnable afﬁne parameter vectors.
The normalized sequences have the same mean and variance, where the non-stationary information is reduced.
As a result, the denormalization layer allows the model to accurately predict the local dynamics within the sequence while receiving inputs of consistent distributions in terms of the mean and variance.
The model then receives the transformed data ˆ x ( i ) as input and forecasts their future values.
However, the input data have different statistics from the original distribution, and only observing the normalized input ˆ x ( i ) , it is difﬁcult to capture the original distribution of the input x ( i ) .
Thus, to make this easier for the model, we explicitly restore the non-stationary properties removed from the input data to the model output by reversing the normalization step (Fig. 2(a-3)).
Such denormalization step can return the model output to the original time series value as well (Ogasawara et al., 2010).
Accordingly, we denormalize the model output y
( i ) by applying the reciprocal of the normalization in Eq.
2 (Fig. 2(a-2))
as
[Equation]
[Figure]
E TT h
E TT m
E TT h
D e n s i t y
D e n s i t y D e n s i t y
[Figure]
where the scaling and shifting variables are the exact instance statistics computed from input data x ( i ) .
Now, ˆ y ( i ) is the ﬁnal prediction of the model instead of y ( i ) .
Simply added to the input and output of a network, RevIN can be easily applied to any network with negligible cost while effectively improving time-series forecasting performance.
Additionally, each step of RevIN, normalization and denormalization, can be applied to any activations of the intermediate layers in the network.
However, our method is most effective when applied to the layers of symmetric encoder-decoder structure.
In a typical time-series forecasting model, the boundary between the encoder and the decoder is often unclear.
Therefore, we apply normalization and denormalization to the input and output of a forecasting network, which can be interpreted as an encoderdecoder structure, generating subsequent values given input data.
Additional analysis on selecting layers to apply RevIN is provided in Fig.
3. E FFECT OF R EVERSIBLE I NSTANCE N ORMALIZATION ON D ISTRIBUTION S HIFT
Here, we verify that our method indeed alleviates the discrepancy in distributions during the forward pass of the network, and then reconstructs it in the network output layer through the denormalization step.
We analyze the distributions of the train and test data before and after each step of our method, i.e., the normalization on input and denormalization on output, as shown in Fig.
As a result, we observe that RevIN signiﬁcantly reduces the discrepancy between the train and test data distributions in the normalization step (Fig. 3(b)) compared to the original data (Fig. 3(a)).
In
Fig. 3(a), the distributions of the train and test data hardly overlap, especially in the ETTh 2 dataset, which is caused by the distribution shift problem.
Also, the distributions have multiple peaks, which can suggest that the sequences in the data have different distributions from each other.
However, in our method, the normalization step transforms each data distribution into mean-centered distributions.
Thus, it can be said that the multimodal distribution in the input data ((Fig. 3(a)) is caused by the different distributions in sequences.
Also, the train and test data distributions become similar to each other.
It demonstrates that the normalization step of RevIN can alleviate the distribution shift problem, reducing the distribution discrepancy between train and test datasets.
Moreover, the baseline model, where RevIN is applied, still maintains the similarly-shaped distributions in the prediction output (Fig. 3(c)).
As expected, these are then returned back to the original distribution by the denormalization step of RevIN (Fig. 3(d)).
Without denormalization, the model needs to re- construct the values that follow the original distributions (Fig. 3(d)) only using the input with the transformed distributions (Fig. 3(c)).
Additionally, we hypothesize that the distribution discrepancy is also reduced in the intermediate layers of the model when RevIN is applied at the input and output layers; RevIN does not need to be applied on the intermediate layers, which will be discussed in Section 4.2.3.
As a result, this procedure of RevIN can be considered as making problems easier ﬁrst and then turning them back rather than directly solving the challenging problem where the distribution shift problem exists.
E XPERIMENTS
This section describes the experimental setup and presents extensive experiments on RevIN.
4. E XPERIMENTAL S ETUP
Datasets.
We evaluate our methods on two large-scale real-world time-series datasets.
(i) Electricity transformer temperature (ETT) 1 is electrical transformer temperature data collected from two different counties in China for two years.
The data consists of seven features: oil temperature and six power load features.
Following the same protocol of Informer (Zhou et al., 2021), we split the dataset into three datasets: ETTh 1 , ETTh 2 , and ETTm 1 .
ETTh 1 and ETTh 2 data are measured every hour at different locations.
ETTm 1 data are measured every 15 minutes.
For all three datasets, we use the ﬁrst 12 months, the middle four months, and the last four months as training, validation, and test data, respectively.
(ii) Electricity Counting Load (ECL) 2 data contains the electricity counting load collected from 321 clients, and the data measured from each client is used as a feature value.
We preprocess the data to have an hourly basis following (Zhou et al., 2021).
For the ECL dataset, we use 15, 3, and 4 months as train, validation, and test data, respectively.
Experimental details.
We set prediction window lengths as 1 day (1d), 2d, 7d, 14d, 30d, and 40d for the datasets measured on an hourly basis, ETTh 1 , ETTh 2 , and ECL.
For the ETTm 1 dataset, which is measured every 15 minutes, we chose 6 hours (6h), 12h, 3d, 7d, and 14d as prediction window lengths.
We evaluate the time-series forecasting performance in terms of mean squared error (MSE) and mean absolute error (MAE).
Following the same evaluation procedure of the previous study (Zhou et al., 2021), we compute the MSE and MAE on normalized data, since the values are not on the same scale.
More details on experimental settings, including training details and hyperparameters, are provided in Appendix A.11.
Compared baselines.
RevIN is a model-agnostic method that can be easily added to any network with further improvements.
In this paper, we evaluate RevIN by adopting it on three time-series forecasting models:
Informer (Zhou et al., 2021), N-BEATS (Oreshkin et al., 2020), and SCINet (Liu et al., 2021).
All the models forecast the time-series sequence non-autoregressively.
The exact training details of the models are reported in Section A.12.
4. R ESULTS AND A NALYSIS
This section presents quantitative analyses and qualitative visualizations of RevIN when adopted to the state-of-the-art time-series forecasting model.
4.2. E FFECTIVENESS OF R EVERSIBLE I NSTANCE N ORMALIZATION ON V ARIOUS T IME -S ERIES F ORECASTING M ODELS
Table 1 demonstrates how much the proposed method improves the forecasting accuracy of the baseline models.
We compare the MSE and MAE between RevIN and the baseline models, including Informer, N-BEATS, and SCINet, on ETT and ECL datasets with varying prediction lengths.
RevIN and the baselines with the same hyperparameter settings, including the input and prediction lengths, are compared.
We report the average value of the errors for ﬁve runs.
Table 1 provides the average errors for ﬁve runs.
Additionally, we report the standard deviation of all results in Appendix A.14 When adopted to various baselines, RevIN consistently outperforms the baselines with
[Table]
ETT m 0.
[Table] [Table]
[Table] [Table]
[Table]
ETTh 1 for ﬁve times and the average errors of ﬁve runs are reported.
Adding RevIN to the baseline consistently decreases both MSE and MAE.
Prediction length
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Informer 0.
+ RevIN 0.
N-BEATS 0.
+ RevIN 0.
SCINet 0.
+ RevIN 0.
a large margin, achieving state-of-the-art performance.
Moreover, RevIN shows signiﬁcantly better results than baselines when forecasting long sequences that are difﬁcult to predict.
Speciﬁcally, for
N-BEATS, the performance of the baseline on ETTh 2 dataset degrades signiﬁcantly from 0.403 (at 24) to 6.408 (at 960) with increasing prediction length, while RevIN performs well consistently, showing the MSE of 0.192 for short sequence prediction (at 24) and 0.471 for long sequence prediction (at 960).
Such tendency appears similarly on other datasets and models.
The results indicate that RevIN is a model-agnostic method that can be easily applied to existing models and improve the model performance.
We further analyze the effect of RevIN on the long prediction sequence quantitatively.
Table 2 shows an error of forecasting long future sequences with the same input sequence length.
RevIN consis-
[Table]
tently outperforms the baselines in forecasting long sequences using relatively short input sequences with a length of 48.
Although the difference in the performance of SCINet and RevIN is not large when predicting short sequences (e.g., 48), RevIN surpasses SCINet with a signiﬁcant margin when predicting long sequences (e.g., 336, 720, and 960).
These results show that our method works robustly to the prediction length.
To further study why RevIN is robust to the prediction length, we analyze the forecasting error for each time step 3 .
Fig.
4 provides the results on ETTh 2 dataset for the baseline and RevIN using
Informer, N-BEATS, and SCINet.
For N-BEATS and SCINET, the error gradually increases for predicting the distant future.
RevIN alleviates such a signiﬁcant increase of error, surpassing them with a large margin.
For Informer, the error is relatively constant compared to the other models regardless of the prediction time step.
However, the average error is still large compared to RevIN.
The results show the effectiveness of RevIN when forecasting long sequences.
4.2. C OMPARISON WITH O THER N ORMALIZATION M ETHODS
We compare RevIN with classical and state-of-the-art normalization methods, including minmax normalization, z-score normalization, layer normalization (Ba et al., 2016), batch normalization (Ioffe & Szegedy, 2015), instance normalization (Ulyanov et al., 2016), and deep adaptive input normalization (DAIN) (Passalis et al., 2019) as shown in Table 3.
The normalization parameters for min-max and z-score normalization methods are calculated for each input instance.
Additionally, we use the batch normalization with the denormalization step of RevIN (reversible batch normaliza-
[Table]
+ + f e a t r u e d
i v e r g e n c e
T r a i n - t e s t d a t a +
LayerLayer-
LayerLayer- LayerLayer-
LayerLayer- + tion, RevBN).
In the case of layer normalization, its standardization operation cannot be reversed, since the input and prediction sequence length differ in our experimental settings.
The MSE is measured on the four datasets using N-BEATS as the baseline for all experiments.
Additional results with other prediction lengths are included in Table 10.
RevIN shows outstanding performance compared to other normalization methods, especially for ETTh 2 and ETTm 1 datasets.
Moreover, RevBN shows improved performance, especially when predicting long sequences (960, 1344), compared to the Batch normalization.
It describes that the denormalization step of RevIN is essential for the input normalization techniques tailored to time-series forecasting.
However, since batch normalization applies an identical normalization to all the input sequences using train data statistics, it can not reduce the discrepancy between the datasets.
As a result, RevIN surpasses RevBN with a large margin, which implies that our method successfully reduces the distribution discrepancy within data.
Additionally, RevIN not only improves the performance of baselines but also has the advantage of being lightweight, only requiring 2 K additional parameters, which is almost negligible compared to the number of parameters of baseline models.
4.2. A NALYSIS ON D ISTRIBUTION S HIFT I N THE I NTERMEDIATE L AYERS
To study whether RevIN also can reduce the distribution shift at an intermediate feature level, we analyze the feature divergence between train and test datasets for baseline and RevIN on ETT and ECL datasets, using Informer as the baseline.
Following previous work (Pan et al., 2018), we calculate the average feature divergence of each feature between train and test datasets using symmetric KL divergence.
The detailed calculation of symmetric KL divergence is described in Appendix A.10.
The analysis results are shown in Fig.
Since Informer consists of two layers for the encoder and one layer for the decoder, we calculate the feature divergence of the intermediate outputs, the ﬁrst encoder layer (Layer-1) output and the second encoder layer (Layer-2) output.
As shown in Fig.
5, the feature divergence between the train and test data is signiﬁcantly reduced with RevIN.
It implies that the normalization applied to the input can resolve the distribution shift in every layer in a model.
Therefore, as a model-agnostic method, our method successfully alleviates the distribution shift problem via normalization and denormalization of the input and output without modifying the model architectures.
C ONCLUSION
We propose a simple yet effective normalization method tailored to time-series forecasting.
RevIN consists of symmetrically structured normalization and denormalization layers, where the shifting and scaling applied on the input data is exactly reversed back on the output prediction, effectively reducing the discrepancy between train and test data distributions.
Easily applied to input and output layers with a negligible number of additional parameters, RevIN signiﬁcantly improves the forecasting performance of a model without any modiﬁcation in the architecture of the model.
By adding
RevIN, we achieve the state-of-the-art performance on four large-scale real-world datasets with a signiﬁcant margin.
The extensive experiments demonstrate the effectiveness of RevIN for accurate time-series forecasting against the distribution shift problem.
R EPRODUCIBILITY S TATEMENT
To ensure reproducibility we provide the source code of our method as the supplementary material, including the pretrained model weights.
In the main manuscript, Section 4.1 describes how we conduct data preprocessing on the dataset used in the experiments.
In the appendix, Section A. explains the experimental details including the seed for the experiments.
Also, we provide a detailed explanation of hyperparameter settings with the reproduction details of the baseline models in
Section A.12 since our method is an add-on module with no additional hyperparameters
A A PPENDIX
The supplementary material presents additional information, visualizations and experimental results that support the main manuscript.
Section A.1 shows experimental results on three real-world datasets along with qualitative analysis on the Nasdaq dataset to verify the effectiveness of our method on obvious non-stationary time-series.
Section A.2 addresses the potential of RevIN on a cross-domain time-series forecasting task.
Section A.3 and Section A.4 conduct the hyperparameter sensitivity analysis and the ablation study on the proposed method, respectively.
Section A. evaluates our method using the complementary metrics that measure the similarity between the two sequences.
Section A.6 and Section A.7 show the comparison results of our method and existing normalization methods.
Section A.8, Section A.9, and Section A.10 provide the algorithm of RevIN, the theoretical justiﬁcation of RevIN, and the calculation details of the feature divergence used in
Section 4.2.3, respectively.
Section A.11 and Section A.12 describe additional implementation details and the reproduction details for baselines where RevIN is applied, respectively.
Section A. illustrates additional quantitative results of RevIN and the baseline models.
Lastly, Section A.14 provides extra quantitative results, including the standard deviation values for ﬁve experiments, which can not be located in the manuscript due to the lack of space.
A. E XPERIMENTAL R ESULTS ON A DDITIONAL R EAL - WORLD T IME - SERIES D ATASETS
The results on the M4 ∗ dataset are multiplied by ten for readability.
The average value and the standard deviation for ﬁve runs are reported
[Table]
q
[Table]
We evaluate our proposed method on four large-scale real-world time-series datasets, the ETTh 1 ,
ETTh 2 , ETTm and ECL datasets
Additionally, this section provides experimental results on three real-world datasets: the two real-world datasets taken from the UCI repository, the air quality dataset and the Nasdaq dataset, and the M4 competition dataset (Makridakis et al., 2020).
Air quality 4 data is hourly data with 13 variables of length 9537.
We set the prediction length as 24, 48, 168, 336, 720 and the corresponding input length as 48, 96, 168, 360 so that their ratio becomes 2x, 2x, 1x, 0.5x, 0.5x.
Nasdaq 5 dataset consists of 82 features, including the closing measured since 2010.
It is measured daily and has 1984 data samples for each variable in total.
We ﬁx the input sequence length to 60 and prolong the prediction length as 30, 60, 120.
M4 6 competition dataset consists of six different hourly, daily, weekly, monthly, quarterly, and yearly sets, containing 100,000 test data.
We follow the original experimental protocol of the M4 competition (Makridakis et al., 2020).
For evaluation, we measure the micro-averaged mean absolute error and mean squared error on the
M4 competition dataset: we ﬁrst compute the metric independently for each set, i.e., hourly, daily, weekly, monthly, quarterly, and yearly sets, and then calculate the weighted average of the metrics using the contributions of each set as weights.
As shown in Table 4, the model adopting RevIN signiﬁcantly improves the forecasting performance of the baseline on all three datasets.
Notably, RevIN shows outstanding performance on the Nasdaq dataset, reducing the prediction errors by more than half compared to the baseline.
To verify the effectiveness of our method on obvious non-stationary time-series, we conduct qualitative analysis on the Nasdaq dataset.
As shown in Fig.
6, the Nasdaq index (the variable ’Close’) has steadily increased since 2002.
Accordingly, when data are divided into the train and test data based on a speciﬁc point in time, the test data values tend to be higher than the train data values.
In other words, the data severely suffers from the distribution shift problem, where the train and test data show a discrepancy in their distribution.
As a result, even existing SOTA networks cannot predict the future values appropriately, as shown in Fig.
The baseline fails to keep up with the trend of data, whose mean value continues to increase, and thus the prediction results are shifted.
However,
RevIN mitigates such distribution discrepancy and remarkably increases the prediction performance of the baseline SOTA model.
Similarly, RevIN shows superior performance on the decreasing data (’DE1’ in Fig. 6), accurately predicting the changing mean of the data.
A. C ROSS -D OMAIN E VALUATION
[Table]
We analyze the cross-domain evaluation results to analyze further the capability of RevIN to alleviate cross-domain distribution discrepancy.
As RevIN reduces the discrepancy in the train and test distributions, it can also reduce the discrepancy in the cross-domain distributions if the input sequences from each dataset have similar local dynamics.
We use ETT datasets for the evaluation
[Table]
N-BEATS +RevIN
Informer +RevIN
SCINet +RevIN MSE MAE
MSE
MAE
MSE MAE MSE MAE
MSE MAE MSE MAE  0.907 0.73 0.
960 1.07 0.802 0.846 0.679 stdev0.026 0.
[Figure]
[Figure] [Figure]
BEATS MSE
N-BEATS MAE +RevIN MSE +RevIN MAE 0.5 0.
168 336 480 720 960 Input sequence length
Informer MSE
Informer MAE +RevIN MSE +RevIN MAE ±. 0.50.
Input sequence length
SCINet
[Figure]
SCINet
[Figure]
since they have the same features. However, their distributions can show a signiﬁcant discrepancy because ETTh 1 and ETTh 2 are measured from different locations, and ETTh and ETTm 1 datasets have different measurement intervals. We train the baseline and RevIN using SCINet as a baseline on ETT datasets and evaluate the trained model on the other datasets.
We report the average MSE of ﬁve runs in Table 5 and variance in Table 11.
Despite the difference between distributions of the datasets, RevIN shows superior performance in cross-domain evaluations.
Especially, RevIN outperforms a SCINet with a large margin when the model needs to reduce the discrepancy in ETThand ETTm 1 datasets.
The results demonstrate that RevIN successfully reduces the discrepancy in the input sequence distributions, leading to better generalization performance.
A. H YPERPARAMETER S ENSITIVITY A NALYSIS
We analyze the hyperparameter sensitivity of the proposed method RevIN in comparison with the baseline SOTA models.
Input sequence length can be a crucial hyperparameter to RevIN since it computes the mean and variance across each input sequence and thus can determine the prediction accuracy and stability of the training process.
In Fig.
7, we increase the model input sequence length from 48 (two days) to 960 (40 days) to see its impact on RevIN when the prediction length is ﬁxed to 960 (40 days).
First, the results demonstrate that the models adopting RevIN consistently outperform the baselines for various input sequence lengths in terms of the MAE and MAE.
More importantly,
RevIN makes the baseline models more robust to the input length.
In other words, RevIN shows stable performance for various input lengths in contrast to the baseline models, which shows a high variance in their performance according to the input length.
Notably, in N-BEATS, the average error as well as the standard deviation of the error signiﬁcantly increase as prolonging the input length.
This is because the trend of data is expressed as a linear function in N-BEATS.
As the input length becomes prolonged, there is a chance that the variance (or non-stationarity) in the time-series values becomes higher, decreasing the accuracy of the linear trend to ﬁt the data unless the data is monotonically increasing or decreasing.
Also, the mispredicted trends linearly increase the error in the future values.
However, when N-BEATS adopts the RevIN layer, it removes the non-stationary statistics from the input, and thus, the model shows robust performance against the input length.
Removing the variability, i.e., normalizing its mean and standard deviation, before feeding it to the model and returning it to the output would make the model learning stable.
A. A BLATION S TUDY
We ablate the afﬁne transformation from RevIN to see its impact on the performance.
We conduct the experiments on the ETTh 1 , ETTh 2 , ETTm 1 , ECL, Nasdaq, and air quality datasets using N-BEATS as the baseline.
The results in Table 6 show that the afﬁne transformation contributes to performance improvement consistently on a variety of datasets.
As mentioned earlier, a model can add RevIN in an intermediate layer, even to several layers.
While existing approaches are a preprocessing-and-postprocessing method applied outside of the main
[Table]
prediction model, RevIN is an end-to-end trainable layer that can be added to any layer in the model as batch normalization and instance normalization, which are the recently proposed deep learningbased normalization layers.
Thus, we verify that adopting RevIN to the intermediate layers instead of the input and output layers can improve the forecasting performance as well.
We add RevIN at the ﬁrst stack of N-BEATS and SCINet and evaluate their performance on the six datasets.
The results in Table 7 demonstrate that even when added to the intermediate layers, RevIN improves the performance of the baselines as a learnable normalization layer.
We mainly focus on adding RevIN to the input and output of a model since it shows robust performance on average.
Nevertheless, the model adopting RevIN in the intermediate layers also shows the best performance frequently, consistently outperforming the baseline without RevIN.
This performance is even better than the dynamic normalization methods, LSTNet ∗ and ES-RNN ∗ , when comparing them where NBEATS is used as the baseline (Table 9).
For example, when the prediction length is 960, the mean squared errors of LSTNet ∗ , ES-RNN ∗ , RevIN (inter.), and RevIN (i/o) are 5.627, 1.338, 0.523, and 0.471, on average.
A. E VALUATION ON S IMILARITY M ETRICS
We additionally evaluate our method using the metrics that measure similarity between the two sequences, dynamic time warping (DTW) and temporal distortion index (TDI) (Le Guen & Thome, 2019; Fr ´ ıas-Paredes et al., 2017).
As shown in Table 8, the results show that our approach significantly improves the baseline models across all datasets in terms of the DTW and TDI.
Notably,
RevIN shows a signiﬁcant margin compared to the baseline for long prediction length.
For example, when RevIN is added, the average DTW decreases from 38.348 to 15.240 for Informer, from 53.148 to 12.766 for N-BEATS, and from 20.498 to 11.080 for SCINet when the prediction length is 960.
There are few cases where the proposed method predicts a less similar sequence than the baseline, but the margin is minimal in terms of either DTW or TDI compared to the signiﬁcant margin
[Table]
ETT m
[Table]
found when our method outperforms the baseline.
These results demonstrate that the model adopting RevIN can generate a sequence more similar to the groundtruth than the baseline, especially showing a better prediction accuracy on the long sequence.
[Table]
A. C OMPARISON WITH E XISTING D YNAMIC N ORMALIZATION M ETHODS
We compare RevIN with the normalization methods proposed in LSTNet (Lai et al., 2017) and
ES-RNN (Smyl, 2020), as shown in Table 9.
Similarly to adding RevIN to the baseline model, we add the autoregressive linear bypass module of LSTNet and the modiﬁed Holt-Winters exponential smoothing of ES-RNN to the baseline model, respectively.
RevIN consistently shows the best performance among the normalization methods adopted on N-BEATS showing a signiﬁcant margin.
The autoregressive linear bypass module of LSTNet (LSTNet ∗ ) also consistently reduces the prediction error compared to the baseline, but the performance improvement is smaller than our method.
For example, when the prediction length is 960 on the ETTh 2 dataset, N-BEATS shows an average error of 6.408 and LSTNet ∗ reduces the error to 5.627, but which is still much higher than RevIN that reduces the error to 0.471.
Similarly, when the prediction length is 1344 on the ETTm 1 dataset, the baseline shows an average error of 14.613 and LSTNet ∗ largely decreases the error to 5.592, but RevIN more signiﬁcantly decreases the error to 0.631.
LSTNet utilizes a linear autoregressive model to make the non-linear neural network robust to scale change.
However, it does not change the statistics of the input sequence.
Thus, the discrepancy between the train and test datasets still exists for the non-linear neural network.
In the case of ES-RNN ∗ , the training of the model is unstable and it often degrades the baseline performance, e.g., when the prediction length is 24 and 48 on the ETTh 1 and ETTm 1 datasets and the prediction length is 672 on the ETTm 1 dataset.
However,
[Table]
[Figure]
E S R
NN *
R e v I
N
L S T
N e t *
D e n s i t y D e n s i t y
D e n s i t y it signiﬁcantly reduces the error for the long prediction length much better than LSTNet ∗ , but still worse than RevIN, e.g., when the prediction length is 960 on the ETTh 2 dataset.
Additionally, we further analyze the data distributions of the methods on the ETTh 2 dataset with the prediction length of 960 as shown in Fig. 8.
ES-RNN ∗ shows the distributions of (a) the original input, (b) the normalized input data where the level and seasonality are removed, (c) the model prediction output c the where the level and seasonality is multiplied back to the original distribution.
LSTNet ∗ shows the distributions of (a) the original input, (b) the same original input since the method does not transform the input data before feeding it to the prediction model, the model prediction output before (c) and after (d) adding the output of the autoregressive model are added.
RevIN shows the distributions of (a) the original input, (b) the input normalized by RevIN, (c the model prediction output, and (d) the denormalized output.
In Fig. 8(a), the train and test data shows discrepancy in their distributions.
Also, they have several peaks, not being centered to the mean.
This implies that time-series in the data would have different means from each other.
In Fig.
8(b), both RevIN and ES-RNN ∗ transform data distributions into mean-centered distributions.
Particularly, ES-RNN ∗ makes the distribution extremely concentrated on the mean with a small variance.
The two methods make sequences have similar statistics, thereby alleviating the distribution shift problem in the input data.
This would lead to competitive performance on the long prediction sequence in contrast to LSTNet ∗ .
LSTNet ∗ cannot resolve the distribution discrepancy because it has no module that can change the input statistics.
Also, the distributions of the model output (Fig. 8(c)) completely differ from the input (Fig. 8(b)).
The methods cannot make the input and output distribution to be consistent.
In addition, its autoregressive model barely affects the model output distribution as shown in Fig.
8(c,d).
There is almost no difference between the model output and the ﬁnal output in LSTNet ∗ .
Also, the distributions of the ﬁnal output (Fig. 8(d)) signiﬁcantly differ from the original data (Fig. 8(a)) in LSTNet ∗ , which implies that they fail to learn the appropriate data distribution, and this would be the main reason for higher prediction error than RevIN.
Similarly, although ES-RNN ∗ alleviates the distribution discrepancy in the input data, it fails to return the model output (Fig. 8(d))
back to the original distribution (Fig. 8(a)), especially for the test data.
On the other hand, in RevIN, the distributions of the ﬁnal output (Fig. 8(d)) return well to the original distributions (Fig. 8(a)).
Also, RevIN makes the input and output of the model to maintain the consistent distributions while the train and test data are overlapped.
Thus, RevIN can show robust performance against the input and output lengths.
A. A DDITIONAL R ESULTS ON C OMPARISON WITH O THER N ORMALIZATION M ETHODS
This section provides the additional results on comparison with other normalization methods, which can not be located in the main manuscript due to the lack of space.
The forecasting error of RevIN and existing normalization methods are evaluated on the ETTh 1 , ETTh 2 , ETTm 1 , and ECL datasets in Table 10.
RevIN consistently outperforms the other normalization methods across all datasets.
Interestingly, when the denormalization step is added to batch normalization (RevBN) as RevIN, the model better forecasts the long sequence, e.g., when the prediction length is 960, than batch normalization (Batch norm).
In the case of DAIN, it learns to transform the mean and variance values of the current input and use the new values to normalize the data.
As a result, the input sequences can have different means and variances from each other, which indicates that the distribution shift can still exist after the normalization.
As shown in our paper, the denormalization step shows a critical role in improving the model performance by recovering the prediction to the original value.
However, a denormalization step cannot be added to DAIN since it has the Hadamard multiplication operation at the last step, which is not reversible when the input and prediction sequence lengths are different.
These differences would be the reason for its inferior performance to RevIN, even requiring more computational costs and a larger amount of model parameters (at least three D × D matrices, where D is the number of variables).
A. A LGORITHM OF R EVERSIBLE I NSTANCE N ORMALIZATION
Algorithm 7 summarizes the RevIN procedure.
Reversible instance normalization is the symmetric structure of the normalization (line 4-5) and denormalization operation (line 7-8).
RevIN simply scales and shifts the input (line 2-5) and output (line 7-8) of a model using identical statistics.
Indeed, g θ in Algorithm 7 (line 6) can be any network.
RevIN is a model-agnostic normalization method easily adopted to any model with negligible cost but effectively improves forecasting performance.
Input : T x , input sequence length; x ( i ) kt , the k -th feature at time step t of the i -th item in a mini-batch; γ, β ∈ R K , parameters to be learned for RevIN; g θ , a module in the model.
Output: γ β , θ
Compute µ T ← 1 T x (cid:80) T x j =1 x ( i ) kj (cid:46) instance mean Compute σ 2 T ← 1 T x (cid:80) T x j =1 ( x ( i ) kj − µ T )(cid:46) instance variance Normalize ˆ x ( i )
[Equation]
← x ( i ) kt
[Equation]
µ
[Equation]
√ σ
T + (cid:15) (cid:46) normalization
Transform ˆ x ( i )
[Equation]
← γ k · ˆ x [Equation] i [Equation]
+ β k ≡ RevIN n
[Equation]
β ( x ( i ) kt ) (cid:46) scale and shift
Predict y ← g θ (ˆ x ) (cid:46) model prediction
[Table]
Retransform ˆ y ( i ) kt ← y
( i ) kt − β k γ k (cid:46) reverse scale and shift [Table]
Denormalize ˆ
[Equation]
i
[Equation]
← µ
[Equation]
y
[Equation]
i )
[Equation] (cid:112) σ [Equation] denormalization
[Table]
E TT h
[Table]
ETT m 0.
E C L 0.
[Table] [Table]
[Table] [Table]
A. T HEORETICAL J USTIFICATION OF R EV IN A GAINST D ISTRIBUTION S HIFT P ROBLEM
Let { x ( i ) } Ni =1 be our data, where x ( i ) ∈
R K × T x denotes a time-series with K variables of length
T x . Consider a simple univariate case where K = 1 without the loss of generality.
Then, x ( i ) ∈ R T x denotes the i -th time-series in the data.
Consider two such data, the training and test data, whose distributions are denoted as P tra and P tst , respectively.
The distribution shift appears when distribution of the data sequences are different (Du et al., 2021).
That is,
[Equation]
In our work, we consider the distribution shift problem in terms of the mean and the variance.
Assume that the given data suffer from the distribution shift problem in terms of the mean and variance, which can be expressed as
[Equation]
In order to solve this problem, the normalization step of RevIN can be applied to the input training data x ( i ) ∼ P tra .
Then, the input data is transformed as
[Equation]
By the laws of expectation and variance,
E [˜ x ( i ) ]
= β and Var[˜ x ( i ) ]
=
γ 2 , for all i (
[Equation]
By symmetry, this also holds for x ( i ) ∼ P tst , for all i .
Therefore, the mean and variance of the training and test data distributions become identical.
Thus, by deﬁnition, the distribution shift problem is solved for the input training and test data by the normalization step.
Given the normalized time-series data, the forecasting model f : R T x → R T y , parameterized by θ , predicts the corresponding future values as ˆ y =
f θ (˜ x ) .
Then, the denormalization step of RevIN returns the dynamics of the original data, E
[ x ( i ) ] and Var[ x ( i ) ] to the prediction output so that model does not have to reconstruct them from the normalized input.
The model prediction ˆ y
( i ) can be denormalized as
[Equation]
By the laws of expectation and variance, the mean and variance of ˜ y
( i ) can be expressed as
[Equation]
The denormalization step allows the mean and variance of the ﬁnal prediction values to be expressed as the difference from the input statistics.
Here, since the input data x ( i ) and the groundtruth future values are consecutive sequences, one can assume that their difference in the mean and variance can be expressed as Eq.
(9) as well.
Under the assumption, when adopting the denormalization step of RevIN, the model only needs to capture the difference from the input statistics, ∆ and λ , to accurately predict the statistics of the future values.
The denormalization step allows the model to focus on capturing the difference in the distribution between input and output by removing all common non-stationary statistics between the input and output to the model.
A. C ALCULATION D ETAILS ON F EATURE D IVERGENCE
This section explains how the feature divergence in Section 4.2.3 is calculated.
Following previous work (Pan et al., 2018), we calculate the average feature divergence of each feature between train and test datasets using symmetric KL divergence, assuming that the output features follow a Gaussian distribution with mean µ and variance σ 2 .
Then, the equation for feature divergence of the k -th feature f k is
[Equation]
10) where KL ( f Ak || f
Bk )
= log σ Bk σ Ak + σ Ak 2 + ( µ Ak − µ Bk )
[Equation]
Bk−2 .
A. E XPERIMENTAL D ETAILS
We exactly follow the experimental settings of the baseline models by using their ofﬁcial code to conduct experiments except for N-BEATS that have no ofﬁcially released code.
We reproduce the model and set hyperparameters as stated in the original N-BEATS paper (Oreshkin et al., 2020).
Additionally, we conduct a grid search for the learning rate of N-BEATS with the range of [1e-5, 1e3] and train the model with the weight decay with the factor of 0.001 to stabilize training.
We train and test the models with the following seeds: 12, 22, 32, 42, and 52.
Experiments using N-BEATS and Informer as backbone are performed on NVIDIA TITAN RTX, and experiments using SCINet are conducted on NVIDIA TITAN Xp.
Following the multivariate time-series forecasting settings of prior work (Zhou et al., 2021; Liu et al., 2021), we select the input window lengths from 2d, 4d, 7d, 14d, 15d, 20d for the hourly datasets, the ETTh 1 , ETTh 2 , and ECL datasets, and from 12h, 1d, 4d, 7d for the ETTm 1 dataset.
Particularly, we set the ratio of input length to prediction length to be from 2.0 to 0.35 as the prediction length becomes longer.
In the case of SCINet, we follow the setting of the original paper for the input length since the method requires the input sequence length to meet a speciﬁc condition.
A. R EPRODUCTION D ETAILS FOR B ASELINE M ODELS
This section describes the implementation details of baselines, Informer, N-BEATS, and SCINet.
Note that we compare each baseline model and RevIN using the same hyperparameters except for the presence of RevIN.
Informer.
We use the ofﬁcial open-source code of Informer 7 .
We follow the provided hyperparameters for the experiments they studied.
In the case of the ECL dataset, we use the same hyperparameter settings for the ETTh 1 dataset since it is not ofﬁcially provided.
N-BEATS.
N-BEATS is reproduced using the PyTorch framework (Paszke et al., 2019).
We follow the hyperparameter settings of the N-BEATS-I model.
We train N-BEATS to minimize the mean squared error between the predicted results and groundtruth values.
To compare with other methods, we report the prediction results of a single model instead of the ensemble method that N-BEATS used in the original paper.
Since N-BEATS is the model tailored to univariate time-series forecasting, we ﬂatten the multivariate input sequence to have a single dimension before feeding it to the model.
SCINet.
We follow the experimental settings provided in the ofﬁcial code 8 of SCINET.
Different from other baseline models, when the prediction length is 720, the input sequence length is set assince SCINet requires the input length to be the multiple of 32 due to its hierarchical architecture (Liu et al., 2021).
A. A DDITIONAL Q UALITATIVE R ESULTS
Informer fails to capture the repeated pattern of increase and decrease on ETTh 2 dataset.
Moreover, the prediction results of Informer are inaccurately scaled and shifted.
With RevIN, Informer predicts reasonable results.
Without any modiﬁcation to the network architecture, RevIN can improve the forecasting performance.
Although the pattern is accurately captured for N-BEATS, the prediction results are inaccurately scaled, resulting in low performance.
RevIN improves the performance of
N-BEATS in that the model predicts accurately.
For SCINet, RevIN improves the baseline to predict more precise results.
A. E XTRA Q UANTITATIVE R ESULTS
Table 11 and Table 12 show the standard deviation for ﬁve independent experiments of cross-domain evaluation and for comparison on long sequence forecasting, respectively.
Table 13 describes the full results of comparison with baselines and RevIN, including the standard deviation for ﬁve runs and comparison with the reported performance in the original papers.
It can be seen as RevIN improves the performance as statistically signiﬁcant than previous state-of-the-art models.
[Table] [Table]
+ RevIN 0.
[Table] [Table]
N
[Table]
BEATS 0.
[Table] [Table]
+ RevIN 0.
SCINet
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
+ RevIN 0.
E
TT m
E C
L
E
TT h
E
TT h
T a b l e 13 :
T i m e - s er i e
s f o rec a s t i n g re s u l t s .
T h
e m ea n a nd s t
a nd a r d d e v
i a ti on o f t
h e
ﬁ v e i nd e p e nd e n t e xp e
r
i m e n t s a r e r ec o r d e d . † d e no t e s t h a t t h e r e s u lt s a r e ob t a i n e d fr o m t
h e o r
i g i n a l p a p e r .
M e t hod s
I n f o r m e r †
I n f o r m e r
I n f o r m e r + R e v
I N
N - B
E A T
S
N - B
E A T
S + R e v
I N
S C
I N e t †
S C
I N e t
S C I N e t +
[Table]
e v I N
M e t r i c
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
[Table]
M A
[Table]
M
[Table]
E
[Table] [Table]
ECL
[Table] [Table]