R EVERSIBLE I NSTANCE N ORMALIZATION FOR A CCURATE T IME -S ERIES F ORECASTING AGAINST D ISTRIBUTION S HIFT
Taesung Kim ∗
KAIST AI zkm @kaist.ac.kr Jinhee Kim * KAIST AI seharanul @kaist.ac.kr Yunwon Tae
VUNO yunwon.tae @vuno.co
Cheonbok Park
NAVER Corporation cbok.park @navercorp.com
Jang-Ho Choi ETRI janghochoi @etri.re.kr Jaegul Choo KAIST AI jchoo @kaist.ac.kr
[Figure]
day
A BSTRACT
Statistical properties such as mean and variance often change over time in time series, i.e., time-series data suffer from a distribution shift problem.
This temporal distribution change is one of the main challenges that prevent accurate time-series forecasting.
To address this issue, we propose a simple yet effective normalization method called reversible instance normalization (RevIN), a generally-applicable normalization-and-denormalization method with learnable afﬁne transformation.
The proposed method is symmetrically structured to remove and restore the statistical information of a time-series instance, leading to signiﬁcant performance improvements in time-series forecasting, as shown in Fig.
We demonstrate the effectiveness of RevIN through extensive quantitative and qualitative results on various real-world datasets, addressing the distribution shift problem.
I NTRODUCTION
Time-series forecasting plays a signiﬁcant role in various daily problems such as health care, economics, and trafﬁc analyses (Kim et al., 2021a; Ahmadi et al., 2019; Park et al., 2020).
Recently, time-series forecasting models have achieved outstanding performance on these problems overcoming several challenges, e.g., long-term forecasting (Zhou et al., 2021; Liu et al., 2021) and missing value imputation (Zhang et al., 2021;
Kim et al., 2021b).
However, the forecasting models often severely suffer from a unique characteristic in time series; their statistical properties, e.g., mean and variance, can change over time, widely known as a distribution shift problem. The distribution shift problem can yield discrepancies between distributions of the training and test data used in the forecasting models.
It is because the data are usually split based on a speciﬁc point in time from the original data in time-series forecasting tasks.
Accordingly, the discrepancy between the training and test data distributions can be highly severe in time series, leading to signiﬁcant degradation in the forecasting performance of the models. Furthermore, it can cause the input sequences of the models to have a crucial discrepancy in the distributions from each other as well. We assume that such discrepancy between different input sequences can signiﬁcantly degrade the model performance.
Under the assumption, if we remove non-stationary information from the input sequences, speciﬁcally, the mean and standard deviation of the instances, the discrepancy in the data distribution will be reduced, thereby improving the model performance.
However, applying such normalization to the model input can cause another problem; it can restrict the models from capturing the original data distribution since it removes non-stationary information that can be important to predict future values in the forecasting task. A model needs to reconstruct the original distribution only using the normalized input, which has inherent limitations. Thus, if we explicitly return the removed information to the model, the model does not have to rebuild the original distribution by itself while enjoying the beneﬁts of normalizing the input.
The eliminated information can be restored by reversing the normalization applied to the input data at the output layer, i.e., by denormalizing the model output.
Inspired by these, we propose a simple yet effective normalization-and-denormalization method, reversible instance normalization (RevIN) , which ﬁrst normalizes the input sequences and then reverses it on the model output sequences to solve the time-series forecasting problems against distribution shift.
RevIN is symmetrically structured to return the original distribution information to the model output by scaling and shifting the output in the denormalization layer as much as the input data is shifted and scaled in the normalization layer.
To verify the effectiveness of RevIN, we conduct extensive quantitative evaluations using various state-of-the-art time-series forecasting methods as the baselines:
Informer (Zhou et al., 2021), N-BEATS (Oreshkin et al., 2020), and SCINet (Liu et al., 2021).
We also provide an in-depth analysis of the behavior of the proposed approach, including the veriﬁcation of the assumptions on reversible instance normalization (See Section 3.2).
RevIN is a ﬂexible, end-to-end trainable layer that can be applied to any arbitrarily chosen layers, effectively suppressing non-stationary information (mean and variance of the instance) from one layer and restoring it on the other layer at a virtually symmetric position, e.g., input and output layers.
Despite its remarkable performance, there has been no work on generalizing and expanding the instance-wise normalization-and-denormalization as a ﬂexibly applicable, trainable layer in the time-series domain.
Recently, deep learning-based time-series forecasting approaches, such as Informer (Zhou et al., 2021) and N-BEATS (Oreshkin et al., 2020), have shown outstanding performance in time-series forecasting.
However, they overlooked the importance of normalization, merely using simple global preprocessing to the model input without further exploration and expecting their end-to-end deep learning model to replace the role.
Despite the simplicity of our method, there have been no cases of using such techniques in modern deep-learning-based time-series forecasting approaches (Zhou et al., 2021; Liu et al., 2021; Oreshkin et al., 2020).
In this sense, we enlight the importance of the appropriate normalization method in deep-learning-based time-series approaches; we propose a carefully designed, deep-learning-friendly module, for time-series forecasting, by combining the learnable afﬁne transformation with the method, which has been widely accepted in recent deep-learning-based normalization work (Ulyanov et al., 2016).
In summary, our contributions are as follows:
• We propose a simple yet effective normalization-and-denormalization method for timeseries, called RevIN, symmetrically structured to remove and restore the statistical information of a time-series instance. It signiﬁcantly improves the forecasting performance by reducing the distribution discrepancy within the data while generally applicable to arbitrary deep neural networks with negligible cost.
• By adding RevIN to the baseline, we achieve the state-of-the-art performance on seven large-scale real-world datasets with a signiﬁcant margin.
• We conduct extensive evaluations on RevIN with quantitative analysis and qualitative visualizations to verify its effectiveness, addressing the distribution shift problem.
R ELATED W ORK
Time-series forecasting.
Time-series forecasting methods are mainly categorized into three distinct approaches: (1) statistical methods, (2) hybrid methods, and (3) deep learning-based methods.
As an example of the statistical models, exponential smoothing forecasting (Holt, 2004; Winters, 1960) is a well-established benchmark to predict future values.
Statistical models have several advantages, e.g., interpretability and theoretically well guaranteed.
In order to boost the performance, recent work proposed a hybrid model (Smyl, 2020) which incorporates a deep learning module with a statistical model.
It achieved better performance over the statistical methods in the M4 timeseries forecasting competition.
The deep learning-based method basically follows the sequenceto-sequence framework to model the time-series forecasting.
Initially, deep learning-based models utilized variations of recurrent neural networks (RNNs).
However, to overcome the limitation of the limited receptive ﬁeld, several studies utilize advanced techniques, such as the dilatation and attention module.
For instance, SCINet (Liu et al., 2021) and Informer (Zhou et al., 2021) modiﬁed the sequence-to-sequence-based model to improve the performance for long sequences.
However, most previous deep learning-based models are hard to interpret compared to statistical models.
Thus, inspired by statistical models, N-BEATS (Oreshkin et al., 2020) designed an interpretable layer for time-series forecasting by encouraging the model to learn trend, seasonality explicitly, and residual components.
This model shows superior performance on the M4 competition dataset.
Distribution shift.
Although there are various models for time-series forecasting, they often suffer from non-stationary time-series, where the data distribution changes over time.
Domain adaptation (Tzeng et al., 2017; Ganin et al., 2016; Wang et al., 2018) and domain generalization (Wang et al., 2021; Li et al., 2018; Muandet et al., 2013) are common solutions for alleviating the distribution shift.
A domain adaptation algorithm attempts to reduce the distribution gap between source and target domains.
A domain generalization algorithm only relies on the source domain and hopes to generalize on the target domain.
Both domain adaptation and generalization have a common objective that bridges the gap between source and target distributions.
However, in non-stationary time-series, deﬁning a domain is not as straightforward since the data distribution shifts over time.
Recently, Du et al.
(Du et al., 2021) proposed Adaptive RNNs to handle the distribution shift problems for non-stationary time-series data.
It ﬁrst characterizes the distribution information by splitting the training data into periods.
Then, it matches distributions of the discovered periods to generalize the model.
However, unlike Adaptive RNNs, which is costly expensive, RevIN is simple yet effective and model-agnostic since the method can be adopted easily to any deep neural network.
P ROPOSED M ETHOD
This section proposes reversible instance normalization to alleviate the distribution shift problem in time-series, which causes a substantial discrepancy between training and test data distributions.
Section 3.1 describes the proposed method in detail, and Section 3.2 discusses how our approach mitigates the distribution discrepancy in time series.
3. R EVERSIBLE I NSTANCE N ORMALIZATION
Given a set of input X = { x ( i ) } Ni =1 and the corresponding target Y = { y ( i ) } Ni =1 , we consider a multivariate time-series forecasting task in discrete time, where N denotes the number of sequences.
Let K, T x , and T y denote the number of variables, the input sequence length, and the model prediction length, respectively.
Given an input sequence x ( i ) ∈ R K × T x , we aim to solve the time-series forecasting problem, which is to predict the subsequent values y ( i
) ∈ R K × T y .
In RevIN, the input [Figure]
sequence length T x and the prediction length
T y can be different since the observations are normalized and denormalized across the temporal dimension, as will be explained below. Our proposed method, RevIN, consists of symmetrically structured normalization-and-denormalization layers, as illustrated in Fig.
First, we normalize the input data x ( i ) using its instance-speciﬁc mean and standard deviation, which is widely accepted as instance normalization (Ulyanov et al., 2016).
The mean and standard deviation are computed for every instance x
( i ) k · ∈ R T x of the input data (Fig. 2(a-3)) as
[Equation]
j = x ( i
[Equation]
kj and
[Equation]
[Equation]
Using these statistics, we normalize the input data x ( i ) (Fig. 2(a-1)) as
[Equation]
where γ, β ∈ R K are learnable afﬁne parameter vectors.
The normalized sequences have the same mean and variance, where the non-stationary information is reduced.
As a result, the normalization layer allows the model to accurately predict the local dynamics within the sequence while receiving inputs of consistent distributions in terms of the mean and variance.
The model then receives the transformed data ˆ x ( i ) as input and forecasts their future values.
However, the input data have different statistics from the original distribution, and only observing the normalized input ˆ x ( i ) , it is difﬁcult to capture the original distribution of the input x ( i ) .
Thus, to make this easier for the model, we explicitly return the non-stationary properties removed from the input data to the model output by reversing the normalization step at a symmetric position, the output layer.
Such denormalization step can return the model output to the original time-series value as well (Ogasawara et al., 2010).
Accordingly, we denormalize the model output ˜
y
( i ) by applying the E TT h E C L D e n s i t y D e n s i t y
D e n s i t y
E TT m
[Figure]
reciprocal of the normalization in Eq.
2 (Fig. 2(a-3)) as
[Equation]
where the same statistics used in the normalization step in Eq.
2 are used for the scaling and shifting.
Now, ˆ y ( i ) is the ﬁnal prediction of the model instead of ˜ y
( i ) .
Simply added to virtually symmetric positions in a network, RevIN can effectively alleviate distribution discrepancy in time-series data, as a generally-applicable trainable normalization layer to arbitrary deep neural networks. Indeed, the proposed method is a ﬂexible, end-to-end trainable layer that can be applied to any arbitrarily chosen layers, even to several layers.
We verify its effectiveness as a ﬂexible layer by adding it to the intermediate layers in the model in Table 7 in Appendix A.4.
Nevertheless, RevIN is most effective when applied to virtually symmetric layers of encoder-decoder structure.
In a typical time-series forecasting model, the boundary between the encoder and the decoder is often unclear.
Thus, we apply RevIN to the input and output layers of a model as they can be interpreted as an encoder-decoder structure, generating subsequent values given input data.
3. E FFECT OF R EVERSIBLE I NSTANCE N ORMALIZATION ON D ISTRIBUTION S HIFT
This section veriﬁes that RevIN can alleviate the distribution discrepancy problem by removing non-stationary information in the input layer and then restoring it in the output layer.
We analyze the distributions of the training and test data at each step of the proposed approach, as shown in Fig.
When comparing the distribution of training and test data in each example (Fig. 3(a-b)), we can observe that RevIN signiﬁcantly reduces their discrepancy.
To be speciﬁc, in the original input (Fig. 3(a)), the training and test data distributions hardly overlap (especially ETTm 1 ), which is caused by the distribution shift problem.
Also, each data distribution has multiple peaks (especially the test data of ETTh 1 and ECL), implying that sequences in the data might have severe discrepancies in their distributions.
However, in the proposed approach, the normalization step transforms each data distribution into mean-centered distributions (Fig. 3(b)).
This result supports that the original multimodal distributions (Fig. 3(a)) are caused by discrepancies in distributions between different sequences in the data.
Even more, the proposed approach makes training and test data distributions overlapped.
This veriﬁes that the normalization step of RevIN can alleviate the distribution shift problem, reducing the distribution discrepancy between training and test data.
Taking the normalized data as the input, the baseline model still maintains the similarly-shaped training and test distributions in the prediction output (Fig. 3(c)).
As expected, these are then returned back to the original distribution by the denormalization step of RevIN (Fig. 3(d)).
Without denormalization, the model needs to reconstruct the values that follow the original distributions (Fig. 3(d)) only using the normalized input following the transformed distributions, where non-stationary information is removed (Fig. 3(b)).
Additionally, we hypothesize that the distribution discrepancy is also reduced in the intermediate layers of the model even when RevIN is applied at the input and output layers only, which will be discussed in Section 4.2.3.
As a result, this procedure of RevIN can be considered as making problems easier ﬁrst and then restoring them to the original state, rather than directly solving the challenging problem where the distribution shift problem exists.
E XPERIMENTS
This section describes the experimental setup and provides extensive experimental results of RevIN.
4. E XPERIMENTAL S ETUP
Datasets.
We evaluate our methods mainly on four large-scale real-world time-series datasets.
Additionally, we provide experimental results on three more datasets, including the air quality and Nasdaq datasets taken from the UCI repository and M4 competition dataset (Makridakis et al., 2020) in Appendix A.1.
(i) Electricity transformer temperature (ETT) 1 data consists of seven features, including power load features and oil temperature. It is collected from two different regions in China for two years.
Following the same protocol as Informer (Zhou et al., 2021), we split the data into three datasets: ETTh 1 , ETTh 2 , and ETTm 1 .
The ETTh 1 and ETTh 2 datasets are hourly data obtained from different regions.
The ETTm 1 dataset has a value every 15 minutes.
For each dataset, we split the ﬁrst 12 months, the middle four months, and the last four months as training, validation, and test data, respectively.
(ii) Electricity Consuming Load (ECL) 2 data contains the electricity consumption (kWh) collected from 321 clients.
Following the prior work (Zhou et al., 2021), data from each client is used as a variable on an hourly basis in the multivariate forecasting setting.
For the ECL dataset, we use 15, 3, and 4 months as train, validation, and test data, respectively.
Experimental details.
We set prediction lengths as one day (1d), 2d, 7d, 14d, 30d, and 40d for the hourly-basis datasets, ETTh 1 , ETTh 2 , and ECL.
For the ETTm 1 dataset, we chose six hours (6h), 12h, 3d, 7d, and 14d as prediction window lengths.
We evaluate the time-series forecasting performance on the mean squared error (MSE) and mean absolute error (MAE).
Following the same evaluation procedure of the previous study (Zhou et al., 2021), we compute the MSE and MAE on zscore normalized data to measure different variables on the same scale.
More details on experimental settings, including training details and hyperparameters, are provided in Appendix A.11.
Compared baselines.
RevIN is a model-agnostic method, generally applicable to any deep neural network.
In this paper, we verify the effectiveness of RevIN by adopting it on three state-of-the-art time-series forecasting models: Informer (Zhou et al., 2021), N-BEATS (Oreshkin et al., 2020), and SCINet (Liu et al., 2021).
These are non-autoregressive forecasting models.
The reproduction details for the baselines are provided in Appendix A.12.
Unless stated otherwise, we compare RevIN and the baselines under the same hyperparameter settings, including the input and prediction lengths.
4. R ESULTS AND A NALYSIS
This section provides the quantitative analysis and qualitative visualization results of RevIN in comparison with the state-of-the-art time-series forecasting baselines
. 4.2. E FFECTIVENESS OF R EVERSIBLE I NSTANCE N ORMALIZATION ON V ARIOUS T IME -S ERIES F ORECASTING M ODELS
Table 1 compares the forecasting accuracy of the baselines and RevIN.
The results show that RevIN consistently outperforms all three baselines, Informer, N-BEATS, and SCINet, with a large margin,
[Table]
ETT h 0.
ETT m 0.
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
analyze the forecasting error of the baselines and RevIN by increasing the prediction length from 48 to 960 while the input length is ﬁxed to 48.
The experiment is conducted on ETTh 1 . The average errors for ﬁve runs are reported, and the complete results, including standard deviation, are provided in Appendix A.14.
[Table]
achieving the state-of-the-art performance on the ETT and ECL datasets.
Moreover, the effectiveness of RevIN is more evident in the long sequence prediction; the proposed method shows stable performance for large prediction lengths in contrast to the baseline models, which show a high increase in error as prolonging the prediction length.
For example, when the prediction length increases fromto 960 on the ETTh 2 dataset, the forecasting error of N-BEATS signiﬁcantly degrades from 0. to 6.408.
In contrast, RevIN shows a much slight increase in the error, i.e., from 0.192 to 0.471.
A similar tendency appears on the other prediction lengths, datasets, and baseline models as well.
These results demonstrate that RevIN makes the baseline model robust to the prediction length. [Table] [Table] [Table]
[Table] We further analyze the effect of RevIN in predicting long sequence quantitatively in Table 2.
When the model prediction length is increased from 48 to 960, RevIN reduces the prediction error compared to the baseline, showing robust performance against the prediction length.
The difference in the forecasting error between SCINet and RevIN is relatively small when the prediction length is short (e.g., 48), but RevIN remarkably surpasses SCINet with a signiﬁcant margin when the prediction length becomes long (e.g., 336, 720, and 960).
These results substantiate that adopting RevIN can make a model robust to the prediction length.
Additionally, in order to study how RevIN can perform well in long sequence prediction, we visualize the forecasting error for each time step in Fig.
The error at the t -th time step is computed as
MSE- t = 1 N (cid:80) Ni =1 1 K (cid:80)
Kk =
(ˆ y ( i )
kt − y ( i ) kt ) 2 .
Overall, RevIN shows superior performance compared to the baselines; the performance degradation is signiﬁcantly slower, showing low error even when forecasting 960 steps ahead.
Speciﬁcally, in N-BEATS and SCINet, the error extremely increases as predicting the distant future values.
RevIN alleviates such a severe increase in error, showing remarkable performance compared to the baselines.
Informer also shows unstable performance for different time steps.
The error is substantial in the early steps and becomes relatively small in the distant steps.
RevIN allows the models to have consistently minor errors at every step, even where the baselines originally show high error (early steps for Informer, distant steps for N-BEATS and SCINet).
The results demonstrate the effectiveness of RevIN on forecasting long sequences.
4.2. C OMPARISON WITH E XISTING N ORMALIZATION M ETHODS
We compare RevIN with classical and state-of-the-art normalization methods, which includes minmax normalization, z-score normalization, layer normalization (Ba et al., 2016), batch normalization (Ioffe & Szegedy, 2015), instance normalization (Ulyanov et al., 2016), and deep adaptive input
+ + f e a t r u e d
i v e r g e n c e
T r a i n - t e s t d a t a +
LayerLayer-
LayerLayer- LayerLayer-
LayerLayer- + normalization (DAIN) (Passalis et al., 2019) in Table 3. Here, we compute the normalization statistics for min-max and z-score normalization methods for each input instance, not for the entire data.
Additionally, we attempt to use batch normalization as the input normalization method in RevIN, named reversible batch normalization (RevBN). Layer normalization cannot be used similarly since it is not reversible when the input and prediction lengths are different as our experimental settings.
As a result, RevIN shows outstanding performance compared to other normalization methods, especially on ETTh 2 and ETTm 1 datasets.
Moreover, RevBN also improves the forecasting performance of batch normalization.
Especially, the error is considerably decreased in the long sequence prediction, e.g., 960 and 1344.
This result supports that the denormalization step of RevIN is essential for improving long sequence forecasting as a key component of the proposed normalization layer.
However, batch normalization applies an identical normalization to all the input sequences using the global statistics obtained from the entire training data; it can not reduce the discrepancy between the training and test data distributions.
Consequently, RevIN, which uses instance-wise normalization, outperforms RevBN with a signiﬁcant margin, demonstrating that reducing the discrepancy between distributions of different input sequences can effectively improve the performance.
Moreover, RevIN not only shows the best performance but also has the advantage of being lightweight.
For example, when K is the number of variables, DAIN requires at least 3 K 2 additional parameters, whereas RevIN requires 2 K additional parameters, which is almost negligible to the baselines.
4.2. A NALYSIS ON D ISTRIBUTION S HIFT I N THE I NTERMEDIATE L AYERS
In Fig. 5, we analyze the feature divergence between the training and test data to verify that RevIN can reduce the distribution shift at the intermediate feature level as well.
We conduct the experiment using Informer as the baseline; it comprises two encoder layers and one decoder layer.
Thus, we analyze the features of the ﬁrst (Layer-1) and the second (Layer-2) encoder layers.
Following the prior work (Pan et al., 2018), we compute the average feature divergence using symmetric KL divergence (See Appendix A.10).
The results show that RevIN signiﬁcantly reduces the feature divergence between the training and test data in both layers, demonstrating that the proposed approach, added only to the input and output layers, successfully alleviates the distribution shift problem in the intermediate layers.
Moreover, this strengthens RevIN as a generally-applicable ﬂexible layer. An arbitrary model can adopt RevIN by adding it to input and output layers without any architectural modiﬁcations.
Note that our approach still can be added to any arbitrarily chosen layers, signiﬁcantly improving the model performance, as shown in Appendix A.4.
C ONCLUSION
This paper aims to address the distribution shift problem in time series, proposing a simple yet effective normalization-and-denormalization method, reversible instance normalization (RevIN). The proposed approach effectively alleviates the discrepancy between training and test data distributions, leading to signiﬁcant performance improvements in time-series forecasting.
As a generallyapplicable layer to arbitrary deep neural networks, the proposed approach achieves state-of-the-art performance on seven real-world time-series datasets with a signiﬁcant margin.
The extensive quantitative and qualitative experiments with in-depth analysis demonstrate the effectiveness of RevIN for accurate time-series forecasting against the distribution shift problem.
R EPRODUCIBILITY S TATEMENT
To ensure reproducibility we provide the source code of our method as the supplementary material, including the pretrained model weights.
In the main manuscript Section 4.1 describes how we conduct data preprocessing on the datasets used in the experiments.
Appendix A.11 explains the experimental details, including random seed values for the experiments.
Appendix A.12 provides a detailed explanation of hyperparameter conﬁgurations with the reproduction details of the baselines
This work was supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2018-0-00219, Spacetime complex artiﬁcial intelligence blue-green algae prediction technology based on direct-readable water quality complex sensor and hyperspectral image, and No.2019-0-00075, Artiﬁcial Intelligence Graduate School Program (KAIST))
A A PPENDIX
The supplementary material provides additional information, visualizations, and experimental results that support the main manuscript.
Section A.1 shows experimental results on additional realworld datasets along with qualitative analysis on one of the datasets to verify the effectiveness of our method on obvious non-stationary time series.
Section A.2 addresses the potential of RevIN on solving the cross-domain time-series forecasting task.
Section A.3 and Section A.4 conduct the hyperparameter sensitivity analysis and the ablation study on the proposed method, respectively.
Section A. evaluates our method using the complementary metrics that measure the similarity between two sequences.
Section A.6 and Section A.7 compare the forecasting performance between the proposed method and existing normalization methods.
Section A.8, Section A.9, and Section A.10 provide the algorithm of RevIN, the theoretical justiﬁcation of RevIN, and the calculation details of the feature divergence used in Section 4.2.3, respectively.
Section A.11 and Section A.12 describe additional implementation details and the reproduction details for the baselines where RevIN is applied, respectively.
Section A.13 illustrates additional quantitative results of RevIN and the baselines.
Lastly, Section A.14 provides complete quantitative results, including the standard deviation values for ﬁve experiments, which can not be located in the main manuscript due to the lack of space.
A. E XPERIMENTAL R ESULTS ON A DDITIONAL R EAL - WORLD T IME - SERIES D ATASETS
The results on the M4 dataset (*) are multiplied by ten for readability.
The average value and the standard deviation value for ﬁve runs are reported.
[Table]
We evaluate the proposed method on the four large-scale real-world time-series datasets, the ETTh 1 ,
ETTh 2 , ETTm 1 , and ECL datasets in the main manuscript.
Additionally, this section provides experimental results on three more datasets, including two real-world datasets taken from the UCI repository, the air quality dataset and the Nasdaq dataset, and the M4 competition dataset (Makridakis et al., 2020).
In total, our proposed method is evaluated on seven datasets in this paper.
Air quality 3 dataset contains hourly averaged responses collected from ﬁve metal oxide chemical sensors located in Italy.
The data consist of 13 variables of length 9537.
We set the prediction length as { 24, 48, 168, 336, 720 } and the corresponding input length as { 48, 96, 168, 168, 360 } so that their ratio becomes { 2x, 2x, 1x, 0.5x, 0.5x } .
Nasdaq 4 dataset consists of 82 variables, including important indices of markets around the world, the price of major companies in the U.S. market, treasury bill rates, etc.
It is measured daily, having a total of 1984 data samples for each variable.
We prolong the prediction length as { 30, 60, 120 } and set the corresponding input length as 60 for all.
M4 5 competition dataset consists of six different hourly, daily, weekly, monthly, quarterly, and yearly sets, containing 100,000 test data.
We follow the original experimental protocol of the M competition (Makridakis et al., 2020).
For evaluation, we measure the micro-averaged mean absolute error and mean squared error:
we ﬁrst compute the metric independently for each set, i.e., hourly, daily, weekly, monthly, quarterly, and yearly sets, and then calculate the weighted average of the metrics the contributions of each set as the weights.
As shown in Table 4, RevIN signiﬁcantly improves the forecasting performance of the baselines on all three datasets.
Notably, RevIN shows outstanding performance on the Nasdaq dataset, reducing the prediction errors by more than half compared to the baselines.
Additionally, we conduct qualitative analysis on the Nasdaq dataset to verify the effectiveness of our method on obvious non-stationary time series.
As shown in Fig.
6, the Nasdaq index (the variable ’Close’) has steadily increased since 2010.
Accordingly, when data are divided into the training and test data based on a speciﬁc point in time (vertical dashed line in Fig. 6), the test data values tend to be higher than the training data values.
In other words, the data severely suffers from the distribution shift problem, where the training and test data show a discrepancy in their distribution.
As a result, even existing state-of-the-art networks cannot predict the future values appropriately, as shown in Fig.
The baseline fails to keep up with the trend of data, whose mean value continues to increase, and thus the prediction results are shifted.
However, RevIN mitigates such distribution discrepancy and remarkably increases the prediction performance of the baseline SOTA model.
Sim- ilarly, RevIN shows superior performance on the other rapidly increasing data (’DTB6’ in Fig. 6) and the decreasing test data (‘DE1’ in Fig. 6), accurately predicting the changing mean of the data.
A. C ROSS -D OMAIN T IME - SERIES F ORECASTING
[Table]
As RevIN can alleviate the distribution discrepancy between the training and test data, we study the capability of RevIN in mitigating distribution discrepancy between different domains, through crossdomain time-series forecasting task.
In time series, a domain can be a location of a sensor where the data is collected.
We use the ETT datasets for the experiment since they have the same feature categories and different data distributions; ETTh 1 and ETTh 2 are collected from other locations, and ETTh and ETTm 1 datasets have different measurement time intervals.
Thus, we alternately use each ETT dataset as a source domain for training and a target domain for testing.
The goal of crossdomain time-series forecasting is to alleviate the data distribution discrepancy between the source and target domains, e.g., between the ETTh 1 and ETTh 2 datasets.
In Table 5, despite the difference in the data distributions, RevIN shows remarkable performance in cross-domain time-series forecasting.
Especially, RevIN outperforms SCINet with a large margin when the model needs to reduce the discrepancy between the ETTh 2 and ETTm 1 datasets.
The results demonstrate that RevIN successfully solves the distribution shift problem by alleviating data distribution discrepancy between different domains, leading to better generalization performance.
A.
H YPERPARAMETER S ENSITIVITY A NALYSIS
We analyze the hyperparameter sensitivity of the proposed method in comparison with the baseline models.
Input sequence length can be a crucial hyperparameter to RevIN since the method computes the mean and the standard deviation across the entire input sequence length and use the statistics at informer - ours (input length 조절 ) scinet - ours (input length 조절 ) 5 run mean stdev 5 runmean stdev 5 runmean stdev inputlength MSE MAE MSE MAE MSE MAE N-BEATS +
[Table]
[Figure]
its normalization and denormalization layers. Thus, the input sequence length would play a crucial role in the prediction accuracy and stability of the training process. Accordingly, we prolong the model input sequence length to see its impact on the forecasting performance of RevIN, as shown in Fig.
As a result, RevIN consistently outperforms the baseline for various input sequence lengths. More importantly, RevIN makes the baseline models more robust to the input length.
In other words,
RevIN shows stable performance for various input lengths in contrast to the baseline models, which shows the high variance in their performance according to the input length.
Notably, in N-BEATS, the average error, as well as the standard deviation of the error, signiﬁcantly increase as prolonging the input length.
This is because the trend of data is expressed as a linear function in N-BEATS.
As the input length becomes prolonged, there is a chance that the variance (or non-stationarity) in the time-series values becomes higher, decreasing the accuracy of the linear trend to ﬁt the data unless the data is monotonically increasing or decreasing.
Also, the mispredicted trends linearly increase the error in the future values.
However, when N-BEATS adopts the RevIN layer, it removes the nonstationary statistics from the input, and thus, the model shows robust performance against the input length.
Removing the variability, i.e., normalizing its mean and standard deviation, before feeding it to the model and returning it to the output would make the model learning stable.
A. A BLATION S TUDY
[Table] We ablate the afﬁne transformation from RevIN to see its impact on the forecasting performance.
We conduct the analysis on the ETTh 1 , ETTh 2 , ETTm 1 , ECL, Nasdaq, and air quality datasets using N-BEATS as the baseline.
The results in Table 6 show that the afﬁne transformation contributes to performance improvement consistently on a variety of datasets.
As mentioned earlier, a model can add RevIN in an intermediate layer, even to several layers.
While existing approaches are a preprocessing-and-postprocessing method applied outside of the main prediction model, RevIN is an end-to-end trainable layer that can be added to any layer in the model as batch normalization (Ioffe & Szegedy, 2015) and instance normalization (Ulyanov et al., 2016), the recently proposed deep learning-based normalization layers.
Thus, we verify that adopting RevIN to the intermediate layers instead of the input and output layers can improve the forecasting performance as well.
We add RevIN at the ﬁrst stack of N-BEATS and SCINet and evaluate their performance on the six datasets.
The results in Table 7 demonstrate that even when added to the intermediate layers, RevIN improves the performance of the baselines as a learnable normalization layer.
We mainly focus on adding RevIN to the input and output of a model since it shows robust performance on average.
Nevertheless, the model adopting RevIN in the intermediate layers also shows the best performance frequently, consistently outperforming the baseline without RevIN.
This performance is even better than the dynamic normalization methods, LSTNet ∗ and ES-RNN ∗ , when comparing them where N-BEATS is used as the baseline (See Table 9 in Appendix A.6).
For example, when the prediction length is 960, the mean squared errors of LSTNet ∗ , ES-RNN ∗ , RevIN (inter.), and RevIN (i/o) are 5.627, 1.338, 0.523, and 0.471, on average.
In conclusion, RevIN is a ﬂexible, end-to-end trainable [Table] layer that can signiﬁcantly increase the performance of a model in time series forecasting, applied to any arbitrarily chosen layers.
A. P ERFORMANCE E VALUATION ON S IMILARITY M EASURE FOR T IME S ERIES
We evaluate the forecasting performance of RevIN mainly on the mean squared error and the mean absolute error.
Additionally, we use the complementary metrics that measure similarity between two sequences, dynamic time warping (DTW) and temporal distortion index (TDI) (Le Guen & Thome, 2019; Fr´ıas-Paredes et al., 2017).
Table 8 shows that our approach signiﬁcantly improves the baseline models across all datasets in terms of the DTW and TDI.
Notably, RevIN shows outstanding performance with a large margin compared to the baselines for long prediction length.
For example, when RevIN is added, the average DTW decreases from 38.348 to 15.240 for Informer, from 53. to 12.766 for N-BEATS, and from 20.498 to 11.080 for SCINet when the prediction length ison ETTh 2 .
There are few cases where the proposed method predicts a less similar sequence than the baseline, but the margin is minimal in terms of either DTW or TDI compared to the signiﬁcant margin found when our method outperforms the baseline.
These results demonstrate that the model adopting RevIN can generate a sequence more similar to the groundtruth than the baseline, especially showing a better prediction accuracy on the long sequence.
[Table] ETT
[Table]
[Table]
[Table]
[Table]
[Figure]
E S R
NN *
R e v I
N
L S T
N e t *
D e n s i t y D e n s i t y D e n s i t y
We compare RevIN with the dynamic normalization methods proposed in LSTNet (Lai et al., 2017) and ES-RNN (Smyl, 2020).
Similar to adding RevIN to the baseline model, we add the autoregressive linear bypass module of LSTNet and the modiﬁed Holt-Winters exponential smoothing of ES-RNN to the baseline model, respectively. As shown in Table 9, RevIN consistently achieves the best performance among the normalization methods adopted on N-BEATS, showing a signiﬁcant margin.
When we replace RevIN with the other normalization methods, the autoregressive linear bypass module of LSTNet (LSTNet ∗ ) also consistently reduces the prediction error compared to the baseline, but the performance improvement is smaller than our method. For example, when the prediction length is 960 on the ETTh 2 dataset, N-BEATS shows an average error of 6.408, and LSTNet ∗ reduces the error to 5.627. But it is still much worse than RevIN that reduces the error to 0.471. Similarly, when the prediction length is 1344 on the ETTm 1 dataset, the baseline shows an average error of 14.613 and LSTNet ∗ largely decreases the error to 5.592, but RevIN more signiﬁcantly decreases the error to 0.631.
In the case of ES-RNN ∗ , the training of the model is unstable, failing to converge for several cases. Also, ES-RNN ∗ often degrades the baseline performance, e.g., when the prediction length is either 24 or 48 on the ETTh 1 and ETTm 1 datasets.
However, it signiﬁcantly reduces the error for the long prediction length much better than LSTNet ∗ , but still worse than RevIN, e.g., when the prediction length is 960 on the ETTh 2 dataset.
Additionally, we further analyze the data distributions of the dynamic normalization methods on the ETTh 2 dataset, as shown in Fig.
We compare the training and test data distributions on each step of the sequential process in each method as follow
ES-RNN ∗ shows the distributions of (a) the original model input, (b) the normalized input where the level and seasonality are removed by its proposed method, (c) the model prediction output, (d) the denormalized output where the level and seasonality is multiplied back to the original distribution.
LSTNet ∗ shows the distributions of (a) the original model input, (b) the same original input since the method does not transform the input data before feeding them to the main prediction model, the model prediction output (c) before, and (d) after adding the output of the autoregressive network.
RevIN shows the distributions of (a) the original model input, (b) the normalized input by RevIN, (c) the model prediction output, and (d) the denormalized output by RevIN.
In Fig. 8(a), the original training and test data show a discrepancy in their distributions.
Also, they have several peaks, not being centered to the mean.
This implies that time series in the data would have different means from each other.
In Fig.
8(b), both RevIN and ES-RNN ∗ transform data distributions into mean-centered distributions.
Particularly, ES-RNN ∗ makes the distribution extremely concentrated on the mean with a small variance.
Both RevIN and ES-RNN ∗ make sequences in the data have similar statistics, thereby alleviating the distribution shift problem in the input data.
This would lead to competitive performance on the long prediction sequence in contrast to LSTNet ∗ .
It cannot resolve the distribution discrepancy because it does not have any module that can change the input statistics.
Also, in LSTNet ∗ , the distributions of the model output (Fig. 8(c)) completely differ from the input (Fig. 8(b)).
The method cannot make the input and output distribution to be consistent.
In addition, its proposed autoregressive model barely affects the model output distributions, as shown in Fig.
8(c-d); there is almost no difference between the model output and the ﬁnal output distributions.
Most importantly, the distributions of the ﬁnal output (Fig. 8(d)) signiﬁcantly differ from the original data (Fig. 8(a)) in LSTNet ∗ .
Similarly, although ES-RNN ∗ alleviates the distribution discrepancy in the input data, it fails to return the model output (Fig. 8(d))
back to the original distribution (Fig. 8(a)), especially for the test data.
These results imply that LSTNet ∗ and ES-RNN ∗ fail to learn the appropriate data distribution, and this would be the main reason for higher prediction error than
RevIN. On the other hand, in RevIN, the distributions of the ﬁnal output (Fig. 8(d)) are successfully restored to the original distributions (Fig. 8(a)).
Also, RevIN makes the input and output of the model maintain the consistent distributions as well as makes the training and test data be overlapped.
As a result, RevIN can show outstanding performance compared to the other normalization methods. [Table] ETT h [Table]
ETT m 0.
E C L 0.
A. A DDITIONAL R ESULTS ON C OMPARISON WITH E XISTING N ORMALIZATION M ETHODS
This section provides the additional results on comparison with other normalization methods, which can not be located in the main manuscript due to the lack of space.
The forecasting error of RevIN and existing normalization methods are evaluated on the ETTh 1 , ETTh 2 , ETTm 1 , and ECL datasets in Table 10.
RevIN consistently outperforms the other normalization methods across all datasets.
Interestingly, when the denormalization step is added to batch normalization (RevBN) as RevIN, the model better forecasts the long sequence than batch normalization (Batch norm), e.g., when the prediction length is 960.
The denormalization step of RevIN plays a critical role in improving the model performance by restoring the model prediction to the original distribution.
However, a denormalization step cannot be added to DAIN since it has the Hadamard multiplication operation at the last step, which is not reversible when the input and prediction sequence lengths are different.
These differences would be the reason for its worse performance compared to RevIN, even requiring more computational costs and a larger amount of model parameters.
A. A LGORITHM OF R EVERSIBLE I NSTANCE N ORMALIZATION
Algorithm 1 summarizes the procedure of the proposed approach.
Reversible instance normalization consists of the normalization (line 3-4) and denormalization layers (line 6-7).
It transforms the input and output of a model using identical statistics.
As RevIN is generally applicable, g θ in Algorithm(line 5) can be any arbitrary deep neural network.
Input : T x ∈ R 1 , the input sequence length; x ( i ) kt ∈ R 1 , the k -th feature at time step t of the i -th item in a mini-batch; γ, β ∈ R K , learnable parameters for RevIN; g θ , a module in the model parameterized by θ .
Output: γ, β , θ .
Compute µ T ← 1 T x (cid:80)
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
A. T HEORETICAL J USTIFICATION OF R EV IN A GAINST D ISTRIBUTION S HIFT P ROBLEM
Let x ( i ) ∈
R K × T x denotes a time series comprising K variables of length T x .
Consider a univariate case where K = 1 without the loss of generality.
Then, x ( i ) ∈ R T x denotes the i -th time series in the data.
Consider training and test data, whose distributions are denoted as P tra and P tst , respectively.
We consider a distribution shift problem where a discrepancy exists between the training and test data distributions (Du et al., 2021).
That is,
[Equation]
In our work, we consider the distribution shift problem in terms of the mean and the variance.
Then, the distribution shift problem can be redeﬁned as
[Equation]
Let’s assume that the given training and test data suffer from the distribution shift problem in terms of the mean and variance.
In order to solve this problem, RevIN ﬁrst normalizes a training sample x ( i ) ∼ P tra .
Mathematically, a training sample is transformed as
[Equation]
By the laws of expectation and variance,
[Equation]
By symmetry, this also holds for x ( i ) ∼ P tst , for all i .
Therefore, the mean and variance of the training and test data distributions become identical.
Thus, by deﬁnition (Eq. 5), the distribution shift problem is solved for the training and test data by the ﬁrst step of RevIN.
Given the normalized time-series data, the forecasting model parameterized by θ , f θ : R T x → R T y , predicts the corresponding subsequent future values as ˜ y
=
f θ (ˆ x ) .
Then, the denormalization step of RevIN returns the non-stationary information of the original data, i.e., E [ x ( i ) ] and Var[ x ( i ) ] in
Eq. 6, to the prediction output so that model does not have to reconstruct them from the normalized input.
In summary, the model prediction ˜
y ( i ) is denormalized as
[Equation]
By the laws of expectation and variance, the mean and variance of ˆ y ( i ) can be expressed as [Equation] and Var [Equation]
x [Equation] i ) ]
[Equation]
The denormalization step allows the mean and variance of the ﬁnal prediction values to be expressed as the difference from the input statistics.
Here, since the input data x ( i ) and the groundtruth future values are consecutive sequences, we can assume that their difference in the mean and variance can be expressed as Eq. 9 as well.
Under the assumption, the model adopting RevIN only needs to capture the difference from the input statistics, ∆ and λ , to accurately predict the statistics of the future values.
In conclusion, through the normalization and denormalization steps of RevIN, a model can focus on learning the offset from the input distribution to the output distribution by removing their common non-stationary statistics.
A. C ALCULATION D ETAILS ON F EATURE D IVERGENCE
This section explains how the feature divergence is computed in Section 4.2.3.
Following the previous work (Pan et al., 2018), we calculate the average feature divergence between training and test data using symmetric KL divergence, assuming that the output features of the model layer will follow a Gaussian distribution with mean µ and variance σ 2 .
Then, the equation for feature divergence of the k -th feature f k can be expressed as
[Equation]
where KL ( f Ak || f
Bk
[Equation]
log σ Bk
[Equation]
Ak
[Equation]
σ Ak
[Equation] A. A DDITIONAL EXPERIMENTAL D ETAILS
We train and evaluate the models using the following seeds: 12, 22, 32, 42, and 52.
The experiments using N-BEATS and Informer as the baseline are performed on NVIDIA TITAN RTX, and the experiments using SCINet are conducted on NVIDIA TITAN Xp.
Following the multivariate timeseries forecasting settings of the previous studies (Zhou et al., 2021; Liu et al., 2021), we select the input sequence length from two days (2d), 4d, 7d, 14d, 15d, 20d, 28d, 30d for the hourly datasets, i.e., the ETTh 1 , ETTh 2 , and ECL datasets, and from half day, 1d, 3.5d, 7d for the ETTm 1 dataset.
Particularly, we set the ratio of input length to prediction length to be smaller from 2.0 to 0.35 as the prediction length becomes longer.
In the case of SCINet, we set its input length as 736, unlike other baselines, when the prediction length is 720. It is because the original paper of the method requires its input sequence length to meet a speciﬁc condition. To be speciﬁc, the input length needs to be the multiple of 32 due to its hierarchical architecture (Liu et al., 2021).
A. R EPRODUCTION D ETAILS FOR B ASELINE M ODELS
This section describes the implementation details of the baselines, Informer, N-BEATS, and SCINet.
Note that we compare each baseline model and RevIN using the same hyperparameters except for the presence of RevIN.
We exactly follow the experimental settings of the baseline models by using their ofﬁcial code to conduct experiments except for N-BEATS that have no ofﬁcially released code.
We reproduce the model and set hyperparameters as stated in the original N-BEATS paper.
Informer.
We use the ofﬁcial open-source code of Informer 6 .
If provided, we follow the same hyperparameter settings in training the network, e.g., hidden dimension of the network or the learning rate.
As experimental details on the ECL dataset is not ofﬁcially provided, we train Informer using the same hyperparameter settings with the ETTh 1 dataset, where the details are given.
N-BEATS.
We reproduce N-BEATS using the PyTorch framework (Paszke et al., 2019).
We follow the same hyperparameter settings of the N-BEATS-I model in the original paper.
We train N-BEATS to minimize the mean squared error between the model prediction and groundtruth values.
For a fair comparison with the other baselines, we use a single model instead of using the ensemble method originally proposed in the N-BEATS paper.
Since N-BEATS is tailored to univariate time-series forecasting, we ﬂatten each multivariate input sequence into a univariate sequence having a single dimension for the feature before feeding it to the model.
Additionally, we conduct a grid search for the learning rate of N-BEATS with the range of [1e-5, 1e-3] and train the model using the weight decay with the factor of 0.001 to stabilize training.
SCINet.
We follow the experimental settings provided in the ofﬁcial code 7 of SCINet.
A. A DDITIONAL Q UALITATIVE R ESULTS
In Fig. 9, we illustrate the additional time-series forecasting results of RevIN and the baselines.
Overall, the prediction results of the baselines are inaccurately scaled and shifted.
However, RevIN shows remarkable performance, consistently improving the predictions of the baselines. Its predictions are precisely aligned with the groundtruth
[Table]
A. C OMPLETE Q UANTITATIVE R ESULTS
Table 11 provides the standard deviation values of ﬁve independent experiments to compare long sequence forecasting performance in Table 2 in the main manuscript.
Table 12 shows the complete results of comparison of the forecasting errors between the baselines and RevIN in Table 1 in the main manuscript. They include the standard deviation of ﬁve runs and the reported performance in the original papers of the baselines.
RevIN shows signiﬁcant performance improvement compared to the state-of-the-art forecasting baselines [Table] [Table] [Table]
E T T m
E C
L
E
TT h
E
TT h
T a b l e 12 :
C o m p l e t e re s u l t s f o r t h e c o m p a r
i s o n o f t h e f o rec a s t i n g err
o r s i n
T a b l ei n t h
e m a i n m a nu s cr i p t .
T h
e m ea n a nd s t
a nd a r d d e v
i a ti on o f t
h e
ﬁ v e i nd e p e nd e n t e xp e
r
i m e n t s a r e r ec o r d e d . † d e no t e s t h e r e po r t e d p e rf o r m a n ce s f o r t h e b a s e li n e s i n t h e i r
o r
i g i n a l p a p e r .
M e t hod s
I n f o r m e r †
I n f o r m e r
I n f o r m e r + R e v
I N
N - B
E A T
S
N - B
E A T
S + R e v
I N
S C
I N e t †
S C
I N e t
S C I N e t + R e
v I N
M e t r i c
M S E
M A E M S E M A E M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
ETTh
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
ETTm
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table]