WaveBound: Preventing Overfitting in Time Series Forecasting with Dynamic Error Bounds
Abstract
Time series forecasting has become a critical task due to its high practicality in real-world applications such as traffic, energy consumption, economics and finance, and disease analysis.
Recent deep-learning-based approaches have shown remarkable success in time series forecasting.
Nonetheless, due to the dynamics of time series data, deep networks still suffer from unstable training and overfitting.
Inconsistent patterns appearing in real-world data lead the model to be biased to a particular pattern, thus limiting the generalization.
In this work, we introduce the dynamic error bounds on training loss to address the overfitting issue in time series forecasting.
Consequently, we propose a regularization method called WaveBound which estimates the adequate error bounds of training loss for each time step and feature at each iteration.
By allowing the model to focus less on unpredictable data,
WaveBound stabilizes the training process, thus significantly improving generalization.
With the extensive experiments, we show that WaveBound consistently improves upon the existing models in large margins, including the state-of-the-art model.
Introduction
Time series forecasting has gained a lot of attention due to its high practicality in real-world applications such as traffic [1], energy consumption [2], economics and finance [3] and disease analysis [4].
Recent deep-learning-based approaches, particularly transformer-based methods [5, 6, 7, 8, 9], have shown remarkable success in time series forecasting.
Nevertheless, inconsistent patterns and unpredictable behaviors in real data enforce the models to fit in patterns, even for the cases of unpredictable incident and induce the unstable training.
In unpredictable cases, the model does not neglect them in training, but rather receives a huge penalty ( i.e. , training loss).
Ideally, small magnitudes of training loss should be presented for unpredictable patterns.
This implies the need for proper regularization of the forecasting models in time series forecasting.
Recently, Ishida et al.
[10] claimed that zero training loss introduces a high bias in training, hence leading to an overconfident model and a decrease in generalization.
To remedy this issue, they propose a simple regularization called flooding and explicitly prevent the training loss from decreasing below a small constant threshold called the flood level .
In this work, we also focus on the drawbacks of zero
[Figure]
training loss in time series forecasting.
In time series forecasting, the model is enforced to fit to an inevitably appearing unpredictable pattern which mostly generates a tremendous error.
However, the original flooding is not applicable to time series forecasting mainly due to the following two main reasons.
(i) Unlike image classification, time series forecasting requires the vector output of the size of the prediction length times the number of features.
In this case, the original flooding considers the average training loss without dealing with each time step and feature individually.
(ii)
In time series data, error bounds should be dynamically changed for different patterns.
Intuitively, a higher error should be tolerated for unpredictable patterns.
To properly address the overfitting issue in time series forecasting, the difficulty of prediction, i.e. , how unpredictable the current label is, should be measured in the training procedure.
To this end, we introduce the target network updated with an exponential moving average of the original network, i.e. , source network .
At each iteration, the target network can guide a reasonable level of training loss to the source network — the larger the error of the target network, the more unpredictable the pattern.
In current studies, a slow-moving average target network is commonly used to produce stable targets in the self-supervised setting [11, 12].
By using the training loss of the target network for our lower bound, we derive a novel regularization method called WaveBound which faithfully estimates the error bounds for each time step and feature.
By dynamically adjusting the error bounds, our regularization prevents the model from overly fitting to a certain pattern and further improves generalization.
Figure 1 shows the conceptual difference between the original flooding and
our
WaveBound method.
The originally proposed flooding determines the direction of the update step for all points by comparing the average loss and its flood level.
In contrast, WaveBound individually decides the direction of the update step for each point by using the dynamic error bound of the training loss.
The difference between these methods is further discussed in Section 3.
Our main contributions are threefold:
• We propose a simple yet effective regularization method called WaveBound that dynamically provides the error bounds of training loss in time series forecasting.
• We show that our proposed regularization method consistently improves upon the existing state-of-the-art time series forecasting model on six real-world benchmarks.
• By conducting extensive experiments, we verify the significance of adjusting the error bounds for each time step, feature, and pattern, thus addressing the overfitting issue in time series forecasting.
Preliminary 2.1 Time Series Forecasting
We consider the rolling forecasting setting with a fixed window size [5, 6, 7].
The aim of time series forecasting is to learn a forecaster g : R L × K → R M × K which predicts the future series y t
= { z t +1 , z t +2 , ..., z t + M : z i ∈ R K } given the past series x t = { z t − L +1 , z t − L +2 , ..., z t : z i ∈ R K } at time t where K is the feature dimension and L and M are the input length and output length, respectively.
We mainly address the error bounding in the multivariate regression problem where the input series x and output series y jointly come from the underlying density p ( x, y ) .
For a given loss function ℓ , the risk of g is R ( g ) :
=
E ( x,y ) ∼ p ( x,y )
[ ℓ ( g ( x ) , y )] .
Since we cannot directly access the distribution p , we instead minimize its empirical version
ˆ R ( g ) :
= 1 N (cid:80) Ni =1 ℓ ( g ( x i ) , y i ) using training data X := { ( x i , y i ) }
Ni =1 .
In the analysis, we mainly consider using the mean squared error (MSE) loss, which is widely used as an objective function in recent time series forecasting models [5, 6, 7].
Then, the risk can be rewritten as the sum of the risk at each prediction step and feature:
[Equation]
where R jk ( g ) :
=
E ( u,v ) ∼ p ( u,v ) (cid:2)
||
g jk ( u ) − v jk || 2 (cid:3) and ˆ R jk ( g ) :
= 1 N (cid:80) Ni =
|| g jk ( x i ) − ( y i )
jk || 2 .
2.2 Flooding
To address the overfitting problem, Ishida et al.
[10] suggested flooding , which restricts the training loss to stay above a certain constant.
Given the empirical risk ˆ R and the manually searched lower bound b , called the flood level , we instead minimize the flooded empirical risk, which is defined as
[Equation]
The gradient update of the flooded empirical risk with respect to the model parameters is performed as that of the empirical risk if ˆ R ( g ) > b and is otherwise performed in the opposite direction.
The flooded empirical risk estimator is known to provide a better approximation of the risk than the empirical risk estimator in terms of MSE if the risk is greater than b .
For the mini-batched optimization, a gradient update of the flooded empirical risk is performed with respect to the mini-batch.
Let ˆ R t ( g ) denote the empirical risk with respect to the t -th mini-batch for t ∈ { 1 , 2 , ..., T } .
Then, by Jensen’s inequality,
[Equation]
Therefore, the mini-batched optimization minimizes the upper bound of the flooded empirical risk.
Method
In this section, we propose a novel regularization called WaveBound which is specially-designed for time series forecasting.
We first deal with the drawbacks of applying original flooding to time series forecasting and then introduce a more desirable form of regularization.
3.1 Flooding in Time Series Forecasting
We first discuss how the original flooding may not effectively work for the time series forecasting problem.
We start with rewriting Equation (2) using the risks at each prediction step and feature:
[Equation]
Flooded empirical risk constrains the lower bound of the average of the empirical risk for all prediction steps and features by a constant value of b .
However, for the multivariate regression model, this regularization does not independently bound each ˆ R jk ( g ) .
As a result, the regularization effect is concentrated on output variables where ˆ R jk ( g ) greatly varies in training.
[Figure] [Figure]
In this circumstance, the modified version of flooding can be explored by considering the individual training loss for each time step and feature.
This can be done by subtracting the flood level b for each time step and feature as follows:
[Equation]
For the remainder of this study, we denote this version of flooding as constant flooding .
Compared to the original flooding that considers the average of the whole training loss, constant flooding individually constrains the lower bound of the training loss at each time step and feature by the value of b .
Nonetheless, it still fails to consider different difficulties of predictions for different mini-batches.
For constant flooding, it is challenging to properly minimize the variants of empirical risk in the batch-wise training process.
As in Equation 3, the mini-batched optimization minimizes the upper bound of the flooded empirical risk.
The problem is that the inequality becomes less tight as each flooded risk term ˆ R t ( g ) − b for t ∈ { 1 , 2 , ..., T } differs significantly.
Since the time series data typically contains lots of unpredictable noise, this happens frequently as the loss of each batch highly varies.
To ensure the tightness of the inequality, the bound for ˆ R t ( g ) should be adaptively chosen for each batch.
3.2 WaveBound
As previously mentioned, to properly bound the empirical risk in time series forecasting, the regularization method should be considered with the following conditions: (i)
The regularization should consider the empirical risk for each time step and feature individually.
(ii) For different patterns, i.e. , mini-batches, different error bounds should be searched in the batch-wise training process.
To handle this, we find the error bound for each time step and feature and dynamically adjust it at each iteration.
Since manually searching different bounds for each time step and feature at every iteration is impractical, we estimate the error bounds for different predictions using the exponential moving average (EMA) model [13].
Concretely, two networks are employed throughout the training phase: the source network g θ and target network g τ which have the same architecture, but different weights θ and τ , respectively.
The target network estimates the proper lower bounds of errors for the predictions of the source network, and its weights are updated with the exponential moving average of the weights of the source network:
[Equation]
where α ∈ [0 , 1] is a target decay rate.
On the other hands, the source network updates their weights θ using the gradient descent update in the direction of the gradient of wave empirical risk ˆ R wb
( g θ ) which is defined as
[Equation]
where ϵ is a hyperparameter indicating how far the error bound of the source network can be from the error of the target network.
Intuitively, the target network guides the lower bound of the training loss for each time step and feature to prevent the source network from training towards a loss lower than that bound, i.e. , overfitting to a certain pattern.
As the exponential moving average model is known to have the effect of ensembling the source networks and memorizing training data visible in earlier iterations [13], the target network can robustly estimate the error bound of the source network against the instability mostly caused by noisy input data.
Figure 2 shows how the source network and the target network perform in our WaveBound method.
A summary of WaveBound is provided in
Appendix B.
Mini-batched optimization.
For t ∈ { 1 , 2 , ..., T } , let ( ˆ R wbt ) jk ( g ) and ( ˆ R t )
jk
( g ) denote the wave empirical risk and the empirical risk at j -th step and k -th feature relative to the t -th mini-batch, respectively.
Given the target network g ∗ , by Jensen’s inequality,
[Equation]
Therefore, the mini-batched optimization minimizes the upper bound of wave empirical risk.
Note that if g is close to g ∗ , the values of ( ˆ R t ) jk ( g ) − ( ˆ R t ) jk ( g ∗ )
+ ϵ are similar across mini-batches, which gives a tight bound in Jensen’s inequality.
We expect the EMA update to work so that this condition is met, giving a tight upper bound for the wave empirical risk in the mini-batched optimization.
MSE reduction.
We show that the MSE of our suggested wave empirical risk estimator can be smaller than that of the empirical risk estimator given an appropriate ϵ .
Theorem 1.
Fix measurable functions g and g ∗ .
Let I :=
{ ( i, j ) : i = 1 , 2 , ..., M, j = 1 , 2 , ..., K } , and let J ( X ) :
= { ( i, j ) ∈
I : ˆ R ij ( g ) <
ˆ R ij ( g ∗ ) − ϵ } .
If the following two conditions hold:
(a) ∀ ( i, j ) , ( k, l ) ∈
I such that
( i, j ) ̸
= ( k, l ) , ˆ R ij ( g ) − ˆ R ij ( g ∗ ) ⊥ ˆ R kl ( g ) (b) ˆ R ij ( g ∗ ) <
R ij ( g ) + ϵ for all ( i, j
) ∈ J ( X ) almost surely
, then MSE ( ˆ R ( g )) ≥ MSE ( ˆ R wb ( g )) .
Given the condition (a), if we have 0 < α such that α <
R ij ( g ) − ˆ R ij ( g ∗ ) + ϵ for all ( i, j
) ∈ J ( X ) almost surely, then
[Equation]
Proof.
Please see Appendix A.
Intuitively, Theorem 1 states that the MSE of the empirical risk estimator can be reduced when the following conditions hold: (i)
The network g ∗ has sufficient expressive power so that the loss difference between g and g ∗ at each output variable is unrelated to the loss at the other output variables in g .
(ii) ˆ R ij ( g ∗ ) − ϵ likely lies in between ˆ R ij ( g ) and R ij ( g ) .
It is preferable to have g ∗ as the
EMA model of g since the training loss of the EMA model cannot be readily below the test loss of the model.
Then, ϵ can be chosen as a fixed small value so that the training loss of the source model at each output variable can be closely bounded below by the test loss at that variable.
Experiments 4.1 WaveBound with Forecasting Models
In this section, we evaluate our WaveBound method on real-world benchmarks using various time series forecasting models, including the state-of-the-art models.
[Table] [Table]
Baselines.
Since our method can be easily applied to deep-learning-based forecasting models, we evaluate our regularization adopted by several baselines including the state-of-the-art method.
For the multivariate setting, we select Autoformer [5], Pyraformer [6], Informer [7], LSTNet [14], and
TCN [16].
For the univariate setting, we additionally include N-BEATS [15] for the baseline.
Datasets.
We examine the performance of forecasting models in six real-world benchmarks.
The
Electricity Transformer Temperature (ETT) [7] dataset contains two years of data from two separate counties in China with intervals of 1-hour level (ETTh1, ETTh2) and 15 minutes level (ETTm1,
ETTm2) collected from electricity transformers.
Each time step contains an oil temperature and 6 additional features.
The Electricity (ECL) 2 dataset comprises 2 years of hourly electricity consumption of 321 clients.
The Exchange [14] dataset provides a collection of eight distinct countries on a daily basis.
The Traffic 3 dataset contains hourly statistics of various sensors in San Francisco Bay provided by the California Department of Transportation.
Road occupancy rate is expressed as a real number between 0 and 1.
The Weather 4 dataset records 4 years of data (2010-2013) of 21 meteorological indicators collected at around 1,600 landmarks in the United
[Figure] [Figure]
States.
The ILI 5 dataset contains data from the Centers for Disease Control and Prevention’s weekly reported influenza-like illness patients from 2002 to 2021, which describes the ratio of patients seen with ILI to the overall number of patients.
As in Autoformer [5], we set L = 36 and
M ∈ { 24 , 36 , 48 , 60 } for the ILI dataset, and set L = 96 and M ∈ { 96 , 192 , 336 , 720 } for the other datasets.
We split each dataset into train/validation/test as follows: 6:2:2 ratio for the ETT dataset and 7:1:2 ratio for the rest.
Multivariate Results.
Table 1 shows the performance of our method in terms of mean squared error (MSE) and mean absolute error (MAE) in the multivariate setting.
We can observe that our method consistently shows improvements for various forecasting models including the state-of-theart methods.
Notably, WaveBound improves both MAE and MSE of Autoformer on the ETTm dataset by 22.13% (0 . 262 → 0 . 204) in MSE and 12.57% (0 . 326 → 0 . 285) in MAE when M = 96 .
In particular, the performance is improved by 41.10% (0 .
→ 0 . 268) in MSE and 27.98% (0 .
→ 0 .
368) in MAE for LSTNet.
For long-term ETTm2 forecasting settings ( M = 720) ,
WaveBound improves the performance of Autoformer by 7.39% (0 . 446 → 0 . 413) in MSE and 6.20% (0 .
→ 0 . 408) in MAE.
In all experiments, our method consistently shows performance improvements with various forecasting models.
The results of full baselines and benchmarks are reported in Appendix D.
Univariate Results.
WaveBound also shows superior results in the univariate setting, as reported in
Table 2.
In particular, for N-BEATS, which is designed especially for univariate time series forecasting, our method improves the performance on the ETTm2 dataset by 8.22% (0 .
→ 0 . 067) in MSE and 5.05% (0 . 198 → 0 . 188) in MAE, when M = 96 .
For the ECL dataset, Informer with WaveBound shows improvements of 40.10% (0 .
631 → 0 . 378) in MSE and 24.35% (0 .
612 → 0 . 463)
in MAE when M = 720 .
The results of the full baselines and benchmarks are reported in Appendix D.
Generalization Gaps.
To identify overfitting, the generalization gap, which is the difference between the training loss and the test loss, can be examined.
To verify that our regularization truly prevents overfitting, we depict both the training loss and test loss for models with and without WaveBound in
Figure 3.
Without WaveBound, the test loss starts to increase abruptly, showing a high generalization gap.
In contrast, when using WaveBound, we can observe that the test loss continued to decrease, which indicates that WaveBound successfully addresses overfitting in time series forecasting.
[Table]
[Figure]
I n f o r m e r
4.2 Significance of Dynamically Adjusting Error Bounds
In WaveBound, the error bound is dynamically adjusted at each iteration for each time step and feature.
To validate the significance of such dynamics, we compare our WaveBound with the original flooding and constant flooding which use the constant values for flood levels, as introduced in Section 3.
Table 3 compares the performance of variants of flooding regularization with different surrogates to empirical risk.
The original flooding bounds the empirical risk by a constant while constant flooding bounds the risk at each feature and time step independently.
We searched the flood level b for the regularization methods with constant value b in space of { 0 . 00 , 0 . 02 , 0 .
As we expected, we cannot achieve the improvements when using a fixed constant value.
The models trained by individually bounding the error in each output variable outperform other baselines by a large margin, which concretely shows the effectiveness of our proposed WaveBound method.
The test error of different methods for each time step is depicted in Figure 4.
For all time steps, WaveBound shows an improved generalization compared to original flooding and constant flooding, which highlights the significance of adjusting the error bounds in time series forecasting.
4.3 Flatness of Loss Landscapes
The visualization of loss landscapes [17] is introduced to evaluate how the model adequately generalizes.
It is known that the flatter the loss landscapes of the model, the better the robustness and generalization [18, 19].
In this section, we depict the loss landscapes of models trained with and without WaveBound.
Figure 5 shows the loss landscapes of Autoformer and Informer.
We visualize the loss landscapes using filter normalization [17] and evaluate the MSE for every model for a fair comparison.
We can observe that the model with WaveBound shows smoother loss landscapes compared to that of the original model.
In other words, WaveBound flattens the loss landscapes of time series forecasting models and stabilizes the training.
Related Work
Time series forecasting.
For time series forecasting tasks, various approaches have been proposed based on different principles.
Statistical approaches can provide interpretability as well as a theoretical guarantee.
Auto-regressive Integrated Moving Average [20] and Prophet [21] are the most representative methods for statistical approaches.
Another important class of time series forecasting is the state space models [22, 23] (SSMs).
SSMs incorporate structural assumptions into the model and learn latent dynamics of the time series data.
However, due to its superior results in long-range forecasting, deep-learning-based approaches are mainly considered as the prominent solution for time series forecasting.
To model the temporal dependencies in time series data, recurrent neural networks (RNN) [24, 25, 26, 1] and convolutional neural networks (CNN) [14, 27] are introduced in time series forecasting.
Temporal convolutional networks (TCN) [28, 16, 29] are also considered for modeling temporal causality.
Approaches combining SSMs and neural networks have also been proposed.
DeepSSM [30] estimates state space parameters using RNN.
Linear latent dynamics have been efficiently modeled using a Kalman filter [31, 32], and methodologies to model non-linear state variables have been proposed [33].
Other recent approaches include using SSMs with Rao-blackwellised particle filters [34] or SSMs with a duration switching mechanism [35].
Recently, transformer-based models have been introduced in time series forecasting due to their ability to capture the long-range dependencies.
However, applying a self-attention mechanism increases the complexity of sequence length L from O ( L ) to O
( L 2 ) .
To alleviate the computational burden, several attempts such as LogTrans [8], Reformer [9], and Informer [7] re-designed the self-attention mechanism to a sparse version and reduced the complexity of the transformer.
Haixu et al.
[5] proposed the decomposition architecture with an auto-correlation mechanism called Autoformer to provide the series-wise connections.
To model the temporal dependencies of different ranges, the pyramidal attention module is proposed in Pyraformer [6].
However, we observe that these models still fail to generalize due to the training strategy that enforces models to fit to all inconsistent patterns appearing in real data.
In this work, we mainly focus on providing the adequate error bounds to prevent models from being overfitted to a certain pattern in the training procedure.
Regularization methods.
Overfitting is one of the critical concerns for the over-parameterized deep networks.
This can be identified by the generalization gap, which is the gap between the training loss and the test loss.
To prevent overfitting and improve generalization, several regularization methods have been proposed.
Decaying weight parameters [36], early stopping [37], and Dropout [38] have been commonly applied to avoid the high bias of deep networks.
In addition to these methods, regularization methods specially designed for time series forecasting have also been proposed [39, 40].
Recently, the flooding [10] has been introduced to explicitly prevent zero training loss .
By providing the lower bound of training loss, called the flood level , flooding allows the model not to completely fit to the training data, thus improving the generalization capacity of the model.
In this work, we also attempt to tackle the zero training loss in time series forecasting.
However, we find that bounding the average loss in time series forecasting does not perform as well as expected.
In time series forecasting, an appropriate error bound for each feature and time step should be carefully chosen.
In addition, a constant flood level may not be suitable to time series forecasting where the difficulty of prediction changes for every iteration in the mini-batch training process.
To handle such issues, we suggest a novel regularization which fully considers the nature of time series forecasting.
Conclusion
In this work, we propose a simple yet effective regularization method called WaveBound for time series forecasting.
WaveBound provides dynamic error bounds for each time step and feature using the slow-moving average model.
With the extensive experiments on real-world benchmarks, we show that our regularization scheme consistently improves the existing models including the state-of-the-art model, addressing overfitting in time series forecasting.
We also examine the generalization gaps and loss landscapes to discuss the effect of WaveBound in the training of over-parameterized networks.
We believe that the significant performance improvements of our method indicate that regularization should be designed specifically for time series forecasting.
We believe that our work will provide a meaningful insight into future research.
Acknowledgments and Disclosure of Funding
This work was supported by the Institute of Information & communications Technology Planning &
Evaluation (IITP) grant funded by the Korean government (MSIT)
(No. 2019-0-00075, Artificial
Intelligence Graduate School Program (KAIST)) and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (No. NRF-2022R1A2B5B02001913).
This work was also partially supported by NAVER Corp.
A Proof of Theorem
Theorem 1.
Fix measurable functions g and g ∗ .
Let I :=
{ ( i, j ) : i = 1 , 2 , ..., M, j = 1 , 2 , ..., K } , and let J ( X ) :
= { ( i, j ) ∈
I : ˆ R ij ( g ) <
ˆ R ij ( g ∗ ) − ϵ } .
If the following two conditions hold: (a) ∀ ( i, j ) , ( k, l ) ∈ I such that
( i, j ) ̸
= ( k, l ) , ˆ R ij ( g ) − ˆ R ij ( g ∗ ) ⊥ ˆ R kl ( g ) (b) ˆ R ij ( g ∗ ) <
R ij ( g ) + ϵ for all ( i, j
) ∈ J ( X ) almost surely, then MSE ( ˆ R ( g )) ≥ MSE ( ˆ R wb ( g )) .
Given the condition (a), if we have 0 < α such that α <
R ij ( g ) − ˆ R ij ( g ∗ ) + ϵ for all ( i, j
) ∈ J ( X ) almost surely, then
MSE (
[Equation] [Equation]
Proof.
We first show that if (a) and (b) hold, then MSE ( ˆ R ( g )) ≥ MSE ( ˆ R wb ( g )) .
MSE ( ˆ R ( g ))
− MSE ( ˆ R wb ( g )) =
[Equation]
−
[Equation]
g
[Equation]
− R
[Equation]
g )) 2 ] =
[Equation]
=
[Equation]
X 
[Equation]
(cid:88)
( i
[Equation]
) ∈
I 1 J (
[Equation] [Equation]
) (cid:16)
[Equation]
[Table]
[Table] [Table]
[Equation] [Equation]
[Equation]
= 0 .
[Equation]
From (b) and by the definition of J ( X ) , ˆ R ij ( g ) − ˆ R ij ( g ∗ ) + ϵ ≤ 0 and ˆ R ij ( g ∗ ) − R ij ( g ) − ϵ ≤for ( i, j ) ∈ J ( X ) with probability 1.
Therefore,
[Equation]
Assume that we have α > 0 such that α < R ij ( g ) − ˆ R ij ( g ∗ )
+ ϵ for all ( i, j
) ∈ J ( X ) with probability 1.
Let A be the event that α < R ij ( g ) − ˆ R ij ( g ∗ )
+ ϵ for all ( i, j
) ∈ J ( X ) .
Then,
[Equation]
B Pseudocode of WaveBound
In Section 3, we describe how wave empirical risk can be obtained in mini-batched optimization process.
In practice, WaveBound can be easily implemented by introducing an additional target network and slightly modifying the training objective in the optimization process as shown in
Algorithm 1.
Algorithm
Input do
{ ( x i , y i ) ∼ X} Bi =
▷ Sample a batch of size B 3: for j = 1 to M do 4: for k
= 1 to K do 5: for ( x i , y i )
∈ B do 6: l ijk ← ℓ (( g θ ) jk ( x i ) , ( y i ) jk )
▷ Compute the loss for g θ 7: r ijk ← ℓ (( g τ ) jk ( x i ) , ( y i ) jk )
▷ Compute the loss for g τ 8: end for 9: end for 10: end for 11: δθ ← 1 MK (cid:80) j,k ∂ θ (cid:12)(cid:12) 1 B (cid:80) i l ijk − 1 B (cid:80) i r ijk + ϵ (cid:12)(cid:12)
▷ Compute the loss gradient w.r.t.
θ 12: θ ← optimizer ( θ, δθ )
▷ update the source network parameter 13: τ ← ατ + (1 − α ) θ ▷ update the target network parameter 14: end for
C Detailed Experimental Settings
In the experiments, WaveBound is applied as the regularization technique in Autoformer [5],
Pyraformer [6], Informer [7], LSTNet [14], and TCN [16] for the multivariate settings.
For the univariate settings, N-BEATS [15] is additionally included.
As described in Section 4, we use six real-world benchmarks: ETT, Electricity, Exchange, Traffic, Weather, and ILI.
In both the multivariate and univariate settings, we train baselines with the batch size of 32 and Adam optimizer with β 1 = 0 .and
β 2 = 0 .
The learning rate is searched in { 0 . 00001 , 0 . 00003 , 0 . 0001 , 0 . 0003 , 0 . 001 } for each model.
In all experiments, we perform early stopping on the validation set.
For WaveBound, the target network g τ is updated with exponential moving average (EMA) of the source network g θ with a decay rate α = 0 .
When applying WaveBound to each baseline, the hyperparameter ϵ is searched in { 0 . 01 , 0 . 001 } .
We use a single TITAN RTX GPU in all experiments.
M u lti v a r i a t e
U n i v a r i a t e 96 predict 192 predict 336 predict 720 predict
When using WaveBound, the results are relatively smoother and have less lag.
D Additional Experimental Results
Quantitative Results.
The additional experimental results of our proposed WaveBound method for full benchamrks and baselines are reported in Table 14, 15, and 16.
Qualitative Results.
To illustrate the effects of WaveBound, we depict the predictions of different models on ETTm2 dataset (96 input).
For the multivariate setting, we plot the predictions for the last feature.
As shown in Figure 6, the models trained with WaveBound consistently predict accurately regardless of forecast step.
E Effects of ϵ in WaveBound
Table 4 shows the results for different hyperparameter values of ϵ which are searched in { 0 . 01 , 0 . 001 } .
The performance can be improved by selecting proper ϵ in each setting.
[Table]
(a) σ = 0.
(b) σ = 0.
(c) σ = 0.
O r i g i n a l w / W a v e B ound
F Performance Analysis on Synthetic Dataset
Synthetic dataset.
To analyze how WaveBound behaves in different noise levels and different dataset sizes, we synthesize a dataset composed of sine wave with the Gaussian noise.
Formally, for a given noise level σ and a time step t , the synthetic dataset is formed as: f ( t ) =
2 sin(2 πt/ 32) + sin(2 πt/ 48) + σz , where z follows the normal distribution.
Analysis on different noise levels and dataset sizes.
In time series forecasting, the overfitting issue becomes significant when the train data is small and the data contains a high level of noise.
Using the synthetic dataset, we examine the effectiveness of WaveBound in different levels of overfitting.
Specifically, we examine Informer [7] trained with and without WaveBound by varying the noise level σ and the datset size |D| of the synthetic dataset.
Table 5 and Table 6 shows the performance of
Informer with and without WaveBound in different noise levels and dataset sizes, respectively.
As the noise level increases and dataset size decreases, Informer struggles with the unpredictable noise and produces noisy predictions, resulting in poor performance while WaveBound improves performance in all noise levels and dataset sizes.
Figure 7 qualitatively shows that WaveBound allows the model to make stable predictions even in the high level of noise.
σ =
[Table]
σ =
[Table]
σ
[Table] [Table]
G Impact of WaveBound on Individual Features and Time Steps
To properly address an overfitting issue, WaveBound estimates adequate error bounds for each feature and time step individually.
To justify the assumptions made by WaveBound, we depict the training curve of each feature and time step of the prediction made by Autoformer on the ETTm2 dataset.
Each row and column in Figure 8 denote one feature and one time step, respectively.
As we assumed, we can observe that different features show different training dynamics, which indicates that the difficulty of prediction varies through the features.
Moreover, the magnitude of error increases as the forecast step increases.
Those observations highlight that different regularization strategies are required for different features and time steps.
We also observe that existing regularization methods such as early-stopping cannot properly handle the overfitting issue, as it considers only the aggregated value rather than considering each error individually.
Red dotted lines in Figure 8 denote the iteration of the best checkpoint found by early-stopping.
Interestingly, even before the best checkpoint is decided by the early-stopping (red dotted lines), the model seems to be overfitted for several features; the train error (green line) is decreasing, but the test error (black line) is increasing.
We conjecture that this is because of the property of the early-stopping, which considers only the mean value of error rather than individual errors of each feature and time step.
In contrast, WaveBound retains the test loss (red lines) at a low level by preventing the train error (blue lines) from falling below a certain level for each feature and time step.
In practice, this results in an overall performance improvement in time series forecasting, as shown in our extensive experiments.
H WaveBound with RevIN
Reversible instance normalization (RevIN) [41] is recently proposed to mitigate the distribution shift problem in time series forecasting.
In this section, we conduct the additional experiments by applying both WaveBound and RevIN in training.
As shown in Table 7, the performance of Autoformer [5] and
Informer [7] is improved with a significant margin when both WaveBound and RevIN are applied.
[Table]
I WaveBound in Short-Term Series Forecasting
In this section
[Table]
we examine WaveBound in short-term series forecasting.
Specifically, we use
Autoformer [5] as the baseline and train the model to forecast 3,
[Table]
and 12 steps on ECL, exchange
12 step
F ea t u r e # F ea t u r e #
F ea t u r e # F ea t u r e #
F ea t u r e
step 48 step 96 step rate, and traffic dataset.
Table 8 shows that WaveBound still achieves the performance improvements even in short-term series forecasting
[Table] [Table]
J WaveBound in Spatial-Temporal Time Series Forecasting
To examine the applicability of our WaveBound method in spatial-temporal time series forecasting
[Table]
we examine Graph WaveNet [42] trained with and without WaveBound on a widely used traffic dataset, METR-LA [1].
As in Wu et al.
[42], Graph WaveNet is trained with the mean absolute error and attempt to predict a future traffic speed of 1 hour given a historical data of 1 hour along with a graph structure.
Table 9 shows that WaveBound improves the performance of Graph WaveNet in terms of all metrics, mean absolute error (MAE), root mean squared error (RMSE), and mean absolute precentage error (MAPE).
[Table]
K Ablation Study on EMA Methods
Ablation study on the effect of EMA methods is shown in the Table 10.
In the experiments, models only with EMA update (Without Bound) shows the slight performance improvements compared to the original model.
However, EMA with dynamic error bound (WaveBound (Indiv.)) outperforms other baselines with a large margin, which concretely shows the effectiveness of individual error bounding in WaveBound.
L WaveBound with Small-Scale Model
We also evaluate the performance of WaveBound with the small-scale model.
Specifically, we train a multi layer perceptron (MLP) model consisting of three linear layers with and without WaveBound in the univariate setting.
Table 11 shows that WaveBound improves the performance of this MLP model.
[Table] [Table]
M WaveBound with Generative Model
To show the applicability of WaveBound when training a model with a loss function other than MSE we mainly considered in Section
[Table]
we conduct an experiment with a generative model DeepAR [43] which uses a negative log likelihood (NLL) as a training objective.
Table 12 shows that WaveBound improves the performance of DeepAR in the univariate setting.
Subscripts denote standard deviations.
[Table]
N Computational Cost Analysis
WaveBound introduces additional computation and memory costs in the training phase as it employs the EMA model during training.
In this section, we analyze the computation and memory cost of
WaveBound on the ETTm2 dataset.
The input and output lengths are fixed to 96 and a single TITAN
RTX GPU is used for all experiments.
Table 13 shows the computation time per epoch and maximum
GPU memory occupied in the training phase.
We also remark that WaveBound does not introduce the additional computation and memory costs in the inference phase.
[Table]
O Limitations
While we observe the success of WaveBound with various baselines on real-world benchmarks, our method inevitably introduces the additional cost in the training procedure since our method utilizes the exponential moving average network for estimating the difficulty of predictions.
Note that our method performs as the regularization technique in training and does not introduce any additional cost during inference.
P Potential Societal Impact
Our proposed method does not introduce any foreseeable societal consequence.
The real-world benchmarks used in this work are widely used for research purposes.
T a b l e 14 :
R e s u lt s o f W a v e
B ound i n t h
e m u lti v a r i a t
e s e tti ng .
A ll r e s u lt s a r e a v e r a g
e d ov e r 3 t r i a l s .
S ub s c r i p t s d e no t e s t a nd a r d d e v i a ti on s .
M od e l s
A u t o f o r m e r [ 5 ]
P y r a f o r m e r [ 6 ]
I n f o r m e r [ 7 ]
L S T N
[Table]
t []
[Table] [Table]
O r i g i n w / O u r s O r i g i n w / O u r
[Table]
O r i g i n w /
[Table]
u r s
O r i g i n w / O u r s
[Table]
r i
[Table]
i n
[Table]
M
[Table]
M
[Table]
M
[Table]
ETTm
[Table]
T a b l e
R e s u lt s o f W a v e
B ound i n t h
e m u lti v a r i a t
e s e tti ng
A ll r e s u lt s a r e a v e r a g
e d ov e r 3 t r i a l s
S ub s c r i p t s d e no t
[Table]
nd
[Table]
d e v i a
[Table]
Traffic
[Table]
ILI
[Table]
T a b l e 16 :
R e s u lt s o f W a v e
B oun
d i n t h e un i v a r i a t e s e tti ng .
A ll r e s u lt s a r e a v e r a g e d ov e rt r i a l s .
S ub s c r i p t s d e no t e s t a nd a r d d e v i a ti on s .
M od e l s
A u t o f o r m e r [ 5 ]
P y r a f o r m e r [ 6 ]
I n f o r m e r [ 7 ]
N - B E
A T S [ 15 ]
L S T N e t []
T C N [ 16 ]
O r i g i n w / O u r s O r i g i n w / O u r s
O r i g i n w / O u r s O r i g i n w / O u r s
O r i g i n w / O u r s O r i g i n w / O u r s
M e t r i c
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
M S E
M A E
ETTm
[Table]