WaveBound: Dynamically Bounding Error for Stable Time Series Forecasting
Anonymous Author(s)
AffiliationAddress email
Abstract
Time series forecasting becomes a critical task due to its high practicality in realworld applications such as traffic, energy consumption, economics and finance, anddisease analysis.
Recent deep-learning-based approaches have shown remarkablesuccess in time series forecasting.
Nonetheless, due to the dynamics of time seriesdata, deep networks still suffer from unstable training and overfitting.
Inconsistentpatterns appearing in real-world data lead the model to be biased to a particularpattern, thus limiting the generalization.
In this work, we introduce the dynamicerror bounds on training loss to address the overfitting issue in time series forecasting.
Consequently, we propose a regularization method called WaveBound whichestimates the adequate error bounds of training loss for each time step and featureat each iteration.
By allowing the model to focus less on unpredictable data, WaveBound stabilizes the training process, hence significantly improving generalization.
With the extensive experiments, we show that WaveBound consistently improvesthe existing models in large margins, including the state-of-the-art model.
Introduction
Time series forecasting gains a lot of attention due to its high practicality in real-world applicationssuch as traffic [1], energy consumption [2], economics and finance [3] and disease analysis [4].
Recent deep-learning-based approaches, particularly Transformer-based methods [5, 6, 7, 8, 9],have shown remarkable success in time series forecasting.
Nevertheless, inconsistent patterns andunpredictable behaviors in real data enforce the models to fit in patterns even for the cases ofunpredictable incident and induce the unstable training.
In unpredictable cases, the model doesnot neglect them in training, but rather receives a huge penalty ( i e. , training loss).
Ideally, smallmagnitudes of training loss should be presented for unpredictable patterns.
This implies the need forproper regularization of the forecasting models in time series forecasting.
Recently, Ishida et al.
[10] claimed that zero training loss introduces a high bias in training, henceleading to an overconfident model and a decrease in generalization.
To remedy this issue, theyproposed a simple regularization called flooding and explicitly prevented the training loss fromdecreasing below the small constant threshold called flood level .
In this work, we also focus on thedrawbacks of zero training loss in time series forecasting.
In time series forecasting, this enforces themodel to fit to an inevitably appearing unpredictable pattern which mostly generates a tremendouserror.
However, the original flooding is not applicable to time series forecasting due to the followingtwo main reasons: (i) Unlike image classification, time series forecasting requires the vector outputof the size of the prediction length times the number of features.
In that case, the original flooding
[Figure]
In time series data, error bounds should be dynamically changed for different patterns.
Intuitively, ahigher error should be tolerated for unpredictable patterns.
To properly address the overfitting issue in time series forecasting the difficulty of prediction, i.e. ,how unpredictable the current label is, should be measured in the training procedure.
To this end, weintroduce the target network updated with an exponential moving average of the original network, i.e. ,source network .
At each iteration, the target network can guide the reasonable level of training loss tothe source network — the larger the error of the target network, the more unpredictable the pattern is.
In recent, a slow-moving average target network is commonly used to produce stable targets in theself-supervised setting [11, 12].
By using the training loss of the target network for our lower bound,we derive a novel regularization method called WaveBound which faithfully estimates the errorbounds for each time step and feature.
By dynamically adjusting the error bounds, our regularizationprevents the model from overly fitting to a certain pattern and further improves the generalization.
Figure 1 shows the conceptual difference between the original flooding and our WaveBound.
Theoriginally proposed flooding determines the direction of the update step for all points by comparingthe average loss and its flood level.
In contrast, our WaveBound individually decides the directionof the update step for each point by using dynamic error bound of the training loss.
The differencebetween these methods will be more discussed in Section 3.
Our main contributions are threefold:
• We propose a simple yet effective regularization that dynamically provides the error bound oftraining loss in time series forecasting.•
We show that our WaveBound consistently improves the existing state-of-the-art time seriesforecasting model on six real-world benchmarks.• With the extensive experiments, we verify the significance of adjusting error bounds for eachtime step, feature, and pattern, thus addressing the overfitting issue in time series forecasting.
Preliminary
2.1 Time Series Forecasting
We consider the rolling forecasting setting with a fixed window size [7, 5, 6].
The aim of timeseries forecasting is to learn a forecaster g : R L × K → R M × K which predicts the future seriesy t = { z t +1 , z t +2 , ..., z t + M : z i ∈ R K } given the past series x t = { z t − L +1 , z t − L +1 , ..., z t : z i ∈ 62 R K } at time t where K is the feature dimension, L and M are the input length and output length,respectively.
We mainly address the error bounding in the multivariate regression problem where theinput series x and output series y jointly come from the underlying density p ( x, y ) .
For given a lossfunction ℓ , the risk of g is R ( g ) :
=
E ( x,y ) ∼ p ( x,y )
[ ℓ ( g ( x ) , y )] .
Since we cannot directly access thedistribution p , we instead minimize its empirical version
ˆ R ( g ) :
= 1 N (cid:80) Ni =1 ℓ ( g ( x i ) , y i ) using trainingdata X := { ( x i , y i ) }
Ni =1 .
In the analysis, we mainly consider to use mean square error (MSE) loss, which is widely used as an objective function in recent time-series forecasting models.
Then, the riskcan be rewritten as the sum of the risk at each prediction step and feature:
[Equation]
where R jk ( g ) :
=
E ( u,v ) ∼ p ( u,v ) (cid:2)
||
g jk ( u ) − v jk || 2 (cid:3) and ˆ R jk ( g ) :
= 1 N (cid:80) Ni =
|| g jk ( x i ) − ( y i )
jk || 2 .
2.2 Flooding
To address the overfitting problem, Ishida et al.
[10] suggested flooding , which restricts the trainingloss to stay above a certain constant.
Given the empirical risk ˆ R and the manually searched lowerbound b , called flood level , we instead minimize the flooded empirical risk, which is defined as
[Equation]
That is, the gradient update of flooded empirical risk with respect to the model parameters is performedas
that of the empirical risk if ˆ R ( g ) > b
and is performed in the opposite direction otherwise.
Theflooded empirical risk estimator is known to provide a better approximation of the risk than theempirical risk estimator in terms of MSE unless the risk is greater than b .
For the mini-batched optimization problem, this perform gradient update of the flooded empirical riskwith respect to the mini-batch.
Let ˆ R t ( g ) denote the empirical risk with respect to t -th mini-batch fort ∈ { 1 , 2 , ..., T } .
Then, by the Jensen’s inequality,
[Equation]
Therefore, the mini-batched optimization minimizes the upper bound of the flooded empirical risk.
Method
In this section, we propose a novel regularization called WaveBound which is specially-designed fortime series forecasting.
We first deal with the drawbacks of applying original flooding to time seriesforecasting and then introduce more desirable form of regularization.
3.1 Flooding in Time Series Forecasting
We first discuss that the original flooding may not effectively work for the time series forecastingproblem.
We start with rewriting Equation (2) using the risks at each prediction step and feature:
[Equation]
That is, flooded empirical risk constraints the lower bound of the average of the empirical risk for allprediction steps and features by a constant value of b .
However, for the multivariate regression model,this regularization does not independently bound each ˆ R jk ( g ) .
As a result, the regularization effect isconcentrated on output variables where ˆ R jk ( g ) greatly varies in training.
In this circumstance, the modified version of flooding can be explored by considering the individualtraining loss for each time step and feature.
This can be done by subtracting the flood level b for each especially useful in the analysis of estimation error.
[Figure]
time step and feature as follows:
[Equation]
In the rest of the paper, we denote this version of flooding as constant flooding .
Compared to theoriginal flooding that considers the average of the whole training loss, constant flooding individuallyconstraints the lower bound of training loss at each time step and feature by the value of b .
Nonetheless, it still fails to consider different difficulties of predictions for different mini-batches.
For constant flooding, it is challenging to properly minimize the variants of empirical risk in thebatch-wise training process.
As in Equation 3, the mini-batched optimization minimizes the upperbound of flooded empirical risk.
The problem is that the inequality becomes less tight as each floodedrisk term ˆ R t ( g ) − b for t ∈ { 1 , 2 , ..., T } are largely different.
Since the time series data typicallycontains lots of unpredictable noise, this happens frequently as the loss of each batch highly varies.
To ensure the tightness of the inequality, bound for ˆ R t ( g ) should be adaptively chosen for each batch.
3.2 WaveBound
As mentioned earlier, to properly bound the empirical risk in time series forecasting, the regularizationmethod should be considered with the following conditions: (i)
The regularization should consider theempirical risk for each time step and feature individually.
(ii) For different patterns, i.e. , mini-batches,different error bounds should be searched in the batch-wise training process.
To handle this, wesearch the error bound for each time step and feature and dynamically adjust it at each iteration.
Sincemanually searching different bounds for each time step and feature at every iteration is impractical,we estimate the error bounds for different predictions using the slow-moving average model [13].
Concretely, we employ two networks throughout the training phase: source network g θ and targetnetwork g τ which have the same architecture, but have different weights θ
and τ .
The target networkestimates the proper lower bounds of errors for the predictions of the source network, and its weightsare updated with exponential moving average of the weights of the source network:
[Equation]
where α ∈ [0 , 1] is a target decay rate.
On the other hands, the source network updates their weightsθ using the gradient descent update in the direction of the gradient of wave empirical risk ˆ R wb
( g θ )which is defined as
[Equation]
where ϵ is a hyperparameter indicating how far the error bound of the source network can be from theerror of the target network.
Intuitively, the target network guides the lower bound of the training lossfor each time step and feature so that prevents the source network from training towards a loss lowerthan that bound, i.e. , overfitting to a certain pattern.
As the slow-moving average model is known tohave the effect of ensembling the source networks and memorizing training data visible in earlieriterations [13], the target network can robustly estimate the error bound of the source network againstthe instability mostly caused by noisy input data.
Figure 2 shows how the source network and thetarget network perform in our WaveBound.
Please see Appendix for the pseudo-code of WaveBound.
Mini-batched optimization.
For t ∈ { 1 , 2 , ..., T } , let ( ˆ R wbt ) jk ( g ) and ( ˆ R t )
jk
( g ) denote the waveempirical risk and the empirical risk at j -th step and k -th feature with respect to t -th mini-batch,respectively.
Given the target network g ∗ , by the Jensen’s inequality,
[Equation]
Therefore, the mini-batch optimization minimizes the upper bound of wave empirical risk.
Notethat if g is close to g ∗ , the values of ( ˆ R t ) jk ( g ) − ( ˆ R t ) jk ( g ∗ ) + ϵ are similar across mini-batches,which gives a tight bound in the Jensen’s inequality.
We expect the EMA update to work so thatthis condition is met, giving a tight upper bound for the wave empirical risk in the mini-batchedoptimization.
MSE reduction.
We show that the mean squared error (MSE) of our suggested wave empirical riskestimator can be smaller than that of the empirical risk estimator for given an appropriate ϵ .
Theorem 1.
Fix measurable functions g and g ∗ .
Let I :=
{ ( i, j ) : i = 1 , 2 , ..., M, j = 1 , 2 , ..., K } ,and let J ( X ) :
= { ( i, j ) ∈
I : ˆ R ij ( g ) <
ˆ R ij ( g ∗ ) − ϵ } .
If the following two conditions hold:
[Equation]
i
[Equation]
( k, l ) ∈ I such that ( i
[Equation]
(b) ˆ R ij ( g ∗ ) <
R ij ( g ) + ϵ for all ( i, j
) ∈ J ( X ) almost surely.
, then MSE ( ˆ R ( g )) ≥ MSE ( ˆ R wb ( g )) .
Given the condition (a), if we have 0 < α such that α <
R ij ( g ) − ˆ R ij ( g ∗ ) + ϵ for all ( i, j
) ∈ J ( X ) almost surely, then
[Equation]
Proof.
Please see Appendix.
Intuitively, Theorem 1 states that the MSE of empirical risk estimator can be reduced when thefollowing conditions hold: (i)
The network g ∗ has sufficient expressive power so that the lossdifference between g and g ∗ at each output variable is not related to the loss at other output variablesin g . (ii) ˆ R ij ( g ∗ ) − ϵ is likely lie in between ˆ R ij ( g ) and R ij ( g ) . It is a nice choice to have g ∗ as the EMA model of g since the training loss of the EMA model cannot be readily below the test loss ofthe model.
Then, ϵ can be chosen as a fixed small value so that the training loss of the source modelat each output variable can be closely bounded below by the test loss at that variable.
Experiments
4.1 WaveBound with Forecasting Models
In this section, we evaluate our WaveBound on real-world benchmarks using various time seriesforecasting models, including the state-of-the-art models.
Baselines.
Since our method can be easily applied to deep-learning-based forecasting models, weevaluate our regularization adopted by several baselines including the state-of-the-art method.
Forthe multivariate setting, we select Autoformer [5], Pyraformer [6], Informer [7], LSTNet [14], and
TCN [16].
For the univariate setting, we additionally include N-BEATS [15] for the baseline. [Table] [Table] [Table]
ETT
[Table]
Datasets.
We examine the performance of forecasting models in six real-world benchmarks.
ETT (Electricity Transformer Temperature) [7] dataset contains two years of data from two separate counties in China with intervals of 1-hour level (ETTh1, ETTh2) and 15 minutes level (ETTm1, ETTm2) collected from electricity transformers.
Each time step contains an oil temperature and other 6 additional features. (2) Electricity 2 dataset comprises 2 years of hourly electricity consumption of 321 clients.
(3) Exchange [14] dataset provides the collection of eight distinct countries on a daily basis.
(4) Traffic 3 dataset contains hourly statistics of various sensors in San Francisco Bay provided by the California Department of Transportation.
Road occupancy rate is expressed as a real number between 0 and 1.
The values are recorded for each hour with a total collection of 48 months (2015-2016).
(5) Weather 4 records 4 years of data (2010-2013) of 21 meteorological indicators collected at around 1,600 landmarks in the United States. (6) ILI 5 contains data from the ca [Figure] [Figure]
Compared with flooding and constant flooding, our Centers for Disease Control and Prevention’s weekly reported influenza-like illness patients from2002 to 2021, which describes the ratio of patients seen with ILI to the overall number of patients.
As in Autoformer [5], we set L = 36 and M ∈ { 24 , 36 , 48 , 60 } for ILI dataset, and set L = 96 and M ∈ { 96 , 192 , 336 , 720 } for other datasets.
We split each dataset into train/validation/test as follows:6:2:2 ratio for the ETT dataset and 7:1:2 ratio for the rest.
Multivariate Results.
The results of our method in the multivariate setting are shown in Table 1.
Wecan observe that our method consistently shows the improvements for various forecasting modelsincluding the state-of-the-art method.
Notably, our WaveBound improves both MAE and MSE of
Autoformer on ETTm1 dataset by 22.13% (0 . 262 → 0 . 204) in MSE and 12.57% (0 .
326 → 0 .
285)in MAE when M = 96 .
In addition, our method consistently shows improvements in variousconfigurations.
For long-term ETTm1 forecasting settings ( M = 720) , WaveBound improves theperformance of Autoformer by 7.39% (0 .
→ 0 . 413) in MSE and 6.20% (0 .
→ 0 . 408) in
MAE.
For LSTNet, the performance is improved by 41.10% (0 .
→ 0 . 268) in MSE and 27.98%(0 . 511 → 0 .
368) in MAE.
On weather dataset, WaveBound improves the performance of Pyraformerby 17.87% (0 .
→ 0 . 193) in MSE and 15.26% (0 . 321 → 0 . 272) in MAE when M = 96 .
Theresults of full baselines and benchmarks are reported in Appendix.
Univariate Results.
WaveBound also shows the superior results in the univariate setting as reportedin Table 2.
In particular, for N-BEATS, which is designed especially for univariate time seriesforecasting, our method improves the performance on ETTm1 dataset by 8.22% (0 .
→ 0 . 067)in MSE and 5.05% (0 . {+
+} → 0 . 188) in MAE, when M = 96 .
For ECL dataset, Informer with
WaveBound shows improvements of 40.10% (0 . 631 → 0 . 378) in MSE and 24.35% (0 .
612 → 0 . 463)in MAE when M = 720 .
The results of full baselines and benchmarks are reported in Appendix.
Generalization Gaps.
To identify overfitting, the generalization gap which is the difference betweenthe training loss and the test loss can be examined.
To verify that our regularization truly preventsoverfitting, we depict both the training loss and test loss for models with and without WaveBound in
Figure 3.
Without WaveBound, the test loss starts to increase abruptly, showing a high generalizationgap.
In contrast, when using WaveBound, we can observe that the test loss continued to decrease,which indicates that WaveBound successfully addresses overfitting in time series forecasting. [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
WaveBound flattens the loss landscapes for both models, improving the generalization of models.
4.2 Significance of Dynamically Adjusting Error Bounds
In WaveBound, the error bound is dynamically adjusted at each iteration for each time step and feature.
To validate the significance of such dynamics, we compare our WaveBound with the original floodingand constant flooding which use the constant values for flood levels as introduced in Section 3.
In Table 3, the training objective, i.e. , the empirical risk to minimize, for each method is declaredwithout the constant term.
Note that the original flooding considers the total value of risk for eachbatch while constant flooding and WaveBound consider each feature and time step independently.
We searched the flood level b for the regularization methods with constant value b in space of{ 0 . 00 , 0 . 02 , 0 .
As we expected, we cannot achieve the improvements when using a fixedconstant value.
The test error of different methods for each time step is depicted in Figure 4.
For alltime steps, our WaveBound shows a better generalization compared to original flooding and constantflooding, which highlights the significance of adjusting error bounds in time series forecasting.
4.3 Flatness of Loss Landscapes
The visualization of loss landscapes [17] is introduced to evaluate how the model generalizes well.
It is known that the flatter the loss landscapes of the model implies the better the robustness andgeneralization [18, 19].
In this section, we depict the loss landscapes of models trained with andwithout WaveBound.
Figure 5 shows the loss landscapes of Autoformer with and without ourregularization.
We visualize the loss landscapes using filter normalization [17] and evaluate meansquared error (MSE) for every model for a fair comparison.
We can observe that the model with
WaveBound shows smoother loss landscapes compared to that of the original model.
That is, our
WaveBound flattens the loss landscapes of time series forecasting models and stabilizes the training.
Related Work
Time series forecasting.
For time series forecasting tasks, various approaches based on differentprinciples have been proposed.
Statistical approaches can provide the interpretability as well as the theoretical guarantee.
Autoregressive integrated moving average (ARIMA) [20] and Prophet [21] arethe most representative methods for statistical approaches.
Another important class of time seriesforecasting is State Space Models [22, 23] (SSMs).
SSMs incorporate structural assumptions intothe model and learn latent dynamics of the time series data.
However, due to its superior resultsin long-range forecasting, deep-learning-based approaches are mainly considered as the prominentsolution for time series forecasting.
To model the temporal dependencies in time series data, recurrentneural networks (RNN) [24, 25, 26, 1] and convolutional neural networks (CNN) [14, 27] areintroduced in time series forecasting.
Temporal convolutional networks (TCN) [28, 16, 29] are alsoconsidered for modeling temporal causality. Approaches combining SSMs and neural networkshave also been proposed. DeepSSM [30] estimates state space parameters using RNN. Linear latentdynamics have been efficiently modeled using Kalman filter [31, 32], and methodologies to modelnon-linear state variables have been proposed [33].
Other recent approaches include using SSMs with
Rao-blackwellised particle filters [34] or SSMs with duration switching mechanism [35].
Recently, Transformer-based models are introduced in time series forecasting due to its abilityto capture the long-range dependencies.
However, applying self-attention mechanism increasesthe complexity of sequence length L from O ( L ) to O
( L 2 ) .
To alleviate the computation burden,several attempts such as LogTrans [8], Reformer [9], and Informer [7] re-design the self-attentionmechanism to a sparse version and reduce the complexity of Transformer.
Haixu et al.
[5] proposedthe decomposition architecture with Auto-Correlation mechanism called Autoformer to provide theseries-wise connections.
To model the temporal dependencies of different ranges, the pyramidalattention module is proposed in Pyraformer [6].
However, we observe that these models still fail togeneralize due to the training strategy that enforces models to fit to all inconsistent patterns appearingin real data.
In this work, we mainly focus on providing the adequate error bounds to prevent modelsfrom being overfitted to a certain pattern in the training procedure.
Regularization methods.
Overfitting is one of the critical concerns for the over-parameterized deepnetworks.
This can be identified by the generalization gaps, which is the gap between the trainingloss and the test loss.
To prevent overfitting and improve generalization, several regularizationmethods are proposed.
Decaying weight parameters [36], early stopping [37], and Dropout [38] havebeen commonly applied to avoid the high bias of deep networks.
In addition to these methods, theregularization methods specially designed for time series forecasting are also proposed [39, 40].
Recently, the flooding [10] is introduced to explicitly prevent zero training loss , thus preventingmodel from training too much .
By providing the lower bound of training loss called flood level ,the flooding allows model not to completely fit to the training data, thus improving generalizationcapacity of the model.
In this work, we also attempt to tackle the zero training loss in time seriesforecasting.
However, we find that bounding the average loss in time series forecasting does notperform well as expected.
In time series forecasting, an appropriate error bound for each feature andtime step should be carefully chosen.
In addition, a constant flood level may not suitable to timeseries forecasting where the difficulty of prediction changes for every iteration in the mini-batchtraining process.
To handle such issues, we suggest a novel regularization which fully considers thenature of time series forecasting.
Conclusion
In this work, we propose a simple yet effective regularization method called WaveBound for timeseries forecasting.
WaveBound provides dynamic error bounds for each time step and feature usingthe slow-moving average model.
With the extensive experiments on real-world benchmarks, we showthat our regularization scheme consistently improves the existing models including the state-of-the-artmodel, addressing overfitting in time series forecasting.
We also examine the generalization gaps andloss landscapes to discuss the effect of WaveBound in the training of over-parameterized networks.
We believe that the significant performance improvements of our method indicate that regularizationshould be designed specifically for time series forecasting.
We believe that our work will provide ameaningful insight to future research.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] We used publicly available datasets.(c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes]The codes are provided in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] In this paper, we used the benchmark datasets widely used for research purpose.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] The benchmark datasets used in this paper are widely used for research purpose and do not contain any offensive contents.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]