TILDE-Q: A T RANSFORMATION I NVARIANT L OSS F UNCTION FOR T IME -S ERIES F ORECASTING
Anonymous authors
Paper under double-blind review
A BSTRACT
Time-series forecasting has caught increasing attention in the AI research field due to its importance in solving real world problems across different domains, such as energy, weather, traffic, and economy.
As shown in various types of data, it has been a must-see issue to deal with drastic changes, temporal patterns, and shapes in sequential data that previous models are weak in prediction.
This is because most cases in time-series forecasting aims to minimize L p norm distances as loss functions, such as mean absolute error (MAE) or mean square error (MSE).
These loss functions are vulnerable to not only consider temporal dynamics modeling, but also capture the shape of signals.
In addition, these functions often make models misbehave and return uncorrelated results to the original time-series.
To become an effective loss function, it has to be invariant to the set of distortions between two time-series data instead of just comparing exact values.
In this paper, we propose a novel loss function, called TILDE-Q (Transformation Invariant Loss function with Distance EQuilibrium), that not only considers the distortions in amplitude and phase but also allows models to capture the shape of time-series sequences.
In addition, TILDE-Q supports modeling periodic and non-periodic temporal dynamics at the same time.
We evaluate the effectiveness of TILDE-Q by conducting extensive experiments with respect to periodic and non-periodic conditions of data, from naive models to state-of-the-art models.
The experiment results indicate that the models trained with TILDE-Q outperforms those trained with other training metrics (e.g., MSE, dynamic time warping (DTW), temporal distortion index (TDI), and longest common subsequence (LCSS)).
I NTRODUCTION
Time-series forecasting has been a core problem across various domains, including traffic domain (Li et al., 2018; Lee et al., 2020), economy (Zhu & Shasha, 2002), and disease propagation analysis (Matsubara et al., 2014).
The crucial part of the time-series forecasting is modeling of the complex temporal dynamics (e.g., non-stationary signal, periodicity).
Temporal dynamics, intuitively, shape, has always been one of the most attention-getting keywords in time-series domains, such as rush hour of traffic data or abnormal usage of the electricity (Keogh et al., 2003; Bakshi & Stephanopoulos, 1994; Weigend & Gershenfeld, 1994; Wu et al., 2021;
Zhou et al., 2022).
Deep learning methods are one of the appealing solutions to model complex non-linear temporal dependencies and nonstationary signals, but recent work reveals that even deep learning is often insufficient to model temporal dynamics.
To properly model the temporal dynamics, Wu et al.
; Zhou et al.
(2022) have proposed a novel deep learning approaches with input sequence decomposition.
Le Guen &
Thome (2019) try to model sudden changes timely and accurately with dynamic time warping (DTW).
Bica et al.
(2020) adopts domain adversarial training to learn balanced representations, which is a treatment invariant representations over time.
Wu et al.
; Zhou et al.
(2022) have less attention to the essence of the problem: a shape, in other words, temporal dynamics.
Le Guen & Thome (2019);
Bica et al.
(2020) try to capture the shape but still have some limitations like Fig.
1 (c).
A shape is a part of patterns in time-series data with a given time interval that could give valuable information, such as rise, drop, trough, peak, and plateau.
We call the prediction is informative when it could properly consider the shape.
In real-world applications like economics, such informative prediction is crucial to make decisions.
To gain informative forecasting, the model should consider the shape rather than only aim to forecast accurate value for each time step.
However, existing models do not consider learning shape (Wu et al., 2021;
Zhou et al., 2022; Bica et al., 2020; Le Guen & Thome, 2019), so the forecasting results are often inaccurate and uninformative, because deep learning model tends to learn in easy way (Karras et al., 2019).
Fig.
1 shows three real forecasting results with same model, different training metrics.
When we utilize mean squared error (MSE) as an objectives, the model only aims to reduce gap between prediction and ground truth for each time-step.
As a results, the model generates relatively easy prediction regardless of temporal dynamics (Fig. 1 (b)).
It rarely gives information about original time-series.
In contrast, if we consider both gap and shape of prediction and ground truth, the model could achieve both accuracy and temporal dynamics, as shown in Fig.
1 (a).
In this work, we aim to design a novel objective function that guides models to improve forecasting performance by learning the shapes in time-series data.
To design such shape-aware loss function, we review existing literature (Esling & Agon, 2012; Bakshi & Stephanopoulos, 1994; Keogh, 2003) and investigate the notions of shapes and distortions that interrupt measurement for recognizing similarity of two time-series data in terms of shapes (Sec. 3.1, Sec. 3.2, and Sec. 3.3).
Based on the investigation, we newly propose required conditions for constructing an objective function for shape-aware time-series forecasting (Sec. 3.4).
We then present a novel loss function, TILDE-Q (Transformation Invariant Loss function with Distance EQualibrium), that enables shape-aware representation learning with three different loss terms, which are invariant to the distortions (Sec. 4).
For evaluation, we conduct extensive experiments with state-of-the-art deep learning models for time-series forecasting with TILDE-Q.
The results indicate that TILDE-Q is model-agnostic and could improve accuracy of existing models, compared to MSE and DILATE.
Contributions We make the following contributions: (1) To understand shape-awareness and distortion invariances in time-series forecasting, we investigate existing distortions in amplitude and phase; (2) we implement TILDE-Q that has invariances to many existing distortions and achieves shape-awareness and informative forecasting in a timely manner; and (3) we show that the proposed
TILDE-Q allows models to have higher accuracy compared to those with existing metrics such as
DTW, TDI, and LCSS on average.
R ELATED W ORK 2. T IME -S ERIES F ORECASTING
There are many methods for time-series forecasting from traditional ones, such as ARIMA model (Box et al., 2015) and hidden markov model (Pesaran et al., 2004) to recent deep learning models.
In this section, we briefly describe the recent deep learning models for time-series forecasting.
Starting with the huge success of the recurrent neural networks (RNNs) (Clevert et al., 2016; Li et al., 2018; Yu et al., 2017), researchers have developed novel deep learning architectures, improving forecasting performance.
To effectively capture long-term dependency, which is a weakness of RNNs,
Stoller et al.
(2020) have proposed convolutional neural networks (CNNs).
However, it is required to stack lots of the same CNNs to capture long-term dependency (Zhou et al., 2021).
Attentionbased approaches have been another popular research direction in time-series forecasting, including
Transformer (Vaswani et al., 2017) and Informer (Zhou et al., 2021).
Although the attention-based models effectively capture temporal dependencies, they require high computational cost and often struggle to find proper temporal information (Wu et al., 2021).
To cope with the problem, Wu et al.
; Zhou et al.
(2022) utilize the input decomposition method that helps models better encode appropriate information.
The other state-of-the-art models adopt neural memory networks (Kaiser et al., 2017; Sukhbaatar et al., 2015; Madotto et al., 2018; Lee et al., 2022), which refer to historical data stored in memory to generate meaningful representation.
2. T RAINING M ETRICS
Conventionally, mean squared error (MSE), L p norm and its variants are the mainstream to optimize forecasting models.
However, they are not the best metric to train forecasting models (Esling & Agon, 2012) because time-series is temporally continuous.
Additionally, L p norm gives less information about temporal correlation among time-series data.
To better model temporal dynamics in timeseries data, researchers have used differentiable, approximated dynamic time warping (DTW), as an alternative metric of MSE (Cuturi & Blondel, 2017; Abid & Zou, 2018; Mensch & Blondel, 2018).
However, using DTW as a loss function results in ignoring temporal localization of changes.
Recently,
Le Guen & Thome (2019) suggests DILATE, a training metric to timely catch sudden changes of non-stationary signals with smooth approximation of DTW and penalized temporal distortion index (TDI).
To guarantee to work in a timely manner, Le Guen & Thome (2019) introduce a loss function that gives a harsh penalty when predictions show high temporal distortion.
However, TDI relies on the DTW path, and DTW often shows misalignment because of its noise- and scale-sensitive.
Thus,
DILATE often loses its advantage with complex data, showing disadvantages at the beginning of the training.
In this work, we discuss distortions and transformation invariances and design a new loss function that allows models to learn shapes in the data and produce noise-robust forecasting results.
P RELIMINARY
In this section, we aim to investigate common distortions without losing the goal of time-series forecasting (i.e., modeling temporal dynamics and accurate forecasting).
To help understand the concepts, we first define notations and terms (Sec. 3.1).
We then discuss common distortions in time-series in transformation perspectives that need to be considered for building a shape-aware loss function (Sec. 3.2) and describe how other loss functions (e.g., DTW and TDI) handle shapes during learning (Sec. 3.3).
Last, we explain the conditions for effective time-series forecasting (Sec. 3.4).
3. N OTATIONS AND D EFINITIONS
Let X t denote a data point at a time step t .
Then, we can define a time-series forecasting problem as:
Definition 1.
Given T -length historical time-series X =
[ X t − T +1 , . .
, X t ] , X i ∈ R F at time t and corresponding T ′ -length
future time-series Y =
[ Y t +1 , . . .
, Y t + T ′ ] , Y i ∈ R C , time-series forecasting aims to learn mapping function f : R T × F → R T ′ × C .
To distinguish the label (i.e., ground-truth) and prediction time-series data, we note the label data as Y and prediction data as ˆY .
Next, we set up two goals for time-series forecasting, which require not only precise, but also informative forecasting Wu et al.
; Zhou et al.
; Le Guen &
Thome (2019) as follow:
• Mapping function f should be learnt to point-wisely reduce distance between ˆY and Y ; and • The output ˆ Y should have similar temporal dynamics with Y .
Temporal dynamics are informative patterns in time-series, such as rise, drop, trough, peak, and plateau.
The optimization for the point-wise distance is the conventional methods utilized in deep learning domain, which could be obtained by using MAE or MSE.
But in the real-world problem, for example, the traffic speed prediction or the economics, accurate forecasting of such “temporal dynamics.”
Esling & Agon (2012) also emphasized the importance of measuring temporal dynamics, as “...allowing the recognition of perceptually similar objects even though they are not mathematically identical.”
In this paper, we define the temporal dynamics as follows:
Definition 2.
Temporal dynamics (or shapes) are the informative periodic and non-periodic patterns in time-series data.
In this work, we aim to design a shape-aware loss function that satisfies both goals.
To this end, we first discuss distortions that two time-series with similar shapes can have.
Definition 3.
Given two time-series F and G having a similar shape but not being mathematically identical, F could be formulated by transformation H ( G ) .
Then, we can call time-series F and G have a distortion, which could be represented by transformation H .
Distortion generally occurs in different aspects.
Distortions are defined as temporal distortion (i.e., warping ) and amplitude distortion (i.e., scaling ) with respect to its relevance of dimension, time and amplitude.
Existing distortion in data leads to misbehavior of the model, as measurements are interrupted by the distortion.
For example, if we have two time-series F and G = F + k , which have a similar shape but different means, G could represent many temporal dynamics of F .
However, measurements often evaluate F and G are different (e.g., measuring with MSE) and causes misguidance of the model in training.
As such, it is important to have measurements that consider similar shape invariant to distortion.
We define a measurement for a distortion as follow:
Definition 4.
Let transformation H represents a distortion H .
Then, we call measurement D invariant to H , if ∃ δ > 0 : D ( T , H ( T )) <
δ for any time-series T .
3. T IME -S ERIES D ISTORTIONS IN T RANSFORMATION P ERSPECTIVES
Distortion, a gap between two similar time-series, affects on capturing shapes in time-series data.
As such, it is important to investigate different distortions and their impact on representation learning aspects.
There are six common time-series distortions that models encounter during learning (Esling & Agon, 2012; Batista et al., 2014; Berkhin, 2006; Warren Liao, 2005;
Kerr et al., 2008)–Amplitude
Shifting, Phase Shifting, Uniform Amplification, Uniform Time Scaling, Dynamic Amplification, and Dynamic Time Scaling.
Next, we explain each common time-series distortion in terms of transformation with n -length time-series F ( t ) =
[ f ( t 1 ) , f ( t 2 ) , . . .
, f ( t n )] , where t =
[ t 1 , t 2 , ..., t n ] .
Fig.
2 presents example distortions, categorized by amplitude and time dimensions.
• Amplitude Shifting describes how much a time-series shifts against another time-series.
This can be described with two time-series and the degree of shifting (k): G ( t ) =
F ( t )
+ k =
[ f ( t 1 ) + k, . .
f ( t n ) + k ] , where k ∈ R is constant.
• Phase Shifting is the same type of transformation (i.e., translation) as amplitude shifting, but it occurs along with the temporal dimension.
This distortion can be represented with two timeseries functions with the degree of shift (k): G ( t ) =
F ( t + k ) =
[ f ( t 1 + k ) , . . .
, f ( t n + k )] , where k ∈ R is constant.
Cross-correlation (Paparrizos & Gravano, 2015; Vlachos et al., 2005) is the most popular measure method that is invariant to this distortion.
• Uniform Amplification is a transformation that changes the amplitude by multiplication of k ∈ R .
This distortion can be described with two functions and a multiplication factor (k): G ( t ) =
k
[Equation]
) =
[Equation]
k
[Equation]
f
[Equation]
k
[Equation]
t n )] .
• Uniform Time Scaling means a uniformly shortened or lengthened F ( t ) on the temporal axis.
This distortion can be represented as G ( t ) =
[ g ( t 1 ) , . .
, g ( t m )] , where g ( t i ) =
f ( t ⌈ k · i ⌉ ) and k ∈ R + .
Although Keogh et al.
(2004) propose uniform time warping methods to handle this distortion, it still remains one of the difficult distortion types to measure, due to the difficulty in finding the scaling factor k without testing all possible cases (Keogh, 2003).
• Dynamic Amplification can be interpreted as any distortion occurred by non-zero multiplication on the amplitude dimension.
This distortion can be described as follows:
G ( t ) =
H ( t ) · F ( t ) =
[ h ( t 1 ) · f ( t 1 ) , . . .
, h ( t n ) · f ( t n )] with function h ( t ) such that ∀ t ∈ T , h ( t ) ̸ = 0 .
Local amplification is a representative distortion of this type of distortions, which still remains challenging to solve.
• Dynamic Time Scaling means any transformation that dynamically lengthens or shortens signals on the temporal dimension including local time scaling (Batista et al., 2014) and occlusion (Batista et al., 2014; Vlachos et al., 2003).
It can be represented as follows:
G ( t ) =
F ( h ( t ))
=
[ f ( h ( t 1 )) , . . .
, f ( h ( t n ))] , where h ( t ) is a positive, strictly increasing function.
Dynamic time warping (DTW) (Bellman & Kalaba, 1959; Berndt & Clifford, 1994; Keogh & Ratanamahatana, 2005) is the most popular technique on this distortion.
Das et
al.
(1997) also introduce the longest common subsequence (LCSS) algorithm to tackle occlusion, noise, and outliers in this distortion.
There are several studies on shape-aware clustering (Bellman & Kalaba, 1959; Batista et al., 2014;
Paparrizos & Gravano, 2015; Berkhin, 2006; Warren Liao, 2005; Kerr et al., 2008) and classification (Xi et al., 2006; Batista et al., 2014; Srisai & Ratanamahatana, 2009) tasks with the consideration of shapes.
On the other hand, only a few studies exist for time-series forecasting tasks, including
Le Guen & Thome (2019) that utilizes dynamic time warping (DTW) and temporal distortion index (TDI) for modeling temporal dynamics.
Next we describe mean square error (MSE) and DILATE, proposed by Le Guen & Thome (2019), and discuss their invariance to the distortions.
3. D ISTORTION H ANDLING IN C URRENT T IME -S ERIES F ORECASTING O BJECTIVES
Many measurement metrics have been used in the time-series forecasting domain, and those based on the L p distance, including Euclidean distance, are widely used to handle time-series data.
However, such metrics do not have invariance to the aforementioned distortions (Ding et al., 2008; Le Guen &
Thome, 2019) due to its point-wise mapping.
Specifically, since L p distance compares the values per time step, it cannot handle temporal distortions appropriately and vulnerable to scaling of the data.
Le Guen & Thome (2019) propose a loss function, called DILATE, to overcome the inadequate characteristic in the L p distance metrics by recognizing temporal dynamics with DTW and TDI.
In terms of transformation, DILATE handles dynamic time scaling, especially, local time scaling with
DTW, and phase shifting with penalized TDI, defined as follows:
[Equation]
where A , δ (ˆ y i , y i ) , Ω are the warping path, cost matrix, and squared penalization matrix, respectively.
While DILATE shows better performance than existing methods, there is a missing point in invariance point of view.
DILATE highly depends on DTW, which allows dynamic alignment of the timeseries for the predefined window.
In such windows, DTW could align the signal regardless of their information, for example, periodicity.
As a result, the model makes misbehavior that could cheat
DTW within the window, as shown in Fig.
1 bottom.
DTW’s scale and noise sensitivity are also problematic.
Basically, DTW computes the Euclidean distance of two time-series after its temporal alignment in dynamic programming and the alignment relies on the distance function.
Consequently, the dynamic alignment of the DTW can be properly achieved only when two time-series have the same range (Esling & Agon, 2012; Bellman & Kalaba, 1959).
That means, it hardly achieves invariance on amplitude distortion without appropriate pre-processing.
Gong & Chen (2017) also show that DTW poorly matches the prediction and target (i.e., ground-truth) time-series with amplitude shifting.
Even when the target time-series is aligned with normalization, we cannot guarantee that the predicted and target time-series are properly aligned due to DTW’s high sensitivity to noise.
As a result, DILATE can generate poor alignment results that can cause wrong optimization of TDI, which produces instability during optimization steps and incorrect results.
To design an effective shape-aware loss function, we have to understand measures and when the measures have transformation invariances.
In the next section, we discuss how we interpret transformations in time-series forecasting point of view and which types of transformations should be considered in objective function design.
3. T RANSFORMATION I NVARIANCES IN T IME -S ERIES F ORECASTING
In the time-series domain, data often have various distortions so measurements are needed to satisfy a number of transformation invariances for meaningfully modeling temporal dynamics.
As discussed in Sec.
3.1, we set the goal of time-series forecasting as (1) point-wisely reducing the gap between prediction and target time-series and (2) preserving temporal dynamics of the target time-series.
To satisfy both of them, we have to consider (1) the method that should not have a negative impact on the traditional goal of accurate time-series forecasting and (2) the distortions that play a crucial role in capturing the temporal dynamics of the target time-series.
In this section, we review all six distortions whether it is a feasible loss function or not, discuss their benefits and trade-offs, and find appropriate distortions to be considered in time-series forecasting.
Amplitude Shifting In a wide range of situations, it is beneficial to capture the the trends of timeseries sequence in spite of shifts in terms of amplitude.
Thus, being invariant to amplitude shifting in a loss function takes many advantages in time-series forecasting: (1) shape-awareness invariant to amplitude shifting, (2) accurate deviation of values in modeling, and (3) effective on-time prediction of the peak or sudden changes.
To guarantee the amplitude shifting invariant in the optimization stage, the loss function should induce an equal gap k between prediction and ground truth data in each step.
Formally speaking, the loss function with consideration of the amplitude shifting should satisfy:
[Equation]
where k ∈ R is an arbitrary and equal gap, and d ( y i , ˆ y i ) is a signed distance with a boundary y i >
ˆ y i .
By allowing tolerance between prediction and target time-series, models can follow trends in time-series instead of tending to predict exact values in point-wise.
In short, unlike existing loss functions that handle only point-wise distance (e.g., DTW), we should deal with both the point-wise distance and its relational distance values to guarantee amplitude shifting.
Phase Shifting There are forecasting tasks, whose main objectives concern accurate forecasting of peaks and periodicity in time-series (e.g., heart beat data and stock price data).
For such tasks, phase shifting invariance is one of the best solutions for (1) modeling periodicity, regardless of translation on temporal axis and (2) having precise statistics with shapes, such as peak and plateau values.
If a loss function is to be invariant to phase shifting, the function should satisfy:
[Equation]
Note Eq.
2 allows a similar shape as target time-series in forecasting, not exactly same shape (e.g., sin( x ) and 2 sin( x + x 0 ) with the same dominant frequency).
Uniform Amplification This proposition will be useful in case of sparse data that contains a significant number of zeros.
By adopting the uniform amplification invariance, models are able to focus non-zero sequences, whereas this proposition allows models to receive less penalty in zero sequences.
Since it guarantees shape-awareness with a multiplication factor in a timely manner as
Fig. 2, invariance for uniform amplification fits well.
To have a model trained with the uniform amplification invariance, the loss function should satisfy:
[Equation]
Uniform Time Scaling, Dynamic Amplification, and Dynamic Time Scaling After careful consideration, we conclude that uniform time scaling, dynamic amplification, and dynamic time scaling are incompatible for optimization.
We describe the reason below.
To achieve invariance for the uniform time scaling, the loss function should satisfy:
[Equation]
This proposition will influence negatively original temporal dynamics, considering that it gives the tolerance of mispredicting periodicity (e.g., daily periodic signals) and even cannot catch events (e.g., abrupt changing values) in timely manner.
In summary, it hinders models from capturing shape and corrupts periodic information.
For both dynamic amplification and dynamic time scaling, loss functions always are zero for all pairs when we do not set the limit of tolerance.
For example, if we do not limit tolerance, the proposition for dynamic amplification invariance is as follow:
[Equation]
If a loss function satisfies the proposition, it is always zero because there always exists c i
= y
i
ˆ y i except ˆ y i = 0 .
Therefore, it is not able to give any information because all random values could be an optimal solution.
The same situation happens with the dynamic time scaling if we do not limit the window.
Consequently, all of the uniform time scaling, dynamic amplification, and dynamic time scaling are unsuitable to be objectives in time-series forecasting.
M ETHODS
In this section, we describe a novel loss function TILDE-Q (a Transformation Invariant Loss function with Distance EQuilibrium), which allows models to perform shape-aware time-series forecasting based on the three distortion invariances.
To build a transformation invariant loss function, we have to design a loss function that satisfies the proposition for amplitude shifting invariance (Eq. 1), phase shifting invariance (Eq. 2), and uniform amplification shifting invariance (Eq. 3), as discussed in
Sec. 3.4.
We select them for our loss function because they help models capture the shape and do not harm the goal of the traditional time-series forecasting (i.e., minimize gap between prediction and target time-series).
Not only the loss function should satisfy these propositions, but also it should consider correlations between the whole sequence of outputs and ground truths rather than pointwisely optimizing the model.
It is not achieved by other loss functions, such as MSE or DILATE.
To handle all three distortions and the whole sequence of correlations, we build three objective functions ( a.shift , phase , and amp losses) that achieve one or more invariance by utilizing softmax, Fourier coefficient, and auto-correlation to design a loss function.
Amplitude Shifting Invariance with Softmax (Amplitude Shifting)
To strengthen amplitude shifting invariance, we design a loss function that satisfies Eq. 1.
This means, d ( y i , ˆ y i ) needs to be the same value for all i .
To satisfy the condition, we utilize the softmax function:
[Equation]
where T ′ , Softmax, and d ( · , · ) are the length of sequence, softmax function, and signed distance function, respectively.
Because the Softmax produces the proportion of each value, it only reaches to the optimal solution when it satisfies Eq. 1.
Also, if we utilize Softmax, there is no need to know arbitrary equal gap k .
Invariances with Fourier Coefficients (Phase Shifting)
As we discussed in Sec. 3.4, one candidate method to obtain phase shifting invariance is to use Fourier coefficients.
As described in prior studies (NG & GOLDBERGER, 2007), we can reconstruct original time-series only with dominant frequencies.
In this way, we utilize the norm of dominant Fourier coefficient of ground truth and prediction sequences as our additional objective function, achieving phase shifting invariance.
When it comes to the other frequencies, we denote the norm of prediction sequence to reduce the value of
Fourier coefficient.
Consequently, with the help of our loss function, this loss function allows model to be noise robustness because the Fourier coefficients of white noises in original time-series are relatively small.
Simply, we optimize the distance between Fourier coefficients of two time-series as:
[Equation]
where || · || p is the L p norm.
To obtain the dominant frequency terms, we calculate the norm of the
Fourier coefficient for each frequency and filter them with the squared root of sequence length, √ T ′ .
We also guarantee the minimum number of dominant frequencies as √ T ′ .
This loss function obtains uniform amplification invariance by utilizing a normalization technique to Fourier coefficients.
For example, sin x and c · sin x have the same Fourier coefficients if properly normalized.
In summary, by Eq. 5, we could obtain (1) invariance for phase shifting, (2) invariance for uniform amplification, and (3) robustness to noise.
Invariances with auto-correlation (Uniform Amplification)
Although Fourier coefficients can be considered as a reasonable solution to catch the periodicity of the target time-series, it is not fully invariant to phase shifting for three reasons–(1) the statistics (e.g., mean and variance) in data keep changing, (2) such changing statistics also cause the changes of Fourier coefficients even in the same frequency, and (3) objectives only with a norm of them cannot fully represent the original time-series.
Thus, we introduce an objective based on normalized cross-correlation, which satisfies Eq. 2 for a periodic signal:
[Equation]
where R ( · , · ) is a normalized cross correlation function.
This loss function helps predicted sequences to mimic label sequences by calculating difference between the auto-correlation of the label sequences and cross-correlation between label and predicted sequences.
Therefore, the label and prediction have similar temporal dynamics regardless of phase shifting and uniform amplification.
In summary, we introduce TILDE-Q (Transformation Invariant Loss Function with Distance Equilibrium), combining Eq.
4, Eq. 5, and Eq.
6 as follows:
[Equation]
where α ∈ [0 , 1] and γ is hyperparameter.
E XPERIMENTS
In this section, we present the results of our comprehensive experiments, demonstrating the effectiveness of TILDE-Q and importance of transformation invariance.
Experimental Setup We conduct the experiments with four state-of-the-art models, including
Informer (Zhou et al., 2021), N-Beats (Oreshkin et al., 2020), Autoformer (Wu et al., 2021), and
FEDformer (Zhou et al., 2022) and one simple sequence-to-sequence gated recurrent unit (GRU) model.
We use seven real-world datasets–ECG5000, Traffic, ETTh2, ETTm2, ECL, Exchange, and
Weather and one synthetic dataset–Synthetic for model training.
We repeat each experiment with a model and dataset 10 times in combinations with three different objective functions.
Appendix A provides detailed explanations on the datasets, hyperparameter setting, and model architectures.
[Table]
5. E XPERIMENT R ESULTS
Evaluation Metrics
In the experiment, we evaluate TILDE-Q with four evaluation metrics: mean squared error (MSE), dynamic time warping (DTW), its corresponding temporal distortion index (TDI), all of which are used in Le Guen & Thome (2019).
As DTW is sensitive to noise and generates incorrect paths when one of the time-series data is noisy (as discussed in Sec. 3.3), we additionally use the longest common subsequence (LCSS) for comparison, which is more robust to outliers and
[Table]
[Figure]
[Table]
[Figure] [Figure]
noise (Esling & Agon, 2012).
The longer the length of matched subsequences is achieved, the better performance LCSS shows in modeling the shapes.
For the state-of-the-art models, we reports the
MSE and LCSS.
For the detailed results including DTW and TDI, please refer to Appendix B.
Results and Analysis Table 1 shows the results of short-term forecasting performance of gated recurrent unit (GRU) optimized with MSE, DILATE, and TILDE-Q metrics.
Synthetic, ECG5000, and Traffic datasets are used for the experiment.
With the Synthetic dataset, every used metric shows its own benefits.
This result indicates that similarity of the shape and MSE measures have a clear advantage when a model is trained and evaluated with themselves.
Also, since the model is evaluated with real-world datasets, it is revealed TILDE-Q outperforms other objective functions in most evaluation metrics.
These results indicate our approach for learning shapes in time-series data works better than existing methods for forecasting.
DILATE does not show impressive performance with
ECG5000 due to its high sensitiveness to noise, as discussed in Sec. 3.3.
Table 2 summarize the experiment results with four state-of-the-art models, N-Beats, Informer,
Autoformer, and FEDformer.
The models make predictions for both short-term ( L =96) and long-term ( L up to 720), so that we can investigate their performances with different forecasting difficulties.
In most of datasets, the models with TILDE-Q outperform those with other training metrics.
Especially for long-term forecasting, we observe that for N-Beats and Informer with TILDE-Q significantly improve the performance with the other metrics.
We provide some visual examples in Appendix B and more detailed analysis, qualitative experiments with example visualizations, ablation study results in Appendix B.
This result implies that TILDE-Q improves performances of the models in learning temporal dynamics, including LCSS of N-Beats (improved over 10%).
C ONCLUSION AND F UTURE W ORK
We propose TILDE-Q, a transformation invariant loss function with distance equilibrium, which allows shape-aware time-series forecasting in a timely manner.
To design TILDE-Q, we review existing transformations in time-series data and discuss the conditions that ensure transformation invariances during optimization tasks.
The designed TILDE-Q ensures a model to be invariant to the amplitude shifting, phase shifting, and uniform amplification so that a model better captures the shape in time-series data.
To prove the effectiveness of TILDE-Q, we conduct comprehensive experiments with state-of-the-art models and real-world datasets.
The results indicate that the model trained with TILDE-Q generates more timely, robust, accurate, and shape-aware forecasting in both short-term to long-term forecasting tasks.
We conjecture that this work can facilitate
A D ETAILED E XPERIMENT S ETUP
Dataset In our experiment, we utilize six datasets – Synthetic, ECG5000, and Traffic dataset for the simple model (i.e., Sequence-to-Sequence Gated Recurrent Unit) and ETTh2, ETTm2, and
Electricity for the state-of-the-art model (i.e., Informer and N-Beats).
For each dataset, we describe some metadata of them and experimental setting, including the input length n and prediction window
L .
Synthetic: As Le Guen & Thome (2019) describe, the Synthetic dataset is an artificial dataset for measuring model performance on sudden changes (step functions) with an input signal composed of two peaks.
The amplitude and temporal position of the two peaks are randomly selected.
Then the selected position and amplitude of the step are determined by a peak position and amplitude.
We use 500 time-series for training, 500 for validation and 500 for testing.
For the Synthetic dataset, we set input length as n = 20 and prediction window as L = 40 .
The generation code is provided in
DILATE Github 1 .
ECG5000:
This dataset is originally a 20-hour long ECG (Electrocardiogram), downloaded from
Physionet 2 and archived in UCR Time Series Classification Archive (Dau et al., 2019).
The data is split by each heartbeat and processed to be in equal lengths (140).
In the training, we use 500 for training, 500 for validation, and 4000 for testing.
We take first n = 84 steps as input and predict last
L = 56 steps.
Traffic: Traffic dataset is a collection of 48 months (2015-2016) hourly road occupancy rate (between 0 to 1) data from the California Department of Transportation 3 .
As Le Guen & Thome (2019) do, we utilize univariate series of the first sensor, a total of 17544 data points.
We set our problem as forecasting L = 24 future occupancy rates with n = 168 historical data (past week).
We use 60% of the data for training, 20% for validation, and the rest for evaluation.
ETT: The ETT (Electricity Transformer Temperature) dataset, published by Zhou et al.
(2021), is 2-year data collected from two separated counties in China, including ETTh2 and ETTm2 datasets.
Each data point has a target value of “oil temperature” and other 6 power load features.
ETTh2 and
ETTm2 datasets have 1-hour and 15-minute intervals, respectively.
As Zhou et al.
(2021) do, we split them into 12/4/4 months for the training/validation/testing.
Detailed settings, such as the input and output length and hyperparameter setting, are based on the information at Informer Github 4 .
ECL: The ECL (Electricity Consuming Load) is a dataset recorded in kWh every 15-minutes from 2012 to 2014, for 321 clients.
In our experiment, we split them into 15/3/4 months for the train/validation/test, as Zhou et al.
do.
Note that we use the same hyperparameter settings in the ETTh2 dataset.
Deep Learning Model Architectures We perform experiments with three different model architectures, including Sequence-to-Sequence GRU, Informer, and N-Beats.
To induce models to predict future time-series in a timely manner, we set α = 0 .
5 and γ = 0 .
01 for TILDE-Q. Other training metrics, including MSE and DILATE, are used as described in their original papers.
All models are trained with Early Stopping and ADAM optimizer.
Sequence-to-Sequence GRU To evaluate TILDE-Q in simple model, we utilize one layer Sequenceto-Sequence GRU model.
For the training of the GRU model, we set learning rate of 1 e − 3 , hidden size of 128, trained by maximum 1000 epochs with Early Stopping and ADAM optimizer.
Informer When we train Informer with ETTh2, ETTm2, and ECL dataset, we utilize the official code and hyperparameter setting.
In the case of ECL dataset, as author answered in their official code 4 , we utilize same hyperparameter and dataset splitting criteria as ETTh2 dataset.
N-Beats For N-Beats, we utilize two generic blocks with the hidden size of 128.
Additionally, we set the learning rate as
Autoformer For Autoformer 5 , we use the official code and hyperparameter setting.
For the ETTh dataset, we utilize hyperparameter settings described in the official code of FEDFormer 6 .
FEDformer For FEDformer 6 , we use the official code and hyperparameter setting.
B A DDITIONAL E VALUATIONS B. D ETAILED E XPERIMENT R ESULTS AND A NALYSIS
[Table]
At first
[Table]
we observe that the model optimized with TILDE-Q outperforms the same model optimized with other objective functions in both short- and long-term forecasting tasks.
An interesting point in the results is the large increased errors of TDI and DTW with long-term forecasting.
For example, TDI of Informer with DILATE shows dramatically increased error with ECL dataset, as the forecasting window increases, while LCSS does not produce such large increased error.
We attribute this to the weakness of DTW-based loss functions, which have a weakness due to high sensitiveness on noise.
In contrast, TILDE-Q does not show such large performance drop and even achieves better performance in the long-term forecasting (e.g., Table 4, ETTh2).
Additionally, we can find that Informer with TILDE-Q on ECL data and N-Beats with TILDE-Q on all three datasets show significant improvements.
It indicates that TILDE-Q success to model shape , but other metrics could not.
We provide additional qualitative results below.
Next, we present qualitative analysis of the results.
Fig.
3 shows how the model with different training metrics forecast with different datasets.
From the figure, we have noticed that TILDE-Q allows the model to generate more robust, shape-aware forecasting, regardless of the amplitude shifting, phase shifting, and uniform amplification.
For example, in the case of N-Beats (Fig. 3 (b) bottom),
TILDE-Q generate forecasting results, which are more robust, shape-aware prediction compared to other metrics.
We also see the strength in the Informer case (Fig.
3 (b), top).
Even when the model has not enough ability to capture shape, TILDE-Q tries to retrieve the shape.
We provide additional qualitative results with visualization below.
When the model have enough ability to capture shape (i.e., except ETTh2, Informer of T ′ ∈ [192 , 336 , 720] ), TILDE-Q shown its noise-robust,
[Table] [Table]
smooth forecasting with correctly modeled temporal dynamics.
In the most of N-Beats results and some of Informer results, TILDE-Q reveals that these models have enough ability to capture the temporal dynamics with proper loss function.
In summary, TILDE-Q proves that it is model-agnostic, noise-robust, and able to capture the shape.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
B. A BLATION S TUDY
To evaluate the effect of the α , γ , and measure the effect of each loss function, we conduct a set of experiment with ETTh2 dataset and N-Beats on the long-term forecasting problem.
As we can see in the Fig.
11, the model tends to predict amplification-free forecasting when α increases.
This results indicate our motivation, “ L a.shift will return the forecasting results with same standard deviation with timely manner but without consideration of proper average value.”
Furthermore, in the top of Fig.
12, we can observe three things: (1) if we utilize L a.shift only, as we intended, it have different average (-1.19 vs. 0.11) but relatively similar standard deviation (0.408 vs.
0.299); (2) In the case of L phase only, they can capture dominant frequency and produce relatively less-noisy forecasting; (3) L amp have relatively similar average value (-1.195 vs. -0.319), but it has far different standard deviation (0.408 vs. 8.592).
In contrast, forecasting results of the model trained with MSE is very noisy and hard to interpret (Fig.
12, bottom).
Note that we normalized the results in
Fig.
12 because of the scale issue.
In Table 7, we provide how model performances vary with respect to hyperparameters of TILDE-Q.
For the default setting, we utilized α = 0 .
5 , γ = 0 .
Because the design of TILDE-Q mainly focuses on shape modeling, we can see that DTW and LCSS are not critically changing for the hyperparameter.
But their trade-offs are revealed in the MSE and TDI.
For example, when we decrease α , we can observe TDI increases.
It indicates the trade-offs of phase shifting invariance, which has tolerance for non-timely forecasting.
Also, we can see that increasing α or γ affects the
MSE.
When we have α = 1 , we have no L phase and less penalty for the statistical differences, and its absence causes the high MSE, as we can see in Fig.
γ also affects the MSE, but L phase reduces
L amp ’s side effect.
[Table]