An Adaptive Temporal Attention Mechanism to Address Distribution Shifts
Abstract
With the goal of studying robust sequence modeling via time series, we propose a robust multi-horizon forecasting approach that adaptively reacts to distribution shifts on relevant time scales.
It is common in many forecasting domains to observe slow or fast forecasting signals at different times.
For example wind and river forecasts are slow changing during drought, but fast during storms.
Our approach is based on the transformer architecture, that across many domains, has demonstrated significant improvements over other architectures.
Several works benefit from integrating a temporal context to enhance the attention mechanism’s understanding of the underlying temporal behavior.
In this work, we propose an adaptive temporal attention mechanism that is capable to dynamically adapt the temporal observation window as needed.
Our experiments on several real-world datasets demonstrate significant performance improvements over existing state-of-the-art methodologies.
The code for reproducing the results is open sourced and available online 1 .
Introduction
Time series forecasting is a vital problem across many domains such as economics [8], retail [4], and healthcare [10] just to name a few which benefits from robust sequence modelling approaches.
The temporal behavior of a time series often changes over time, sometimes slowly and other times rapidly which results in distribution shifts.
Building a robust and adaptive model in response to such dynamic shifts can be challenging.
Transformers [15] have gained popularity in modeling the temporal behavior of a time series.
However, the basic attention mechanism in transformers estimates the similarity based on a point-wise vector of the query and key, each representing individual time steps, thereby ignoring the underlying distribution of the surrounding temporal context, we find that adding multiple layers does not fix this insufficiency, see Section 4.1.
Lie et al [9] suggest to integrate temporal context by using a convolutional neural network (CNN) to inform the attention mechanism with temporal information.
However, our experimental results demonstrate a weakness in this approach, as it is based on a single fixed-length temporal window, which limits the degree of flexibility to respond to distribution shifts on different time scales.
We dive into this problem and investigate the importance of models that can adaptively adjust the attention’s window size and how this leads to a better forecasting model.
Problem Definition
Given the input data prior to time step t 0 , the task is to predict the variables of interest for multiple steps into the future from t 0 to t 0 + τ .
Given the historical observations denoted t
T a r ge t V a r i ab l e
Time * * ** ** ** *** *** ***
The attention mechanism identifies the most similar points for the query at time step t 0 , (depicted in orange), and predicts a similar trajectory observed in keys.
In the case of point-wise attention this would be the dark blue point (denoted as *) which would result in an erroneous forecast following the dashed dark line (denoted as *).
In the case of temporal attention with a fixed-length temporal
, the light denoted as **) is determined to exhibit the most similar behavior which also results in an erroneous forecast the dashed light denoted as of temporal the similarity is apparent on different scales the big and small would result in an accurate in as x 0: t 0 =
[ x 0 , . .
, x t 0 ] , we predict the variables of interest for the next τ steps denoted as y t 0 :
t
+ τ =
[ y t 0 , . . .
, y t
+ τ ] .
Where x i ∈ R d x , and y i ∈ R d y .
Related Work
One of the most dominant classical time series forecasting methods is autoregressive integrated moving average (ARIMA) [3].
ARIMA and other traditional models assumes that the time series is stationary, therefore are not suitable for modeling distribution shifts.
Recurrent neural networks (RNNs) have been widely used to model the temporal behavior of time series [12, 2, 11].
The
DeepAr [13] model uses long short-term memory networks (LSTMs) to generates parameters of a
Gaussian distribution.
Transformers have shown superior performance in modeling temporal behavior compared to RNNs [9, 5].
To enhance the forecasting quality several works benefit from employing a
CNN layer to integrate the temporal context [8, 9, 14].
Recently, new approaches developed more generalized alternatives to the attention mechanism in transformers to obtain more robust models, from which Autoformer [6] and Informer [16] have demonstrated the best performance.
We compare our proposed model to these methods in the experimental section.
Methodology 4.1 Background: Point-wise Attention
Given time series data, a single-layer transformer with point-wise scaled dot-product attention predicts the output y
i at time step i as y i = (cid:80) j ≤ i a ij x j with attention a ij = softmax ( q ⊺ i · k j / √ d ) .
Typically the query and key vectors are derived via projection q i = proj ( x i ) , and k j = proj ( x j ) using two different multi-layer perceptron-style projections of inputs to vectors with dimensionality d .
We call this attention point-wise as it estimates the similarity between query q i and key k j based on information at time step i and j , and this does not inform the attention mechanism with the underlying temporal context that the query q i and key k j are representing.
This point-wise attention overlooks the temporal behavior required for more robust predictions, see Figure 1.
The canonical approach to integrate the temporal context is to use a multi-layer model.
The hypothesis is that from a previous layer, the temporal context can be absorbed into the queries and keys.
However, none of the layers have an innate understanding of temporal context, which is a hurdle to overcome during training.
[Table]
E n
[Table] [Table]
4.2 Background: Temporal attention
CNN-trans includes the temporal context into the attention mechanism to build a more robust attention model.
It is accomplished by deriving query and key vectors from the temporal window of observations preceding the time step i .
Denoting this temporal window i <W
=
[ i − W, . .
, i ] , the transformer’s attention has only needs to be modified by how query and key vectors are obtained:
q i = proj
( x i<W ) and k j = proj ( x j<W ) .
It has been shown that deriving the query q i and key k j vectors from a temporal window of observations enables the attention mechanism to better understand the underlying temporal behavior of the query q i and key k j [9].
While the incorporation of the temporal context can be helpful, a limitation is that he temporal window is of fixed size W .
Even after tuning the window size W , or choosing a larger temporal window issues remain: (1) noise induced by an excessively large window may hinder good performance.
(2) dynamic time periods (e.g. storms) may need a shorter window than stagnant time periods (e.g. droughts).
(3) some temporal patterns may be stretched to slightly different time scales, such would not be recognized by the CNN-trans approach (see Figure 1).
In the following, we address this shortcoming with an adaptive approach.
4.3 Adaptive Temporal-aware Attention (Ours)
The temporal behavior of a time series can change from fast and dynamic to slow and steady.
Hence, our goal is to build a robust and adaptive model that respond to distribution shifts on different time scales as needed.
Furthermore, we want the model to have this ability to identify similar behaviors regardless of their time scales.
For example, if the unseen data exhibits fast and dynamic behavior, the desired model should be able to identify a similar temporal pattern even if it is slightly stretched in the past observations.
Considering the storm forecasting domain, storms can be short and extreme or long and steady, where both of which generally produce similar outcomes, therefore identifying and matching these temporal behaviors is critical for accurate forecasting.
To achieve this goal, we will consider multiple temporal window sizes W =
[ w 1 , . .
. , w n ] to derive query and key vectors using MLP-projections:
For all temporal windows w ∈ W we obtain: query q
i,w = proj ( x i<w ) and key k j,w = proj ( x j<w ) .
With this approach at each position i and j , there exists temporal-aware query and key vectors that represent the temporal behavior across different time scales via different window sizes.
Therefore, it allows the attention mechanism to match queries with keys that represent similar temporal behaviors regardless of the pace at which they occur.
This comparison enables the attention mechanism to identify the similarity on different time scales from stretched to compressed.
(see
Figure 1).
This would result in a more robust model under distribution shifts, and hence a better forecasting model.
Our experimental evaluation will demonstrate the validity of this claim.
After deriving query and key vectors for all temporal windows, a time series with Q queries and
K keys would contain |W| · Q query and |W| · K key vectors.
Note that this approach leads to redundancies, since any input x i and x j are subsumed into multiple overlapping context windows of query q i,w , q i +1 ,w , . . .
and of key k j,w , k j +1 ,w , . .
vectors.
We exploit these redundancies by selecting an optimal subset of query and key vectors with total number of Q and K queries and keys.
Then use query and key vectors at indexes
i ∈
I s and j ∈ J s to calculate the attention scores a i,j = softmax ( q ⊺ i · k j ) / √ d , where I s ⊂ { 1 , 2 , . .
, | W |· Q } and J s ⊂ { 1 , 2 , . . .
, | W |· K } denote the selected subset of query and key positions.
These positions are selected to represent the top
Q and K highest valued query and key vectors.
The chosen subsets meet the training objective to minimize the forecasting loss.
The intuition is that during training the back-propagation encourages higher weights on the subset of query and a key vectors that contribute to a better overall forecasting performance.
Experiments
Datasets : We empirically perform experiment on three datasetes that have been widely used for evaluations by previous studies: (1) Traffic 2 is a collection of hourly occupancy rate of 440 SF Bay
Area freeways.
(2) Air quality 3 is a collection of hourly air pollution of ten cities in China.
(3) Solar energy 4 is a collection of hourly solar power in different locations in America.
Implementation details : All neural models are trained and evaluated three times.
We use Optuna [1] for hyperparameter optimization.
The model size is chosen from {16, 32} for all neural models, and the batch size is set to 256.
Baselines : (1) ARIMA [3] (2) LSTM [7] (3) Transformer [15]: a multi-layer encoder-decoder with basic attention.
(4) Conv-trans [9]: an encoder-decoder with convolutional attention.
(5) Autoformer [6] and (6) Informer [16].
5.1 Main Results
Results are reported as mean and standard error of MSE and MAE scores.
Table 1 summarizes the evaluation results on three datasets when generating predictions for horizons of 24, 48, 72, and 96.
Across all datasets and settings our model outperforms other methods.
Our model exhibits an MSE reduction compared to the next best model by 6% (at 96 horizons), 5% (at 72 horizons), and 5% (at 72 horizons) on traffic, air quality, and solar energy datasets.
The difference in performance to CNN-trans demonstrates the gain when providing the model with the flexibility to choose the right temporal context in response to distribution shifts.
Conclusion
In this paper, we study the multi-horizon time series forecasting problem in the presence of distribution shifts in the timescale of temporal patterns and introduce the adaptive temporal attention, which adaptively respond to such changes, and can detect patterns if temporarily stretched or compressed.
Our experimental evaluation on three datasets demonstrates performance improvements over stateof-the-art methods, including temporal CNN-trans, and generalized transformer-based models such as Autoformer and Informer.
Given the complementary nature of our adaptive temporal approach, it can be integrated into future forecasting methods whenever relevant temporal patters need to be recognized across shifting timescales.