a r
X i v : .
08897v [ c s .
L
G ] S e p FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting
Abstract
Recent studies have shown that deep learning models such as RNNs and Transformers have brought signiﬁcant performance gains for long-term forecasting of time series because they effectively utilize historical information.
We found, however, that there is still great room for improvement in how to preserve historical information in neural networks while avoiding overﬁtting to noise presented in the history.
Addressing this allows better utilization of the capabilities of deep learning models.
To this end, we design a F requency i mproved L egendre M emory model, or FiLM : it applies Legendre Polynomials projections to approximate historical information, uses Fourier projection to remove noise, and adds a low-rank approximation to speed up computation.
Our empirical studies show that the proposed FiLM signiﬁcantly improves the accuracy of stateof-the-art models in multivariate and univariate long-term forecasting by ( 20.3% , 22.6% ), respectively.
We also demonstrate that the representation module developed in this work can be used as a general plug-in to improve the long-term prediction performance of other deep learning modules.
Code is available at https:/ github.com/tianzhou2011/FiLM/.
Introduction
Long-term forecasting refers to making predictions based on the history for a long horizon in the future, as opposed to short-term forecasting.
Long-term time series forecasting has many key applications in energy, weather, economics, transportation and so on.
It is more challenging than ordinary time series forecasting.
Some of the challenges in long-term forecasting include long-term time dependencies, susceptibility to error propagation, complex patterns, and nonlinear dynamics.
These challenges make accurate predictions generally impossible for traditional learning methods such as ARIMA.
Although RNN-like deep learning methods have made breakthroughs in time series forecasting Rangapuram et al.
; Salinas et al.
, they often suffer from problems such as gradient vanishing/exploding Pascanu et al.
(2013), which limits their practical performance.
Following the success of Transformer Vaswani et al.
(2017) in the NLP and CV communities Vaswani et al.
; Devlin et al.
; Dosovitskiy et al.
; Rao et al.
(2021), it also shows promising performance in capturing long-term dependencies Zhou et al.
; Wu et al.
; Zhou et al. Wen et al. (2022) for time series forecasting.
We provide an overview of this line of work (including deep recurrent networks and efﬁcient Transformers) in the appendix A.
In order to achieve accurate predictions, many deep learning researchers increase the complexity of their models, hoping that they can capture critical and intricate historical information.
These methods, however, cannot achieve it.
Figure 1 compares the ground truth time series of the real-world ETTm1 dataset with the predictions of the vanilla Transformer method and the LSTM model Vaswani et al.
; Hochreiter & Schmidhuber (1997a)
Zhou et al.
It is observed that the prediction is completely off from the distribution of the ground truth.
We believe that such errors come from these models miscapturing noise while attempting to preserve the true signals.
We conclude that two keys in accurate forecasting are: 1) how to capture critical historical information as complete as possible; and 2) how to effectively remove the noise.
Therefore, to avoid a derailed forecast, we cannot improve a model by simply making it more complex.
Instead, we shall consider a robust representation of the time series that can capture its important patterns without noise.
This observation motivates us to switch our view from long-term time series forecasting to long-term sequence compression.
Recursive memory model Voelker et al.
; Gu et al.
(2021a,b, 2020) has achieved impressive results in function approximation tasks.
Voelker et al. (2019) designed a recursive memory unit (LMU) using Legendre projection, which provides a good representation for long time series.
S 4 model
Gu et al.
(2021a) comes up with another recursive memory design for data representation and signiﬁcantly improves state-ofthe-art results for Long-range forecasting benchmark (LRA) Tay et al.
However, when coming to long-term time series forecasting, these approaches fall short of the Transformer-based methods’ state-of-the-art performance.
A careful examination reveals that these data compression methods are powerful in recovering the details of historical data compared to LSTM/Transformer models, as revealed in Figure 2.
However, they are vulnerable to noisy signals as they tend to overﬁt all the spikes in the past, leading to limited long-term forecasting performance.
It is worth noting that, Legendre Polynomials employed in LMP Voelker et al.
(2019) is just a special case in the family of Orthogonal Polynomials (OPs).
OPs (including Legendre, Laguerre, Chebyshev, etc.) and other orthogonal basis (Fourier and Multiwavelets) are widely studied in numerous ﬁelds and recently brought in deep learning Wang et al.
; Gupta et al.
; Zhou et al.
Voelker et al.
; Gu et al.
A detailed review can be found in Appendix A.
The above observation inspires us to develop methods for accurate and robust representations of time series data for future forecasting, especially longterm forecasting.
The proposed method signiﬁcantly outperforms existing long-term forecasting methods on multiple benchmark datasets by integrating those representations with powerful prediction models.
As the ﬁrst step towards this goal, we directly exploit the Legendre projection, which is used by LMU Voelker et al.
(2019) to update the representation of time series with ﬁxed-size vectors dynamically.
This projection layer will then be combined with different deep learning modules to boost forecasting performance.
The main challenge with directly using this representation is the dilemma between information preservation and data overﬁtting, i.e., the larger the number of Legendre projections is, the more the historical data is preserved, but the more likely noisy data will be overﬁtted.
Hence, as a second step, to reduce the impact of noisy signals on the Legendre projection, we introduce a layer of dimension reduction by a combination of Fourier analysis and low-rank matrix approximation.
More speciﬁcally, we keep a large dimension representation from the Legendre projection to ensure that all the important details of historical data are preserved.
We then apply a combination of Fourier analysis and low-rank approximation to keep the part of the representation related to low-frequency Fourier components and the top eigenspace to remove the impact of noises.
Thus, we can not only capture the long-term time dependencies, but also effective reduce the noise in long-term forecasting. We refer to the proposed method as F requency i mproved L egendre M emory model, or FiLM for short, for long-term time series forecasting.
In short, we summarize the key contributions of this work as follows:
We propose a Frequency improved Legendre Memory model (FiLM) architecture with a mixture of experts for robust multiscale time series feature extraction.
We redesign the Legendre Projection Unit (LPU) and make it a general tool for data representation that any time series forecasting model can exploit to solve the historical information preserving problem.
We propose Frequency Enhanced Layers (FEL) that reduce dimensionality by a combination of Fourier analysis and low-rank matrix approximation to minimize the impact of noisy signals from time series and ease the overﬁtting problem.
The effectiveness of this method is veriﬁed both theoretically and empirically.
We conduct extensive experiments on six benchmark datasets across multiple domains (energy, trafﬁc, economics, weather, and disease).
Our empirical studies show that the proposed model improves the performance of state-of-the-art methods by 19.2% and 26.1% in multivariate and univariate forecasting, respectively.
In addition, our empirical studies also reveal a dramatic improvement in computational efﬁciency through dimensionality reduction.
Time Series Representation in Legendre-Fourier Domain 2.1 Legendre Projection
Given an input sequence, the function approximation problem aims to approximate the cumulative history at every time t .
Using Legendre Polynomials projection, we can project a prolonged sequence of data onto a subspace of bounded dimension, leading to compression, or feature representation, for evolving historical data.
Formally, given a smooth function f observed online, we aim to maintain a ﬁxed size compressed representation of its history f
( x ) [ t − θ ,t ] , where θ speciﬁes the window size.
At every time point t , the approximation function g ( t ) ( x ) is deﬁned with respect to the measure µ ( t ) =
1 θ [ t − θ ,t ] ( x ) .
In this paper, we use Legendre Polynomials of degree at most N −to build the function g ( t ) ( x ) , i.e.
[Equation]
where P n ( · ) is the n -th order Legendre Polynomials.
Coefﬁcients c n ( t ) are captured by the following dynamic equation:
[Equation]
where the deﬁnition of A and B can be found in Voelker et al.
Using Legendre Polynomials as a basis allows us to accurately approximate smooth functions, as indicated by the following theorem.
Theorem 1 ( Similar to Proposition 6 in Gu et al.
If f ( x ) is L -Lipschitz, then " f [ t − θ ,t ] ( x ) − g ( t ) ( x ) " µ ( t ) ≤ O ( θ L/ √ N ) .
Moreover, if f ( x ) has k -th order bounded derivatives, we have
[Equation]
According to Theorem 1, without any surprise, the larger the number of Legendre Polynomials basis, the more accurate the approximationwill be, which unfortunately may lead to the overﬁtting of noisy signals in the history.
As shown in Section 4, directly feeding deep learning modules, such as MLP, RNN, and vanilla Attention without modiﬁcation, with the above features will not yield state-ofthe-art performance, primarily due to the noisy signals in history.
That is why we introduce, in the following subsection, a Frequency Enhanced Layer with Fourier transforms for feature selection.
Before we close this subsection, we note that unlike Gu et al.
(2021a), a ﬁxed window size is used for function approximation and feature extraction.
This is largely because a longer history of data may lead to a larger accumulation of noises from history.
To make it precise, we consider an autoregressive setting with random noise.
Let { x t } ∈ d be the time sequence with x t +
=
Ax t + b + " t for t = 1 , 2 , ... , where A ∈ d × d , b ∈ d , and " t ∈ d is random noise sampled from N (0 , σ 2 I ) .
As indicated by the following theorem, given x t , the noise will accumulate in x t − θ over time at the rate of √ θ , where θ is the window size.
Theorem 2.
Let A be an unitary matrix and " t be σ 2 -subgaussian random noise.
We have x t =
A θ x t − θ + $ θ − 1 i =1 A
i b + O ( σ √ θ ) .
2.2 Fourier Transform
Because white noise has a completely ﬂat power spectrum, it is commonly believed that the time series data enjoys a particular spectral bias, which is not generally randomly distributed in the whole spectrum.
Due to the stochastic transition environment, the real output trajectories of the forecasting task contain large volatilises and people usually only predict the mean path of them.
The relative more smooth solutions thus are preferred.
According to Eq.
(1), the approximation function g ( t ) ( x ) can be stabilized via smoothing the coefﬁcients c n ( t ) in both t and n .
This observation helps us design an efﬁcient data-driven way to tune the coefﬁcients c n ( t ) .
As smoothing in n can be simply implemented via multiplying learnable scalars to each channel, we mainly discuss smoothing c n ( t ) in t via Fourier transformation.
The spectral bias implies the the spectrum of c n ( t ) mainly locates in low frequency regime and has weak signal strength in high frequency regime.
To simplify our analysis, let us assume the Fourier coefﬁcients of c n ( t ) as a n ( t ) .
Per spectral bias, we assume that there exists a s, a min > 0 , such that for all n we have t > s , | a n ( t ) | ≤ a min .
An idea to sample coefﬁcients is to keep the the ﬁrst k dimensions and randomly sample the remaining dimensions instead of the fully random sampling strategy.
We characterize the approximation quality via the following theorem:
Theorem 3.
Let A ∈ d × n be the Fourier coefﬁcients matrix of an input matrix X ∈ d × n , and µ ( A ) , the coherence measure of matrix A , is Ω ( k/n ) .
We assume there exist s and a positive a min such that the elements in last d − s columns of A is smaller than a min .
If we keep ﬁrst s columns selected and randomly choose O ( k 2 / " 2 − s ) columns from the remaining parts, with high probability
[Equation]
where P ( A ) denotes the matrix projecting A onto the column selected column space.
Theorem 3 implies that when a min is small enough, the selected space can be considered as almost the same as the original one.
Model Structure 3.1 FiLM: Frequency improved Legendre Memory Model
The overall structure of FiLM is shown in Figure
The FiLM maps a sequence X &→ Y , where
X, Y ∈ R T × D , by mainly utilizing two sub-layers: Legendre Projection Unit (LPU) layer and
Fourier Enhanced Layer (FEL).
In addition, to capture history information at different scales, a mixture of experts at different scales is implemented in the LPU layer.
An optional add-on data normalization layer RevIN Kim et al.
(2021) is introduced to further enhance the model’s robustness.
It is worth mentioning that FiLM is a simple model with only one layer of LPU and one layer of FEL.
LPU: Legendre Projection Unit LPU is a state space model: C t =
AC t − 1 + Bx t , where x t ∈ R is the input signal, C t ∈ R N is the memory unit, and N is the number of Legendre Polynomials.
LPU contains two untrainable preﬁxed matrices A and B deﬁned as follows:
[Equation]
LPU contains two stages, i.e., Projection and Reconstruction.
The former stage projects the original signal to the memory unit: C =
LPU( X ) .
The later stage reconstructs the signal from the memory unit: X re =
LPU _ R( C ) .
The whole process in which the input signal is projected/reconstructed to/from memory C is shown in Figure 4.
FEL: Frequency Enhanced Layer
Low-rank Approximation The FEL is with a single learnable weight matrix ( W ∈ R M ! × N × N ) which is all we need to learn from the data.
However, this weight could be large.
We can decompose
W into three matrices W 1 ∈ R M !
× N ! × N ! , W 2 ∈ R N ! × N and W 3 ∈ R N ! × N to perform a lowrank approximation ( N $ << N ).
Take Legendre Polynomials number N = 256 as default, our model’s learnable weight can be signiﬁcantly reduced to 0.4% with N $ = 4 with minor accuracy deterioration as shown in Section 4.
The calculation mechanism is described in Figure 5.
Mode Selection
A subset of the frequency modes is selected after Fourier transforms to reduce the noise and boost the training speed.
Our default selection policy is choosing the lowest M mode.
Various selection policies are studied in the experiment section.
Results show that adding some random high-frequency modes can give extra improvement in some datasets, as supported by our theoretical studies in Theorem 3.
Implementation source code for LPU and FEL are given in Appendix B.
3.2 Mixture of Multiscale Experts Mechanism
Multiscale phenomena is a unique critical data bias for time series forecasting.
Since we treat history sequence points with uniform importance, our model might lack such prior.
Our model implemented a simple mixture of experts strategy that utilizes the input sequence with various time horizons { T, 2 T, ...
nT } to forecast the predicted horizon T and merge each expert prediction with linear layer as show in Figure 3.
This mechanism improves the model’s performance consistently across all datasets, as shown in Table 7.
3.3 Data Normalization
As Wu et al.
; Zhou et al.
(2022) point out, time series seasonal-trend decomposition is a crucial data normalization design for long-term time series forecasting.
We ﬁnd that our LMU projection can inherently play a normalization role for most datasets, but lacking an explicate normalization design still hurt the robustness of performance in some cases.
A simple reversible instance normalization (RevIN)
Kim et al.
(2021) is adapted to act as an add-on explicate data normalization block.
The mean and standard deviation are computed for every instance x
( i ) k ∈ R T of the input data as E t % x ( i ) kt ’ = 1 T $
Tj =1 x ( i ) kj and
Var % x ( i ) kt ’ = 1 T $
Tj = ) x ( i ) kj − E t % x ( i ) kt ’* 2 .
Using these statistics, we normalize the input data x ( i ) as ˆ x ( i ) kt = γ k   x ( i ) kt − E t !
x ( i ) kt " # Var !
x ( i ) kt " + " 
 + β k , where γ , β ∈ R K are learnable afﬁne parameter vectors.
Then the normalized input data is sent into the model for forecasting.
In the end, we denormalize the model output by applying the reciprocal of the normalization performed at the beginning.
RevIN slows down the training process by 2-5 times, and we do not observe consistent improvement on all datasets by applying RevIn.
Thus, it can be considered an optional stabilizer in model training.
Its detailed performance is shown in the ablation study in Table 7.
Experiments
To evaluate the proposed FiLM, we conduct extensive experiments on six popular real-world benchmark datasets for long-term forecasting, including trafﬁc, energy, economics, weather, and disease.
Since classic models such as ARIMA and simple RNN/TCN models perform inferior as shown in
Zhou et al.
(2021) and Wu et al.
(2021), we mainly include ﬁve state-of-the-art (SOTA) Transformerbased models, i.e., FEDformer, Autoformer Wu et al.
Informer Zhou et al.
, LogTrans Li et al.
(2019), Reformer Kitaev et al.
(2020), and one recent state-space model with recursive memory S4 Gu et al.
(2021a), for comparison.
FEDformer is selected as the main baseline as it achieves SOTA results in most settings.
More details about baseline models, datasets, and implementations are described in Appendix.
4.1 Main Result
For better comparison, we follow the experiment settings of Informer Zhou et al.
(2021) where the input length is tuned for best forecasting performance, and the prediction lengths for both training and evaluation are ﬁxed to be 96, 192, 336, and 720, respectively.
Multivariate Results In multivariate forecasting tasks, FiLM achieves the best performance on all six benchmark datasets at all horizons, as shown in Table 17.
Compared with SOTA work (FEDformer), our proposed FiLM yields an overall 20.3% relative MSE reduction.
It is worth noting that the improvement is even more signiﬁcant on some of the datasets, such as Exchange (over 30%). [Table] [Figure]
The Exchange dataset does not exhibit apparent periodicity, but FiLM still achieves superior performance.
The improvements made by FiLM are consistent with varying horizons, demonstrating its strength in long-term forecasting.
More results on the ETT full benchmark are provided in Appendix C.5.
Univariate Results
The benchmark results for univariate time series forecasting are summarized in Appendix C.4, Table 10.
Compared with SOTA work (FEDformer), FiLM yields an overall 22.6% relative MSE reduction.
And on some datasets, such as Weather and Electricity, the improvement can reach more than 40%.
It again proves the effectiveness of FiLM in long-term forecasting.
[Table]
LPU Boosting Results
A set of experiments are conducted to measure the boosting effect of LPU when combined with various common deep learning modules (MLP,
LSTM, CNN, and Atten- tion), as shown in Figure 6.
In the experiment, we compare the LPU and the comparable-sized linear layer.
It is worth mentioning that LPU does not contain any learnable parameter.
The results are shown in Table 19.
For all the modules, LPU signiﬁcantly improves their average performance for long-term forecasting: MLP: 119.4% , LSTM: 97.0% , CNN: 13.8% , Attention: 8.2% .
Vanilla Attention has a relative poor performance when combining the LPU, which worth further digging.
[Table]
Low-rank Approximation for FEL Low-rank approximation of learnable matrix in Frequency Enhanced Layer can signiﬁcantly reduce our parameter size to 0.1% ∼ 0.4% with minor accuracy deterioration.
The experiment details are shown in Table 13.
Compared to Transformer-based baselines, FiLM enjoys 80% learnable parameter reduction and 50% memory usage reductions, as shown in Appendix I, J.
[Table]
mode selection policy is studied in Table 4.
The
Lowest mode selection method shows the most robust performance.
The results in Low random column shows that randomly adding some high-frequency signal gives extra improvement in some datasets, as our theoretical studies in
Theorem 3 support.
4.2 Ablation Study
This subsection presents the ablation study of the two major blocks (FEL & LPU) employed, the multiscale mechanism, and data normalization (RevIN). original parameters and trainable.
Random Init means the matrix is initialized randomly and trainable).
The experiments are performed on ETTm1 and Electricity with different input lengths.
The metric of variants is presented in relative value (‘+‘ indicates degraded performance, ‘-‘ indicates improved performance)
[Table]
E l ec t r i c it y
[Table] Ablation of LPU To prove the effectiveness of LPU layer, in Table 5, we compare the original
LPU layer with six variants.
The LPU layer consists of two sets of matrices (Projection matrices &
Reconstruction matrices).
Each has three variants: Fixed, Trainable, and Random Init.
In Variant 6, we use comparable-sized linear layers to replace the LPU layer.
We observe that Variant 6 leads to a 32.5% degradation on average, which conﬁrms the effectiveness of Legendre projection.
The projection matrix in LPU is recursively called N times ( N is the input length).
So if the projection matrix is randomly initialized (Variant 5), the output will face the problem of exponential explosion.
If the projection matrix is trainable (Variants 2, 3, 4), the model suffers from the exponential explosion as well and thus needs to be trained with a small learning rate, which leads to a plodding training speed and requires more epochs for convergence.
Thus, the trainable projection version is not recommended, considering the trade-off between speed and performance.
The variant with trainable reconstruction matrix (Variant 1) has comparable performance and less difﬁculty in convergence.
[Table] Ablation of FEL To prove the effectiveness of the FEL, we replace the FEL with multiple variants (MLP, LSTM, CNN [Table] and Transformer).
S4 is also introduced as a variant since it has a version with
Legendre projection memory.
The experimental results are summarized in Table 6.
FEL achieves the best performance compared with its LSTN, CNN, and Attention counterpart.
MLP has comparable performance when input length is 192, [Table] and 720.
However, MLP suffers from insupportable memory usage, which is N 2 L (while FEL is N 2 ).
S4 achieves similar results as our LPU+MLP variant.
Among all, the LPU+CNN variant shows the poorest performance
[Table]
[Table]
Ablation of Multiscale and Data Normalization (RevIN)
The multiscale module leads to signiﬁcant improvement on all datasets consistently.
However, the data normalization achieves mixed performance, leading to improved performance on Trafﬁc and Illness but a slight improvement on the rest.
The Ablation study of RevIN data normalization and the mixture of multiscale experts employed is shown in Table 7.
4.3 Other Studies
Other supplemental studies are provided in Appendix. 1.
A parameter sensitivity experiment (in
Appendix F) is carried out to discuss the choice of hyperparameter for M and N, where M is the frequency mode number and N is the Legendre Polynomials number.
A noise injection experiment (in Appendix G) is conducted to show the robustness of our model.
A Kolmogorov-Smirnov (KS) test (in Appendix H) is conducted to discuss the similarity between the output distribution and the input.
Our proposed FiLM has the best results on KS test, which supports our motivation in model design.
At last, (in Appendix J), though failing short in training speed compared to some MLP based models like N-HiTSChallu et al.
(2022), FiLM is an efﬁcient model with shorter per-step training time and smaller memory usage compared to baseline models in univariate forecasting tasks.
However, it is worth mentioning that the training time will considerably prolong for multivariate forecasting tasks with many target variables.
Discussions and Conclusion
In long-term forecasting, the critical challenge is the trade-off between historical information preservation and noise reduction for accurate and robust forecasting.
To address this challenge, we propose a Frequency improved Legendre Memory model, or FiLM , to preserve historical information accurately and remove noisy signals.
Moreover, we theoretically and empirically prove the effectiveness of the Legendre and Fourier projection employed in out model.
Extensive experiments show that the proposed model achieves SOTA accuracy by a signiﬁcant margin on six benchmark datasets.
In particular, we would like to point out that our proposed framework is rather general and can be used as the building block for long-term forecasting in future research.
It can be modiﬁed for different scenarios.
For example, the Legendre Projection Unit can be replaced with other orthogonal functions such as Fourier, Wavelets, Laguerre Polynomials, Chebyshev Polynomials, etc.
In addition, based on the properties of noises, Fourier Enhanced Layer is proved to be one of the best candidate in the framework.
We plan to investigate more variants of this framework in the future.