FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recent studies have shown that deep learning models such as RNNs and Transformers have brought signiﬁcant performance gains for long-term forecasting of timeseries because they effectively utilize historical information.
We found, however,that there is still great room for improvement in how to preserve historical information in neural networks while avoiding overﬁtting to noise presented in the history.
Addressing this allows better utilization of the capabilities of deep learning models.
To this end, we design a F requency i mproved L egendre M emory model, or FiLM :it applies Legendre Polynomials projections to approximate historical information, uses Fourier projection to remove noise, and adds a low-rank approximationto speed up computation.
Our empirical studies show that the proposed FiLMsigniﬁcantly improves the accuracy of state-of-the-art models in multivariate andunivariate long-term forecasting by ( 19.2% , 22.6% ), respectively.
We also demonstrate that the representation module developed in this work can be used as a generalplug-in to improve the long-term prediction performance of other deep learningmodules.
Code is available at https://anonymous.4open.science/r/FiLM-65E0/.
Introduction
Long-term forecasting refers to making predictions based on the history for a long horizon in thefuture, as opposed to short-term forecasting.
Long-term time series forecasting has many keyapplications in energy, weather, economics, transportation and so on.
It is more challenging thanordinary time series forecasting.
Some of the challenges in long-term forecasting include long-termtime dependencies, susceptibility to error propagation, complex patterns and nonlinear dynamics.
These challenges make accurate predictions generally impossible for traditional learning methodssuch as ARIMA.
Although RNN-like deep learning methods have made breakthroughs in time seriesforecasting Rangapuram et al.
; Salinas et al.
, they often suffer from problems suchas gradient vanishing/exploding Pascanu et al.
(2013), which limits their practical performance.
Following the success of Transformer Vaswani et al.
(2017) in the NLP and CV communities Vaswaniet al.
; Devlin et al.
; Dosovitskiy et al.
; Rao et al.
(2021), it also shows promisingperformance in capturing long-term dependencies Zhou et al.
; Wu et al.
; Zhou et al.(2022); Wen et al. (2022) for time series forecasting.
We provide an overview of this line of work(including deep recurrent networks and efﬁcient Transformers) in the appendix A.
In order to achieve accurate predictions, many deep learning researchers increase the complexityof their models, hoping that they can capture critical and intricate historical information.
Thesemethods, however, cannot achieve it.
Figure 1 compares the ground truth time series of thereal-world ETTm1 dataset with the predictions of the vanilla Transformer method and the LSTMmodel Vaswani et al.
; Hochreiter & Schmidhuber (1997a)
Zhou et al.
It is observedthat the prediction is completely off from the distribution of the ground truth.
We believe that such errors come from these models miscapturing noise while attempting to preserve the true signals.
Weconclude that two keys in accurate forecasting are: 1) how to capture critical historical informationas complete as possible; and 2) how to effectively remove the noise.
Therefore, to avoid a derailedforecast, we cannot improve a model by simply making it more complex.
Instead, we shall considera robust representation of the time series that can capture its important patterns without noise.
This observation motivatesus to switch our viewfrom long-term time series forecasting to longterm sequence compression.
Recursive memorymodel Voelker et al.
Gu et al.
(2021a,b, 2020)has achieved impressive results in function approximation tasks.
Voelker et al.(2019) designed a recursive memory unit (LMU) using Legendre projection, which provides agood representation for long time series.
S 4 model
Gu et al.
(2021a) comes up with another recursivememory design for data representation and signiﬁcantly improves state-of-the-art results for Longrange forecasting benchmark (LRA) Tay et al. (2020b).
However, when coming to long-term timeseries forecasting, these approaches fall short of the Transformer-based methods’ state-of-the-artperformance.
A careful examination reveals that these data compression methods are powerful inrecovering the details of historical data compared to LSTM/Transformer models, as revealed in
Figure 2.
However, they are vulnerable to noisy signals as they tend to overﬁt all the spikes in the past,leading to limited long-term forecasting performance.
It is worth noting that, Legendre Polynomialsemployed in LMP Voelker et al.
(2019) is just a special case in the family of Orthogonal Polynomials(OPs).
OPs (including Legendre, Laguerre, Chebyshev, etc.) and other orthogonal basis (Fourier and
Multiwavelets) are widely studied in numerous ﬁelds and recently brought in deep learning Wanget al.
; Gupta et al.
; Zhou et al.
; Voelker et al.
; Gu et al.
A detailedreview can be found in Appendix A.
The above observation inspires us to develop methods for accurate and robustrepresentations of time series data for future forecasting, especially long-termforecasting.
The proposedmethod signiﬁcantly outperforms existing long-termforecasting methods on multiple benchmark datasets byintegrating those representations with powerful prediction models.
As the ﬁrst step towards this goal, we directly exploit the
Legendre projection, which is used by LMU Voelker et al.
(2019) to update the representation oftime series with ﬁxed-size vectors dynamically.
This projection layer will then be combined withdifferent deep learning modules to boost forecasting performance.
The main challenge with directlyusing this representation is the dilemma between information preservation and data overﬁtting, i.e.,the larger the number of Legendre projections is, the more the historical data is preserved, but themore likely noisy data will be overﬁtted.
Hence, as a second step, to reduce the impact of noisysignals on the Legendre projection, we introduce a layer of dimension reduction by a combination of
Fourier analysis and low-rank matrix approximation.
More speciﬁcally, we keep a large dimensionrepresentation from the Legendre projection to ensure that all the important details of historical dataare preserved.
We then apply a combination of Fourier analysis and low-rank approximation to keepthe part of the representation related to low-frequency Fourier components and the top eigenspace toremove the impact of noises.
Thus, we can not only capture the
In short, we summarize the key contributions of this work as follows:
We propose a Frequency improved Legendre Memory model (FiLM) architecture with a
mixture of experts for robust multiscale time series feature
extraction
We redesign the Legendre Projection Unit (LPU) and make it a general tool for datarepresentation that any time series forecasting model can exploit to solve the historicalinformation preserving problem.
We propose Frequency Enhanced Layers (FEL) that reduce dimensionality by a combinationof Fourier analysis and low-rank matrix approximation to minimize the impact of noisysignals from time series and ease the overﬁtting problem.
The effectiveness of this methodis veriﬁed both theoretically and empirically.4.
We conduct extensive experiments on six benchmark datasets across multiple domains(energy, trafﬁc, economics, weather, and disease).
Our empirical studies show that theproposed model improves the performance of state-of-the-art methods by 19.2% and 26.1%in multivariate and univariate forecasting, respectively.
In addition, our empirical studies
also reveal a dramatic improvement in computational efﬁciency through dimensionality
reduction.
Time Series Representation in Legendre-Fourier Domain
2.1 Legendre Projection
Given an input sequence, the function approximation problem aims to approximate the cumulativehistory at every time t .
Using Legendre Polynomials projection, we can project a prolonged sequenceof data onto a subspace of bounded dimension, leading to compression, or feature representation, forevolving historical data.
Formally, given a smooth function f observed online, we aim to maintaina ﬁxed size compressed representation of its history f
( x ) [ t  ✓ ,t ] , where ✓ speciﬁes the window size.
At every time point t , the approximation function g ( t ) ( x ) is deﬁned with respect to the measureµ ( t )
= 1 ✓ [ t  ✓ ,t ] ( x ) .
In this paper, we use Legendre Polynomials of degree at most N  1 to buildthe function g ( t ) ( x ) , i.e.
[Equation]
where P n ( · ) is the n -th order Legendre Polynomials.
Coefﬁcients c n ( t ) are captured by the followingdynamic equation:
[Equation]
where the deﬁnition of A and B can be found in Voelker et al.
Using Legendre Polynomials asa basis allows us to accurately approximate smooth functions, as indicated by the following theorem.
Theorem 1 ( Similar to Proposition 6 in Gu et al.
If f ( x ) is L -Lipschitz, then k f [ t  ✓ ,t ] ( x ) g ( t ) ( x ) k µ ( t )  O ( ✓ L/ p N ) .
Moreover, if f ( x ) has k -th order bounded derivatives, we havek f [
[Equation]
✓ ,
[Equation] [Equation]
x
[Equation]
[Equation]
According to Theorem 1, without any surprise, the larger the number of Legendre Polynomialsbasis, the more accurate the approximation will be, which unfortunately may lead to the overﬁttingof noisy signals in the history.
As shown in Section 4, directly feeding deep learning modules,such as MLP, RNN, and vanilla Attention without modiﬁcation, with the above features will notyield state-of-the-art performance, primarily due to the noisy signals in history.
That is why weintroduce, in the following subsection, a Frequency Enhanced Layer with Fourier transforms forfeature selection.
Before we close this subsection, we note that unlike Gu et al.
(2021a), a ﬁxed window size is usedfor function approximation and feature extraction.
This is largely because a longer history of datamay lead to a larger accumulation of noises from history.
To make it precise, we consider an autoregressive setting with random noise.
Let { x t }d be the time sequence with x t +
=
Ax t + b + ✏ t for t = 1 , 2 , ... , where Ad
[Equation]
d , bd , and ✏ td is random noise sampled from N (0 ,  2 I ) .
As indicated by the following theorem, given x t , the noise will accumulate in x t  ✓ over time at therate of p ✓ , where ✓ is the window size.
Theorem 2.
Let A be an unitary matrix and ✏ t be  2 -subgaussian random noise.
We have x t =
A ✓ x t
[Equation]
2.2 Fourier Transform
Because white noise has a completely ﬂat power spectrum, it is commonly believed that the timeseries data enjoys a particular spectral bias, which is not generally randomly distributed in the wholespectrum.
Due to the stochastic transition environment, the real output trajectories of the forecastingtask contain large volatilises and people usually only predict the mean path of them.
The relativemore smooth solutions thus are preferred.
According to Eq.
(1), the approximation function g ( t ) ( x )can be stabilized via smoothing the coefﬁcients c n ( t ) in both t and n .
This observation helps usdesign an efﬁcient data-driven way to tune the coefﬁcients c n ( t ) .
As smoothing in n can be simplyimplemented via multiplying learnable scalars to each channel, we mainly discuss smoothing c n ( t ) int via Fourier transformation.
The spectral bias implies the the spectrum of c n ( t ) mainly locates in lowfrequency regime and has weak signal strength in high frequency regime.
To simplify our analysis,let us assume the Fourier coefﬁcients of c n ( t ) as a n ( t ) .
Per spectral bias, we assume that there existsa s, a min > 0 , such that for all n we have t > s , | a n ( t ) |  a min .
An idea to sample coefﬁcients is tokeep the the ﬁrst k dimensions and randomly sample the remaining dimensions instead of the fullyrandom sampling strategy.
We characterize the approximation quality via the following theorem:
Theorem 3.
Let Ad ⇥ n be the Fourier coefﬁcients matrix of an input matrix
Xd ⇥ n , andµ ( A ) , the coherence measure of matrix A , is ⌦ ( k/n ) .
We assume there exist s and a positive a minsuch that the elements in last d  s columns of A is smaller than a min .
If we keep ﬁrst s columnsselected and randomly choose O ( k 2 / ✏ 2  s ) columns from the remaining parts, with high probability
[Equation]
where P ( A ) denotes the matrix projecting A onto the column selected column space.
Theorem 3 implies that when a min is small enough, the selected space can be considered as almostthe same as the original one.
Model Structure
3.1 FiLM: Frequency improved Legendre Memory Model
The overall structure of FiLM is shown in Figure
The FiLM maps a sequence X 7!
Y , where 168 X, Y 2 R T ⇥ D , by mainly utilizing two sub-layers: Legendre Projection Unit (LPU) layer and Fourier
Enhanced Layer (FEL).
In addition, to capture history information at different scales, a mixture of experts at different scales is implemented in the LPU layer.
An optional add-on data normalizationlayer RevIN Kim et al.
(2021) is introduced to further enhance the model’s robustness.
It is worthmentioning that FiLM is a simple model with only one layer of LPU and one layer of FEL.
LPU: Legendre Projection Unit LPU is a state space model: C t =
AC t  1 + Bx t , where x t 2 R 174 is the input signal, C t 2 R N is the memory unit, and N is the number of Legendre Polynomials.
LPU contains two untrainable preﬁxed matrices A and B deﬁned as follows:
[Equation]
[Equation]
LPU contains two stages, i.e., Projection and Reconstruction.
The former stage projects the originalsignal to the memory unit: C =
LPU( X ) .
The later stage reconstructs the signal from the memory unit: X re =
LPU _ R( C ) .
The whole process in which the input signal is projected/reconstructedto/from memory C is shown in Figure 4.
FEL: Frequency Enhanced Layer
Low-rank Approximation The FEL is with a single learnable weight matrix ( W 2 R M 0 ⇥ N ⇥ N )which is all we need to learn from the data.
However, this weight could be large.
We can decompose
W into three matrices W 1 2 R M 0 ⇥ N 0 ⇥ N 0 , W 2 2 R N 0 ⇥ N and W 3 2 R N 0 ⇥ N to perform a low-rankapproximation ( N 0 << N ).
Take Legendre Polynomials number N = 256 as default, our model’slearnable weight can be signiﬁcantly reduced to 0.4% with N 0 = 4 with minor accuracy deteriorationas shown in Section 4.
The calculation mechanism is described in Figure 5.
Mode Selection
A subset of the frequency modes is selected after Fourier transforms to reducethe noise and boost the training speed.
Our default selection policy is choosing the lowest M mode.
Various selection policies are studied in the experiment section.
Results show that adding somerandom high-frequency modes can give extra improvement in some datasets, as supported by ourtheoretical studies in Theorem 3.
Implementation source code for LPU and FEL are given in Appendix B.
3.2 Mixture of Multiscale Experts Mechanism
Multiscale phenomena is a unique critical data bias for time series forecasting.
Since we treat historysequence points with uniform importance, our model might lack such prior.
Our model implementeda simple mixture of experts strategy that utilizes the input sequence with various time horizons{ T, 2 T, ...
nT } to forecast the predicted horizon T and merge each expert prediction with linearlayer as show in Figure 3.
This mechanism improves the model’s performance consistently across alldatasets, as shown in Table 7.
3.3 Data Normalization
As Wu et al.
; Zhou et al.
(2022) point out, time series seasonal-trend decomposition is a crucialdata normalization design for long-term time series forecasting.
We ﬁnd that our LMU projection caninherently play a normalization role for most datasets, but lacking an explicate normalization designstill hurt the robustness of performance in some cases.
A simple reversible instance normalization(RevIN)
Kim et al.
(2021) is adapted to act as an add-on explicate data normalization block.
Themean and standard deviation are computed for every instance x
( i ) k 2 R T of the input data as
E t h x ( i ) kt i = 1 T
P
Tj =1 x ( i ) kj and
Var h
x ( i ) kt i
= 1 T
P
Tj = ⇣ x ( i ) kj  E t h x ( i ) kt i⌘ 2 .
Using these statistics, we normalizethe input data x ( i ) as ˆ x ( i ) kt =
k@ x
( i ) kt
E t
h x ( i ) kt
i r
Var h x ( i ) kt
i + ✏
A +  k , where  ,  2 R K are learnable afﬁne parametervectors.
Then the normalized input data is sent into the model for forecasting.
In the end, wedenormalize the model output by applying the reciprocal of the normalization performed at thebeginning.
RevIN slows down the training process by 2-5 times, and we do not observe consistent improvementon all datasets by applying RevIn.
Thus, it can be considered an optional stabilizer in model training.
Its detailed performance is shown in the ablation study in Table 7.
Experiments
[Table]
To evaluate the proposed FiLM
[Table]
we conduct extensive experiments on six popular real-world benchmark datasets for long-term forecasting, including trafﬁc, energy, economics, weather
[Table]
and disease.
Since classic models such as ARIMA and simple RNN/TCN models perform inferior as shown in
Zhou et al.
(2021) and Wu et al.
(2021), we mainly include ﬁve state-of-the-art (SOTA) Transformerbased models, i.e., FEDformer, Autoformer Wu et al.
Informer Zhou et al.
, LogTrans Li et al.
(2019), Reformer Kitaev et al.
(2020), and one recent state-space model with recursive memory
S4 Gu et al.
(2021a), for comparison.
FEDformer is selected as the main baseline as it achieves
SOTA results in most settings.
More details about baseline models, datasets, and implementations aredescribed in Appendix C.
4.1 Main Result
For better comparison, we follow the experiment settings of Informer Zhou et al.
(2021) where theinput length is tuned for best forecasting performance, and the prediction lengths for both trainingand evaluation are ﬁxed to be 96, 192, 336, and 720, respectively.
Multivariate Results In multivariate forecasting tasks, FiLM achieves the best performance on allsix benchmark datasets at all horizons, as shown in Table 1.
Compared with SOTA work (FEDformer),our proposed FiLM yields an overall 19.2% relative MSE reduction.
It is worth noting that theimprovement is even more signiﬁcant on some of the datasets, such as Exchange (over 30%).
The Exchange dataset does not exhibit apparent periodicity, but FiLM still achieves superior performance.
The improvements made by FiLM are consistent with varying horizons, demonstrating its strength inlong-term forecasting.
More results on the ETT full benchmark are provided in Appendix C.5.
Univariate Results
The benchmark results for univariate time series forecasting are summarized in
Appendix C.4, Table 10.
Compared with SOTA work (FEDformer), FiLM yields an overall 22.6%relative MSE reduction.
And on some datasets, such as Weather and Electricity, the improvement canreach more than 40%.
It again proves the effectiveness of FiLM in long-term forecasting.
[Table]
indicates degraded performance
[Table]
LPU Boosting Results
Aset of experiments are conducted to measure the boosting effect of LPU whencombined with various common deep learning modules(MLP, LSTM, CNN, and
Attention), as shown in Figure 6.
In the experiment,we compare the LPU andthe comparable-sized linearlayer.
It is worth mentioning that LPU does not contain any learnable parameter.
The results are shown in Table 2.
For all themodules, LPU signiﬁcantly improves their average performance for long-term forecasting: MLP:119.4% , LSTM: 97.0% , CNN: 13.8% , Attention: 8.2% .
Vanilla Attention has a relative poorperformance when combining the LPU, which worth further digging.
[Table]
Low-rank Approximation for FEL Low-rank approximation of learnablematrix in Frequency Enhanced Layercan signiﬁcantly reduce our parametersize to 0.1% ⇠ 0.4% with minor accuracy deterioration.
The experimentdetails are shown in Table 3.
Compared to Transformer-based baselines,
FiLM enjoys 80% learnable parameter reduction and 50% memory usagereductions, as shown in Appendix G, H.
[Table]
and
[Table]
[Table]
M
[Table] frequency mode [Table] Mode selection policy for FEL Frequencymode selection policy is studied in Table 4.
The
Lowest mode selection method shows the mostrobust performance.
The results in Low random column shows that randomly adding somehigh-frequency signal gives extra improvementin some datasets, as our theoretical studies in
Theorem 3 support.
4.2 Ablation Study
This subsection presents the ablation study ofthe two major blocks (FEL & LPU) employed,the multiscale mechanism, and data normalization (RevIN).
[Table]
Ablation of LPU To prove the effectiveness of LPU layer, in Table 5, we compare the original
LPU layer with six variants.
The LPU layer consists of two sets of matrices (Projection matrices &
Reconstruction matrices).
Each has three variants: Fixed, Trainable, and Random Init.
In Variant6, we use comparable-sized linear layers to replace the LPU layer.
We observe that Variant 6 leadsto a 32.5% degradation on average, which conﬁrms the effectiveness of Legendre projection.
Theprojection matrix in LPU is recursively called N times ( N is the input length).
So if the projectionmatrix is randomly initialized (Variant 5), the output will face the problem of exponential explosion.
If the projection matrix is trainable (Variants 2, 3, 4), the model suffers from the exponential explosionas well and thus needs to be trained with a small learning rate, which leads to a plodding training speedand requires more epochs for convergence.
Thus, the trainable projection version is not recommended,considering the trade-off between speed and performance.
The variant with trainable reconstructionmatrix (Variant 1) has comparable performance and less difﬁculty in convergence.
[Table]
Ablation of FEL To prove the effectiveness of the FEL, we replace the FEL with multiple variants(MLP, LSTM, CNN, and Transformer).
S4 is also introduced as a variant since it has a version with
Legendre projection memory.
The experimental results are summarized in Table 6.
FEL achieves thebest performance compared with its LSTN, CNN, and Attention counterpart.
MLP has comparableperformance when input length is 192, 336, and 720.
However, MLP suffers from insupportablememory usage, which is N 2 L (while FEL is N 2 ).
S4 achieves similar results as our LPU+MLPvariant.
Among all, the LPU+vanilla Attention variant shows the poorest performance
[Table]
[Table]
Ablation of Multiscale and Data Normalization (REvIN)
The multiscale module leads to signiﬁcant improvement on all datasets consistently.
However, the data normalization achieves mixedperformance, leading to improved performance on Trafﬁc and Illness but a slight improvement on therest.
The Ablation study of RevIN data normalization and the mixture of multiscale experts employedis shown in Table 7.
4.3 Other Studies
Other supplemental studies are provided in Appendix. 1.
A parameter sensitivity experiment (in
Appendix D) is carried out to discuss the choice of hyperparameter for M and N, where M is thefrequency mode number and N is the Legendre Polynomials number.
A noise injection experiment(in Appendix E) is conducted to show the robustness of our model.
A Kolmogorov-Smirnov (KS)test (in Appendix F) is conducted to discuss the similarity between the output distribution and theinput.
Our proposed FiLM has the best results on KS test, which supports our motivation in modeldesign.
At last, (in Appendix H), we show that FiLM is an efﬁcient model with shorter trainingtime and smaller memory usage compared to baseline models.
Discussions and Conclusion
In long-term forecasting, the critical challenge is the trade-off between historical information preservation and noise reduction for accurate and robust forecasting.
To address this challenge, we proposea Frequency improved Legendre Memory model, or FiLM , to preserve historical information accurately and remove noisy signals.
Moreover, we theoretically and empirically prove the effectivenessof the Legendre and Fourier projection employed in out model.
Extensive experiments show thatthe proposed model achieves SOTA accuracy by a signiﬁcant margin on six benchmark datasets.
Inparticular, we would like to point out that our proposed framework is rather general and can be usedas the building block for long-term forecasting in future research.
It can be modiﬁed for differentscenarios.
For example, the Legendre Projection Unit can be replaced with other orthogonal functionssuch as Fourier, Wavelets, Laguerre Polynomials, Chebyshev Polynomials, etc.
In addition, basedon the properties of noises, Fourier Enhanced Layer is proved to be one of the best candidate in theframework.
We plan to investigate more variants of this framework in the future.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Discussions and Conclusionfor future works
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 23.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Abstract
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix C.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Appendix C.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See Abstract
(d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
) Did you describe any potential participant risks, with links to Institutional Review [Table] Board ( ) approvals, if applicable
[N/A]
[Table]
) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]