FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting
Abstract
Recent studies have shown that deep learning models such as RNNs and Transformers have brought significant performance gains for long-term forecasting of time series because they effectively utilize historical information.
We found, however, that there is still great room for improvement in how to preserve historical information in neural networks while avoiding overfitting to noise presented in the history.
Addressing this allows better utilization of the capabilities of deep learning models.
To this end, we design a F requency i mproved L egendre M emory model, or FiLM : it applies Legendre Polynomials projections to approximate historical information, uses Fourier projection to remove noise, and adds a low-rank approximation to speed up computation.
Our empirical studies show that the proposed
FiLM significantly improves the accuracy of state-of-the-art models in multivariate and univariate long-term forecasting by ( 20.3% , 22.6% ), respectively.
We also demonstrate that the representation module developed in this work can be used as a general plug-in to improve the long-term prediction performance of other deep learning modules.
Code is available at https://github.com/tianzhou2011/FiLM/.
Introduction
Long-term forecasting refers to making predictions based on the history for a long horizon in the future, as opposed to short-term forecasting.
Long-term time series forecasting has many key applications in energy, weather, economics, transportation, and so on.
It is more challenging than ordinary time series forecasting.
Some of the challenges in long-term forecasting include long-term time dependencies, susceptibility to error propagation, complex patterns and nonlinear dynamics.
These challenges make accurate predictions generally impossible for traditional learning methods such as ARIMA.
Although RNN-like deep learning methods have made breakthroughs in time series forecasting Rangapuram et al.
; Salinas et al.
, they often suffer from problems such as gradient vanishing/exploding Pascanu et al.
(2013), which limits their practical performance.
Following the success of Transformer Vaswani et al.
(2017) in the NLP and CV communities Vaswani et al.
; Devlin et al.
; Dosovitskiy et al.
; Rao et al.
(2021), it also shows promising performance in capturing long-term dependencies for time series forecasting Wen et al.
(2022b);
Zhou et al.
; Wu et al.
; Zhou et al.
We provide an overview of this line of work (including deep recurrent networks and efficient Transformers) in the appendix A.
In order to achieve accurate predictions, many deep learning researchers increase the complexity of their models, hoping that they can capture critical and intricate historical information.
These methods, however, cannot achieve it.
Figure 1 compares the ground truth time series of the real-world
ETTm1 dataset with the predictions of the vanilla Transformer model Vaswani et al.
(2017) and the
LSTM model Hochreiter & Schmidhuber (1997a).
It is observed that the prediction is completely off from the distribution of the ground truth.
We believe that such errors come from these models miscapturing noise while attempting to preserve the true signals.
We conclude that two keys in accurate forecasting are: 1) how to capture critical historical information as completely as possible, and 2) how to effectively remove the noise.
Therefore, to avoid a derailed forecast, we cannot improve a model by simply making it more complex.
Instead, we shall consider a robust representation of the time series Wen et al.
(2022a) that can capture its important patterns without noise.
[Figure]
This observation motivates us to switch our view from long-term time series forecasting to longterm sequence compression.
Recursive memory model Voelker et al.
Gu et al.
(2021a,b, 2020) has achieved impressive results in function approximation tasks.
Voelker et al.
(2019) designed a recursive memory unit (LMU) using Legendre projection, which provides a good representation for long time series.
S 4 model
Gu et al.
(2021a) comes up with another recursive memory design for data representation and significantly improves state-of-the-art results for Long-range forecasting benchmark (LRA) Tay et al.
However, when coming to long-term time series forecasting, these approaches fall short of the Transformer-based methods’ state-of-the-art performance.
A careful examination reveals that these data compression methods are powerful in recovering the details of historical data compared to LSTM/Transformer models, as revealed in Figure 2.
However, they are vulnerable to noisy signals as they tend to overfit all the spikes in the past, leading to limited long-term forecasting performance.
It is worth noting that, Legendre Polynomials employed in LMP Voelker et al.
(2019) is just a special case in the family of Orthogonal Polynomials (OPs).
OPs (including
Legendre, Laguerre, Chebyshev, etc.) and other orthogonal bases (Fourier and Multiwavelets) are widely studied in numerous fields and recently brought in deep learning Wen et al.
(2021b)
; Wang et al.
; Gupta et al.
; Zhou et al.
; Gao et al.
Gu et al.
A detailed review can be found in Appendix A.
[Figure]
The above observation inspires us to develop methods for accurate and robust representations of time series data for future forecasting, especially long-term forecasting.
The proposed method significantly outperforms existing long-term forecasting methods on multiple benchmark datasets by integrating those representations with powerful prediction models.
As the first step towards this goal, we directly exploit the
Legendre projection, which is used by LMU Voelker et al.
(2019) to update the representation of time series with fixed-size vectors dynamically.
This projection layer will then be combined with different deep learning modules to boost forecasting performance.
The main challenge with directly using this representation is the dilemma between information preservation and data overfitting, i.e., the larger the number of Legendre projections is, the more the historical data is preserved, but the more likely noisy data will be overfitted.
Hence, as a second step, to reduce the impact of noisy signals on the
Legendre projection, we introduce a layer of dimension reduction by combining Fourier analysis and low-rank matrix approximation.
More specifically, we keep a large dimension representation from the Legendre projection to ensure that all the important details of historical data are preserved.
We then apply a combination of Fourier analysis and low-rank approximation to keep the part of the representation related to low-frequency Fourier components and the top eigenspace to remove the impact of noises.
Thus, we can not only capture the long-term time dependencies but also effectively reduce the noise in long-term forecasting.
We refer to the proposed method as F requency i mproved
L egendre M emory model, or FiLM for short, for long-term time series forecasting.
In short, we summarize the key contributions of this work as follows:
We propose a Frequency improved Legendre Memory model (FiLM) architecture with a mixture of experts for robust multiscale time series feature extraction.
We redesign the Legendre Projection Unit (LPU) and make it a general tool for data representation that any time series forecasting model can exploit to solve the historical information preserving problem.
We propose Frequency Enhanced Layers (FEL) that reduce dimensionality by combining Fourier analysis and low-rank matrix approximation to minimize the impact of noisy signals from time series and ease the overfitting problem.
The effectiveness of this method is verified both theoretically and empirically.
We conduct extensive experiments on six benchmark datasets across multiple domains (energy, traffic, economics, weather, and disease).
Our empirical studies show that the proposed model improves the performance of state-of-the-art methods by 19.2% and 26.1% in multivariate and univariate forecasting, respectively.
In addition, our empirical studies also reveal a dramatic improvement in computational efficiency through dimensionality reduction.
Time Series Representation in Legendre-Fourier Domain 2.1 Legendre Projection
Given an input sequence, the function approximation problem aims to approximate the cumulative history at every time t .
Using Legendre Polynomials projection, we can project a prolonged sequence of data onto a subspace of bounded dimension, leading to compression, or feature representation, for evolving historical data.
Formally, given a smooth function f observed online, we aim to maintain a fixed size compressed representation of its history f
( x ) [ t − θ,t ] , where θ specifies the window size.
At every time point t , the approximation function g ( t ) ( x ) is defined with respect to the measure µ ( t )
=
1 θ
I [ t − θ,t ] ( x ) .
In this paper, we use Legendre Polynomials of degree at most N − 1 to build the function g ( t ) ( x ) , i.e.
[Equation]
where P n ( · ) is the n -th order Legendre Polynomials.
Coefficients c n ( t ) are captured by the following dynamic equation:
[Equation]
where the definition of A and B can be found in Voelker et al.
Using Legendre Polynomials as a basis allows us to accurately approximate smooth functions, as indicated by the following theorem.
Theorem 1 ( Similar to Proposition 6 in Gu et al.
If f ( x ) is L -Lipschitz, then ∥ f
[ t − θ,t ] ( x ) − g
( t ) ( x ) ∥ µ ( t ) ≤ O ( θL/ √ N ) .
Moreover, if f ( x ) has k -th order bounded derivatives, we have ∥ [Equation] [Equation] x
[Equation]
−
[Equation]
x
[Equation]
O
[Equation] N − [Equation] According to Theorem 1, without any surprise, the larger the number of Legendre Polynomials basis, the more accurate the approximation will be, which unfortunately may lead to the overfitting of noisy signals in history.
As shown in Section 4, directly feeding deep learning modules, such as MLP, RNN, and vanilla Attention without modification, with the above features will not yield state-of-the-art performance, primarily due to the noisy signals in history.
That is why we introduce, in the following subsection, a Frequency Enhanced Layer with Fourier transforms for feature selection.
Before we close this subsection, we note that unlike Gu et al.
(2021a), a fixed window size is used for function approximation and feature extraction.
This is because a longer history of data may lead to a larger accumulation of noises from history.
To make it precise, we consider an auto-regressive setting with random noise.
Let { x t } ∈ R
d be the time sequence with x t +
=
Ax t + b + ϵ t for t = 1 , 2 , ... , where A ∈ R d × d , b ∈ R d , and ϵ t ∈ R
d is random noise sampled from N (0 , σ 2 I ) .
As indicated by the following theorem, given x t , the noise will accumulate in x t − θ over time at the rate of √ θ , where θ is the window size.
Theorem 2.
Let A be an unitary matrix and ϵ t be σ 2 -subgaussian random noise.
We have x t =
A θ x t
[Equation]
+
[Equation]
2.2 Fourier Transform
Because white noise has a completely flat power spectrum, it is commonly believed that the time series data enjoys a particular spectral bias, which is not generally randomly distributed in the whole spectrum.
Due to the stochastic transition environment, the real output trajectories of the forecasting task contain large volatilization and people usually only predict the mean path of them.
The relatively more smooth solutions thus are preferred.
According to Eq.
(1), the approximation function g ( t ) ( x ) can be stabilized via smoothing the coefficients c n ( t ) in both t and n .
This observation helps us design an efficient data-driven way to tune the coefficients c n ( t ) .
As smoothing in n can be simply implemented via multiplying learnable scalars to each channel, we mainly discuss smoothing c n ( t ) in t via Fourier transformation.
The spectral bias implies the spectrum of c n ( t ) mainly locates in the low-frequency regime and has weak signal strength in the high-frequency regime.
To simplify our analysis, let us assume the Fourier coefficients of c n ( t ) as a n ( t ) .
Per spectral bias, we assume that there exists a s, a min > 0 , such that for all n we have t > s , | a n ( t ) | ≤ a min .
An idea to sample coefficients is to keep the first k dimensions and randomly sample the remaining dimensions instead of the fully random sampling strategy.
We characterize the approximation quality via the following theorem:
Theorem 3.
Let A ∈ R d
× n be the Fourier coefficients matrix of an input matrix X ∈ R d × n , and µ ( A ) , the coherence measure of matrix A , is Ω( k/n ) .
We assume there exist s and a positive a min such that the elements in last d − s columns of A is smaller than a min .
If we keep the first s columns selected and randomly choose O ( k 2 /ϵ 2 − s ) columns from the remaining parts, with high probability
[Equation]
where P ( A ) denotes the matrix projecting A onto the column selected column space.
Theorem 3 implies that when a min is small enough, the selected space can be considered almost the same as the original one.
Model Structure 3.1 FiLM: Frequency improved Legendre Memory Model
The overall structure of FiLM is shown in Figure
The FiLM maps a sequence X
(cid:55)→ Y , where
X, Y ∈ R T × D , by mainly utilizing two sub-layers: Legendre Projection Unit (LPU) layer and Fourier
Enhanced Layer (FEL).
In addition, to capture history information at different scales, a mixture of experts at different scales is implemented in the LPU layer.
An optional add-on data normalization layer RevIN Kim et al.
(2021) is introduced to further enhance the model’s robustness.
It is worth mentioning that FiLM is a simple model with only one layer of LPU and one layer of FEL.
LPU: Legendre Projection Unit LPU is a state space model: C t =
AC t − 1 + Bx t , where x t ∈ R is the input signal, C t ∈ R N is the memory unit, and N is the number of Legendre Polynomials.
LPU contains two untrainable prefixed matrices A and B defined as follows:
[Equation]
LPU contains two stages, i.e., Projection and Reconstruction.
The former stage projects the original signal to the memory unit: C =
LPU( X ) .
The later stage reconstructs the signal from the memory unit: X re =
LPU _ R( C ) .
The whole process in which the input signal is projected/reconstructed to/from memory C is shown in Figure 4.
FEL: Frequency Enhanced Layer
Low-rank Approximation The FEL is with a single learnable weight matrix ( W ∈ R M ′ × N × N ) which is all we need to learn from the data.
However, this weight could be large.
We can decompose
W into three matrices W 1 ∈ R M ′ × N ′ × N ′ , W 2 ∈ R N ′ × N and W 3 ∈ R N ′ ×
N to perform a low-rank approximation ( N ′ << N ).
Take Legendre Polynomials number N = 256 as default, our model’s learnable weight can be significantly reduced to 0.4% with N ′ =
4 with minor accuracy deterioration as shown in Section 4.
The calculation mechanism is described in Figure 5.
Mode Selection
A subset of the frequency modes is selected after Fourier transforms to reduce the noise and boost the training speed.
Our default selection policy is choosing the lowest M mode.
Various selection policies are studied in the experiment section.
Results show that adding some random high-frequency modes can give extra improvements in some datasets, as supported by our theoretical studies in Theorem 3.
Implementation source code for LPU and FEL are given in Appendix B.
3.2 Mixture of Multiscale Experts Mechanism
The multiscale phenomenon is a unique critical data bias for time series forecasting.
Since we treat history sequence points with uniform importance, our model might lack such prior.
Our model implemented a simple mixture of experts strategy that utilizes the input sequence with various time horizons { T, 2 T, ...
nT } to forecast the predicted horizon T and merge each expert prediction with linear layer as shown in Figure 3.
This mechanism improves the model’s performance consistently across all datasets, as shown in Table 7.
3.3 Data Normalization
As Wu et al.
; Zhou et al.
(2022) point out, time series seasonal-trend decomposition is a crucial data normalization design for long-term time series forecasting.
We find that our LMU projection can inherently play a normalization role for most datasets, but lacking an explicate normalization design still hurts the robustness of performance in some cases.
A simple reversible instance normalization (RevIN)
Kim et al.
(2021) is adapted to act as an add-on explicate data normalization block.
The mean and standard deviation are computed for every instance x
( i ) k ∈ R T of the input data as E
[Equation]
[Equation]
=
[Equation]
and
Var (cid:104)
[Equation]
( i ) [Equation] (cid:105) = 1 T (cid:80) Tj =
[Equation]
(cid:16) x ( i ) kj − E t (cid:104) x ( i ) kt (cid:105)(cid:17) 2 . Using these statistics, we normalize the input data x ( i ) as ˆ x ( i ) kt = γ k 
[Equation]
[Table] [Equation] Var
[Equation]
[Table]
  + β k , where γ, β ∈ R K are learnable affine parameter vectors.
Then the normalized input data is sent into the model for forecasting.
In the end, we denormalize the model output by applying the reciprocal of the normalization performed at the beginning.
RevIN slows down the training process by 2-5 times, and we do not observe consistent improvement in all datasets by applying RevIn.
Thus, it can be considered an optional stabilizer in model training.
Its detailed performance is shown in the ablation study in Table
[Table]
Experiments
[Table]
To evaluate the proposed FiLM, we conduct extensive experiments on six popular real-world benchmark datasets for long-term forecasting, including traffic, energy, economics, weather, and disease.
Since classic models such as ARIMA and simple RNN/TCN models perform inferior as shown in
Zhou et al.
(2021) and Wu et al.
(2021), we mainly include five state-of-the-art (SOTA) Transformerbased models, i.e., FEDformer, Autoformer Wu et al.
Informer Zhou et al.
, LogTrans Li et al.
(2019), Reformer Kitaev et al.
(2020), and one recent state-space model with recursive memory
S4 Gu et al.
(2021a), for comparison.
FEDformer is selected as the main baseline as it achieves
SOTA results in most settings.
More details about baseline models, datasets, and implementations are described in Appendix.
4.1 Main Result
For better comparison, we follow the experiment settings of Informer Zhou et al.
(2021) where the input length is tuned for best forecasting performance, and the prediction lengths for both training and evaluation are fixed to be 96, 192, 336, and 720, respectively.
Multivariate Results In multivariate forecasting tasks, FiLM achieves the best performance on all six benchmark datasets at all horizons, as shown in Table 17.
Compared with SOTA work (FEDformer), our proposed FiLM yields an overall 20.3% relative MSE reduction.
It is worth noting that the improvement is even more significant on some of the datasets, such as Exchange (over 30%).
The Exchange dataset does not exhibit apparent periodicity, but FiLM still achieves superior performance.
The improvements made by FiLM are consistent with varying horizons, demonstrating its strength in long-term forecasting.
More results on the ETT full benchmark are provided in Appendix C.5.
Univariate Results The benchmark
results for univariate time series forecasting are summarized in
Appendix C.4, Table 10.
Compared with SOTA work (FEDformer), FiLM yields an overall 22.6% relative MSE reduction.
And on some datasets, such as Weather and Electricity, the improvement can reach more than 40%.
It again proves the effectiveness of FiLM in long-term forecasting.
[Table]
[Figure]
LPU Boosting Results
A set of experiments are conducted to measure the boosting effect of LPU when combined with various common deep learning modules (MLP, LSTM, CNN, and
Attention), as shown in Figure 6.
In the experiment, we compare the LPU and the comparable-sized linear layer.
It is worth mentioning that LPU does not contain any learnable parameter.
The results are shown in Table 19.
For all the modules, LPU significantly improves their average performance for long-term forecasting: MLP: 119.4% , LSTM: 97.0% , CNN: 13.8% , Attention: 8.2% .
Vanilla Attention has a relatively poor performance when combining the LPU, which is worth further digging.
[Table]
Low-rank Approximation for FEL
Low-rank approximation of learnable matrix in Frequency Enhanced Layer can significantly reduce our parameter size to 0.1% ∼ 0.4% with minor accuracy deterioration.
The experiment details are shown in Table 13.
Compared to Transformer-based baselines,
FiLM enjoys 80% learnable parameter reduction and 50% memory usage reductions, as shown in Appendix I, J.
[Table]
Mode selection policy for FEL Frequency mode selection policy is studied in Table 4.
The
Lowest mode selection method shows the most robust performance.
The results in Low random column show that randomly adding some high-frequency signal gives extra improvements in some datasets, as our theoretical studies in
Theorem 3 support.
4.2 Ablation Study
This subsection presents the ablation study of the two major blocks (FEL & LPU) employed, the multiscale mechanism, and data normalization (RevIN).
The metric of variants is presented in relative value (‘+‘ indicates degraded performance, ‘-‘ indicates improved performance)
[Table]
Ablation of LPU To prove the effectiveness of LPU layer, in Table 5, we compare the original
LPU layer with six variants.
The LPU layer consists of two sets of matrices (Projection matrices &
Reconstruction matrices).
Each has three variants: Fixed, Trainable, and Random Init.
In Variant 6, we use comparable-sized linear layers to replace the LPU layer.
We observe that Variant 6 leads to a 32.5% degradation on average, which confirms the effectiveness of Legendre projection.
The projection matrix in LPU is recursively called N times ( N is the input length).
So if the projection matrix is randomly initialized (Variant 5), the output will face the problem of exponential explosion.
If the projection matrix is trainable (Variants 2, 3, 4), the model suffers from the exponential explosion as well and thus needs to be trained with a small learning rate, which leads to a plodding training speed and requires more epochs for convergence.
Thus, the trainable projection version is not recommended, considering the trade-off between speed and performance.
The variant with trainable reconstruction matrix (Variant 1) has comparable performance and less difficulty in convergence.
[Table] Ablation of FEL To prove the effectiveness of the FEL [Table] we replace the FEL with multiple variants (MLP, LSTM, CNN, and Transformer).
S4 is also introduced as a variant since it has a version with
Legendre projection memory.
The experimental results are summarized in Table 6.
FEL achieves the best performance compared with its LSTN, CNN, and Attention counterpart.
MLP has comparable performance when the input length is 192, 336, and 720.
However, MLP suffers from insupportable memory usage, which is N 2 L (while FEL is N 2 ).
S4 achieves similar results as our LPU+MLP variant.
Among all, the LPU+CNN variant shows the poorest performance
[Table] [Table]
Ablation of Multiscale and Data Normalization (RevIN)
The multiscale module leads to significant improvement on all datasets consistently.
However, the data normalization achieves mixed performance, leading to improved performance on Traffic and Illness but a slight improvement on the rest.
The Ablation study of RevIN data normalization and the mixture of multiscale experts employed is shown in Table 7.
4.3 Other Studies
Other supplemental studies are provided in Appendix. 1.
A parameter sensitivity experiment (in
Appendix F) is carried out to discuss the choice of hyperparameter for M and N, where M is the frequency mode number and N is the Legendre Polynomials number.
A noise injection experiment (in Appendix G) is conducted to show the robustness of our model.
A Kolmogorov-Smirnov (KS) test (in Appendix H) is conducted to discuss the similarity between the output distribution and the input.
Our proposed FiLM has the best results on KS test, which supports our motivation in model design.
At last, (in Appendix J), though failing short in training speed compared to some MLP based models like N-HiTS Challu et al.
(2022), FiLM is an efficient model with shorter per-step training time and smaller memory usage compared to baseline models in univariate forecasting tasks.
However, it is worth mentioning that the training time will considerably prolong for multivariate forecasting tasks with many target variables.
Discussions and Conclusion
In long-term forecasting, the critical challenge is the trade-off between historical information preservation and noise reduction for accurate and robust forecasting.
To address this challenge, we propose a Frequency improved Legendre Memory model, or FiLM , to preserve historical information accurately and remove noisy signals.
Moreover, we theoretically and empirically prove the effectiveness of the Legendre and Fourier projection employed in our model.
Extensive experiments show that the proposed model achieves SOTA accuracy by a significant margin on six benchmark datasets.
In particular, we would like to point out that our proposed framework is rather general and can be used as the building block for long-term forecasting in future research.
It can be modified for different scenarios.
For example, the Legendre Projection Unit can be replaced with other orthogonal functions such as Fourier, Wavelets, Laguerre Polynomials, Chebyshev Polynomials, etc.
In addition, based on the properties of noises, Fourier Enhanced Layer is proved to be one of the best candidates in the framework.
We plan to investigate more variants of this framework in the future.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Discussions and Conclusion for future works (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section(b)
Did you include complete proofs of all theoretical results?
[Yes] See Section3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Abstract (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix C. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix C. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See Abstract (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Related Work
In this section, we will give an overview of the related literature in time series forecasting.
Traditional Time Series Models
The first generation of well-discussed time series model is the autoregressive family.
ARIMA Box & Jenkins (1968); Box & Pierce (1970) follows the Markov process and build recursive sequential forecasting.
However, a plain autoregressive process has difficulty in dealing non-stationary sequences.
Thus, ARIMA employed a pre-process iteration by differencing, which transforms the series to stationary.
Still, ARIMA and related models have the linear assumption in the autoregressive process, which limits their usage in complex forecasting tasks.
Deep Neural Network in Forecasting With the bloom of deep neural networks, recurrent neural networks (RNNs) were designed for tasks involving sequential data.
However, canonical RNN tends to suffer from gradient vanishing/exploding problems with long input because of its recurrent structure.
Among the family of RNNs, LSTM Hochreiter & Schmidhuber (1997b) and GRU Chung et al.
(2014) proposed gated structure to control the information flow to deal with the gradient vanishing/exploration problem.
Although recurrent networks enjoy fast inference, they are slow to train and not parallelizable.
Temporal convolutional network (TCN) Sen et al.
(2019) is another family for sequential tasks.
However, limited to the reception field of the kernel, the long-term dependencies are hard to capture.
Convolution is a parallelizable operation but expensive in inference.
Transformers With the innovation of Transformers in natural language processing Vaswani et al.
(2017); Devlin et al.
(2019) and in computer vision tasks Dosovitskiy et al.
; Rao et al.
(2021), they are also discussed, renovated, and applied in time series forecasting Wen et al.
(2022b), especially the main attention module.
Some works use temporal attention Qin et al.
(2017) to capture long-range dependencies for time series.
Others use the backbone of Transformer.
Transformers usually employ an encoder-decoder architecture, with the self-attention and cross-attention mechanisms serving as the core layers.
Li et al.
(2019) invents a logsparse attention module to deal with the memory bottleneck for Transformer model.
Kitaev et al.
(2020) uses locality-sensitive hashing to replace the attention module for less time complexity.
Zhou et al.
(2021) proposes a probability sparse attention mechanism to deal with long-term forecasting.
Wu et al.
(2021) designs a decomposition Transformer architecture with an Auto-Correlation mechanism as an alternative for the attention module.
Liu et al.
(2022) designs a low-complexity Pyramidal Attention for the long-term time forecasting tasks.
Zhou et al.
(2022) proposes two attention modules that operate in the frequency domain using Fourier or wavelet transformation.
Orthogonal Basis and Neural Network Orthogonal basis project arbitrary functions onto a certain space and thus enable the representation learning in another view.
Orthogonal basis family is easy to be discretized and serves as a plug-in operation in neural networks.
Recent studies began to realize the efficiency and effectiveness of the Orthogonal basis, including the polynomial family and others (Fourier basis & Multiwavelet basis).
Fourier basis is first introduced for acceleration due to the Fast Fourier algorithm, for example, acceleration of computing convolution Gu et al.
(2020) or Auto-correlation function Wen et al.
(2021a)
; Wu et al.
Fourier basis also serves as a performance boosting block: Fourier with Recurrent structure Zhang et al.
(2018), Fourier with MLP Li et al.
(2020); Lee-Thorp et al.
(2021) and Fourier in Transformer Zhou et al.
Multiwavelet transform is a more local filter (compared with Fourier) and a frequency decomposer.
Thus, neural networks which employ multiwavelet filter usually exhibit a hierarchical structure and treat different frequency in different tunnels, e.g., Wang et al.
; Gupta et al.
; Zhou et al.
Orthogonal Polynomials are naturally good selections of orthogonal basis.
Legendre
Memory Unit (LMU) Voelker et al.
(2019) uses Legendre Polynomials for an orthogonal projection of input signals for memory strengthening with the backbone of LSTM.
The projection process is mathematically derived from delayed linear transfer function.
HiPPO Gu et al.
(2020), based on
LMU, proposes a novel mechanism (Scaled Legendre), which involves the function’s full history (LMU uses rolling windowed history).
In the subsequent work of HiPPO, the authors propose S model Gu et al.
(2021a)
and give the first practice on time series forecasting tasks.
However, LMU and HiPPO share the same backbone (LSTM), which may limit their performance.
B Algorithm Implementation
[Table]
.weights0)
a = torch.einsum("bjhx,hkx->bjkx", a, self.weights1)
out_ft
[:, :, :, :self.modes2]
= torch.einsum("bjkx,ko->bjox", a, self.weights2)
# Return to physical space x = torch.fft.irfft(out_ft, n=x.size(-1)) return x
[Table]
[Table]
[Table]
C Dataset and Implementation Details C.1 Dataset Details
In this subsection, we summarize the details of the datasets used in this paper as follows: 1) ETT Zhou et al.
(2021) dataset contains two sub-dataset: ETT1 and ETT2, collected from two separated counties.
Each of them has two versions of sampling resolutions (15min & 1h).
ETT dataset contains multiple time series of electrical loads and one time sequence of oil temperature.
2) Electricity dataset contains https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams
[Table]
the electricity consumption for more than three hundred clients with each column corresponding to one client.
3) Exchange Lai et al.
(2018) dataset contains the current exchange of eight countries.
4) Traffic dataset contains the occupation rate of freeway systems in California, USA. 5)
Weather dataset contains 21 meteorological indicators for a range of one year in Germany.
6) Illness dataset contains the influenza-like illness patients in the United States.
Table 8 summarizes all the features of the six benchmark datasets.
They are all split into the training set, validation set, and test set by the ratio of 7:1:2 during modeling.
C.2 Implementation Details
Our model is trained using ADAM Kingma & Ba (2017) optimizer with a learning rate of 1 e − 4 to 1 e − 3 .
The batch size is set to 32 (It depends on the GPU memory used in the experiment.
In fact, a batch size up to 256 does not deteriorate the performance but with faster training speed if larger memory GPU or multiple GPUs is used).
The default training process is 15 epochs without any early stopping.
We save the model with the lowest loss on the validation set for the final testing.
The mean square error (MSE) and mean absolute error (MAE) are used as metrics.
All experiments are repeated 5 times and the mean of the metrics is reported as the final results.
All the deep learning networks are implemented using PyTorch Paszke et al.
and trained on NVIDIA V100 32GB GPUs/NVIDIA
V100 16GB GPUs.
C.3 Experiment Error Bars
We train our model 5 times and calculate the error bars for FiLM and SOTA model FEDformer to compare the robustness, which is summarized in Table 9.
It can be seen that the overall performance of the proposed FiLM is better than that of the SOTA FEDformer model
[Table] [Table]
C.4 Univariate Forecasting Results
The univariate benchmark results are summarized in Table 10.
C.5 ETT Full Benchmark
We present the full-benchmark on four ETT datasets Zhou et al.
(2021) in Table 11 (multivariate forecasting) and Table 12 (univariate forecasting).
The ETTh1 and ETTh2 are recorded hourly while http://pems.dot.ca.govhttps://www.bgc-jena.mpg.de/wetter/ https://gis.cdc.gov/grasp/fluview/fluportaldashboard html
[Table]
ETTm1 and ETTm2 are recorded every 15 minutes.
The time series in ETTh1 and ETTm1 follow the same pattern, and the only difference is the sampling rate, similarly for ETTh2 and ETTm2.
On average, our FiLM yields a 14.0% relative MSE reduction for multivariate forecasting, and a 16.8% reduction for univariate forecasting over the SOTA results from FEDformer
[Table] [Table]
E TT h
[Table]
E TT h
[Table]
D Low-rank Approximation for FEL
With the low-rank approximation of learnable matrix in Fourier Enhanced Layer significantly reducing our parameter size, here we study its effect on model accuracy on two typical datasets as shown in
Table 13.
[Table]
TT
[Table] [Table]
E TT
[Table] [Table]
[Table]
E Theoretical Analysis E.1 Theorem
The proof is a simple extension of Proposition 6 in Gu et al.
We omit it for brevity.
E.2 Theorem
As we have x t =
Ax t − 1 + b − ϵ t − 1 for t = 2 , 3 , ...θ , we recursively use them and the following result holds:
[Equation]
Following Hoeffding inequality, for µ > 0 we have
[Equation]
where ∥ · ∥ ψ 2 is the Orlicz norm defined as
[Equation]
Since we require A being unitary, we will have ∥
Aϵ ∥ 22 = ∥ ϵ ∥
and it implies ∥
A i ϵ t − 1 ∥ 2 ψ 2 ∥ =
∥
ϵ t − 1 ∥ 2 ψ 2 =
O ( σ 2 ) for i
= 1 , 2 , ..., θ .
The desirable result follows by setting µ = O ( √ θσ ) .
E.3 Theorem
As we keep first s columns selected, P ( A ) − A has all 0 elements in first s columns.
We thus ignore them and consider the approximation quality on ˜ A ∈ R d × ( n − s ) with the sampled columns.
Via the similar analysis in Appendix C of Zhou et al.
, with high probability we have ∥ ˜
A − P ( A ) ∥ F ≤ (1 + ϵ ) ∥
˜
A − ˜
A k ∥ F , where ˜
A k is the “best” rank- k approximation provided by truncating the singular value decomposition (SVD) of ˜ A , and where ∥ · ∥ F is the Frobenius norm.
As we assume the element in last n − s columns of A is smaller than a min , one can verify ∥ ˜
A − P ( A ) ∥ F ≤ (cid:112) d × ( n − s ) a min and desirable result follows immediately.
F Parameter Sensitivity
[Figure]
Influence of Legendre Polynomial number N and Frequency mode number M
The experimental results on three different datasets (ETTm1, Electricity, and Exchange) in Figure 7 show the optimal choice of Legendre Polynomials number ( N ) when we aim to minimize the reconstruction error (in MSE) on the historical data.
The MSE error decreases sharply at first and saturates at an optimal point, where N is in proportion to the input length.
For input sequences with lengths of 192, 336, and 720, N ≈ 40, 60, and 100 gives the minimal MSE, respectively.
Figure 8 shows the MSE error of time series forecasting on the Electricity dataset, with different
Legendre Polynomials number ( N ), mode number, and input length.
We observe that, when enlarging
N , the model performance saturates at an optimal point.
For example, in Figure 8 Left (input length=192), the best performance is reached when N > 64 .
While in Figure 8 Right (input length=720), the best performance is reached when N is larger than 128.
Another influential parameter is the mode number.
From Figure 8 we observe that a small mode number will lead to better performance, as a module with a small mode number works as a denoising filter.
64 Modes Number 0.
M S E input =    64 Modes Number 0.
input =    64 Modes Number 0.
input =
N=4 N=8 N=
N=32 N=64 N=
N=256 N=
G Noise Injection Experiment
Our model’s robustness in long-term forecasting tasks can be demonstrated using a series of noise injection experiments as shown in Table 14.
As we can see, adding Gaussian noise in the training/test stage has a limited effect on our model’s performance, since the deterioration is less than 1.5% in the worst case.
The model’s robustness is consistent across various forecasting horizons.
Note that adding the noise in the testing stage other than the training stage will even improve our performance by 0.4%, which further supports our claim of robustness.
[Table]
H Distribution Analysis of Forecasting Output H.1 Kolmogorov-Smirnov Test
We adopt the Kolmogorov-Smirnov (KS) test to check whether our model’s input and output sequences come from the same distribution.
The KS test is a nonparametric test for checking the equality of probability distributions.
In essence, the test answers the following question “Are these two sets of samples drawn from the same distribution?”.
The Kolmogorov-Smirnov statistic is defined as:
[Equation]
where sup is the supremum function, F 1 ,n and F 2 ,m are the empirical distribution functions of the two compared samples.
For samples that are large enough, the null hypothesis would be rejected at level α if
[Equation]
where n and m are the first and second sample sizes.
H.2 Distribution Analysis
In this section, we evaluate the distribution similarity between models’ input and output sequences using the KS test.
In Table 15, we applied the Kolmogrov-Smirnov test to check if the output sequences of various models that trained on ETTm1/ETTm2 are consistent with the input sequence.
On both datasets, by setting the standard P-value as 0.01, various existing baseline models have much smaller P-values except FEDformer and Autoformer, which indicates their outputs have a high probability of being sampled from different distributions compared to their input signals.
Autoformer and
FEDformer have much larger P-values mainly due to their seasonal-trend decomposition mechanism.
The proposed FiLM also has a much larger P-value compared to most baseline models.
And its null hypothesis can not be rejected in most cases for these two datasets.
It implies that the output sequence generated by FiLM shares a similar pattern as the input signal, and thus justifies our design motivation of FiLM as discussed in Section 1.
Though FiLM gets a smaller P-value than FEDformer, it is close to the actual output, which indicates that FiLM makes a good balance between recovering and forecasting.
[Table]
I Learnable Parameter Size
Compared to Transformer-based baseline models, FiLM enjoys a lightweight property with 80% learnable parameter reduction as shown in Table 16.
It has the potential to be used in mobile devices, or, in some situations where a lightweight model is preferred
[Figure]
[Table]
J Training Speed and Memory Usage
[Figure]
Memory Usage As shown in Figure 9 (Left), FiLM has good memory usage with the prolonging output length.
For a fair comparison, we fix the experimental settings of Xformer , where we fix the input length as 96 and prolong the output length.
From Figure 9 (Left), we can observe that FiLM
[Figure] [Figure]
R e l a t i v e S c
o r e ( S c
o r e
= 1 /
M S E )
FiLM
FiLMFEDformer
[Figure]
Transformer nHits has a quasi-constant memory usage.
Note that the memory usage of FiLM is only linear to the input length.
Furthermore, FiLM enjoys a much smaller memory usage than others because of the simple architecture and compressed parameters with low-rank approximation as discussed in Appendix I.
Training Speed Experiments are performed on one NVIDIA V100 32GB GPU.
As shown in
Figure 9 (Right), FiLM has a faster training speed than others with the prolonging output length.
For a fair comparison, we fix the experimental setting of Xformer , where we fix the input length as 96 and prolong the output length.
However, in the real experiment settings, we use a longer input length (much longer than 96).
Thus, the experiment in Figure 9 (Right) is merely a toy case to show the tendency.
In Figure 10, we show the average epoch time vs average performance under the settings of benchmarks.
The experiment is performed on ETTm2 dataset with output lengths = 96, 192, 336, and 720.
Because of the extremely low memory usage of FiLM, it can be trained with a larger batch size (batch size = 256) on only one GPU compared with baselines (batch size = 32).
In Figure 10, FiLM-256 is the FiLM models trained with batch size = 256, it exhibits significant advantages on both speed and accuracy.
Furthermore, due to the shallow structure and smaller amount of trainable parameters, FiLM is easy to converge and enjoys smaller performance variation and smaller performance degradation when using a large batch size.
It is observed that the models with
Fourier enhanced block (FiLM & FEDformer) have better robustness.
It is also worth noting that the vanilla Transformer has good training speed because of the not-so-long sequence length.
Only a sequence length over one thousand will distinguish the advantage of efficient Transformers.
K Additional Benchmarks K.1 Multivariate long-term series forecasting with extra baseline models
For the additional benchmarks for multivariate experiments, we add some non-Transformer methods for comparison.
N-BEATSOreshkin et al.
and
N-HiTSChallu et al.
(2022) are two recent proposed powerful non-Transformer methods.
As N-HiTS is the latest development from the research group, which also published N-BEATS, we add N-HiTS to our empirical comparison.
Here, we adopt the results in the N-HiTS paper to prevent inappropriate parameter tuning problems.
We also add a seasonal-naive model in the comparison.
FiLM outperforms N-HiTS in most cases(33/48).
Moreover,
Simple Seasonal-naiveMakridakis et al.
(1982) is a solid baseline on exchange datasets better than
N-hits, Fedformer, and Autoformer, but FiLM still surpasses its performance, as shown in Table 17.
[Table]
K.2 Ablation univariate forecasting experiments for FEL layers with all six datasets
As shown in Table 18, although LPU+MLP combining all boosting tricks has slightly better performance than FiLM for the ETTm1 dataset, FiLM remains the most consistent and effective model among all variants across all six datasets.
FEL is a much better backbone structure than MLP, LSTM,
CNN, and vanilla attention modules.
K.3 Boosting experiments of LPU with common deep learning backbones for all six datasets
As shown in Table 19, LPU shows a consistent boosting effect across all selected common deep learning backbones for most datasets.
It can be used as a simple and effective build add-on block for long-term time series forecasting tasks.
Although without data normalization, pure LPU negatively boosts performance for some cases.
K.4 Ablation univariate forecasting experiments for Low rank approximation with all six datasets
As shown in Table 20, with the low-rank approximation of learnable matrix in the Fourier Enhanced
Layer significantly reduces our parameter size, and even improves our model’s performance for some datasets.
K.5 Ablation univariate forecasting experiments for frequency mode selection with all six datasets
Three different mode selection policies are studied for frequency enhanced layer: 1) lowest mode selection: we select m lowest frequency modes to retain.
2) random model selection: we select m frequency modes randomly to retain. 3) lowest with extra high mode selection: we select 0 .
8 × m lowest frequency modes and 0 .
2 × m high-frequency modes randomly to retain.
The experimental results are summarized in Table 21 with m = 64 for both experiments.
The lowest mode selection is the most stable frequency mode selection policy through adding some randomness mode can improve the results for some datasets.
[Table] [Table]
[Table] [Table]
TT m
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
E l e c t r i c i t y
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
E
[Table]
h a n
[Table]
e
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
a ff i c 0.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]