DEPTS: D EEP E XPANSION L EARNING FOR P ERIODIC T IME S ERIES F ORECASTING
A BSTRACT
Periodic time series (PTS) forecasting plays a crucial role in a variety of industries to foster critical tasks, such as early warning, pre-planning, resource scheduling, etc.
However, the complicated dependencies of the PTS signal on its inherent periodicity as well as the sophisticated composition of various periods hinder the performance of PTS forecasting.
In this paper, we introduce a deep expansion learning framework, DEPTS, for PTS forecasting.
DEPTS starts with a decoupled formulation by introducing the periodic state as a hidden variable, which stimulates us to make two dedicated modules to tackle the aforementioned two challenges.
First, we develop an expansion module on top of residual learning to perform a layer-by-layer expansion of those complicated dependencies.
Second, we introduce a periodicity module with a parameterized periodic function that holds sufﬁcient capacity to capture diversiﬁed periods.
Moreover, our two customized modules also have certain interpretable capabilities, such as attributing the forecasts to either local momenta or global periodicity and characterizing certain core periodic properties, e.g., amplitudes and frequencies.
Extensive experiments on both synthetic data and real-world data demonstrate the effectiveness of DEPTS on handling PTS.
In most cases, DEPTS achieves signiﬁcant improvements over the best baseline.
Speciﬁcally, the error reduction can even reach up to 20% for a few cases.
All codes are publicly available at https://github.com/weifantt/DEPTS .
I NTRODUCTION
Time series (TS) with apparent periodic (seasonal) oscillations, referred to as periodic time series (PTS) in this paper, is pervasive in a wide range of critical industries, such as seasonal electricity spot prices in power industry (Koopman et al., 2007), periodic trafﬁc ﬂows in transportation (Lippi et al., 2013), periodic carbon dioxide exchanges and water ﬂows in sustainability domain (Seymour, 2001; Tesfaye et al., 2006).
Apparently, PTS forecasting plays a crucial role in these industries since it can foster their business development by facilitating a variety of capabilities, including early warning, pre-planning, and resource scheduling (Kahn, 2003; Jain, 2017).
Given the pervasiveness and importance of PTS, two obstacles, however, largely hinder the performance of existing forecasting models.
First, future TS signals yield complicated dependencies on both adjacent historical observations and inherent periodicity.
Nevertheless, many existing studies did not consider this distinctive periodic property (Salinas et al., 2020; Toubeau et al., 2018;
Wang et al., 2019; Oreshkin et al., 2020).
The performance of these methods has been greatly restrained due to its ignorance of periodicity modeling.
Some other efforts, though explicitly introducing periodicity modeling, only followed some arbitrary yet simple assumptions, such as additive or multiplicative seasonality, to capture certain plain periodic effects (Holt, 1957; 2004; Vecchia, 1985b; Taylor & Letham, 2018).
These methods failed to model complicated periodic dependencies beyond much simpliﬁed assumptions.
The second challenge lies in that the inherent periodicity of a typical real-world TS is usually composed of various periods with different amplitudes and frequencies.
For example, Figure 1 exempliﬁes the sophisticated composition of diversiﬁed periods via a real-world eight-years hourly TS of electricity load in a region of California.
However, existing methods (Taylor & Letham, 2018; Smyl, 2020) required the pre-speciﬁcation of periodic frequencies before estimating other parameters from data, which attempted to evade this obstacle by transferring the burden of periodicity coefﬁcient initialization to practitioners.
To better tackle the aforementioned two challenges, we develop a deep expansion learning framework, DEPTS, for PTS forecasting.
The core idea of DEPTS is to build a deep neural network that conducts the progressive expansions of the complicated dependencies of PTS signals on periodicity to facilitate forecasting.
We start from a novel decoupled formulation for PTS forecasting by introducing the periodic state as a hidden variable.
This new formulation stimulates us to make more customized and dedicated designs to handle the two speciﬁc challenges mentioned above.
For the ﬁrst challenge, we develop an expansion module on top of residual learning (He et al., 2016; Oreshkin et al., 2020) to conduct layer-by-layer expansions between observed TS signals and hidden periodic states.
With such a design, we can build a deep architecture with both high capacities and efﬁcient parameter optimization to model those complicated dependencies of TS signals on periodicity.
For the second challenge, we build a periodicity module to estimate the periodic states from observational data.
We represent the hidden periodic state with respect to time as a parameterized periodic function with sufﬁcient expressiveness.
In this work, for simplicity, we instantiate this function as a series of cosine functions.
To release the burden of manually setting periodic coefﬁcients for different data, we develop a data-driven parameter initialization strategy on top of Discrete Cosine Transform (Ahmed et al., 1974).
After that, we combine the periodicity module with the expansion module to perform end-to-end learning.
To the best of our knowledge, DEPTS is a very early attempt to build a customized deep learning (DL) architecture for PTS that explicitly takes account of the periodic property.
Moreover, with two delicately designed modules, DEPTS also owns certain interpretable capabilities.
First, the expansions of forecasts can distinguish the contributions from either adjacent TS signals or inherent periodicity, which intuitively illustrate how the future TS signals may vary based on local momenta and global periodicity.
Second, coefﬁcients of the periodicity module have their own practical meanings, such as amplitudes and frequencies, which provide certain interpretable effects inherently.
We conduct experiments on both synthetic data and real-world data, which all demonstrate the superiority of DEPTS on handling PTS.
On average, DEPTS reduces the error of the best baseline by about 10%.
In a few cases, the error reduction can even reach up to 20%.
Besides, we also include extensive ablation tests to verify our critical designs and visualize speciﬁc model components to interpret model behaviors.
R ELATED W ORK
TS forecasting is a longstanding research topic that has been extensively studied for decades.
After a comprehensive review of the literature, we ﬁnd three types of paradigms in developing TS models.
At an early stage, researchers developed simple yet effective statistical modeling approaches, including exponentially weighted moving averages (Holt, 1957; 2004; Winters, 1960), auto-regressive moving averages (ARMA)
(Whittle, 1951; 1963), the uniﬁed state-space modeling approach as well as other various extensions (Hyndman & Khandakar, 2008).
However, these statistical approaches only considered the linear dependencies of future TS signals on past observations.
To handle highorder dependencies, researchers attempted to adopt a hybrid design that combines statistical modeling with more advanced high-capacity models (Montero-Manso et al., 2020; Smyl, 2020).
At the same time, with the great successes of DL in computer vision (He et al., 2016) and natural language processing (Vaswani et al., 2017), various DL models have also been developed for TS forecasting (Rangapuram et al., 2018; Toubeau et al., 2018;
Salinas et al., 2020; Zia & Razzaq, 2020; Cao et al., 2020).
Among them, the most representative one is N-BEATS (Oreshkin et al., 2020), which is a pure DL architecture that has achieved state-of-the-art performance across a wide range of benchmarks.
The connections between DEPTS and N-BEATS have been discussed in Section 4.2.
As for PTS forecasting, many traditional statistical approaches explicitly considered the periodic property, such as periodic ARMA (PARMA) (Vecchia, 1985a;b) and its variants (Tesfaye et al., 2006; Anderson et al., 2007; Dudek et al., 2016).
However, as discussed in Sections 1 and 3, these methods only followed some arbitrary yet simple assumptions, such as additive or multiplicative seasonality, and thus cannot well handle complicated periodic dependencies in many real-world scenarios.
Besides, other recent studies either followed the similar assumptions for periodicity or required the pre-speciﬁcation of periodic coefﬁcients (Taylor & Letham, 2018; Smyl, 2020).
To the best of our knowledge, we are the ﬁrst work that develops a customized DL architecture to model complicated periodic dependencies and to capture diversiﬁed periodic compositions simultaneously.
P ROBLEM F ORMULATIONS
We consider the point forecasting problem of regularly sampled uni-variate TS.
Let x
t denote the time series value at time-step t , and the classical auto-regressive formulation is to project the historical observations x t − L : t =
[ x t − L , . . .
, x t − 1 ] into its subsequent future values x t : t + H
=
[ x t , . . .
, x t + H − 1 ] : x t : t + H = F Θ ( x t − L : t ) + (cid:15) t : t + H
[Equation]
where H is the length of the forecast horizon, L is the length of the lookback window, F Θ : R L → R H is a mapping function parameterized by Θ , and (cid:15) t : t + H
=
[ (cid:15) t , . .
, (cid:15) t
+ H − 1 ] denotes a vector of independent and identically distributed Gaussian noises.
Essentially, the fundamental assumption behind this formulation is the Markov property x t : t + H ⊥ x 0: t − L | x t − L : t , which assumes that the future values x t : t + H are independent of all farther historical values x 0: t − L given the adjacent shortterm observations x t − L : t .
Note that most existing DL models (Salinas et al., 2020; Toubeau et al., 2018; Wang et al., 2019; Oreshkin et al., 2020) directly follow this formulation to solve TS.
Even traditional statistical TS models (Holt, 1957; 2004; Winters, 1960) are indeed consistent with that if omitting those long-tail exponentially decayed dependencies introduced by moving averages.
To precisely formulate PTS, on the other hand, this assumption needs to be slightly modiﬁed such that the dependency of x t : t + H on x t − L : t is further conditioned on the inherent periodicity, which can be anchored by associated time-steps.
Accordingly, we alter the equation (1) into
[Equation]
where other than x t − L : t , F (cid:48)
Θ : R L × R → R H takes an extra argument t , which denotes the forecasting time-step.
Existing methods for PTS adopt a few different instantiations of F (cid:48) Θ .
For example, Holt (1957; 2004) developed several exponentially weighted moving average processes with additive or multiplicative seasonality.
Vecchia (1985a;b) adopted the multiplicative seasonality by treating the coefﬁcients of the auto-regressive moving average process as time dependent.
Smyl (2020) also adopted the multiplicative seasonality and built a hybrid method by coupling that with recurrent neural networks (Hochreiter & Schmidhuber, 1997), while Taylor & Letham (2018) chose the additive seasonality by adding the periodic forecast with other parts as the ﬁnal forecast.
DEPTS
In this section, we elaborate on our new framework, DEPTS.
First, we start with a decoupled formulation of (2) in Section 4.1.
Then, we illustrate the proposed neural architecture for this formulation in Sections 4.2 and 4.3.
Last, we discuss the interpretable capabilities in Section 4.4.
4. T HE D ECOUPLED F ORMULATION
To explicitly tackle the two-sided challenges of PTS forecasting, i.e., complicated periodic dependencies and diversiﬁed periodic compositions, we introduce a decoupled formulation (3) that reﬁnes
DEPTS 𝒙 𝑡−𝐿:𝑡 𝒛 𝑡−𝐿:𝑡+𝐻
Layer
Layer ℓ
Layer 𝑵 𝒙 𝑡−𝐿:𝑡(𝑁) 𝒛 𝑡−𝐿:𝑡+𝐻(𝑁) ෝ𝒙 𝑡:𝑡+𝐻(𝑁) 𝑓 𝜃
Local Block 𝑓 𝜃 𝑙 (ℓ)𝑙 𝒙 𝑡−𝐿:𝑡(ℓ−1)
Periodic Block 𝑓 𝜃 𝑝 (ℓ) 𝑝 𝒛 𝑡−𝐿:𝑡+𝐻(ℓ−1) 𝒙 𝑡−𝐿:𝑡(ℓ) ෝ𝒙 𝑡:𝑡+𝐻(ℓ−1) 𝒛 𝑡−𝐿:𝑡+𝐻(ℓ) 𝒗 𝑡−𝐿:𝑡(ℓ) 𝒗 𝑡:𝑡+𝐻(ℓ) 𝒖 𝑡−𝐿:𝑡(ℓ) 𝒖 𝑡:𝑡+𝐻(ℓ)
ෝ𝒙 𝑡:𝑡+𝐻(ℓ) 𝒗 𝑡−𝐿:𝑡+𝐻(ℓ)
Layer ℓ
Time 𝑡 𝑡−𝐿 𝑡+𝐻
The Expansion Module 𝑓 𝜃
The Periodicity Module 𝑔 𝜑 𝒙 𝑡−𝐿:𝑡 𝒛 𝑡−𝐿:𝑡+𝐻 𝒙 𝑡−𝐿:𝑡(𝑁) 𝒛 𝑡−𝐿:𝑡+𝐻(𝑁) ෝ𝒙 𝑡:𝑡+𝐻 (𝑁) ෝ𝒙 𝑡:𝑡+𝐻
[Equation]
where we treat z t ∈ R 1 as a scalar value to be consistent with the uni-variate TS x t ∈ R 1 , we use f θ : R L × R L + H → R H to model complicated dependencies of the future signals x t : t + H on the local observations x t − L : t and the corresponding periodic states z t − L : t + H within the lookback and forecast horizons, and g [Equation]
1 is to produce a periodic state z t for a speciﬁc time-step t .
The right part of Figure 2 depicts the overall data ﬂows of this formulation, in which the expansion module f θ and the periodicity module g φ are responsible for handling the two aforementioned PTSspeciﬁc challenges, respectively.
4. T HE E XPANSION M ODULE
To effectively model complicated periodic dependencies, the main challenge lies in the trade-off between model capacity and generalization.
To avoid the over-ﬁtting issue, many existing PTS approaches relied on the assumptions of additive or multiplicative seasonality (Holt, 1957; Vecchia, 1985b; Anderson et al., 2007; Taylor & Letham, 2018; Smyl, 2020), which however can hardly express periodicity beyond such simpliﬁed assumptions.
Lately, residual learning has shown its great potentials in building expressive and generalizable DL architectures for a variety of crucial applications, such as computer vision (He et al., 2016) and language understanding (Vaswani et al., 2017).
Speciﬁcally, N-BEATS (Oreshkin et al., 2020) conducted a pioneer demonstration of introducing residual learning to TS forecasting.
Inspired by these successful examples and with full consideration of PTS-speciﬁc challenges, we develop a novel expansion module f θ on top of residual learning to characterize the complicated dependencies of x t : t + H on x t − L : t and z t − L : t + H .
The proposed architecture for f θ , as shown in the middle part of Figure 2, consists of N layers in total.
As further elaboration in the left part of Figure 2, each layer (cid:96) , share an identical residual structure consisting of three residual branches, which correspond to the recurrence relations of z ( (cid:96) )
t − L : t + H , x ( (cid:96) )
t − L : t , and ˆ x ( (cid:96) ) t : t + H , respectively.
Here x ( (cid:96) )
t − L : t and z ( (cid:96) )
t − L : t + H denote the residual terms
In layer (cid:96) , three residual branches are speciﬁed by two parameterized blocks, a local block f lθ l ( (cid:96) ) and a periodic block f pθ p ( (cid:96) ) , where θ l ( (cid:96) ) and θ p ( (cid:96) ) are their respective parameters.
First, we present the updating equation for z ( (cid:96) − 1) t − L : t + H , which aims to produce the forecasts from periodic states and exclude the periodic effects that have been used.
To be more concrete, f pθ p ( (cid:96) ) takes in z ( (cid:96) − 1) t − L : t + H and emits the (cid:96) -th expansion term of periodic states, denoted as v ( (cid:96) )
t − L : t + H ∈ R L
+ H .
v ( (cid:96) )
t − L : t + H has two components, a backcast component v ( (cid:96) )
t − L : t and a forecast one v ( (cid:96) )
t : t + H .
We leverage v ( (cid:96) )
t − L : t to exclude the periodic effects from x ( (cid:96) − 1) t − L : t and adopt v ( (cid:96) ) t : t + H as the portion of forecasts purely from the (cid:96) -th periodic block.
Besides, when moving to the next layer, we exclude v ( (cid:96) )
t − L : t + H from z ( (cid:96) − 1) t − L : t + H as z ( (cid:96) )
t − L : t + H
=
z ( (cid:96) − 1) t − L : t + H − v ( (cid:96) )
t − L : t + H to encourage the subsequent periodic blocks to focus on the unresolved residue z ( (cid:96) )
t − L : t + H .
Then, since v ( (cid:96) ) t − L : t is related to the periodic components that have been used to produce a part of forecasts, we construct the input to f lθ l ( (cid:96) ) as ( x ( (cid:96) − 1) t − L : t − v ( (cid:96) )
t − L : t ) .
Here the purpose is to encourage f lθ l ( (cid:96) ) to focus on the unresolved patterns within x ( (cid:96) − 1) t − L : t .
f lθ l ( (cid:96) ) emits u ( (cid:96) )
t − L : t and u ( (cid:96) )
t : t + H , which correspond to the local backcast and forecast expansion terms of the (cid:96) -th layer, respectively.
After that, we update x ( (cid:96) ) t − L : t by further subtracting u ( (cid:96) )
t − L : t from ( x ( (cid:96) − 1) t − L : t − v ( (cid:96) )
t − L : t ) as x ( (cid:96) )
t − L : t = x ( (cid:96) − 1) t − L : t − v ( (cid:96) )
t − L : t − u ( (cid:96) )
t − L : t .
Here the insight is also to exclude all analyzed patterns of this layer to let the following layers focus on unresolved information.
Besides, we update ˆ x ( (cid:96) ) t : t + H by adding both u ( (cid:96) ) t : t + H and v ( (cid:96) )
t : t + H as ˆ x ( (cid:96) )
t : t + H
= ˆ x ( (cid:96) − 1) t : t + H + u ( (cid:96) )
t : t
+ H + v ( (cid:96) )
t : t + H .
The motivation of such expansion is to decompose the forecasts from the (cid:96) -th layer into two parts, u ( (cid:96) )
t : t + H and v ( (cid:96) ) t : t + H , which correspond to the part from local observations excluding redundant periodic information and the other part purely from periodic states, respectively.
Note that before the ﬁrst layer, we have x (0) t − L : t = x t − L : t , z (0) t − L : t + H =
z t − L : t + H , and ˆ x (0) t : t + H
=
Besides, we collect the cumulative forecasts ˆ x ( N ) t : t + H of the N -th layer as the overall forecasts ˆ x t : t + H .
Therefore, after stacking N layers of z ( (cid:96) ) t − L : t + H , x ( (cid:96) )
t − L : t , and ˆ x ( (cid:96) ) t : t + H , we have the following triply residual expansions that encapsulate the left and middle parts of Figure 2:
[Equation]
) where z ( N ) t − L : t + H and x ( N ) t − L : t are deemed to be the residues irrelevant to forecasting.
Connections and differences to N-BEATS.
Our design of f θ shares the similar insight with NBEATS (Oreshkin et al., 2020), which is stimulating a deep neural network to learn expansions of raw TS signals progressively, whereas N-BEATS only considered the generic TS by modeling the dependencies of x t : t + H on x t − L : t .
In contrast, our design is to capture the complicated dependencies of x t : t + H on x t − L : t and z t − L : t + H for PTS.
Moreover, to achieve periodicity modeling, N-BEATS produces coefﬁcients solely based on the input signals within a lookback window for a group of predeﬁned seasonal basis vectors with ﬁxed frequencies and phases.
However, our work can capture diversiﬁed periods in practice and model the inherent global periodicity.
Inner architectures of local and periodic blocks.
The local block f lθ l ( (cid:96) ) aims to produce a part of forecasts based on the local observations excluding redundant periodic information as ( x t − L : t − (cid:80) (cid:96) −i
=1 v ( i ) t − L : t ) .
Thus, we reuse the generic block developed by Oreshkin et al.
(2020), which consists of a series of fully connected layers.
As for the periodic block f pθ p ( (cid:96) ) , which handles the relatively stable periodic states, we can adopt a simple two-layer perception.
Due to the space limit, we include more details of inner block architectures in Appendix A.
4. T HE P ERIODICITY M ODULE
To represent the sophisticated periodicity composed of various periodic patterns, we estimate z t via a parameterized periodic function g φ ( t ) that holds sufﬁcient capacities to incorporate diversiﬁed periods.
In this work, for simplicity, we instantiate this function as a series of cosine functions as g φ ( t ) =
A 0 + (cid:80) Kk =
A k cos(2 πF k t + P k ) , where K is a hyper-parameter denoting the total number of periods, A 0 is a scalar parameter for the base scale, A k , F k , and P k are the scalar parameters for the amplitude, the frequency, and the phase of the k -th cosine function, respectively, and φ represents the set of all parameters.
Coupling g φ with f θ illustrated in Section 4, we can effectively model the periodicity-aware auto-regressive forecasting process in the equation (2).
However, it is extremely challenging to directly conduct the joint optimization of φ and θ from random initialization.
The reason is that in such a highly non-convex condition, the coefﬁcients in φ are easily trapped into numerous local optima, which do not necessarily characterize our desired periodicity.
Parameter Initialization.
To overcome the optimization obstacle mentioned above, we formalize a two-stage optimization problem based on raw PTS signals to ﬁnd good initialization for φ .
First, we construct a surrogate function, g Mφ ( t ) =
A 0 + (cid:80)
Kk =1 M k · A k cos (2 πF k t + P k ) , to enable the selection of a subset of periods via M = { M k , k ∈ { 1 , · · · , K }} , where each M k ∈ { 0 , 1 } is a mask variable to enable or disable certain periods.
Note that g φ ( t ) is equivalent to g Mφ ( t ) when every
M k is equal to one.
Then, we construct the following two-stage optimization problem:
[Equation]
where L D train and L D val denote the discrepancy losses on training and validation, respectively; the inner stage is to obtain φ ∗ that minimizes the discrepancy between z t and x t on the training data D train ; the outer stage is a binary integer programming on the validation data D val to ﬁnd
M ∗ that can select certain periods with good generalization, and the hyper-parameter J controls the maximal number of periods being selected.
With the help of such two-stage optimization, we are able to estimate generalizable periodic coefﬁcients from observational data as a good starting point for φ to be jointly optimized with θ .
Nevertheless, it is still costly to perform exact optimization of equations (5) in practice.
Thus, we develop a fast approximation algorithm to obtain an acceptable solution with affordable costs.
Our approximation algorithm contains the following two steps: 1) conducting Discrete Cosine Transform (Ahmed et al., 1974) of PTS signals on D train to select topK cosine bases with the largest amplitude as an approximated solution of φ ∗ ; 2) iterating over the selected K cosine bases from the largest amplitude to the smallest one and greedily select J periods that generalize well on the validation set.
Due to the space limit, we include more details of this approximation algorithm in Appendix B. After obtaining approximated solutions ˜ φ ∗ and ˜ M ∗ , we ﬁx M = ˜
M ∗ to exclude those unstable periodic coefﬁcients and initialize φ with ˜ φ ∗ to avoid being trapped into bad local optima.
Then, we follow the formulation (3) to perform the joint learning of φ and θ in an end-to-end manner.
4. I NTERPRETABILITY
Owing to the speciﬁc designs of f θ and g φ , our architecture is born with a degree of interpretability.
First, for f θ , as shown in equations (4), we decompose ˆ x t : t + H into two types of components, u ( (cid:96) )
t : t + H and v ( (cid:96) ) t : t + H .
Note that v ( (cid:96) )
t : t
+ H is conditioned on z t − L : t
+ H and independent of x t − L : t .
Thus, (cid:80) N(cid:96)
=1 v ( (cid:96) ) t : t + H represents the portion of forecasts purely from periodic states.
Meanwhile, u ( (cid:96) ) t : t + H depends on both x t − L : t and z t − L : t + H , and it is transformed by feeding the subtraction of v ( (cid:96) )
t − L : t from x ( (cid:96) − 1) t − L : t into the (cid:96) -th local block.
Thus, we can regard (cid:80) N(cid:96)
=1 u ( (cid:96) )
t − L : t as the forecasts from the local historical observations excluding the periodic effects, referred to as the local momenta in this paper.
In this way, we can differentiate the contribution to the ﬁnal forecasts into both the global periodicity and the local momenta.
Second, g φ , the periodicity estimation module in our architecture, also has interpretable effects.
Speciﬁcally, the coefﬁcients in g φ ( t ) have practical meanings, such as amplitudes, frequencies, and phases.
We can interpret these coefﬁcients as the inherent properties of the series and connect them to practical scenarios.
Furthermore, by grouping various periods together, g φ provides us with the essential periodicity of TS ﬁltering out various local momenta.
E XPERIMENTS
Our empirical studies aim to answer three questions.
1) Why is it important to model the complicated dependencies of PTS signals on its inherent periodicity?
2) How much beneﬁt can DEPTS gain for
PTS forecasting compared with existing state-of-the-art models?
What kind of interpretability can
[Figure]
DEPTS offer based on our two customized modules, f θ and g φ ?
To answer the ﬁrst two questions, we conduct extensive experiments on both synthetic data and real-world data, which are illustrated in Section 5.1 and 5.2, respectively.
Then, Section 4.4 answers the third question by comparing and interpreting model behaviors for speciﬁc cases.
Baselines.
We adopt the state-of-the-art DL architecture, N-BEATS (Oreshkin et al., 2020), as our primary baseline since it has been shown to outperform a wide range of DL models, including
MatFact (Yu et al., 2016), Deep State (Rangapuram et al., 2018), Deep Factors (Wang et al., 2019), and DeepAR (Salinas et al., 2020), and many competitive hybrid methods (Montero-Manso et al., 2020; Smyl, 2020).
Besides, we also include PARMA as a reference to be aware of the positions of conventional statistical models.
For this baseline, we leverage the AutoARIMA implementation provided by L ¨ oning et al.
(2019) to search for the best conﬁgurations automatically.
Evaluation Metrics.
To compare different models, we utilize the following two metrics, normalized deviation, abbreviated as nd , and normalized root-mean-square error, denoted as nrmse , which are conventionally adopted by Yu et al.
; Rangapuram et al.
; Salinas et al.
Oreshkin et al.
(2020) on PTS-related benchmarks.
[Equation]
where i is the index of TS in a dataset, t is the time index, and Ω denotes the whole evaluation space.
5. E VALUATION ON S YNTHETIC D ATA
To intuitively illustrate the importance of periodicity modeling, we generate synthetic data with various periodic dependencies and multiple types of periods.
Speciﬁcally, we generate a simulated
TS signal x t by composing an auto-regressive signal l t , corresponding to the local momentum, and a compounded periodic signal p t , denoting the global periodicity, via a function f c as x t = f c ( l t , p t ) , which characterizes the dependency of x t on l t and p t .
First, we produce l t via an auto-regressive process, l t = (cid:80) L i =
1 α i l t − i + (cid:15) lt , in which α i is a coefﬁcient for the i -lag dependency, and the error term (cid:15)
lt ∼ N (0 , σ l ) follows a zero-mean Gaussian distribution with standard deviation σ l .
Then, we produce p t by sampling from another Gaussian distribution N ( z t , σ p ) , in which z t is characterized by a periodic function (instantiated as g φ ( t ) in Section 4.3), and σ p is a standard deviation to adjust the degree of dispersion for periodic samples.
Next, we take three types of f c ( l t , p t ) , ( l t + p t ) , ( l t + p t ) 2 , and ( l t + p t ) 3 , to characterize the linear, quadratic, and cubic dependencies of x t on l t and p t , respectively.
Last, after data generation, all models only have access to the ﬁnal mixed signal x t for training and evaluation.
Due to the space limit, we include the main results in Figure 3 and leave ﬁner grained parameter speciﬁcations and more experimental details to Appendix C. For each setup (linear, quadratic, cubic) in Figure 3, we have searched for the best lookback length ( L ) for N-BEATS and the best number of periods ( J ) for DEPTS on the validation set and re-run the model training with ﬁve different random seeds to produce robust results on the test set.
We can observe that for all cases, even with an exhaustive search of proper lookback lengths for N-BEATS, there exists a considerable performance gap between it and DEPTS, which veriﬁes the utility of explicit periodicity modeling.
Moreover, as the periodic dependency becomes more complex (from linear to cubic), the average error reduction of DEPTS over N-BEATS keeps increasing (from 7% to 11%), which further demonstrates the importance of modeling high-order periodic effects.
[Table] [Table]
5. E VALUATION ON R EAL - WORLD D ATA
Other than simulation experiments, we further demonstrate the effectiveness of DEPTS on realworld data.
We adopt three existing PTS-related datasets, E LECTRICITY 1 , T RAFFIC 2 , and M (H OURLY ) 3 , which contain various long-term (quarterly, yearly), mid-term (monthly, weekly), and short-term (daily, hourly) periodic effects corresponding to regular economic and social activities.
These datasets serve as common benchmarks for many recent studies (Yu et al., 2016;
Rangapuram et al., 2018; Salinas et al., 2020; Oreshkin et al., 2020).
For E LECTRICITY and T RAFFIC , we follow two different test splits deﬁned by Salinas et al.
(2020) and Yu et al.
(2016), and the evaluation horizon covers the ﬁrst week starting from the split date.
As for M4 (H OURLY ), we adopt the ofﬁcial test set.
Besides, we note that the time horizons covered by these three benchmarks are still too short, which results in very limited data being left for periodicity learning if we alter the time split too early.
This drawback of lacking enough long PTS limits the power of periodicity modeling and thus may hinder the research development in this ﬁeld.
To further verify the importance of periodicity modeling in real-world scenarios, we construct two new benchmarks with sufﬁciently long
PTS from public data sources.
The ﬁrst one, denoted as C AISO , contains eight-years hourly actual electricity load series in different zones of California 4 .
The second one, referred to as NP, includes eight-years hourly energy production volume series in multiple European countries 5 .
Accordingly, we deﬁne four test splits that correspond to all four seasons of the last year for robust evaluation.
For all benchmarks, we search for the best hyper-parameters of DEPTS on the validation set.
Similar to N-BEATS (Oreshkin et al., 2020), we also produce ensemble forecasts of multiple models trained with different lookback lengths and random initialization seeds.
Tables 1 and 2 show the overall performance comparisons.
On average, the error reductions ( nd ) of DEPTS over N-BEATS on
E LECTRICITY , T RAFFIC , M4 (H OURLY ), C AISO , and NP are 12.5%, 3.5%, 8.7%, 13.3%, and 9.9%, respectively.
Interestingly, we observe some prominent improvements in a few speciﬁc cases,
[Figure]
such as 18.7% in E LECTRICITY (2014-09-01), 23.1% in C AISO (2020-10-01), and 15.2% in
NP (2020-10-01).
At the same time, we also observe some tiny improvements, such as 2.6% in
T RAFFIC (2008-06-15) and 3.3% in C AISO (2020-07-01).
These observations imply that the predictive abilities and the complexities of periodic effects may vary over time, which corresponds to the changes in performance gaps between DEPTS and N-BEATS.
Nevertheless, most of the time,
DEPTS still brings stable and signiﬁcant performance gains for PTS forecasting, which clearly demonstrate the importance of periodicity modeling in practice
[Figure]
Due to the space limit
[Figure]
we leave more details about datasets and hyper-parameters used in real-world experiments to Appendix D. Moreover, to achieve effective periodicity modeling, we have made several critical designs, such as the triply residual expansions in Section 4.2 and the composition of diversiﬁed periods in Section 4.3.
We also conduct extensive ablation tests to verify these critical designs, which are included in Appendix E.
5. I NTERPRETABILITY
In Figure 4, we illustrate the interpretable effects of DEPTS via two cases, the upper one from
E LECTRICITY and the bottom one from T RAFFIC .
First, from subplots in the left part, we observe that DEPTS obtains much more accurate forecasts than N-BEATS and PARMA.
Then, in the middle and right parts, we can visualize the inner states of DEPTS to interpret how it makes such forecasts.
As Section 4.4 states, DEPTS can differentiate the contributions to the ﬁnal forecasts ˆ x t : t + H into the local momenta (cid:80) N(cid:96)
=1 u ( (cid:96) )
t : t
+ H and the global periodicity (cid:80)
N(cid:96)
=1 v ( (cid:96) ) t : t + H .
Interestingly, we can see that DEPTS has learned two different decomposition strategies: 1) for the upper case, most of the contributions to the ﬁnal forecasts come from the global periodicity part, which implies that this case follows strong periodic patterns; 2) for the bottom case, the periodicity part just characterizes a major oscillation frequency, while the model relies more on the local momenta to reﬁne the ﬁnal forecasts.
Besides, the right part of Figure 4 depicts the hidden periodic state z t estimated by our periodicity module g φ ( t ) .
We can see that g φ ( t ) indeed captures some inherent periodicity.
Moreover, the actual PTS signals also present diverse variations at different time, which further demonstrate the importance of leveraging f θ to model the dependencies of x t : t + H on both x t − L : t and z t − L : t + H .
We include more case studies and interpretability analysis in Appendix F.
C ONCLUSION
In this paper, we develop a novel DL framework, DEPTS, for PTS forecasting.
Our core contributions are to model complicated periodic dependencies and to capture sophisticated compositions of diversiﬁed periods simultaneously.
Extensive experiments on both synthetic data and real-world data demonstrate the effectiveness of DEPTS on handling PTS.
Moreover, periodicity modeling is actually an old and crucial topic for traditional TS modeling but is rarely studied in the context of DL.
Thus we hope that the new DL framework together the two new benchmarks with evident periodicity and sufﬁciently long observations can facilitate more future research on PTS.
A B LOCK A RCHITECTURES
As illustrated in Section 4.2, the local block f lθ l ( (cid:96) ) produces forecasts based on local observed PTS signals excluding redundant periodic effects.
This goal aligns with that of N-BEATS to extract informative representations from generic TS signals.
Therefore, we reuse the generic block design of
N-BEATS to instantiate f lθ l ( (cid:96) ) .
Here we include a brief description of the local block for completeness.
Please refer to Section 3.1 in (Oreshkin et al., 2020) for more details.
[Equation]
[Figure]
Linear
Linear
[Figure]
Local Block.
The left part of Figure 5 shows the detailed architecture within a local block, where we use ˜ x
( (cid:96) )
t − L : t = x ( (cid:96) − 1) t − L : t − v ( (cid:96) )
t − L : t to denote the portion of the local observations x ( (cid:96) − 1) t − L : t excluding the periodic effects v ( (cid:96) )
t − L : t for the (cid:96) -th layer.
After taking in ˜ x ( (cid:96) ) t − L : t , we pass it through four fullyconnected layers and then obtain the backcast coefﬁcients c ( (cid:96) )
b and the forecast coefﬁcients c ( (cid:96) )
f via two linear projections:
[Equation]
[Equation]
−
[Equation]
[Equation]
[Equation] ( (cid:96) ) , [Equation] − L :
[Equation]
=
FC (cid:96)
[Equation]
2 ( u ( (cid:96) ) , 1 t − L : t ) , u ( (cid:96) ) , 3 t − L : t = FC (cid:96), 3 ( u ( (cid:96) ) , 2 t − L : t )
[Equation]
( (cid:96) )
[Equation]
−
[Equation]
: t ) , where FC is a standard fully-connected layer with ReLU activation (Nair & Hinton, 2010), and
LINEAR denotes a linear projection function.
Then, we pass these coefﬁcients to the basis layers, h b ( · ) and h f ( · ) , to obtain the backcast term u
( (cid:96) )
t − L : t and the forecast term u ( (cid:96) )
t : t + H , respectively.
The generic choice for h b ( · ) can simply be another linear projection function, which is also adopted by us since it produces more competitive and stable performance on PTS-related benchmarks than other interpretable basis layers, as shown by (Oreshkin et al., 2020) in Appendix C.4.
Periodic Block.
The periodic block f pθ p ( (cid:96) ) aims to extract predictive information from associated periodic states, which are relatively simple and stable compared with rapidly shifting PTS signals.
Therefore, we can adopt a simple architecture while still maintain desired effects.
In this work, we use one-layer standard fully-connected layer to encode z ( (cid:96) − 1) t − L : t and leverage another two linear projection functions to obtain the backcast term v ( (cid:96) )
t − L : t and the forecast term v ( (cid:96) )
t : t + H as the periodic effects of the (cid:96) -th layer.
[Equation]
where FC and LINEAR share the same meanings mentioned above.
Moreover, when training for multiple series simultaneously, we use a series-speciﬁc scalar parameter α i ( i is the series index) to take account of differences in the strengths of periodicity by updating v ( (cid:96) )
t − L : t + H as α i · v ( (cid:96) )
t − L : t + H .
B P ARAMETER I NITIALIZATION FOR THE P ERIODICITY M ODULE
As illustrated in Section 4.3, we leverage a fast approximation approach to obtain an acceptable solution of the two-stage optimization problem (5) with affordable costs in practice.
Algorithmsummarizes the overall procedure for this fast approximation.
Input: D train = x 0: T v , D val =
x T v : T , K , and J Conduct DCT over x 0: T v .
Sort the top- K cosine bases by amplitudes in descending order to obtain ˜ φ ∗ =
˜
A ∗ 0 } ∪ { ˜ A ∗ k , ˜ F ∗ k , ˜
P ∗ k } Kk =1 .
Initialize ˜ M ∗ = 0 .
for j in [1 , · · · , K ] do if (cid:107) ˜
M ∗ (cid:107) 1 < J then
Update ˜ M ∗ j by arg min M j ∈{ 0 , 1 } L D val ( g M j ˜ φ ∗ ( t )) else return ˜ φ ∗ and ˜ M ∗ end end
Output: ˜ φ ∗ and ˜ M ∗
First, we divide the whole PTS signals x 0: T into the training part D train = x 0: T v and the validation part D val =
x T v : T , where T v is the split time-step.
Then, the inner optimization stage is to identify the optimal parameter set φ ∗ that can best ﬁt the training data:
[Equation]
where the hyper-parameter K controls the capacity of g φ ( t ) and the discrepancy training loss
L D train can be instantiated as the mean square error (cid:80) T v − 1 t =
(cid:107) g φ ( t ) − x t (cid:107) 22 .
Directly optimizing (7) via gradient descent from random initialization is inefﬁcient and time-consuming since it involves numerous gradient updates and is easily trapped into bad local optima.
Fortunately, our instantiation of g φ ( t ) as a group of cosine functions shares the similar format with Discrete Cosine Transform (DCT) (Ahmed et al., 1974).
Accordingly, we conduct DCT over x 0: T v and select top- K cosine bases with the largest amplitudes, which characterize the major periodic oscillations of this series, as the approximated solution ˜ φ ∗ of (7).
Next, we enter the outer optimization stage to select certain periods with good generalization:
[Equation]
where the hyper-parameter J further constrains the expressiveness of g M φ ( t ) for good generalization.
Conducting exact optimization of this binary integer programming is also costly since it involves an exponentially growing parameter space.
Similarly, to capture the major periodic oscillations as much as possible, we develop a greedy strategy that iterates the selected K cosine bases from the largest amplitude to the smallest and greedily assigns 1 or 0 to M k depending on whether the k -th period further reduces the discrepancy loss on the validation data.
Speciﬁcally, assuming K periods are already sorted by their amplitudes descendingly and are indexed by k , we construct another surrogate function g M j φ ∗ ( t ) for the j -th greedy step:
[Equation]
where { ˜ M ∗ k } j − 1 k =1 is determined in previous steps, M j is an integer parameter to be set in the current step.
Thus, for the j -th step, we are actually updating ˜ M ∗ j by
[Equation]
Besides, to tolerate the approximation errors introduced by ˜ φ ∗ , which may result in shifted periodic oscillations, we use Dynamic Time Warping to measure the discrepancy of g M j φ ∗ ( t ) and x t on D val .
We continue this greedily updating steps until selecting J periods in total or completing the traverses of all K selected periods.
Finally, we obtain an approximated solution ˜
M ∗ of (8).
Complexity Analyses.
We also provide the complexity analyses of Algorithm 1, which runs very fast in practice and takes up negligible time compared with training neural networks.
Let us denote the length of training series as L t and the length of validation series as L v .
First, the complexity of conducting DCT over training series is O
( L t log ( L t )) .
Then, the complexity of selecting top-K frequencies with the largest amplitudes is O
( L t log ( K )) , which can be ignored since K << L t .
Next, we need to select at most J frequencies greedily based on the generalization errors on the validation set.
Since we measure the generalization errors via dynamic time warping, the total worst complexity for this selection procedure is O ( KL 2 v ) .
In total, the worse complexity of our approximation algorithm for a series is O
( L t log ( L t )
+ KL 2 v ) .
In practice, L v , the length of the validation series, is relatively small, and K , the maximum number of frequencies, can be regarded as a constant.
So, the squared complexity term O ( KL 2 v ) is not a big trouble.
C M ORE D ETAILS ON S YNTHETIC E XPERIMENTS
As Section 5.1 states, we produce a TS l t via an auto-regressive process, l t = (cid:80) Li
=
α i l t − i + (cid:15) lt , in which α i is a coefﬁcient for the i -lag dependence, and the error term (cid:15) lt ∼ N (0 , σ l ) follows a zero-mean Gaussian distribution with standard deviation σ l .
Speciﬁcally, we set L as 3 and σ l as 1.
We leverage uniform samples from [ − 1 , 1] to initialize { α i } 3 i =1 and also uniformly sample three values from [0 , 5 for the initial points, l − 3 , l − 2 , and l − 1 .
Then, we produce p t by sampling from another Gaussian distribution N ( z t , σ p ) , in which z t is characterized by a periodic function (instantiated as g φ ( t ) in Section 4.3), and σ p is a standard deviation to adjust the degree of dispersion for periodic samples.
Speciﬁcally, we also set σ p as 1 and produce z t via a composition of three cosine bases, 8 cos (2 π ( t + 2) / 50) , 4 cos (2 π ( t + 3) / 10) , 2 cos (2 πt/ 4) , and a base level, 30 .
These three cosine bases represent long-term, mid-term, short-term periodic effects, respectively, which are very similar to the circumstance in practice.
Next, we take three types of f c ( l t , p t ) , ( l t + p t ) , ( l t + p t ) 2 , and ( l t + p t ) 3 , to characterize the linear, quadratic, and cubic dependencies of x t on l t and p t , respectively.
We repeat the above procedure for 5000 time steps and divide them into 4000, 100, and 900 for training, validation, and evaluation, respectively.
Figure 6 shows the ﬁrst 1000 time steps of these synthetic series.
Note that after data generation, all models only have access to the ﬁnal mixed signal x t for training and evaluation.
Moreover, as illustrated in Section 5.1, we search for the best loobkack length ( L ) for N-BEATS and the best number of periods ( J ) for DEPTS.
The lookback length for DEPTS is ﬁxed as 3, which is also determined by hyper-parameter tuning on the validation set.
Figure 7 shows detailed comparisons of N-BEATS and DEPTS for different conﬁgurations of L and J .
We can see that
N-BEATS always needs a relatively long lookback window, such as 48 or 96 time steps, to capture those periodic patterns effectively.
Besides, further increasing the lookback length will introduce more irrelevant noises, which overwhelm effective predictive signals and thus result in more worse performance.
In contrast, with effective periodicity modeling, DEPTS can achieve better performance by using a short lookback window, which is also consistent with the auto-regressive process that governs the local momenta.
[Figure]
D M ORE D ETAILS ON R EAL - WORLD E XPERIMENTS D. D ATASETS
Table 3 includes main statistics of the ﬁve datasets used by our experiments.
We can see that the existing datasets (E LECTRICITY , T RAFFIC , and M4 (H OURLY )) utilized by recent studies usually have a large number of series but with relatively short lengths.
Therefore, it is hard to identify or evaluate yearly or quarterly periods on these benchmarks.
In contrast, C AISO and NP contain tens of series with the lengths of several years, which can better illustrate the inherent periodicity of these series and serve as complementary benchmarks for PTS modeling
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table]
rate 1e [Table]
(Kingma & Ba, 2014)
D. H YPER - PARAMETERS
For N-BEATS, we use its default hyper-parameters 6 for E LECTRICITY , T RAFFIC , and M (H OURLY ), and we report its hyper-parameters searched on C AISO and NP in Table 4.
Besides,
N-BEATS used multiple loss functions, such as sMAPE or MASE, for model training, and we also follow these setups.
Tables 5 and 6 include the hyper-parameters of DEPTS for all ﬁve datasets.
Note
[Table]
[Table]
that, all these hyper-parameters are searched on a validation set
[Table]
which is deﬁned as the last week before the test split.
Moreover, for a typical dataset with multiple series
[Table]
we build an independent periodicity module g φ for each series and perform respective parameter initialization procedures as described in Appendix B.
Then, for all datasets (splits), we train 30 models (6 lookback lengths × 5 random seeds) for both N-BEATS and DEPTS and then produce ensemble forecasts for fair and robust evaluation.
E A BLATION T ESTS
As Figure 8 shows, we adopt three ablated variants of DEPTS to demonstrate our critical designs in the expansion module (Section 4.2):
• DEPTS-1 : removing the residual connection of ( x ( (cid:96) − 1) t − L : t − v ( (cid:96) )
t − L : t ) so that the outputs of the local block u ( (cid:96) ) t − L : t are only conditioned on the raw PTS signals x t − L : t , which correspond to the mixed observations of local momenta and global periodicity.
• DEPTS-2 : removing the residual connection of (ˆ x ( (cid:96) − 1) t : t
+ H + v ( (cid:96) )
t : t + H ) so that the contributions to the forecasts only come from the local block, which takes in the signals excluding periodic effects progressively.
• DEPTS-3 : removing the residual connection of ( z ( (cid:96) − 1) t − L : t + H ) − v ( (cid:96) )
t − L : t + H so that the inputs to the periodic block of each layer are the same hidden variables z t − L : t + H .
We also construct another four baselines to demonstrate the importance of our customized periodicity learning:
• NoPeriod : removing the periodic blocks by directly feeding ( x t − z t ) to N-BEATS.
• RandInit : randomly initializing periodic coefﬁcients ( φ ) and directly applying the end-toend learning.
• FixPeriod : ﬁxing the periodic coefﬁcients ( φ ) after the initialization stage and only tuning θ during end-to-end optimization.
• MultiVar : treating z t as a covariate of x t and feeding ( x t , z t ) into an N-BEATS-style model via two channels.
Moreover, as illustrated in Section 4.3 and Appendix B, the maximal number of selected periods
J is a critical hyper-parameter to balance expressiveness and generalization of g φ ( t ) .
Thus, we conduct experiments with different J to verify its sensitivity on different datasets.
Tables 7 andinclude experimental results of these model variants on E LECTRICITY , T RAFFIC , C AISO , and NP with different J .
Since we only identify one reliable period via Algorithm 1 on M4 (H OURLY ), we report its results separately in Table 9.
Local Block 𝑓 ! !
[Equation]
ℓ)% 𝒙 &’(:&(ℓ’*)
Periodic Block 𝑓 ! "
(ℓ)+ 𝒛 &’(:&,-(ℓ’*) 𝒙 &’(:&(ℓ) 𝒙$ &:&,-(ℓ’*) 𝒛 &’(:&,-(ℓ) 𝒗 &’(:&(ℓ) 𝒗 &:&,-(ℓ) 𝒖 &’(:&(ℓ) 𝒖 &:&,-(ℓ) 𝒙$ &:&,-(ℓ) 𝒗 &’(:&,-(ℓ)
[Table]
(a) DEPTS
Local Block 𝑓 ! !
(ℓ)% 𝒙 &’(:&(ℓ’*)
Periodic Block 𝑓 ! "
(ℓ)+ 𝒛 &’(:&,-(ℓ’*) 𝒙 &’(:&(ℓ) 𝒙$ &:&,-(ℓ’*) 𝒛 &’(:&,-(ℓ) 𝒗 &:&,-(ℓ) 𝒖 &’(:&(ℓ) 𝒖 &:&,-(ℓ) 𝒙$ &:&,-(ℓ) 𝒗 &’(:&,-(ℓ)
Layer ℓ (b) DEPTSLocal Block 𝑓 ! !
(ℓ)% 𝒙 &’(:&(ℓ’*)
Periodic Block 𝑓 ! "
(ℓ)+ 𝒛 &’(:&,-(ℓ’*) 𝒙 &’(:&(ℓ) 𝒙$ &:&,-(ℓ’*) 𝒛 &’(:&,-(ℓ) 𝒗 &’(:&(ℓ) 𝒖 &’(:&(ℓ) 𝒖 &:&,-(ℓ) 𝒙$ &:&,-(ℓ) 𝒗 &’(:&,-(ℓ)
Layer ℓ (c) DEPTSLocal Block 𝑓 ! !
(ℓ)% 𝒙 &’(:&(ℓ’*)
Periodic Block 𝑓 ! "
(ℓ)+ 𝒛 &’(:&,-(ℓ’*) 𝒙 &’(:&(ℓ) 𝒙$ &:&,-(ℓ’*) 𝒛 &’(:&,-(ℓ) 𝒗 &’(:& (ℓ) 𝒗 &:&,-(ℓ) 𝒖 &’(:&(ℓ) 𝒖 &:&,-(ℓ) 𝒙$ &:&,-(ℓ) 𝒗 &’(:&,-(ℓ)
Layer ℓ (d) DEPTS-
First, as Tables 7 and 8 show, J is a crucial hyper-parameter that has huge impacts on forecasting performance.
The reason is that if J is too small, the periodicity module g φ cannot produce effective representations of the inherent periodicity to boost the predictive ability.
While if it is too large, g φ has a high risk of over-ﬁtting to the irrelevant noises contained by the training data, which also results in poor predictive performance.
Moreover, the interactions between local momenta and global periodicity may vary over time.
Therefore, it is critical to search for a proper J for each PTS and
[Table]
DEPTS0.
[Table] [Table]
DEPTS [Table] [Table] [Table]
[Table]
DEPTS
[Table]
[Table]
DEPTS
[Table]
DEPTS
[Table]
[Table]
RandInit 0.
FixPeriod 0.
MultiVar 0.
DEPTS 0.
NoPeriod 0.
RandInit 0.
FixPeriod 0.
MultiVar 0.
DEPTS 0.
NoPeriod 0.
RandInit 0.
FixPeriod 0.
MultiVar 0.
DEPTS 0.
NoPeriod 0.
RandInit 0.
FixPeriod 0.
MultiVar 0.
DEPTS 0.
each split point to pursue better performance.
Fortunately, we demonstrate that the hyper-parameter tuning of J on the validation set can ensure its good generalization abilities on the subsequent test horizons.
Then, let us focus on Table 7 to compare DEPTS with DEPTS-1, DEPTS-2, and DEPTS-3.
First, we can see that DEPTS-1 usually produces the worst performance in most cases, which demon-
[Table]
strates that excluding periodic effects from raw PTS signals can stably and signiﬁcantly boost the performance of PTS forecasting.
Second, for most cases, DEPTS outperforms DEPTS-2, and the performance gaps can be remarkable, such as 0 .
139 vs. 0 .
148 on E LECTRICITY and 0 .
020 vs. 0 .on
C AISO .
These results verify the importance of including the portion of forecasts solely from the periodicity module.
Third, DEPTS-3 can produce competitive results compared with DEPTS in many cases.
Nevertheless, after selecting the best J for each dataset, DEPTS still slightly outperforms DEPTS-3 in most cases.
Besides, from Table 9, we also observe that DEPTS performs much better than DEPTS-3 on M4 (H OURLY ).
Thus we retain the residual connection to reduce the periodic effects leveraged by previous layers.
Next, let us focus on Table 8 to compare DEPTS with other four baselines, NoPeriod, RandInit,
MultiVar, and FixPeriod.
First, we observe that NoPeriod usually produces the worst preformance.
The reason is that ( x t – z t ) denotes the raw time-series signal subtracting the periodic effect, so it is challenging for the model to forecast the future signals, x t : t + H , solely based on the periodicityagnostic inputs, ( x t − L : t – z t − L : t ) .
Second, RandInit also produces much more worse results than
DEPTS, which demonstrate the importance of initializing periodic coefﬁcients (Section 4.3).
Third,
DEPTS performs much better than FixPeriod in most cases, which demonstrate the effectiveness of ﬁne-tuning periodic coefﬁcients after the initialization stage.
Last, we observe that sometimes MultiVar can produce comparable and even slightly better results than DEPTS.
However, after selecting the best J on each dataset for these two models, we ﬁnd that DEPTS still outperforms MultiVar consistently and signiﬁcantly, which also demonstrates the superiority of our expansion learning.
Moreover, as Table 9 shows, DEPTS outperforms all these baselines by a large margin on M (H OURLY ), which containing very short PTS with 854 observations on average.
Given limited data, all our critical designs, such as properly initializing periodic coefﬁcients, ﬁne-tuning periodic coefﬁcients, and conducting expansion learning to decouple the dependencies of x t on z t , play much crucial roles in producing accurate forecasts.
F M ORE C ASE S TUDIES AND I NTERPRETABILITY A NALYSES
In the following, we further study the interpretable effects of DEPTS with more cases.
Figures 9, 10, 11 and 12 show the additional two cases on E LECTRICITY , T RAFFIC , C AISO , and NP, respectively.
Following Figure 4, we compare the forecasts of N-BEATS and DEPTS in the left side, differentiate the forecasts of DEPTS into the local part (DEPTS-L) and the periodic part (DEPTS-P) in the middle side, and plot the hidden state z t together with the PTS signals in the right side.
The general observations are that with the help of explicit periodicity modeling, DEPTS achieves better performance than N-BEATS in PTS forecasting, and DEPTS has learned diverse behaviors for different cases.
Besides, we also include their critical periodic coefﬁcients (amplitude A k , frequency
F k , and phase P k ) in Tables 10, 11, 12, and 13.
We ﬁnd that DEPTS can learn many meaningful periods that are consistent with practical domains.
[Figure]
c [Figure] r i [Figure]
i t
[Figure]
i
[Figure]
l e c t r i c i t
y i d
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table] [Table]
[Figure]
[Table]