H UBER
A DDITIVE M ODELS FOR N ON - STATIONARY
T IME S ERIES A NALYSIS
Anonymous authors
Paper under double-blind review
A BSTRACT
Sparse additive models have shown promising ﬂexibility and interpretability in processing time series data.
However, existing methods usually assume the time series data to be stationary and the innovation is sampled from a Gaussian distribution.
Both assumptions are too stringent for heavy-tailed and non-stationary time series data that frequently arise in practice, such as ﬁnance and medical ﬁelds.
To address these problems, we propose an adaptive sparse
Huber additive model for robust forecasting in both non-Gaussian data and (non)stationary data.
In theory, the generalization bounds of our estimator are established for both stationary and non-stationary time series data, which are independent of the widely used mixing conditions in learning theory of dependent observations.
Moreover, the error bound for non-stationary time series contains a discrepancy measure for the shifts of the data distributions over time.
Such a discrepancy measure can be estimated empirically and used as a penalty in our method.
Experimental results on both synthetic and real-world benchmark datasets validate the effectiveness of the proposed method.
I NTRODUCTION
Additive model has become one of the most powerful tools for time series analysis due to the exemplary monograph (Stone, 1985; Hastie & Tibshirani, 1990) and companion software (Chambers & Hastie, 1992).
For the past two decades, the growing importance of algorithmic ﬂexibility and interpretability motivates the development of various additive models along with theoretical explorations (Huang & Yang, 2004; Wang & Yang, 2007; Chu & Glymour, 2008; Song & Yang, 2010;
Yang et al., 2018) and practical applications (Dominici et al., 2002; Wang & Brown, 2011; Ravindra et al., 2019; Bussmann et al., 2020).
Although these aforementioned works have shown promising behaviours, the methods proposed in these works require some stringent assumptions on the stochastic process, e.g., various mixing conditions (Doukhan, 1994), stationary distribution and Gaussian innovation.
A number of attempts have been made to relax such stringent assumptions.
For the purpose of dealing with non-Gaussian innovation, Qiu et al.
(2015) develop an elliptical vector autoregressive model for estimating heavy-tailed stationary processes with parametric convergence analysis that reduces the inﬂuence of non-Gaussian innovation.
Moreover, under stationarity and decaying β -mixing condition, Wong et al.
(2020) derive nonasymptotic estimation error of lasso without assuming special parametric form of the data generating process.
Stationarity and various mixing conditions are commonly adopted in many previous studies, see, e.g., (Mohri & Rostamizadeh, 2009; 2010; Kock & Callot, 2015; Wong et al., 2020).
However, the mixing and stationary conditions are too stringent and not always valid in practice (Baillie, 1996; Kuznetsov & Mohri, 2020a).
To relax the mixing and stationary conditions, Adams & Nobel (2010) prove asymptotic guarantees for stationary ergodic sequences.
Agarwal & Duchi (2013) establish generalization bounds for asymptotically stationary (mixing) processes in the case of stable on-line learning algorithms.
Kuznetsov & Mohri (2014) establish learning guarantees for fully non-stationary and mixing processes.
Recently, Kuznetsov & Mohri (2020a) provide data-dependent generalization risk bounds for non-stationary and non-mixing stochastic processes.
However, both theoretically and practically, the exploration on robust additive models for nonstationary time series analysis is still limited.
In this paper, we propose a class of sparse
Huber additive models with theoretical guarantees.
Our main contributions are summarized as follows:
• Algorithm design and theoretical guarantees : In Section 3.1, we ﬁrst propose a novel sparse
Huber additive model (SpHAM) by integrating Huber loss and sparsity-inducing (cid:96) 2 , 1 -norm regularizer into an additive data-dependent hypothesis space.
This proposed method can achieve robust forecasting and satisfactory inference (e.g., Granger causal discovery) simultaneously.
In theory, Section 3.2 establishes the upper bound of the function approximation error of SpHAM by developing error decomposition technique (Wu et al., 2006; Chen et al., 2020) and employing sequential Rademacher complexity (Rakhlin et al., 2010; 2015; Kuznetsov & Mohri, 2020a).
With properly selected scale parameters, the theoretical ﬁndings indicate that: a) For stationary time series, the function approximation consistency with convergence rate O ( n − 12 ) can be pursued, even if the innovation is nonGaussian distribution (see Theorem 1 and Corollary 1 for more details).
Moreover, this consistency analysis appears to be novel because no explicit data dependence assumptions are not imposed here, e.g., various mixing conditions used in (Doukhan, 1994; Mohri & Rostamizadeh, 2010; Zou et al., 2009; Wong et al., 2020); b) For non-stationary time series, the function approximation error is bounded by a discrepancy measure, which characterizes the drifts of the data distributions along with time (see Theorem 2 for more details).
By penalizing such a discrepancy measure, Section 3.3 further proposes an adaptive SpHAM for dealing with non-stationary time series and provides its theoretical upper bound correspondingly (see Theorem 3 for more details).
• Optimization and empirical evaluations : The proposed SpHAM and adptive SpHAM can be implemented efﬁciently by Difference of Convex programming (DC programming)
(Tao & An, 1998) and Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) (Beck & Teboulle, 2009).
Experimental results on both synthetic and real-world benchmark CauseMe (Runge et al., 2019) validate the effectiveness of the proposed method.
Related works : There exist some interesting studies towards sparse additive model for time series analysis from both theoretical and practical viewpoints; see, e.g., (Chu & Glymour, 2008; Song &
Yang, 2010; Yang et al., 2018).
Although they show promising interpretability and modeling capacity, all of them require Gaussian innovation, stationarity and various mixing conditions.
Moreover, these assumptions are not always valid in heavy-tailed and non-stationary time series data.
In contrast to these previous additive models, we are seeking to formulate our method and investigate its asymptotic properties without resorting to such strict assumptions.
Robust forecasting is a vibrant area of research in time series analysis (Qiu et al., 2015; Wong et al., 2020).
As one of the triumphs and milestones of robust statistics, the success stories of Huber algorithms are mainly on robust prediction tasks with i.i.d datasets (Huber, 1964; Huber & Ronchetti, 2009; Loh, 2017; Feng & Wu, 2020), and their extensions to time series prediction are fairly sparse.
To our best knowledge, this is the ﬁrst work that considers Huber additive models for non-stationary and dependent time series data.
Our theoretical analyses are inspired from the successful usage of sequential Rademacher complexity in Kuznetsov & Mohri (2020b;a). The generalization risk bounds are established in their works for a general scenario of non-stationary and non-mixing stochastic processes.
However, it should be mentioned that we are chieﬂy concerned about the function approximation analysis, which is essentially different from theirs and is very crucial for Huber algorithms, since the convergence of generalization risk cannot imply the convergence of function approximation (Sun et al., 2019; Feng & Wu, 2020).
Moreover, the development of analysis techniques (e.g., error decomposition and sequential Rademacher complexity) for assessing our method may shed light on other robust models for nonstationary time series analysis.
To better highlight the novelty of our method, Table 1 summarizes the algorithmic properties of our method and other related works, e.g., Sparse additive models for time series (TS-SpAM)
(Yang et al., 2018), Discrepancy-based Forecasting (DBF) (Kuznetsov & Mohri, 2020a) and Robust Dantzigselector-type estimator (R-Dantzig) (Qiu et al., 2015). [Table]
The remainder of this paper is organized as follows.
Section 2 recalls the background of additive models.
Section 3 mainly provides our methods and theoretical guarantees.
We provide empirical evaluations in Section 4.
Finally, Section 5 concludes this paper.
P RELIMINARY
Let { Z t } ∞ t =
−∞ be a stochastic time series with time index t , where variable Z t =
( X t , Y t ) takes values in the compact input space
X ⊂ R p and the output space Y ⊂ R .
We consider a common nonparametric model
[Equation]
where f ∗ ( · ) is the ground truth function, and the innovation ε t is i.i.d.
across time t ∈ Z .
For the sake of simplicity, we denote ρ t and ρ t X as the jointed distribution of ( X t , Y t ) and the corresponding marginal distribution with respect to X t , respectively.
This setup actually covers a large number of scenarios commonly used in practice.
For instance, the case X t that contains p lagged values of Y t (e.g., X t =
Y ( t − p ) × · · · × Y ( t − 1) ) corresponds to the p -order autoregressive models.
Moreover, this case can be viewed as a vector autoregressive model in the sense that input X t includes the historical information of multiple variables.
Although such a nonparametric model (1) makes very few assumptions on data generation, the related nonparametric algorithms suffer so-called “curse of dimensionality”, see Fan & Gijbels (1996) for further discussion.
An effective strategy for solving this problem is additive model.
Usually, the additive structure is obtained by decomposing the input space X ∈ R p into X =
X 1 × ...
× X p .
Under the assumption that the ground truth admits an additive structure f ∗ = (cid:80) pj =1 f ∗ j , the additive model can be deﬁned as
[Equation]
where each component f ∗ j : X j → R is a smooth function.
In linear time series analysis, a weak stationarity condition (i.e., the ﬁrst two moments of time series are time invariant) is preferred (Han et al., 2015; Qiu et al., 2015).
In contrast, strict stationarity is primarily used if our focus is on nonlinear relationships (Fan & Yao, 2005).
Deﬁnition 1.
A stochastic process { Z t } ∞ t =
−∞ is strictly stationary if ( Z 1 , ..., Z t ) and ( Z 1+ k , ..., Z t + k ) have the same joint distributions for any t ∈ Z and k ∈ Z .
Note that, if not otherwise stated, the stationarity in this paper refers to strict stationarity.
Suppose that we are given T size time series data { ( x t , y t ) }
Tt =1 ∈ Z T which are drawn from an additive data-generating model (2).
Under stationarity condition and zero-mean Gaussian innovation with ﬁnite variance, the most existing methods that learn the ground truth f ∗ usually integrate squared loss and a smoothness- or sparsity-inducing regularizer Ω( · ) into a structural risk minimization scheme:
[Equation]
where H :=
{ f 1 + ... + f p : f j ∈ H j , j = 1 , ..., p } is an additive hypothesis space.
Commonly, each subspace H j can be reproducing kernel Hilbert space (Chen et al., 2017; Kandasamy & Yu, 2016;
Raskutti et Yang et al., 2018), or the composite function space with the neural network as a typical (Agarwal et al., 2020; Bussmann et al., 2020).
However, when facing heavy-tailed innovation, these methods may have degraded performance due to the ampliﬁcation of the squared loss to large residuals.
In statistic learning community, as one commonly used robust statistic, Huber loss is deﬁned as
[Equation]
where σ is a positive hyper-parameter.
Note that in the previous studies (Huber & Ronchetti, 2009;
Loh, 2017), the hyper-parameter σ is set to be ﬁxed according to the 95% asymptotic efﬁciency rule.
However, Huber regression with a ﬁxed scale parameter may not be able to learn the ground truth when the noise is asymmetric, as argued recently in Feng & Wu (2020); Sun et al.
In this paper, we choose the scale parameter σ by relating it to the moment condition of the noise distribution and the sample size so that the resulting regression estimator can asymptotically converge to the ground truth function.
M ETHOD 3. S PARSE H UBER ADDITIVE MODELS
In this paper, we choose reproducing kernel Hilbert space (RKHS)
H K j , j = 1 , .., p, to form the the additive hypothesis space H , where each H K j is associated with a symmetric and positive semi-deﬁnite Mercer kernel K j : X j × X j → R .
An additive RKHS is deﬁned as
[Equation]
with kernel norm (cid:107) f (cid:107) 2 K = [Equation] f [Equation]
+ ... +
[Equation]
f [Equation] p } . By integrating the Huber loss (3), additive
RKHS and kernel norm inducing regularizer into a Tikhonov regularization scheme, the kernel-based
Huber additive model can be formulated as
[Equation]
where η is positive regularization parameter and τ j is the weight for j -th kernel norm.
The representer theorem (Wahba, 1990) ensures that ˆ f η can be represented as
[Equation]
To offer the method with sparsity, we consider the following sparsity-inducing penalty
[Equation]
Let H Z be an additive data dependent hypothesis space deﬁned by
[Equation]
Then the SpHAM can be formulated as
[Equation]
=
[Equation]
+
[Equation]
Denote K tj =
( K j ( x 1 j , x tj ) , ..., K j ( x Tj , x tj )) (cid:48) ∈ R T , α = ( α (cid:48) 1 , ..., α (cid:48) p ) (cid:48) ∈ R Tp and α j = ( α 1 j , ..., α Tj ) (cid:48) ∈ R T , where α (cid:48)
j here refers to the transposition of α j for avoiding confusion with time T .
The SpHAM can be represented as
[Equation]
with
[Equation]
p (cid:88) j = τ j (cid:107) α j (cid:107) 2 } .
The optimization problem (7) can be solved by Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) (Beck & Teboulle, 2009).
We provide the detailed optimization procedure in Appendix E due to space limitation.
Remark 1.
Inspired by the studies on group additive models (Yin et al., 2012; Pan & Zhu, 2017; Chen et al., 2017), the modeling capacity of our proposed method can be improved by embedding the (Hierarchical) group structure. For instance, group SpHAM can be easily formulated by replacing the direct decomposition {X j } pj =1 , X j ∈ R with the subgroups decomposition {X d } Dd =1 , where each
X d ⊂ X may cover more than one variable.
3. A SYMPTOTIC THEORY ANALYSIS
In this section, we provide the theoretical analysis of SpHAM.
It is natural and necessary to investigate the following three problems:
• The function estimation consistency and convergence rate under stationary time series data (See Theorem 1 in Section 3.2.1); • The upper bound of the function estimation error under non-stationary time series data (See Theorem 2 in Section 3.2.2); • Whether the theoretical ﬁndings can inspire an adaptive SpHAM for dealing with nonstationary time series (See Theorem 3 in Section 3.3)
Due to space limitation, the high-level outline and detailed proofs are provided in the Appendix A-D.
3.2. F UNCTION APPROXIMATION ANALYSIS FOR STATIONARY TIME SERIES
Through this paper, the marginal distribution w.r.t Y t is assumed to be almost everywhere supported on [ − M, M ] for some M ≥ 0 .
Assumption 1.
Assume that | Y t | , ∀ t ∈ Z , is bounded and there exists a constant c > 0 such that E [Equation] < ∞ , [Equation]
t ∈ Z . The moment condition in Assumption 1 is rather weak in the sense that the response variable Y t possesses inﬁnite variance.
The same condition also applies to the distributions of the innovation ε t , implying that heavy-tailed innovation is allowed.
Assumption 2.
Let κ = sup x ∈X (cid:112) K j ( x, x ) < ∞ , ∀ j = 1 , ..., p .
Assumption 2 only requires that the kernel is bounded under compact X , which holds for all Mercer kernels (e.g., Gaussian kernel) and has been used in many literatures; see, e.g., (Wu et al., 2006;
Steinwart & Christmann, 2008; Wu & Zhou, 2008).
The following deﬁnitions are needed for Assumption 3.
For any j = 1 , ..., p , we deﬁne a kernel integral operator L K j ,T +1 :
L 2 ( ρ T +1 X j )
→ L 2 ( ρ T +1 X j ) associated with the kernel K j by
[Equation]
Note that L K j ,T +1 is a compact and positive operator on L 2 ( ρ T +1 X j ) .
According to Mercer theorem, we can ﬁnd the corresponding normalized eigenpairs { ( ζ ji , ψ ji ) }
i ≥ 1 such that { ψ ji } i ≥ 1 is an orthonormal basis of L 2 ( ρ T +1 X ) and ζ ji → 0 as i → ∞ .
Then for given r > 0 , we deﬁned the r -th power L rK j ,T + by L rK j ,T +1 ( (cid:88) i ≥β ji
[Equation]
ji )
= (cid:88)
i ≥β ji ( ζ ji ) r ψ ji .
Assumption 3.
We assume that f ∗ j : X j → R , ∀ j = 1 , ..., p is a function of the form f ∗ j =
L rK j ,T +1 ( g ∗ j ) , ∀ r ∈ (0 , 12 ] with some g ∗ j ∈ L 2 ( ρ T +1 X j ) , ∀ h ∈ Z .
Assumption 3 is a natural extension from i.i.d setting (Assumption 1 in (Wu et al., 2006; Christmann & Zhou, 2016)) to non-i.i.d time series.
Indeed, this assumption stands in most practical cases.
For instance, if r = 0 . 5 , the ground truth function f ∗ needs to be a real-valued function in the RKHS
H K .
The RKHS admits a large class of bounded and continuous real-valued functions that are in
Hilbert space.
Moreover, this assumption has been widely used in learning theory; please refer to (Smale & Zhou, 2003; Cucker & Zhou, 2007) for more practical discussions (e.g., 0 < r < 0 . 5 ).
Theorem 1.
Suppose that the process { Z t } ∞ t =
−∞ is stationary.
Let Assumptions 1-3 be true.
By taking σ = T m , η = T − 14 r and λ = T − 14 r − m , we have for any 0 < δ <
[Equation]
with conﬁdence at least 1 − δ , where (cid:101) C is a positive constant independently of T, λ, η, δ and σ and Ψ( m, c, [Equation] =
[Equation]
Remark 2.
In the stationary case, our bound appears to be novel for the following reasons: a) the result is completely independent of the various mixing conditions which are widely used in the theory analysis of non-i.i.d dependent time series (Mohri & Rostamizadeh, 2009; Yang et al., 2018; Mohri & Rostamizadeh, 2010; Guo & Shi, 2011;
Qiu et al., 2015); b) compared with Yang et al.
(2018), the innovation assumption is rather weak in the sense that the innovation possesses inﬁnite variance and thus admits a heavy-tailed distribution.
Corollary 1.
Suppose that the process { Z t } ∞ t =
−∞ is stationary.
Let all the conditions in Theorembe true.
We then have for any 0 < δ <
[Equation]
with conﬁdence at least 1 − δ , where
[Equation]
[Figure]
Figure 1 summaries the convergence rates in Corollary 1 by taking different σ and c .
If the innovation is Gaussian distribution with ﬁnite variance (i.e., Assumption 1 holds for any c > 1 ), one can arbitrarily select a σ =
T m (0 < m < 12 ) to obtain convergence rates O ( n − 12 ) .
Moreover, we can see that the convergence rate will decrease as m increases.
Combined with the conclusion in Lemma 1 (i.e., the equivalence relation between Huber loss based empirical risk and MSE as σ → ∞ ), it indicates that σ indeed plays a tradeoff role between algorithmic robustness and variance-reduction.
For the weak moment condition (e.g., 0 < c < 1 ), one may get slower convergence rates (e.g., O ( n (1 − c ) m − 12 ) or
O ( n (2 − c ) m − 1 ) ), which also coincides with our intuitive understanding that small σ may be conducive to robust forecasting.
Note that our method will not converge when σ and c are both located in the white area in Figure 1.
As a comparison, due to the non-robustness of the squared loss, the most existing convergence rates are established under the assumption that the innovation is Gaussian distribution with ﬁnite variance (see, e.g., Yang et al.
; Han et al.
; Wang & Yang (2007); Kock & Callot (2015)).
However, from Theorem 1 and Corollary 1, the asymptotic convergence of SpHAM can be obtained under weaker moment condition, which veriﬁes the robustness of our method.
Recall the learning rate O ( n 2 d 2 d +1 ) derived in Yang et al.
(2018), where d is the order of smoothness of the component function f j , j = 1 , ..., p .
Relatively slow convergence rate O ( n − 12 ) we obtained indicates the sacriﬁce for the absence of mixing condition.
3.2. F UNCTION APPROXIMATION ANALYSIS FOR NON - STATIONARY TIME SERIES
In nonstationary time series setting, different Z t s may follow different distributions.
Thus we introduce the weights s = { s 1 , ..., s T } and assign them to the losses made on different sample points in terms of their relevance to forecasting the future Z T +1 .
For given weights { s t } Tt =1 , we deﬁne the estimator ˆ f s = (cid:80) pj
=
ˆ f s j as the minimizer of the following weighted objective
[Equation]
Next, we introduce a discrepancy measure to characterize the discrepancy between the target distribution and the distributions of observations (Kuznetsov & Mohri, 2020a).
Deﬁnition 2.
For any f ∈ H Z , the discrepancy measure with respect to Huber loss is deﬁned as
[Equation]
The discrepancy measure is a natural measure of the non-stationarity of the stochastic process { Z t } ∞ t =
−∞ with respect to both the loss function (cid:96) σ and the hypothesis set H Z .
Theorem 2.
Let Assumptions 1-3 be true.
We assume that f ∗ j ∈ H K j , ∀ j = 1 , ..., p .
By taking σ = T 12 c , λ = T − 1 and η = T − 12 , we have for any 0 < δ <
[Equation]
with conﬁdence at least 1 − δ , where (cid:101) C 1 , (cid:101) C 2 are two positive constants independently of T, λ, η, δ and σ .
Note that there are several existing studies towards analyzing nonstationary and non-mixing time series, see, e.g., Kuznetsov & Mohri (2020a).
Different from them, the error bound we derived is with respect to the function approximation rather than generalization risk, which is very crucial for Huber regression problem, since the convergence of generalization risk cannot imply the convergence of function approximation (Sun et al., 2019; Feng & Wu, 2020).
3. A DAPTIVE SPARSE H UBER ADDITIVE MODEL FOR NONSTATIONARY TIME SERIES
Theorem 2 illustrates that we shall minimize the following optimization problem for non-stationary time series forecasting:
[Equation]
where λ 1 and λ 2 are two positive regularization parameters, and c is a positive constant introduced in
Assumption 1.
Although the discrepancy measure disc ( s ) is crucial for such an optimization problem, we cannot obtain its exact value since we do not have access to the distributions of Z t , t ∈ Z .
Hence, we need to estimate the approximated discrepancy from given data.
Inspired by (Kuznetsov & Mohri, 2020a), one natural and necessary assumption is that there is an underlying representation relationship between distribution ρ T +1 and distributions ρ t , t = 1 , ..., T .
Assumption 4.
Denote by a probability set q ∗ = { q ∗ t } Tt =1 with (cid:80) Tt =1 q ∗ t = 1 .
We assume that the following term is sufﬁciently small:
[Equation]
Note that the priori q ∗ can be any distribution, which shall be given empirically according to the trend of the time series. For instance, in a particular scenario of the distribution ρ T +1 does not change drastically compared with the distributions ρ t , t = 1 , ..., T , Kuznetsov & Mohri (2020a) have proven that the Assumption 4 holds if the s ∗ is an uniform distribution over last l > 0 observations, where l is a hyper-parameter that can be tuned in practical applications.
Theorem 3.
Let Assumption 4 and the conditions in Theorem 2 be true.
We have for any 0 < δ <
[Equation]
with conﬁdence at least 1 − δ , where (cid:101) C 1 , (cid:101) C 2 are two constants independently of T, λ, η, δ and σ .
Finally, the optimization problem of adaptive SpHAM can be formulated as following two stages:
Step A: ﬁnding the weight ˆ s :
[Equation]
Step B: forecasting :
[Equation]
where
[Equation]
The optimization problems (9) can be solved by standard gradient descent, where, for each step, we employ DC programming (Tao & An, 1998) to evaluate the supremum over f ∈ H Z .
Similarly to the strategy for optimization problem (7), we use Fast Iterative Shrinkage-Thresholding Algorithm (FISTA)
(Beck & Teboulle, 2009) for Step B .
We provide the detailed procedure in Appendix E.
E XPERIMENT
This section validates the effectiveness of SpHAM and adaptive SpHAM.
In all experiments, the
Gaussian kernel K j ( u, v ) = exp( − (cid:107) u − v (cid:107)
2 d 2 ) , where j = 1 , ..., p and bandwidth d > 0 , is employed for constructing the additive data dependent hypothesis space.
Due to limited space, we only present the evaluations of part of the synthetic time series data.
The real-world experiments can be found in
Appendix F.
In the following, we consider three synthetic examples as below:
Example A: Inspired by (Kuznetsov & Mohri, 2020a), a stationary time series is generated according to the non-linear additive autoregressive model:
[Equation]
where the innovation ε t s are i.i.d.
drawn from Gaussian distribution N (0 , 0 . 1) and Student distribution with freedom 2 , respectively.
Example B: A stochastic time series with interaction term is generated by
[Equation]
Example B considers the data generating process with interaction term Y t − 1 Y t − 2 .
In order to improve the modeling capacity of our SpHAM, according to Remark 1, Group SpHAM is introduced here by dividing the input space into 3 subgroups {X 1 } , {X 2 } and {X 1 , X 2 } .
Example C: Inspired by (Kuznetsov & Mohri, 2020a), a time series with smooth drift is generated by
[Equation]
Hyper-parameter selection and evaluation criterions: Recall that SpHAM algorithm requires three hyper-parameters: regularization parameter λ , bandwidth of kernel d and Huber parameter σ .
For Huber parameter σ selection, we set σ = T 148 based on the suggestion in Theorem 1 and Corollary 1.
Moreover, we grid search the hyper-parameters: the bandwidth d ∈ { 0 . 1 , 0 . 5 , 1 , 5 } , the regularization parameters λ, λ 1 , λ 2 ∈ { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 10 , 100 , 1000 } and l ∈ { 100 , 150 , 200 , ..., 350 } .
The evaluation criterions for forecasting used here contains Average Sample Error(ASE)= 1 N (cid:113)(cid:80)
Nt
=
( ˆ f ( x t ) − y t ) 2 and True Deviation (TD)= 1 N (cid:113)(cid:80)
Nt
=
( ˆ f ( x t ) − f ∗ ( x t )) 2 , where N is the number of test samples.
For each example, we generate time series with 4000 sample points.
The samples at time t = { 1500 , 1501 , ..., 1899 } are used as a training set, and the samples at time { 1900 , 1901 , 1902 } are considered as the test set.
The competitors include standard Autoregression (AR), TS-SpAM (Yang et al., 2018), Gaussian processes regression (GP) and Vanilla Long Short-Term Memory (LSTM).
The tuning parameters of other competing methods such as LSTM and GP, are chosen according to their original sklearn package in python (Pedregosa et al., 2011).
All the evaluations are repeated for 50 times.
The average results for Examples A-C are presented in Table 2.
Moreover, in Appendix F.1, we investigate the impact of the σ on the performance of our method.
Table 2 veriﬁes that robust forecasting of our SpHAM in presence of heavy-tailed t noise.
Moreover, when facing the stochastic time series with interaction term, the result (Example B) indicates the modelling capacity of our SpHAM can be further improved by embedding group structure.
The result (Example C) shows the promising performance of additive SpHAM on nonstationary time series forecasting
[Table]
[Table]
C ONCLUSION
We propose an adaptive sparse
Huber additive model by integrating Huber loss and (cid:96) 2 , 1 -norm regularizer into an additive data dependent hypothesis space.
We theoretically explore the asymptotic properties of our method for both non-Gaussian and (non)stationary time series.
Experimental results on both synthetic and real-world data validate the effectiveness of the proposed method.
A P ROOF SKETCH
This section sketches the proof focusing on the conceptual aspects.
The full proofs are provided in
Appendix B-D.
[Figure]
Figure 2 summaries the important ingredients for the proof of Theorem 1.
Lemma 1 describes the properties of ˆ f and Lemma 2 illustrates the relation between huber-based risk and MSE.
Following these two lemmas, we then decompose the function error into three important parts: sample error, hypothesis error and approximation error.
To bound the sample error, we need to employ the sequential Rademacher complexity (Kuznetsov & Mohri, 2020a), which is the key to measuring the capacity of the data-dependent hypothesis space for non-i.i.d data.
Lemmas 3-4 give an important concentration inequality and the upper bound of sequential Rademacher complexity, respectively.
Finally, Theorem 1 is obtained by combining Propositions 2-4.
The proof of Theorems 2 is similar to the above process, except that we develop a different error decomposition in Proposition 5.
On the basis of Theorem 2, by applying an assumption on data distribution, we derive Theorem 3.
The full proofs of Theorems 1-3 are provided in Sections C-E, respectively.
B P ROOF OF T HEOREM
We ﬁrst illustrate two key properties of ˆ f .
Lemma 1.
Let Assumptions 1-2 be true.
From the deﬁnition of ˆ f in Eq.
(6), there hold (cid:107)
[Equation]
j . and Ω( ˆ f ) = p (cid:88)
j = τ j (cid:107) α λj (cid:107) [Equation] ≤ κ − [Equation]
M ˆ f = 2 MλT min j =1 ,...,p
[Equation]
where M is a positive constant such that | Y t | ≤ M, ∀ t ∈ Z .
Proof.
Denote by E σ T ( f ) :
= (cid:80) T t =1 (cid:96) σ ( f ( x ( t ) )
− y ( t ) ) for notational convenience.
From the deﬁnition of ˆ f , we know that
[Equation]
It means that
[Equation]
From the deﬁnition of Huber loss (3), we obtain
[Equation]
and
[Equation]
Therefore,
[Equation]
According to the property of RKHS
[Equation]
we conclude that
[Equation]
This completes the proof by denoting M ˆ f =
[Equation]
Let ¯ H Z be a data-dependent hypothesis space constrained by sparsity-inducing regularizer Ω( · ) , i
[Equation]
e., ¯ H Z = { f ( x ) = p (cid:88)
j =
T (cid:88) i =
α ij K j
( x ( t ) j , x j ) : p (cid:88)
j = τ
[Equation]
α j (cid:107) 2 ≤
[Equation]
λT
[Equation]
j =1 ,..,p τ j } .
For any h ∈ Z , denote by E σT +1 ( f ) :
= E (cid:96) σ ( f ( x T +1 ) − y T +1 ) and E T +1 ( f ) :
=
E ( f ( x T +1 ) − y T +1 ) 2 , respectively.
In the following, we describe the relationship between E σT +1 ( f ) and E T +1 ( f ) , ∀ f ∈ ¯ H Z .
Note that a similar proof is given in (Feng & Wu, 2020) for the i.i.d case.
Lemma 2.
Let Assumptions 1-2 be true.
For any h ∈ Z and f ∈ ¯ H Z , there holds
[Equation]
where both c and M c = 2 3+ c ( M f ∗ + M ˆ f + 1) 2 E ( | Y T +1 | 1+ c ) are positive constants.
Proof.
For any σ > max { M + M ˆ f , 1 } , we denote two events I Y and II Y as follows
[Equation]
and
[Equation]
Then, for any f ∈ ¯ H Z and (cid:107) f ∗ (cid:107) ∞ =
M f ∗ < ∞ , we have
[Equation]
[Table]
[Equation]
The last inequality is based on the fact that
[Equation]
and hence
[Equation]
Recalling that the Huber loss is 2 σ -Lipschitz continuous, then we get
[Equation]
According to Markov’s inequality
[Equation]
for any constant c >
[Equation]
we further have
[Equation]
Then we have
[Equation]
Moreover, the second term in the right-hand side can be bounded by
[Equation]
According to Holder inequality
[Equation]
we have
[Equation]
Then we can deduce that
[Equation]
Finally
[Equation]
there holds
[Equation]
with
[Equation]
This completes the proof.
Deﬁne a stepping-stone function with respect to the distribution ρ T +1 :
[Equation]
To establish the bound of function approximation for stationary time series setting
[Equation]
we make the following error decomposition.
Proposition 1.
Let Assumptions 1-2 be true.
For any f ∈ ¯ H Z , there holds
[Equation]
where
[Equation]
and
[Equation]
Proof.
According to Lemma
[Equation]
for any f ∈ ¯ H Z
[Equation]
we can make following error decomposition
[Equation]
j =
[Table]
τ j
[Table]
f
[Table] [Table]
[Table]
(cid:124) (cid:123)(cid:122) (cid:125)
[Table]
+ {E σT ( f ) + λ Ω( f ) − E σT ( f ∗ η,T
[Table]
1 ) − η p (cid:88) j =
[Table]
j (cid:107) f ∗
[Table] [Table]
1 ,j (cid:107)
[Table]
j } + 2 M c
[Table]
c ≤ E 1 + E 2 + {E σT ( f ) + λ Ω( f ) − E σT (
[Table]
f η ) − η
[Table] [Table]
+
[Table]
σ
[Table]
≤ E 1 + E 2 + E 3 + 2 M c σ c .
In statistical machine learning community, we call E 1 , E 2 and E 3 sample error, approximation error and hypothesis error, respectively.
The sample error E 1 describes the divergence between the empirical risk E σ T ( f ) and the expected risk E σ ( f ) .
The hypothesis error E 2 characterizes the difference between the empirical regularized risks with H K and H Z .
The approximation error measures the approximation ability of RKHS H K to H .
B. S AMPLE E RROR
In this section, we focus on providing the bound of sample error E 1 .
Unfortunately, the traditional tools for complexity analysis such as covering number (Ron, 2000; Guo & Shi, 2011), Rademacher complexity (Mohri & Rostamizadeh, 2009)) and concentration inequalities ((Wu et al., 2007)), cannot be applied into this non-i.i.d.
setting directly.
To solve this problem, we employ the sequential
Rademacher complexity developed in (Rakhlin et al., 2010; Kuznetsov & Mohri, 2020a).
We deﬁne a function-based random variable as
[Equation]
Now, we turn to establish the bound of
[Equation]
A necessary ingredient needed for our analysis is data-dependent sequential Rademacher complexity
Rakhlin et al.
(2010), which we review in the following.
We adopt the following deﬁnition of a complete binary tree: a Z -valued complete binary tree v is a sequence ( v 1 , ..., v T ) of T mappings v t : {± 1 } t − 1 → Z , t ∈ [1 , T ] .
A path in the tree is γ = ( γ 1 , ..., γ T − 1 ) ∈ {± 1 } T − 1 .
To simplify the notation, we will write v t ( γ ) instead v t ( γ 1 , ..., γ t − 1 ) , even though v t depends only on the ﬁrst t −elements of γ .
The following deﬁnition generalizes the classical notion of Rademacher complexity to sequential setting.
Deﬁnition 3.
The sequential Rademacher complexity R T ( G ) of a function class G is deﬁned by
[Equation]
where the supremum is taken over all complete binary trees of depth T with values in Z and γ is a sequence of Rademacher random variables.
Based on the deﬁnition of sequential Rademacher complexity, we have the following concentration inequality:
Lemma 3.
Suppose that the time series { Z t } ∞ t =
−∞ is strictly stationary.
For any δ > 0 , with probability at least 1 − δ , the following inequality holds for all f ∈ ¯ H Z and all α >
[Equation]
where G := { (cid:96) σ ( f, z ) + (cid:96) σ ( f
[Equation]
, z ) : f ∈ ¯ H Z } and M (cid:96) σ is a positive constant such that (cid:96) σ ( f ( x ( t ) )
− y ( t ) ) ≤ M (cid:96) σ for any f ∈ ¯ H Z and t ∈ Z .
According to the property of sequential Rademacher complexity (see Proposition 14 in (Rakhlin et al., 2015)), we have
[Equation]
with the 2 σ -Lipschitz constant of (cid:96) σ ( · ) .
Lemma 4.
Under Assumptions 1-2, there holds
[Equation]
Proof.
In fact, to every kernel K j , we can associate a feature map φ with inner product < φ ( x j ) , φ ( x (cid:48) j ) >
= K ( x j , x (cid:48) j ) for any x j , x (cid:48)
j ∈ X j and j = 1 , ..., p .
Then the constrained datadependent hypothesis space ¯ H Z can be rewritten as
[Equation]
By direct computation, we have
[Equation]
Thus, combining above result with Eq.
We get the desirable result.
Combining the results in Lemma 4 with the inequality in Lemma 3, we then obtain the bound of sample error E 1 for the strictly stationary time series setting.
Proposition 2.
Under Assumptions 1-2, for any δ > 0 , the following inequality holds for all f ∈ ¯ H Z and all α >
[Equation]
with probability at least 1 − δ , where
[Equation]
E RROR
Since the output function f ∗ η,T +1 is the optimal estimator in RKHS H K for time T + 1 , the learning rates of the learning algorithm indeed depend on the approximation ability of the hypothesis space H K with respect to the optimal risk E ( f ∗ ) measured by the approximation error E 2 .
For any j = 1 , ..., p and h ∈ Z , we ﬁrst deﬁne the kernel integral operator L K j ,T +1 :
L 2 ( ρ T +1 X j )
→ L 2 ( ρ T +1 X j ) associated with the kernel K j by
[Equation]
Note that L K j ,T +1 is a compact and positive operator on L 2 ( ρ T +1 X j ) .
According to Mercer theorem, we can ﬁnd the corresponding normalized eigenpairs { ( ζ jh,i , ψ jh,i ) }
i ≥ 1 such that { ψ jh,i } i ≥ 1 is an orthonormal basis of L 2 ( ρ T +1 X ) and ζ jh,i → 0 as i → ∞ .
Then for given r > 0 , we deﬁned the r -th power L rK j ,T +1 by
[Equation]
We introduce an intermediate function as follows:
[Equation]
Lemma 5.
Under Assumption 3, for the intermediate function ˜ f η,T +1 ,j deﬁned above, there holds
[Equation]
Proof.
Under Assumption 3, for any h ∈ Z , we know that f ∗ j =
L rK j ,h ( g ∗ j,h ) for some g ∗ h,j = (cid:80) i ≥ 1 β h,i ψ jh,
i ∈
L 2 ( ρ T +1 X j ) .
Then we have
[Equation]
= (cid:88)
i ≥( ζ jh,i ) r β h,i ψ jh,i .
The case r = 12 means each f ∗ j lies in the RKHS H K j .
Then we have ˜ f
[Equation]
[Equation]
[Equation] I [Equation] K j
[Equation]
h
[Equation]
f ∗ j = (cid:88) i ≥ητ j ( ζ jh,i ) r β jh,i ψ jh,i ζ ji + ητ j .
Then we have (cid:107) ˜
f
η,T +1 ,j − f ∗ j (cid:107)= ( (cid:88)
i ≥ητ j ( ζ {+jh,i ) r β jh,i ζ jh,i + ητ j ) 
+}
= ( ητ j ) 2 r (cid:88)
i ≥( ητ j ζ jh,i + ητ j ) 2 − 2 r ( ζ jh,i ζ jh,i + ητ j ) 2 r ( β jh,i )≤ ( ητ j ) 2 r (cid:88)
i ≥( β jh,i )
= ( ητ j ) 2 r (cid:107) L − rK j ,T +1 f ∗ j (cid:107) 22 .
Similarly, we also have ητ j (cid:107) ˜
f η,T +1 ,j (cid:107)= ( ητ j ) 2 (cid:88)
i {+≥( ( ζ jh,i ) 
+}
+ r ζ jh,i + ητ j ) {+2 ( β jh,i ) 
+}
= ( ητ j ) 2 r (cid:88)
i ≥( ζ jh,i ζ jh,i + ητ j ) 2+2 r ( ητ j ζ jh,i + ητ j ) 2 − 2 r ( β jh,i )≤ ( ητ j ) 2 r (cid:107) L −
rK j ,T +1 f ∗ j (cid:107)
[Equation]
Proposition 3.
Under Assumptions
[Equation]
there holds
[Equation]
Proof.
According to Lemma
[Equation]
we have
[Equation]
Then we have
[Equation] [Equation]
K
[Equation]
j = [ ˜ f η,h,j − f ∗ j ] (cid:107) 22 + η p (cid:88)
j
= τ j (cid:107) ˜
f
η,h,j (cid:107) 2 K ≤ p (cid:88)
j = p (cid:107) ˜
f
η,T +1 ,j
[Equation]
f
[Equation]
j (cid:107)
[Equation]
η p (cid:88) j
= τ j (cid:107) ˜
f
η,T +1 ,j (cid:107) 2 K ≤ ( p
[Equation]
1) p (cid:88)
j = ( ητ j ) 2 r (cid:107) L
[Equation]
rK j ,T +1 f ∗ j (cid:107) 22 .
Combing above results, we get the desired result.
B. H YPOTHESIS E RROR
This section focuses on bounding the hypothesis error
[Equation]
We ﬁrst give the properties of ˆ f and then use them to bridge ˆ f and f η .
Lemma 6.
For all j = 1 , ..., p , there holds
[Equation]
Proof.
Recall the represent theorem which ensures that
[Equation]
For notation simplicity, denote α ηj =
( α η 1 j , ..., α ηTj ) (cid:48) ∈ R T and α η =
(( α η 1 ) (cid:48) , ..., ( α ηp ) (cid:48) )
(cid:48) ∈ R Tp .
From the deﬁnition of Eq.
(7), we deduce that
[Equation]
α
[Equation]
where K j = { K ( x sj , x ij ) } ni,s =1 ∈ R T × T .
we then have
T T
[Equation]
t =
[Equation]
It is easy to deduce that
[Equation]
Then we obtain that, for any j = 1 , ..., p ,
[Equation]
This completes the proof.
Proposition 4.
Under Assumptions 3, there holds
[Equation]
Proof.
[Equation]
Combining the above inequality with Lemma 6, we get that
[Equation]
Proof of Theorem 1: Combining Propositions 1-4, for Tη ≤ 1 , we have with conﬁdence 1 − δ
[Equation]
where (cid:101) C is a positive constant independently of T, λ, η, δ, σ .
By taking σ = T m , η = T β and λ =
T γ , we then have with conﬁdence 1 − δ
[Equation]
, where
[Equation]
= max {− 1 2 , m − 1 , 2 rβ, γ − 1 2 − β + m, − cm − 2 γ − 2 , − cm − γ − 1 } .
By taking σ = 12 c , β = − 14 r and γ = − 14 r −
12 c
Direct computation shows that
[Equation]
This completes the proof.
C P ROOF OF T HEOREM
Denote by s = { s t } Tt =1 a probability set with (cid:80)
Tt =1 s t = 1 .
We deﬁne following functions associated with the probability set s : f
[Equation]
= p (cid:88)
j =
[Equation]
f
[Equation]
p (cid:88) j =
[Equation]
f ∗
[Equation]
s T
[Equation]
T (cid:88) t =
s t (cid:96) σ ( f ( x t ) − y t ) , ˆ f s η = p (cid:88)
j
= ˆ f s η,j = arg min f = (cid:80) pj
=1 f j ,f j ∈H Kj {
T (cid:88) t =
s t (cid:96) σ ( y t − p (cid:88)
j = f j ( x tj ))
+ η p (cid:88)
j = τ j (cid:107) f j (cid:107) 2 K j } , and ˆ f s = p (cid:88)
j = ˆ f s j = arg min f = (cid:80) pj
=1 f j ,f j ∈H Kj {
T (cid:88) t =
s t (cid:96)
[Equation]
Proposition 5.
Let Assumptions 1-2 be true.
For any f ∈ ¯ H Z , there holds
[Equation]
where
[Equation]
and
[Equation]
Proof.
According to Lemma
[Equation]
for any f ∈ ¯ H Z
[Equation]
we can make following error decomposition
[Equation]
≤ E
[Equation]
+ {E
[Equation]
( f
[Equation]
+ λ
[Equation]
f
[Equation]
( f
[Equation]
) − η
[Equation]
(cid:88)
[Equation]
= τ
[Equation]
(cid:107) f
[Equation]
(cid:107)
[Equation]
η
[Equation]
(cid:88)
[Equation]
=
[Table]
τ
[Equation]
(cid:107) f
[Equation]
f
[Equation]
η
[Equation] [Equation]
[Equation]
η
[Equation]
τ j (cid:107) f
[Equation]
(cid:107)
[Equation]
[Table]
[Equation]
σ c ≤ E
[Equation]
+ {E σ,
[Equation]
( f ) + λ Ω( f ) − E σ,
[Equation]
( ˆ f s η ) − η p (cid:88)
j
= τ j (cid:107) ˆ f s
[Table]
,j (cid:107)
[Equation]
K j } (cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
E
[Table]
[Equation]
( f
[Table]
[Equation] [Equation]
η
[Equation]
(cid:88)
[Equation]
= τ j (cid:107) f ∗ ,
[Equation]
[Table]
,j (cid:107)
[Equation]
K j + E
[Table]
[Equation]
( f
[Table] [Table]
+1 ) − E
[Table]
[Equation]
[Table]
( f
[Table]
[Equation]
η p (cid:88) j =
[Table]
j (cid:107) f
[Table]
[Equation]
[Table]
[Equation]
j (cid:107)
[Equation]
[Table]
j }
[Equation]
M c σ c
[Table]
+ E
[Table]
+ E T
[Equation]
( f
[Table] [Table]
+1 ) − E
[Table]
[Equation]
1 ( f
[Table]
+ η p (cid:88)
j =
[Table]
j (cid:107) f
[Table] [Table]
[Table]
j (cid:124) (cid:123)(cid:122)
[Table]
+ E σ, s T ( f ∗ , s T ) + η p (cid:88) j
= τ j (cid:107) f ∗ , s T,j (cid:107) 2 K j + 2 M c σ c ≤ E 1 + E 2 + E 3 + E σ, s T ( f ∗ , s T ) + η p (cid:88) j = τ j (cid:107) f ∗ , s T,j (cid:107) 2 K j + 2 M c σ c ≤ E 1 + E 2 + E 3 + min f ∈H Z {E σ, s T ( f ) + λ Ω( f ) } + η p (cid:88) j
= τ j (cid:107) f ∗ , s T,j (cid:107) 2 K j + 2 M c σ c .
Following the generalization analysis in (Kuznetsov & Mohri, 2020a), we need to discrepancy measure to measure the discrepancy of the non-stationarity of the stochastic process { Z t } ∞ t =
−∞ with respect to both the loss function (cid:96) σ and the hypothesis set ¯ H Z .
Deﬁnition 4.
The discrepancy describes the discrepancy between target distribution and the distribution of the sample.
For any f ∈ ¯ H Z , the discrepancy measure with respect to Huber loss is deﬁned as
[Equation]
Lemma 7.
For any δ > 0 , with probability at least 1 − δ , the following inequality holds for all f ∈ ¯ H Z and all α >
[Equation]
where G := { (cid:96) σ ( f, z ) : f ∈ ¯ H Z } and M (cid:96) σ is a positive constant such that (cid:96) σ ( f ( x t ) − y t ) ≤ M (cid:96) σ for any f ∈ ¯ H Z and t ∈ Z .
Proof of Theorem 2 The proofs of bounding errors E 1 , E 2 and E 3 proceeds similarly to the proof of Theorem 1 and are omitted for brevity.
According to Lemma 3, we have
[Equation]
where
[Equation]
Moreover, from Proposition 4, we can get
[Equation]
Similarly, to bound the hypothesis error
[Equation]
we ﬁrst give the properties of ˆ f and then use them to bridge ˆ f and f η .
Lemma 8.
For all j = 1 , ..., p , there holds
[Equation]
Proof.
Recall the represent theorem which ensures that
[Equation]
For notation simplicity, denote α η, s j =
( α η, s 1 j , ..., α η, s Tj ) (cid:48) ∈ R T and α η, s =
(( α η, s 1 ) (cid:48) , ..., ( α η, s p ) (cid:48) )
(cid:48) ∈
R Tp .
We then deduce that
[Equation]
where K j = { K ( x sj , x ij ) } ni,s =1 ∈ R T × T .
We get
T (cid:88) t =
s t (cid:96) (cid:48)
σ ( y t − p (cid:88) j
=
K (cid:48)
jt α η, s j ) K (cid:48) ji
= ητ
j K j α [Table] , s j .
It is easy to deduce that
[Equation]
Then we obtain that, for any j = 1 , ..., p ,
[Equation]
This completes the proof.
Under Assumptions 1-
[Equation]
there holds
[Equation]
By combining the above results
[Equation]
we have with conﬁdence 1 − δ
[Equation]
T − 1 + η ) , where (cid:101) C is a positive constant independently of T, λ, η, δ, σ and p .
By taking σ = T 12 c , λ = T −and
η
= T − 12 , we completes the proof.
D P ROOF OF T HEOREM
According to the deﬁnition of disc ( T + 1) , we have
[Equation]
disc
[Equation]
Furthermore, it is easy to deduce that
[Equation]
) − y t ) − (cid:96)
[Equation]
( f ( x t ) − y t )] .
Proof of Theorem
According to the proof of Theorem 1 in Kuznetsov & Mohri (2020a), we have sup f ∈H Z
T (cid:88) t =
[Equation]
By combining above results, we have with conﬁdence 1 − δ
[Equation]
By taking σ = T 12 c , λ = T − 1 and η = T − 12 , we complete the proof.
Algorithm
[Equation]
Variable selection: { j : (cid:107) α s j (cid:107) 2 ≥ v, j = 1 , ..., p } .
E E XPERIMENT O PTIMIZATION
The optimization problem (10) reduces to problem 7 when taking s ∗ = 1 T I T .
Recall the optimization problem in Step A :
[Equation]
The above optimization problem can be equivalently rewritten as a common type of bilevel optimization problem (Colson et al., 2007), i.e,
Outer problem
[Equation]
Inner problem
[Equation]
where the outer (min) problem parameterized by s , is nested within the inner (max) problem.
The outer problem can be solved by standard gradient descent, where in each step, we need to optimize the inner (max) problem with last updated s .
We denote by k the iteration time.
Then for k + 1 -th iteration, we have the following gradient update rule
[Equation] [Equation]
where γ is learning rate,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
and
[Equation]
is obtained by the following weighted optimization problem
[Equation]
Note that the inner problem is subjected to f ∈ H K , i.e., (cid:107) α
[Equation]
≤
[Equation]
A widely-used method for solving this constrained inner problem is DC programming (Tao & An, 1998).
After obtaining the solution ˆ s
[Equation]
we turn to solve the following weighted optimization problem in Step B :
[Equation]
We can see that this problem contains non-smooth function λ (cid:80) pj =
τ j || α
j || 2 which makes the standard gradient descent method inapplicable.
To conquer this challenge, we leverage fast iterative shrinkage-thresholding algorithm (FISTA)(Beck & Teboulle, 2009).
Our optimization problem becomes
[Equation]
= (cid:32)(cid:18)
[Equation]
− λ |
[Equation]
(cid:19)
[Equation]
, (14) where v
[Equation]
L
[Equation]
) and ∇ α j (cid:96) σ ( y t − ( K t ) (cid:48)
α k − 1 ))
=
  2(( K
[Equation] [Equation]
| ( K
[Equation]
) (cid:48)
α − y
[Equation]
| < σ 2 σ ( K t ) (cid:48)
[Equation]
y
[Equation]
σ − 2 σ ( K t ) (cid:48)
α − y t ≤ − σ and L = max 1 T (cid:80)
Tt =1 (cid:107) ( K t ) (cid:48)
K t (cid:107) is the lipschitz constant 1 T (cid:80)
Tt =1 ∇ (cid:96) σ ( y t − ( K t ) (cid:48)
α k − 1 ) .
Denote
Proj( s ) as a projection of s and v > 0 as the threshold value.
To focus on the weights of samples which are really useful for forecasting, we consider Proj ( s ) as a box projection such that s t >
q ∗ t , ∀ t = 1 , ..., p ).
Finally, the optimization procedure for adaptive SpHAM can be summarized in
Algorithm 1.
Note that if we only run Step B with ˆ s t = 1 T , ∀ t = 1 , ..., T , we can further obtain the optimization procedure for SpHAM.
Remark 3.
The computational complexity of Algorithm 1 depends on the optimization strategy for
DC programming, the training size T , dimension p and the iteration times Z .
We denote by the computational complexity of DC programming O ( DC ( T, p )) .
Then, the computational complexity of
Step A is O ( ZDC ( T, p ) + ZT ) and the computational complexity of Step B is O ( ZT 3 p 3 ) .
Thus, the total computational complexity of Algorithm 1 is O ( ZO ( DC )
+ ZT 3 p 3 ) .
For large scale data, we can further speed up Algorithm 1 by random Fourier features technique [Rahimi and Recht 2007], which is leaved for future work.
This has been carefully discussed in the revised manuscript.
F A DDITIONAL E XPERIMENT F. E VALUATION ON HYPER - PARAMETERS
We investigate the impact of the selection of σ on our method.
For ﬁxed sample size T = 400 , the empirical results over different σ =
T α are summarized in Table 4.
This table shows that, for heavy-tailed t noise, our method tends to perform better with smaller α .
This result also validates
Theorem 1 and Figure 1.
F. E VALUATION ON BENCHMARK DATA
We test our algorithm on nonlinear dataset from CauseMe.
The hyper-parameter selection is the same as the one in Section 4.
The results in Table 3 verify the effectiveness of our method.
F. E XPERIMENTS ON A IR Q UALITY DATASET
We use the Air Quality dataset obtained from UCI Machine Learning Repository ( https:// archive.ics.uci.edu/ml/datasets/Air+quality ) to test our model’s ability to detect the Granger causality.
This dataset includes 9358 hourly air quality data in an Italian city, collected from March 2004 to February 2005.
The details of the dataset can be obtained on the UCI website.
The Granger causal network we detect is shown in Figure 3.
Given the network, we observe that the temperature (T) inﬂuences ozone (O 3 ) and nitrogen dioxide (NO 2 ) which is validated in Kalisa et al.
Mwaniki et al.
(2014) conﬁrm the relationship between relative humidity (RH) and nitrogen dioxide (NO 2 ).
Yan et al.
(2018) verify the relationship between humidity and nitrogen dioxide (NO 2 ).
F. E XPERIMENTS ON C ORONAL M ASS E JECTIONS DATASET
Coronal Mass Ejections (CMEs) are the most violent eruptions in the Solar System.
Despite machine learning approaches have been applied to these tasks recently Wang et al.
Liu et al.
(2018), there is no any work for interpretable prediction with Granger causal network.
CMEs data are provided in The Richardson and Cane List ( http://www.srl.caltech.edu/ACE/
ASC/DATA/level3/icmetable2.htm ).
From this link, we collect 152 ICMEs observations from 1996 to 2016.
The features of CMEs are provided in SOHO LASCO CME Catalog ( https: //cdaw.gsfc.nasa.gov/CME_list/ ).
In-situ solar wind parameters can be downloaded from OMNIWeb Plus ( https://omniweb.gsfc.nasa.gov/ ).
A total of 9 features are chosen as input, including: (1) Central PA (CPA), (2) Angular Width, (3) three approximated speeds ( Linear
Speed, 2nd-order Speed at ﬁnal height,and 2nd-order Speed at 20 Rs), (4) Mass, (5) Kinetic Energy, (6) MPA and (7)CMEs arrival time.
Figure 4 shows that Granger causal network when the output is CMEs Arrival time.
Some interesting ﬁndings are concluded from this Granger causal network.
For instance, Speed and Mass, as the signiﬁcant variables causing CMEs arrival time, have been also screened out in Liu et al.
Morover, the CMEs Angular Width does not cause the CMEs arrival time forecasting, while Liu et al.
(2018) state that they have a signiﬁcant correlation.
This indicates that the CMEs arrival time is affected by the Angular Width at current time, but not by the historical
Angular Width.
[Table] [Table]