Causal Forecasting: Generalization Bounds for Autoregressive Models
Abstract
Despite the increasing relevance of forecasting methods, causal implications of these algorithms remain largely unexplored.
This is concerning considering that, even under simplifying assumptions such as causal sufficiency, the statistical risk of a model can differ significantly from its causal risk .
Here, we study the problem of causal generalization —generalizing from the observational to interventional distributions—in forecasting.
Our goal is to find answers to the question: How does the efficacy of an autoregressive (VAR) model in predicting statistical associations compare with its ability to predict under interventions?
To this end, we introduce the framework of causal learning theory for forecasting.
Using this framework, we obtain a characterization of the difference between statistical and causal risks, which helps identify sources of divergence between them.
Under causal sufficiency, the problem of causal generalization amounts to learning under covariate shifts albeit with additional structure (restriction to interventional distributions under the VAR model).
This structure allows us to obtain uniform convergence bounds on causal generalizability for the class of VAR models.
To the best of our knowledge, this is the first work that provides theoretical guarantees for causal generalization in the time-series setting.
INTRODUCTION
Forecasting algorithms are increasingly relevant in a variety of applications including meteorology, climatology, economics, and business.
While traditional economic mod- not know ground that
Since at most one of them can be right, we conclude that at least the other one makes a notable forecasting error under the intervention.
elling relies on relatively simple time series models (Brockwell et al. 1991), e.g., autoregressive models, or methods like co-integration, modern business planning heavily uses neural networks for forecasting (Faloutsos et al.
Januschowski et al.
2020; Salinas et al. 2020).
Despite the advancements of forecast quality, causal implications are not yet well understood.
There has been notable progress in ‘explainable’ models in the sense of feature relevance (Lundberg et al. 2017; Molnar 2019;
Janzing et al.
Wang et al. 2020) with potential applications in forecasting.
Furthermore, specialized models (Hatt et al. 2021; Bica et al.
2020; Lim et al. 2018) have shown remarkable success for causal inference in forecasting.
It is common practice in business and econometrics to learn statistical forecasting models and interpret them causally.
In practice, while forecasting models tend to agree on their statistical predictions, they can differ substantially on their causal predictions (see Figure 1 for an example).
In particular, this practice is considered justified under simplifying assumptions such as causal sufficiency and the absence of contemporaneous effects (see for instance Hyvärinen et al. (2010, Section 1)).
Here, we are interested in the funda- mental question: what is the relation between the statistical predictability of a forecasting model and its causal generalizability — ability to predict under interventions.
We argue that even for very simple models and even under simplifying assumptions such as causal sufficiency and absence of contemporaneous influence, causal interpretation of forecasting models is non-trivial.
To appreciate the challenges, consider a simple example of a process with strongly correlated observations where x t ≈ x t − 1 , and hence x t ≈ x t − 2 .
These observations can be explained either by a causal model with a strong influence of x t − 1 on x t or a causal model with a strong influence from x t − 2 on x t .
The difference between the models gets apparent when an intervention randomizes x t − 1 and x t − 2 independently.
Then, predictions become hard, particularly when x t − 1 and x t −are set to significantly different values.
While both models are similar in their statistical predictions, they differ substantially in their causal predictions .
This example already shows that, even in a simple setting, causal and statistical predictability can differ significantly.
The question of causal generalization is thus practically relevant and non-trivial and begs for a better theoretical understanding.
Specifically, we consider the simple class of vector autoregressive models (VAR) and ask the question
How does the efficacy of an autoregressive model in predicting statistical associations compare with its ability to predict under interventions?
These models are widely applied in domains ranging from econometrics (Lütkepohl 2009; Grabowski et al. 2020) and finance (Zivot et al. to neuroscience (Valdés-Sosa et al. 2005).
Connection to Covariate Shift.
The problem of causal generalization is closely related to the problem of covariate shift.
To see this, we first ignore the time series setting and consider the scenario where a variable Y should be predicted from a variable X , which is known not to be an effect of Y .
If there is no common cause of X and Y , that is, we assume causal sufficiency (Spirtes et al. 1993), the statistical relation between X and Y is entirely due to the influence of X on Y .
Therefore, the observational and interventional conditionals coincide ( P Y | x = x ∗ =
P Y | do ( x = x ∗ ) in Pearl’s language (Pearl 2009)) and the true parameters would be optimal both from a statistical and causal perspective.
However, due to estimation bias , a prediction model learned using finite samples from P x may perform poorly when randomized interventions draw x -values from a different distribution ˜
P X , which is the usual covariate shift scenario (Sugiyama et al.
In our setting, X and Y are represented by the past and the present values of a (possibly multivariate) time series, respectively.
Accordingly, we focus on interventional distributions that are natural for this setting: independent interventions at different time points and components of the multivariate process.
Hence, we have additional structure in comparison with the standard covariate shift problem.
We are not aware of any theoretical work on covariate shift in the time-series setting.
Nevertheless, we describe the connections to learning theory in the standard covariate shift setting and other related work in Section 6.
Our Contributions.
Our central goal in this work is to develop a formal and thorough understanding of causal generalization for the class of VAR models.
a. To this end, we introduce a framework of causal learning theory for forecasting to analyze when forecasting models can generalize from the observational to the interventional distributions (Section 2).
This is closely related to the setting of learning under domain adaptation.
b. Using this framework, we provide a characterization of the difference in the statistical and causal risks (Section 3).
Such a characterization allows us to identify the sources of divergence between the two quantities.
Our results show that the strength of correlation of the underlying process plays a key role in determining causal generalizability.
They also highlight that already for simple models, causal and statistical errors can even diverge.
c.
Further, we provide finite-sample, uniform convergence bounds on causal generalization for the class of VAR models (Section 3).
Our simulations demonstrate that our bounds indeed capture the key drivers of causal generalization.
To the best of our knowledge, this is the first work that provides theoretical guarantees for causal generalization of any kind in the time-series setting.
d. As a by-product of our analysis, we provide an explicit characterization of the powers of a companion matrix (see
Section 2) using symmetric Schur polynomials (Macdonald 1998) of its eigenvalues (Lemma 2) which, to the best of our knowledge, has not been noted in the literature.
This result could be of independent interest in theoretical endeavors that build upon companion matrices which, for instance, are ubiquitous in stochastic processes and in
Linear-Time-Invariant dynamical systems (Davison 1976;
Melnyk et al. 2016).
e. We conduct experiments with a variety of deep neural networks on real data
Our experiments approach causal risks in this setting and explore its relationship to uncertainty.
CAUSAL LEARNING THEORY FOR FORECASTING
In this section, we introduce a framework to formally evaluate the quality of a forecasting model with respect to prediction and the validity of its causal implications.
We refer to this framework as causal learning theory for forecasting.
First, we introduce some relevant notation.
Notation.
For any stochastic process { x t } t ∈ Z ∈ R d , we use x nt − ω
= { x t − ω − n +1 , ·
· · , x t − ω − 1 , x t − ω } to denote the set of x t − ω and the n − 1 variables in the past of x t − ω .
We distinguish this from y nt which denotes the vector (cid:0) x t , x t − 1 , · · · , x t − n +1 (cid:1) T ∈ R nd .
When it is clear from context, to reduce cumbersome notation, we simply use y t .
For any random variable x , E [ x ] denotes its expectation.
For any matrix A , we use A i : and A : j to denote the i th row and j th column of A respectively.
We use A j 1 k to denote the (1 , k ) th element of A j .
For any vector x t at time t , we use x t,i to denote the i th element of x t .
We use λ max ( A ) , λ min ( A ) , κ ( A )
=
λ max ( A ) /λ min ( A ) to denote the maximum and minimum eigenvalues and the condition number of A respectively.
I p denotes the identity matrix of size p , N , Z denote the set of natural numbers and integers respectively and [ n ] denotes the set { 1 , 2 , · · · n } .
To evaluate the statistical and causal efficacy of an estimator we introduce the notions of statistical and causal forecast risks.
To define statistical forecast risk, we consider the setting of ω − step forecasting where the goal is to predict x t from observations x nt − ω drawn from a stochastic process { x t } t ∈ Z for some ω ∈ N .
To define the causal forecast risk, we consider interventions on x t − ω,i for some
i ∈
[ d ] .
Definition 2.1 ( Statistical forecast error ) .
The statistical forecast error of an estimator ˆ f in the prediction of a target variable x t from x nt − ω , drawn from the observational distribution , can be defined as
[Equation]
The empirical counterpart ( ˆ S ω ), is defined naturally by replacing the expectation by the empirical mean.
For causal questions, we want to investigate the behavior of a model under interventions.
Here, we consider atomic interventions.
Using Pearl’s do notation (Pearl 2009), an atomic intervention do ( x = x ∗ ) refers to setting the variable x to some value x ∗ .
Definition 2.2 ( Causal errors ) .
The interventional forecast error of ˆ f in predicting the effect of an intervention do ( x t − ω,i =
x ∗ t − ω,i ) , on target variable x t is defined as
[Equation]
where do ω,i is shorthand for do ( x t − ω,i
= x ∗ t − ω,i ) and P do ω,i denotes the distribution induced by the intervention do ( x t − ω,i = x ∗ t − ω,i ) .
To isolate from the dependence on specific values that the intervened variables are set to, we present our results via the notion of average causal error .
It is defined as the expected interventional error for interventions drawn from the marginal distribution of x t − ω,i since it provides a natural scale at which the statistical and causal errors can be compared.
[Equation]
Statistical and Causal Learning Theory.
Consider the standard framework of statistical learning in time-series prediction.
For any stochastic process { x t } t ∈
Z taking values in X , given a loss function l : X × X → R + , the goal of statistical learning is to learn a function f
∗S that achieves the optimal statistical risk S ω ( f ) : Since the true process is unknown, the empirical average ( (cid:98) S ω ) of generalization risk is used to estimate S ω .
Statistical generalization bounds of the form: S ω ( f ) < (cid:98) S ω ( f ) + C ( F , n ) are then used to provide guarantees on the uniform deviation of empirical risk from expected risk given sufficiently many samples and when the “complexity” of the function class is small.
Analogously, the goal of causal learning is to find a function f ∗G that achieves the optimal causal risk G ω ( f )
In contrast to statistical learning, the empirical averages of the causal error cannot be utilized to estimate G ω since we often do not have access to data from the interventional distributions.
Instead, we are only provided with data from the observational/statistical distribution of the stochastic process and the goal of causal learning theory is to understand, to what extent is it possible to provide causal generalization guarantees of the form: G ω ( f ) < (cid:98) S ω ( f ) + C ( F , n ) .
To summarize, we ask: Can the predictors in F generalize from the empirical observational distribution to the true interventional distribution assuming that we control the complexity of F and that we observe sufficiently many samples drawn from the observational distribution?
One cannot address this question in a very general setting and would need model assumptions to make any meaningful statements.
To this end, we now formally introduce our problem setup and some preliminaries.
We provide additional relevant background in the Appendix 1.
Statistical and Causal Models.
We assume that the stochastic process { x t } t ∈ Z ∈ R
d follows a weakly stationary vector autoregressive model(VAR(p)) of order p for some p, d ∈ N which is defined as
[Equation]
where x t ∈ R d is a vector-valued time-series, for all i ∈
[ p ] ,
[Equation] are the coefficients of the VAR model, and ϵ t ∈ R d denotes the noise vector such that E [ ϵ t ] = 0 and
E [ ϵ t ϵ Tt + h ] =
Σ ϵ if h = 0 and 0 otherwise .
For some σ 2 ϵ > 0 , we simply set Σ ϵ
= σ 2 ϵ I for enhanced readability.
Our results can be easily generalized to arbitrary covariance matrices by means of the spectral properties ( λ min , λ max ) of Σ ϵ .
The autocovariance matrix of { x t } t ∈ Z plays a central role in our results and analysis.
For any n ∈ N , we use Σ n to denote the autocovariance matrix of size n defined as E [( y n t − E
[ y n t ])( y n t − E
[ y n t ]) T ] .
It is convenient to rewrite a VAR model of order p in Equation (4) as a VAR(1) model, y t =
Ay t − 1 + e t , where y t ∈ R dp , e t ∈ R dp are defined as y t = (cid:0) x t , x t − i , · · · , x t − p
+
(cid:1) T , e t
= (cid:0) ϵ t , 0 , · · · , 0 (cid:1) T , and
A ∈ R dp × dp is a (multi) companion matrix defined as:
[Equation]
The eigenvalues of the multi-companion matrix A fully characterize the stability and stationarity of the VAR process.
For a VAR(p) process to be weakly stationary, that is for the mean and the covariance of the process to not change over time, the eigenvalues of A , which satisfy
[Equation]
are constrained to not lie on the unit circle.
If the magnitude of all the eigenvalues are | λ i | < 1 , then the process is stable, that is, its values do not diverge (Lütkepohl 2013).
Causal Models.
Under the assumptions of causal sufficiency and absence of contemporaneous influences, a causal interpretation of the VAR model in (4) as structural equations naturally yields the corresponding causal model.
We consider the family of all VAR models as our function class
F of statistical and causal estimators.
CAUSAL GENERALIZATION FOR VAR
In this section, we present causal generalization bounds for the family of VAR models under atomic interventions.
We first provide an overview of our results in the more general case of VAR(p) models and later provide a thorough interpretation of the results, often by deriving simplified versions of the results for AR(p) models.
We begin by providing an exact characterization of the difference in statistical and causal errors in terms of the model and estimated parameters and the autocovariance matrix of the underlying process.
Lemma 1 ( Difference in Causal and Statistical errors (VAR) ) .
Consider a vector-valued time series { x t } t ∈ Z ∈ R d , following a VAR(q) process parameterized by { A 1 , A 2 , · · · A q } .
Let ν = max { p, q } .
For any VAR(p) model f with parameters { ˆ A 1 , ˆ A 2 , · · · ˆ A p } ,
[Equation]
where Σ ν denotes the autocovariance matrix of x t of size ν ,
A is a multi-companion matrix of the form described in (5) with the first d rows populated by { A ′ 1 , A ′ 2 , · · · A ′ ν } , with
A ′ l defined as A l for all l ≤ p and as 0 d × d for all l > p .
ˆ A is analogously defined.
Building on Lemma 1, we establish that the condition number of the autocovariance matrix of the underlying process controls causal generalizability from the observational to interventional distributions .
Proposition 1 ( Stability Controls Causal Generalization (VAR) ) .
Let { x t } t ∈
Z follow a VAR(q) process for some q ∈ N .
For any VAR(p) model,
[Equation]
where κ (Σ ν ) denotes the condition number of the autocovariance matrix Σ ν .
Further, one can construct processes where equality holds upto a small constant factor.
The result states that the difference in expected causal and statistical errors is controlled by the condition number of the autocovaraince matrix of size max { p, q } .
It also states that without incorporating additional information, one cannot obtain a much tighter bound which is also verified by our experiments in Section 4.
The condition number of the autocovariance matrix can get arbitrarily large as the process gets closer to the boundary of the stability domain.
This result therefore shows that even for very simple classes of forecasting models, causal interpretations can get challenging.
We later provide a detailed interpretation of this result and provide an explicit bound on κ (Σ ν ) in terms of the stability parameter for AR(p) models (Corollary 2).
Proposition 1 allows us to employ generalization bounds for time-series (Yu 1994; Meir 2000; Mohri et al. 2009;
McDonald et al. 2017) to derive finite-sample causal generalization bounds for VAR models.
In particular, we utilize
Rademacher complexity bounds for generalization in timeseries under mixing conditions (Mohri et al. 2009) to derive
Theorem 1.
Theorem 1 ( Finite sample bounds for VAR(p) models ) .
Let F denote the family of all VAR models of dimension d and order p .
For any n > max { p, q } ∈ N , let µ, m > 0 be integers such that 2 µm = n and δ > 2( µ − 1) ρ m for a fixed constant 0 < ρ < 1 determined by the underlying process.
Let { x 1 , x 2 , · · · x n } ∈ R d be a finite sample drawn from a VAR(q) process.
Then, simultaneously for every f ∈ F , under the square loss truncated at M , with probability at least 1 − δ ,
[Equation]
where ζ = 2 κ (Σ ν ) , δ
[Equation]
= δ − 2( µ − 1) ρ m , and (cid:98) R µ ( F ) denotes the empirical Rademacher complexity of F .
Our causal generalization bound in Theorem 1 suggests that, given sufficiently many samples, the true causal error can be guaranteed to be close to empirical statistical error
if our
VAR models come from a class with a small Rademacher complexity, particularly when the process is associated with a small stability parameter.
We now focus on providing a detailed interpretation of our results.
First, we take a minor detour to present a technical result (Lemma 2) which is useful both in deriving some of our main results as well as in interpreting them.
Lemma 2 ( Expressing powers of a companion matrix using symmetric polynomials ) .
For a companion matrix
A with distinct eigenvalues, for any k ∈ [ p ] , the (1 , k ) th element of A j , can be expressed using Schur polynomials of the eigenvalues
λ
= { λ 1 , λ 2 , · · · λ p } of A , that is, A j 1 ,k =
S j,k ( λ ) , where S j,k ( λ ) refers to the Schur polynomial indexed by K = { j, 1 , · · · k − 1 times · · · , 1 , 0 , · · · , 0 } .
Lemma 2 shows that the coefficients of the powers of a companion matrix can be fully characterized using
symmetric
Schur polynomials of its eigenvalues.
A good overview of these polynomials can be found in Chaugule et al.
An advantage of expressing the coefficients using symmetric Schur polynomials is that these polynomials have been a subject of extensive research in combinatorics and an equivalence between several alternate definitions has been established.
To name a few, Cauchy’s bialternant expression, (Cauchy 1815; Jacobi 1841), the combinatorial formula (Macdonald 1998) or Jacobi–Trudi identity (Jacobi 1841) are all equivalent ways to define Schur polynomials.
It is therefore possible and often beneficial to choose the definition that yields the most useful notion for the context.
We utilize this connection to interpret our results.
First, for easier interpretation, we simplify Lemma 1 to the following result for scalar AR models.
Corollary 1 ( Difference in Causal and Statistical errors (AR) ) .
Let { x t } t ∈
Z follow an AR(q) process.
Then, for any
AR(p) model with parameters ˆ A ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] where, for any k ∈ N , γ k denotes the autocovariance of { x t } t ∈
Z with lag k .
A and (cid:98) A are the corresponding companion matrices of the model and estimated parameters as defined in Lemma 1.
Lemma 1 identifies factors that control causal generalizability.
We now describe them.
Correlations control causal generalizability.
Recall our motivating example of the two highly correlated time-series where the casual and statistical errors diverge.
Intuitively, one would therefore expect that large correlations among time series potentially induce large differences between observational and interventional distributions.
The quantitative dependence of causal generalizability on the correlation structure of the process is, however, less obvious.
Lemmaconfirms the intuition and shows that correlations between the intervened time-series x t − ω,i across both the components and time instances in x t − ω control generalizability from observational to the interventional distributions.
High-dimensional and higher-order processes can hurt generalization.
For high-dimensional processes it is not unlikely to have strong correlations across components, which may obscure causal relations in the same way as strong correlations across time does for univariate processes.
Lemma 1 also supports this intuition and shows that strong correlations across components as well as time instances play a role.
With increasing order or dimension of the processes, larger orders of covariances across time and dimensions could entail poor causal generalizability.
Dependence on ω .
The dependence of the error on ω arises through the elements of the matrix power A k .
A simple computation shows that, even for an AR(2) model, the dependence of these coefficients on the model parameters is asymmetric and highly intricate.
However, using the Cauchy’s bialternant formulation of Schur polynomials, we have that for any AR(p) model, the coefficients A ω 1 k can be expressed as A ω 1 k = ( − 1)
{+k +1 (cid:80) pi =
 λ p + ω − 1 i e k ( λ i ) det (cid:12)(cid:12)(cid:8) λ p − k ′ k+}
(cid:9) k,k ′ ∈
[ p ] (cid:12)(cid:12) , where e k ( λ i ) refers to the elementary symmetric polynomial of order k and with variables { λ 1 , · · · λ i − 1 , λ i +1 , · · · , λ p } .
While this is not the most interpretable definition per se, the dependence of the coefficients on ω is easily understood and it is easy to verify that if the underlying model as well as the estimated model are both stable ( | λ | < 1 ), the coefficients and hence the difference in errors exponentially decays with interventions arbitrarily in the past of the target variable and if either of the process is not stable ( | λ | > 1 ), the difference can indeed diverge.
Proposition 1 allows us to obtain a high-level perspective on causal generalizability.
It states that the condition number of the autocovariance matrix controls causal generalizability.
Both the maximum and the minimum eigenvalue of the autocovariance matrix (and hence the condition number) can be used as a measure of stability and hence determine the strength of correlation of the underlying process (Basu et al. 2015;
Melnyk et al. 2016).
As the process gets closer to the boundary of stability domain, the autocovariance matrix gets singular and hence the condition number of the autocovariance matrix can get arbitrarily large.
Proposition 1, therefore, can be interpreted as if the underlying process gets closer to the boundary of the stability domain the causal and statistical errors can diverge.
For intuition, let us revisit our motivating example from the introduction with strongly correlated observations in an AR ( p ) process.
Let, without loss of generality p = q .
Introducing the vectors a :=
( a 1 , a 2 , . .
. , a p ) and ˆ a :=
(ˆ a 1 , ˆ a 2 , . . .
, ˆ a p ) and the covariance matrix Σ p = Σ max { p,q } .
Then the quotient between causal and statistical error for predicting one time step ahead i.e. ( ω = 1) reads:
[Equation]
Where we have assumed X t to have unit variance without loss of generality.
The quotient is maximized if (ˆ a − a ) is a multiple of the eigenvector to the smallest eigenvalue of Σ p .
This aligns with the intuition that causal loss diverges when the auto-covariance matrix gets singular.
Moreover, we see that the vector (ˆ a − a ) can be large with little observable effect when it mainly consists of eigenvectors with small eigenvalues of Σ p .
In the extreme case, if the minimum eigenvalue of the autocovariance matrix is 0 , it is possible to arbitrarily deviate from the true model parameters along the direction of the corresponding eigenvector which can significantly affect the causal error without affecting the statistical error at all.
For an AR (2) process, for instance, we obtain Σ p = (cid:18) a 1 / (1 − a 2 ) a 1 / (1 − a 2 ) (cid:19) , which becomes singular for a 1 = ± (1 − a 2 ) which indeed is the boundary of the stability domain (see for example, Lütkepohl (2009)).
This is the limit in Section 1 where X t = ±
X t − 1 .
The eigenvector for eigenvalue 0 reads (1 , ∓ 1) .
Accordingly, the quotient (10) diverges when ˆ a differs from a by (1 , ∓ 1) .
This further highlights that even for simple classes of forecasting models and with simplifying assumptions such as causal sufficiency, causal risks may even diverge from statistical risks.
To show this formally, by means of Lemma 2, we can derive an explicit upper bound on the condition number of the autocovariance matrix κ (Σ max { p,q } ) for AR(p) models and arrive at Corollary 2.
Corollary 2 ( Stability Controls Causal Generalization (AR) ) .
Consider an AR(q) process, such that eigenvalues of its companion matrix satisfy | λ | < δ < 1 .
For any AR(q) model f ,
[Equation]
where K p is some finite constant that depends on the order p of the underlying process.
The bound in Corollary 2 is elegant due to its simplicity and generality.
However, the cost of generality of the bound that relies only on the stability parameter is clearly that it cannot explain the variations in behavior exhibited by individual processes with the same stability parameter.
For instance, consider an AR(2) model with parameters a 1 and a 2 with a 2 ≈
so that it is essentially an AR(1) model.
Then, it is easy to verify that λ 2 ≈ 0 .
The combinatorial definition of the Schur polynomials (Macdonald 1998) allows us to express the coefficients as follows:
[Equation]
ing this with Corollary 1, it is easy to see that if the estimated model is also close to AR(1), then the coefficients A ω 12 and
[Equation]
A ω 12 and hence the difference in statistical and causal errors is close to 0 .
The bound in (11) which relies on the stability parameter does not capture this.
For tighter bounds that utilize additional information about the spectrum of the companion matrix, we can exploit the connection to Schur polynomials to arrive at the following bound.
[Equation]
where K p,q is a constant that depends on p, q , δ and (cid:98) δ are the stability parameters of the true and estimated processes respectively and λ and (cid:98) λ denote the set of eigenvalues of A and (cid:98) A respectively.
SIMULATIONS
To verify the practical behavior of causal and statistical risks, we provide some simple simulations to study the errors of different estimators under AR processes.
For each presented plot, we draw parameters for 10,000 stationary AR ( p ) processes using rejection sampling.
We draw the coefficients of each process independently and uniformly from [ − 2 , 2] and reject sets of parameters that yield a non-stationary process.
For each process, we draw a training sample withtimesteps and a test sample with 1000 timesteps.
For all figures in the main paper we set ω = 1 .
To estimate the coefficients we use Ordinary Least Squares (OLS).
In Appendix 5 we provide additional plots with hidden confounder, as well as varying order, sample size, ω and other estimators: Ridge, Lasso, and Elastic Net regressors.
OLS minimizes the empirical statistical error, that is, (cid:80)
y i , ˆ y i
( y i − ˆ y i ) 2 , where ˆ y i denotes the model prediction with estimated parameters ˆ a .
In line with our theoretical results, we find that even for simple scalar AR processes of small orders, the causal error of the estimators is often several times larger than the statistical error (see Figure 2).
In Figure 3 we sorted the randomly drawn datasets by their autocorrelation (measured by the condition number κ of the autocorrelation matrix) and split the sorted list into buckets of 500 dataset.
For each we calculated the maximum, mean and 90% quantile of the difference in causal and statistical error for the OLS and Ridge estimators.
The plots corresponding to the other estimators are provided in Appendix 5 We can see that upto constant factors, our theoretical finite sample causal generalization bound matches the difference in causal and statistical risks observed empirically.
EXPERIMENTS ON REAL DATA
Data.
We conduct experiments on the m4 hourly dataset (Makridakis et al. 2018) that includes timeseries from a diverse set of sources.
The datasets has a hourly frequency and a prediction length of 48.
To create an interventional distribution without a generative model, for each time series we replace the last time step prior to the evaluation window by sampling at random either from all time-series at that time step (referred to as across-ts ) or from previous values of the same time series (referred to as within-ts ). Section 5. records results for additional datasets.
Models.
We include three popular deep neural network architectures in our evaluation.
DeepAR consists of an RNN that takes the previous time steps as inputs and predicts the parameters of an auto-regressive model (Gasthaus et al.
Wavenet is a hierarchical CNN developed for speechto-text (Oord et al. 2016).
Transformer is an attention-based deep neural network widely applied to NLP tasks including translation (Vaswani et al. 2017).
For all these models we use AutoGluonTS’s default hyperparameters.
Metrics.
For the observational distribution, we compute the root-mean-square error (RMSE) comparing average prediction for each time point with the ground truth in the evaluation set.
For the interventional distribution we are lacking ground truth.
Therefore, we train two separate models and compute their disagreement.
Definition 5.1.
The disagreement is the average root-meansquare deviation of the mean forecasts of two models.
The average is taken over a set of time-series.
If the time-series come from the original dataset, we call it the statistical disagreement.
If they come from one of the interventional datasets, we call it causal disagreement and specify the type of intervention as across time-series or within time-series.
This disagreement is a measure of uncertainty introduced by the randomness in the training and evaluation procedure.
Here, however, we use it to approach the causal risk, that we cannot compute directly.
If the disagreement is high on the interventional distribution at least one of the models must have a high causal risk.
For comparison, we also included this disagreement measure for examples from the observational distribution.
Finally, to explore the relationship between causal forecasting error and uncertainty, we also compute the width of the 80% prediction interval for both the observational and interventional distribution.
Definition 5.2.
The 80% prediction width of a forecasting model is the absolute distance between the 0.9 quantile and 0.1 quantile of the forecast distribution.
It is averaged over a set of time-series that can come from the observational or the interventional distritibutions.
The experiments were conducted using GluonTS (Alexandrov et al. 2019) with default hyperparameters on instances with 4 virtual CPUs and a 2.9 GHz processor.
Limitations.
The dataset and models have clear shortcomings.
Likely, the dataset is not causally sufficient.
Also, we did not tune the models.
Moreover, we are lacking samples from the marginal distribution for the interventions and groundtruth on what happens under these interventions.
Nevertheless, we hope to get a sense for how popular deep learning networks can behave on real data for relevant prediction tasks under interventions.
Results.
Figure 4 shows the results of the metrics when we evaluate the models on the datsets for both observation and interventional distributions.
We see that the causal disagreement between two models of the same architecture and hyper-parameters can be much higher than their disagreement on the observational distribution.
While there are only smaller differences in the statistical risk between the model architectures, their causal disagreement differs more.
Overall, the the causal disagreement can be high, which implies high causal risk, but it varies across datasets and model architectures.
Wavenet’s disagreement is an order of magnitude larger when sampling interventions from other time-series.
For transformer models their interventional disagreement is close to the observational one.
Uncertainty.
When we compare the width of the 80% interval of predictions in Table 1 we see that this uncertainty measure is higher for the interventional distribution compared to the observational one.
Moreover, directionally it relates to the causal disagreement across models.
Unlike the disagreement [Table] that requires a second model to be trained, this uncertainty measure is readily available from the predicted forecasts.
5.1 FURTHER EXPERIMENTS ON REAL DATA
We conduct additional experiments on the traffic dataset that records the occupancy rates of car lanes on freeways in the San Francisco Bay Area (Dua et al. 2017) and the electricity dataset (Dua et al. 2017) that records the electricity consumption of 370 customers hourly. The additional results on these two datasets in Figure 5 confirm our previous discussion, that the causal disagreement between two models of the same architecture and hyperparameters can be much higher than their disagreement on the observational distribution. While there are only smaller differences in the statistical risk between the model architectures, their causal disagreement differs more.
Wavenet continues to have a high causal disagreement.
The disagreement can be viewed as an uncertainty measure over the model training.
An additional uncertainty measure can be derived from the forecasts themselves which represent a distribution over future time-series continuations.
Tablereports the average width that captures 80% of the samples drawn from the forecast distribution.
We see that this it is yields similar results to those of Figure 5: The prediction width is wider for the interventional distributions and varies across datasets and model architectures.
The causal disagreement can be high for some models which implies a high causal risk.
This cautions against the use of statistical deep learning models to forecast what will happen under interventions.
The difference we observe in causal disagreement across models motivates further development of specific model architectures suitable for causal forecasting.
For existing models, the uncertainty measure considering the width of the prediction interval can be an indicator for causal risk.
Our work intersects with domain adaption, RL, and treatment effect estimation, reviewed separately below.
Domain Adaptation.
The literature that is perhaps most relevant to our context is that of learning theory for domain adaptation, in particular, for covariate shift.
Theoretical analysis of domain adaptation when labelled samples from the source distribution and unlabelled samples from the target [Table] distribution are generated i.i.d was initiated by Ben-David et al.
(2007), who provided VC bounds for binary classification under covariate shifts based on a discrepancy measure d F between source and target distributions that depends on the hypothesis class F and is estimable from finite samples.
Mansour et al.
(2009) extended the work to the context of regression in the i.i.d setting by adapting the discrepancy measure for more general loss functions and by providing tighter, data-dependent Rademacher bounds.
Estimation of Treatment Effects.
A related problem is that of estimating treatment effects in the potential outcomes framework (Hill et al. 2006; Shi et al. 2019), where the goal is to estimate the effects of binary-valued treatments from observational data under a multivariate confounding model.
Our setting is more general in that variables in the multivariate process can take a continuum of interventions and play a multiplicity of roles — each variable plays the role of treatment, confounder, and the target variable.
Of particular relevance is the work of Shalit et al.
(2017) and Johansson et al.
(2020), who prove generalization error bounds on estimating individual-level treatment effects in terms of standard generalization error and a distance measure between the treated and control distributions.
This result is similar to domain adaptation bounds in Ben-David et al.
and Mansour et al.
and may be interpreted as causal learning theory in the sense of our paper.
Reinforcement Learning.
The ratio of observational versus interventional densities in our setting play a similar role as the state density ratio in off-policy evaluation in reinforcement learning(RL)
(Bennett et al. 2021).
In RL, however, the clear separation between the state of actions and the state space acted on admits techniques that we do not see for our problem, e.g., deconfounding (Hatt et al. 2021), or learning representations of the history that are independent of the actions (Bica et al. 2020), which overcomes the problem of high inverse probability weightings (Lim et al. 2018).
DISCUSSION AND CONCLUSION
Our work highlights that even for very simple models and even under simplifying assumptions such as causal sufficiency, causal and statistical errors can diverge.
It emphasizes the need for providing guarantees for causal generalization in a similar vein as providing guarantees for statistical learning.
To this end, we initiate a first analysis in this direction by introducing a framework for causal learning theory for forecasting and providing conditions under which one can guarantee generalization in the causal sense for the class of VAR models.
We hope that this work inspires more theoretical work that allows certifying the validity of the causally interpreting forecasting models.
Our theoretical as well as empirical results challenge the causal interpretation of forecasting models used in practice which are typically far more complex.
Our experiments show that causal disagreement can be high for some models which implies a high causal risk.
This cautions against the use of statistical deep learning models for causal forecasting.
The difference we observe in causal disagreement across models motivates further development of specific model architectures suitable for causal forecasting.
For existing models, the uncertainty measure considering the width of the prediction interval can be an indicator for causal risk.
[-+
-]
ACKNOWLEDGMENTS
This work has been supported by the German Research
Foundation (Research Training Group GRK 2428) and the
[-+
-]
Baden-Wurttemberg Stiftung (Eliteprogram for Postdocs project “Clustering large ¨ evolving networks”).
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Leena
Chennuru Vankadara
[-( y i − ˆ y i ) 2 + λ Ω(ˆ a ) , where ˆ y i denotes the model prediction with estimated parameters ˆ a and λ > 
-]