Interpolation and Regularization for Causal Learning
Abstract
Recent work shows that in complex model classes, interpolators can achieve statistical generalization and even be optimal for statistical learning.
However, despite increasing interest in learning models with good causal properties, there is no understanding of whether such interpolators can also achieve causal generalization .
To address this gap, we study causal learning from observational data through the lens of interpolation and its counterpart—regularization.
Under a simple linear causal model, we derive precise asymptotics for the causal risk of the min-norm interpolator and ridge regressors in the high-dimensional regime.
We ﬁnd a large range of behavior that can be precisely characterized by a new measure of confounding strength .
When confounding strength is positive, which holds under independent causal mechanisms—a standard assumption in causal learning—we ﬁnd that interpolators cannot be optimal.
Indeed, causal learning requires stronger regularization than statistical learning.
Beyond this assumption, when confounding is negative, we observe a phenomenon of self-induced regularization due to positive alignment between statistical and causal signals.
Here, causal learning requires weaker regularization than statistical learning, interpolators can be optimal, and optimal regularization can even be negative.
Introduction
We consider the problem of learning the causal inﬂuence of multivariate covariates x ∈ R d on a scalar target variable y ∈ R purely from observational data and under the presence of hidden confounders.
Formally, given ﬁnite samples { ( x i , y i ) } n i =1 drawn independently and identically (i.i.d) from the joint observational distribution p ( x, y ) = p ( x ) p
( y | x ) , the goal of causal learning is to predict the effects on the target variable y under interventions on the covariates x .
In other words, the goal is to learn a predictive model that minimizes the expected loss on a random draw from the interventional distribution p do ( x, y )
= p ( x ) p ( y | do ( x )) , which can be different from the observational distribution.
Recently, Janzing (2019) established a close analogy between statistical and causal learning (albeit under a highly constructed confounded model).
As a consequence, Janzing (2019) suggested that standard statistical learning-theoretic techniques (such as norm-based regularization) may also help learn good causal models.
However, the classical statistical principles of bias-variance trade-off have been challenged in recent years by highly complex classes of models that are trained to interpolate the data and yet achieve remarkable generalization properties across a broad range of problem domains (Zhang et al., 2021).
A large volume of recent work suggests that interpolation can be compatible with and may even be necessary to achieve optimal statistical generalization in the high-dimensional regime (Belkin et al., 2018; Belkin et al., 2019a; Liang et al., 2020; Feldman, 2020).
Despite the surge in interest, causal properties of such interpolators have not yet been explored.
In this work, we consider a simple linear causal model in the high-dimensional regime ( n, d → ∞ , d/n ∈
O (1) ) and ask: can interpolators achieve good causal generalization?
1.1 Motivation and Related Work
Resemblance between statistical and causal generalization Causal learning can be regarded as an instance of the general problem of learning under distribution shifts, where the training (observational) distribution is shifted from the test (interventional) distribution.
In the framework of out-of-distribution generalization, an interesting proposition for causal learning arises from the following high-level idea.
Observing small sample sizes may induce a similar bias as distribution shifts.
Therefore, techniques for learning models with good out-of-sample generalization (such as regularization) may also help to learn models with good out-of-distribution generalization and vice-versa.
The literature provides plentiful evidence to support this general principle for different classes of distribution shifts.
For instance, under a broad class of distribution shifts, distributionally robust optimization is equivalent to norm-based regularization (Xu et al., 2009; Shaﬁeezadeh Abadeh et al., 2015; Gao et al., 2017; Shaﬁeezadeh-Abadeh et al., 2019; Blanchet et al., 2019; Kuhn et al., 2019).
Analogously, distributionally robust optimization techniques are also employed for statistical learning under limited samples (Zhu et al., 2020).
Particularly relevant to our work is Janzing (2019), which formally establishes a close analogy between “generalizing from empirical to observational distributions ” and “generalizing from observational to interventional distributions ” under a highly constructed confounding model.
This analogy suggests that standard norm-based regularization such as lasso or ridge, typically used for statistical learning, may also help learn better causal models.
Interpolation can be compatible with statistical learning Explicit norm-based regularization techniques were initially motivated by the classical learning theory principle of bias-variance tradeoff, which is characterized by a U-shaped generalization curve.
This principle recommends to avoid interpolation and instead suggests to balance data ﬁtting with the complexity of the hypothesis class.
Recently, however, these classical principles have been challenged by deep learning models.
Despite being highly complex with the ability to ﬁt even random labels and often trained to interpolate the training data, they achieve state-of-the-art out-of-sample generalization across many domains (Zhang et al., 2021).
A partial explanation is provided by the double-descent phenomenon (Belkin et al., 2019b; Belkin, 2021).
Extending the generalization curve beyond the interpolation threshold reveals two regimes: the classical U-curve in the underparameterized regime and a decreasing curve in the overparameterized regime.
This behaviour is not limited to deep neural networks, but extends to other settings such as random feature models and random forests (Belkin et al., 2019b; Hastie et al., 2022; Mei et al., 2021).
Follow-up work suggests that in the overparameterized regime, interpolators can indeed achieve low statistical risk (Belkin et al., 2019a; Liang et al., 2020; Bartlett et al., 2020; Tsigler et al., 2020; Muthukumar et al., 2020).
Is interpolation compatible with causal learning?
On account of the parallels between statistical (out-of-sample) and causal (out-of-distribution) learning, it is therefore natural to ask: can interpolators also learn good causal models?
One line of empirical work suggests that naively applying distributionally robust learning techniques such as importance reweighting or distributionally robust optimization approaches (which are equivalent to certain forms of regularization) offers vanishing beneﬁts over empirical risk minimization in overparameterized model classes (Byrd et al., 2019; Sagawa et al., 2020; Gulrajani et al., 2021).
However, there is also empirical evidence that suggests that augmenting such techniques with additional explicit norm-based regularization may help to learn distributionally robust models in the overparameterized regime (Sagawa et al., 2020; Donhauser et al., 2021).
In the context of causal learning, it has been suggested that explicit regularization can be beneﬁcial and might even need to be stronger than for statistical learning (Janzing, 2019; Vankadara et al., 2021).
Existing work, therefore, remains unclear about the role of explicit regularization in causal learning, or correspondingly, whether interpolation is compatible with causal learning.
In this work, we take a theoretical approach to systematically address these questions.
1.2 Our Contributions
We provide a ﬁrst analysis of causal generalization from observational data in the modern, overparameterized and interpolating regime under a simple linear causal model.
Speciﬁcally, we consider the interpolating minimum l 2 norm least-squares estimator and the family of regularized ridge regression estimators in the proportional asymptotic
regime. We seek answers to the following questions is there a regime where the optimal causal regularization parameter is 0, that is, can we observe benign causal
overﬁtting Furthermore, if the optimal causal regularization parameter is positive, how strongly do we need to regularize?
How does the optimal causal regularization compare to the optimal statistical regularization
While our analysis is exhaustive, we emphasize the results under the assumption of independent causal mechanisms (Janzing et al ), a standard assumption in causal learning
• Precise asymptotics of the causal risk (Section 3).
We provide precise asymptotics of the causal risk of the ridge regression estimator as well as the minimum l 2 norm interpolator in the high-dimensional setting: n, d → ∞ , d/n → γ ∈ (0 , ∞ ) .
Our results conﬁrm that, similar to the statistical setting, the causal generalization curve of the min-norm estimator exhibits the double-descent phenomenon.
This is because the variance term diverges at the interpolation threshold and is decreasing in the overparameterized regime ( γ > 1 ).
• A measure of confounding strength ζ
(Section 2.1).
We introduce a new measure of confounding strength ζ that quantiﬁes the relative contribution of the “confounding signal” to the “causal signal”.
It can be interpreted as the strength of the distribution shift between the observational and interventional distributions.
While ζ can take any real value in general, it is restricted to [0 , 1] under the assumption of independent causal mechanisms.
There, it induces a strict, model-independent ordering of all causal models that entail the same observational distribution.
• Benign causal overﬁtting (Section 4).
When the causal signal dominates the statistical signal ( ζ < 0 ), we observe a phenomenon of self-induced regularization due to the confounding signal.
As a consequence, the optimal causal regularization can be 0 or negative even if the optimal statistical regularization is strictly positive.
Under the assumption of independent causal mechanisms, however, we show that there is no benign causal overﬁtting.
This is in contrast to benign statistical overﬁtting, which can occur in the highly underparameterized regime ( γ → 0 ).
• Optimal causal vs. statistical regularization (Section 5).
We show that causal learning requires weaker regularization than statistical learning when the confounding strength ζ is negative.
However, when ζ > 0 and in particular under the principle of independent causal mechanisms, we show that causal learning requires stronger regularization than statistical learning.
More speciﬁcally, the optimal causal regularization is strictly increasing in confounding strength.
Problem Setup
[Equation]
[Figure] We consider a linear causal model with parameters M ∈ R d × l , α ∈ R l , β ∈ R d with l ≥ d and σ 2 > 0 described via the structural equations
[Equation]
The covariates x ∈ R d and the target y ∈ R are confounded through z ∈ R l , which follows a standard normal distribution.
This structure implies that E x = 0 and the covariance of x is Σ :=
Cov x = MM T .
A graphical representation of this causal model is given in Figure 1a.
The observational joint distribution of this causal model is given by p ( x, y )
= p ( x ) p ( y | x ) , where x ∼ N (0 , Σ) and y | x ∼ N ( x T ˜ β, ˜ σ 2 ) .
Here, the statistical parameter ˜ β := β + Γ consists of the causal parameter β and a confounding parameter Γ :=
M + T α , where M + T is shorthand for ( M + ) T and M + denotes the Moore-Penrose inverse of M .
The statistical noise is given by ˜ σ 2 := σ 2 + (cid:107) α (cid:107) 2 − (cid:107) Γ (cid:107) 2Σ , where (cid:107) x
(cid:107) 2Σ :
= x T Σ x denotes the generalized norm.
Note that the observational distribution alone cannot distinguish the causal model from the one in Figure 1b
The goal of statistical learning is to predict y after observing x , which is captured by the conditional distribution p ( y | x ) .
In contrast, the goal of causal learning is to predict y after manipulating or intervening on x .
This is formally captured by Pearl’s do -calculus (Pearl, 2009), which describes interventions on random variables as a shift in the joint distribution.
Intervening on x with the value x 0 , denoted as do ( x = x 0 ) , removes all arrows to x and sets x = x 0 .
In our causal model (1), the intervention do ( x = x 0 ) removes the arrow z
→ x and yields the updated structural causal equations z ∼ N (0 , I l ) ,
[Equation]
The corresponding distribution of y after intervening on x is therefore given by y | do ( x = x 0 ) ∼
N ( x T 0 β, ˜ σ 2 + (cid:107) Γ (cid:107) 2Σ ) .
Since arbitrary interventions can introduce arbitrary distribution shifts, we consider the natural class of interventions drawn from the observational marginal distribution on x .
This yields the interventional joint distribution p do ( x, y )
= p ( x ) p ( y | do ( x )) with the slight abuse of notation do ( x ) in which the random variable x and its value coincide.
Causal learning from observational data Assume we are given i.i.d.
samples { ( x i , y i ) } ni =1 from the observational joint distribution p ( x, y ) , which we collect in X ∈ R n × d and Y ∈ R n .
The usual statistical learning aims for the observational conditional p ( y | x ) , which means that train and test distributions coincide.
Causal learning aims for the interventional conditional p ( y | do ( x )) , a distribution shift problem for which train and test distributions differ.
We deﬁne the corresponding causal risk R C and statistical risk R S of any linear regressor ˆ β ∈ R d under the squared loss as R [Equation] do
[Equation]
x [Equation]
x T ˆ
[Equation]
and R S (ˆ β ) := E x E y | x ( x T
[Equation]
The following proposition (proven in Appendix A) characterizes the risks under the model (1).
Proposition 2.1 (Causal and Statistical Risk).
For any ˆ β ∈ R d , the risks deﬁned in Eq.
(2) satisfy
[Equation]
Therefore, β is the optimal causal parameter and ˜ β is the optimal statistical parameter.
In the following, we simply refer to them as causal and statistical parameters.
2.1 A New Measure of Confounding Strength
Since the interventional distribution generally differs from the observational distribution, we require a measure that quantiﬁes how this shift inﬂuences causal learning from observational data.
Signal-to-noise ratios (SNRs) Before we deﬁne our measure of confounding strength, we ﬁrst deﬁne the statistical and causal signal-to-noise ratios, which help to intuitively understand our confounding strength measure.
Recall that every causal model entails a statistical model since the causal parameter β and the confounding parameter Γ jointly specify the statistical parameter ˜ β = β + Γ .
The statistical SNR is deﬁned as usual by SNR S :
= (cid:107) ˜
β (cid:107) 2 / ˜ σ 2 .
For the causal SNR, a natural notion would be (cid:107) β (cid:107) 2 / (˜ σ 2 + (cid:107) Γ (cid:107) 2Σ ) if the learning algorithm had access to data from the interventional distribution y | do ( x ) ∼ N ( x T β, ˜ σ 2 + (cid:107) Γ (cid:107) 2 Σ ) ; but since we are constrained to data from the observational conditional y | x ∼ N ( x T ˜ β, ˜ σ 2 ) , the corresponding causal SNR, which quantiﬁes the hardness of the learning problem, needs to take this into consideration.
Accordingly, we consider the causal SNR as the ratio of the alignment between the statistical and causal parameters and the variance of the observational conditional.
Formally, we deﬁne it as SNR C := (cid:104) β, ˜ β (cid:105) / ˜ σ 2 .
In what follows, we therefore often refer to (cid:104) β, ˜ β (cid:105) as the causal signal and (cid:107)
˜ β (cid:107) 2 as the statistical signal .
Correspondingly, we refer to (cid:104) ˜ β − β, ˜ β (cid:105) =
(cid:104) Γ , ˜ β (cid:105) as the confounding signal , which is the alignment between the confounding parameter Γ and the statistical parameter ˜ β .
Confounding strength Regression on observational data implicitly assumes that the interventional distribution coincides with the observational distribution, while it can be shifted in general.
To quantify the impact of this distribution shift on the corresponding causal risk, we introduce a new confounding strength measure ζ .
It measures the relative contribution of the confounding signal to the statistical signal and is deﬁned by
[Equation]
Other notions of confounding strength are possible, but we will see later that this deﬁnition is well-suited to capture the shift strength for causal learning from observational data.
Without further restrictions, ζ can take any value in R .
This measure divides the causal models into the following three regimes, depending on the relationship between causal and statistical signal:
• ζ ≥ 1 : the causal signal (cid:104) β, ˜ β (cid:105) is non-positive, which implies that causal and statistical parameters are orthogonal or negatively aligned.
Statistical learning is adversarial to causal learning.
• 0 < ζ < 1 : causal and statistical parameters are positively aligned but the causal signal is weaker than the statistical signal (cid:107) ˜
β (cid:107) 2 , for example β = ˜ β/ 2 .
• ζ ≤ 0 : the causal signal dominates the statistical signal, for example β = 2˜ β .
The SNRs are related to the confounding strength measure via SNR C
= (1 − ζ ) SNR S .
In particular, the causal signal decreases as the confounding strength increases.
The regime 0 ≤ ζ ≤ 1 is practically most relevant Causal learning often requires strong assumptions because causal models cannot be uniquely identiﬁed by their observational distribution.
A standard assumption is the principle of independent causal mechanisms (ICM) (Janzing et al., 2010; Lemeire et al., 2013; Peters et al., 2017), which informally asserts that causal mechanisms share no information.
In our causal model (1), a corresponding assumption could be that the causal mechanisms α and β are drawn from rotationally invariant distributions.
This implies that (cid:104) β, Γ (cid:105) →as d → ∞ , which in turn falls in the regime 0 ≤ ζ ≤ 1 .
While our following analysis covers all possible causal models, we pay special attention to this regime because it might be of highest practical relevance.
Note that for (cid:104) β, Γ (cid:105) = 0 , our measure of confounding strength coincides with the measure ζ (cid:48)
= (cid:107) Γ (cid:107) 2 / ( (cid:107) Γ (cid:107) 2 + (cid:107) β
(cid:107) 2 ) introduced by Janzing et al.
It measures the relative contribution of causal and confounding signal in terms of lengths rather than inner products.
Causal and Statistical Risk of High-Dimensional Regression Models
Causal learning is extremely challenging, because it requires scarcely available interventional data or has to rely on other information such as exogenous (Rothenhäusler et al., 2021) or instrumental variables (Angrist et al., 1991).
In our setting where only observational data are available, causal learning requires additional model assumptions.
One such approach has been followed by the Concorr method (Janzing, 2019) which leverages the ICM assumption to make an improved choice of regularization parameter under a linear regression model.
To fully characterize the effect of regularization on causal generalization, we consider two estimators for learning causal models from observational data ( X, Y ) ∈ R n × d × R n : the min-norm interpolator and ridge regressors.
The min-norm interpolator is the minimum l 2 norm solution to the least squares regression problem
[Equation]
A closed form is given by ˆ β 0 ( X, Y ) =
( X T X ) + X T Y .
For λ > 0 , the ridge regressor solves
[Equation]
which has the explicit solution ˆ β λ ( X, Y ) =
( X T X + nλI d ) − 1 X T Y .
The min-norm interpolator can be obtained as a limiting case from the ridge regression solution via ˆ β 0 ( X, Y ) =
lim λ
→ 0 + ˆ β λ ( X, Y ) .
Whenever it is clear from the context, we drop the dependence of the predictors on X and Y .
Before proceeding with the analysis, we motivate the idea that appropriate regularization can help to learn causal models from purely observational data.
To this end, we compare regularization chosen by statistical cross validation to regularization based on an interventional validation set in Figure 2.
Since cross validation implicitly assumes that there is no confounding, it is close to Bayes optimal for ζ = 0 when n (cid:29) d .
However, as confounding increases, it falls behind regularization based on the interventional validation set.
The latter even yields Bayes optimal risk again in the purely confounded setting ζ = 1 , where the lack of causal signal ( β = 0) is encoded by inﬁnite regularization. While we might not have access to an interventional validation set in practice, our theory will show that knowledge of confounding strength is sufﬁcient for choosing appropriate regularization. Finally, we want to caution that even though regularization can help, it does not remove the hardness of causal learning. Reliable causal inference still requires stronger assumptions or additional data.
Confounding strength ζ
C a u s a l e x ce ss r
i s k
||
ˆ
β − β
|| [Figure] 3.1 Precise Asymptotics of the Causal and Statistical Risks In this section, we provide precise asymptotics for the causal and statistical risks of the min-norm interpolator and ridge regression solutions in the high-dimensional regime.
This regime is characterized by both n, d → ∞ such that d/n → γ ∈ (0 , ∞ ) , where γ is called the overparameterization ratio .
We distinguish between the underparameterized regime ( γ < 1 ) and the overparameterized regime ( γ > 1 ).
All proofs for this section are deferred to Appendix B. Since the predictors ˆ β = ˆ β ( X, Y ) are random variables in the training data X and Y , so is their corresponding causal risk.
We consider the expectation of this risk under Y conditioned on X .
According to Proposition 2.1, it is given by
R CX (ˆ β ) :=
E Y | X R C (ˆ β ) =
E Y | X (cid:107) ˆ β − β (cid:107) 2Σ + ˜ σ 2 + (cid:107) Γ (cid:107) 2Σ .
Due to its simple form, similar to the usual statistical risk, the causal excess risk can be decomposed into bias and variance:
[Equation]
The next theorem is one of our main results.
It gives a closed-form expression for the limiting causal bias and variance of the min-norm interpolator and ridge regression estimators.
We make the simplifying assumption of isotropic covariance Σ
= I d .
The proof relies on recent techniques from random matrix theory.
It employs arguments similar to Dicker (2016), Dobriban et al. (2018), and
Hastie et al.
and can correspondingly be extended to arbitrary covariances under boundedness assumptions on the spectrum.
We leave such extensions for future work and focus on thoroughly understanding the isotropic causal model, because it already exhibits rather rich behavior.
Theorem 3.1 (Limiting Causal Bias-Variance Decomposition for the Ridge Estimator).
Let (cid:107) β (cid:107) 2 = r 2 , (cid:107) Γ (cid:107) 2 = ω 2 , (cid:104) Γ , β (cid:105) =
η , and ﬁx ˜ σ 2 .
Then as n, d → ∞ such that d/n → γ ∈ (0 , ∞ ) , it holds almost surely in X for every λ > 0 that
[Equation]
where m ( λ ) =
((1 − γ − λ ) − (cid:112) (1 − γ − λ ) 2 − 4 γλ ) / (2 γλ ) and ˜ r 2 = r
+ ω 2 + 2 η .
Therefore
R CX (ˆ β λ ) → R Cλ = B Cλ + V Cλ + ˜ σ 2 + ω 2 .
The corresponding limiting quantities for the min-norm interpolator can be obtained by taking the limit λ
→ 0 + in (7) and (8) .
From these limiting expressions we can see that the causal risk curve of the min-norm interpolator exhibits the double descent phenomenon: it diverges at the interpolation threshold γ = 1 due to the variance term and decreases again for γ > 1 .
A corresponding visualization is given in Figure 4.
Explicit regularization dampens the divergence of the variance term.
While we are primarily interested in the causal risk, the corresponding statistical risk serves as a natural baseline.
An analogue set of results for the statistical risk is given in Appendix C. These results have already been derived by Hastie et al.
and can also be recovered as a special case of our causal results: for ﬁxed statistical parameters ˜ β and ˜ σ 2 , the statistical risk coincides with the causal risk of an unconfounded causal model deﬁned with β = ˜ β , σ 2 = ˜ σ 2 , and α = 0 .
In particular, the corresponding statistical limiting expressions are the same as in Theorem 3.1 after setting η = ω 2 = 0 .
Optimal statistical and causal regularization By directly optimizing the closed form expressions for limiting causal and statistical risks we can ﬁnd the optimal causal and statistical regularization. Overparameterization ratio γ C a u s a l r i s k R C
S = S = 0 . S = − Overparameterization ratio γ
Risk
Bias
Variance
For any γ ∈ (0 , ∞ ) , the optimal statistical regularization λ ∗ S ( γ ) := arg inf λ ∈ (0 , ∞ ) R Sλ can be expressed in closed-form as λ ∗ S ( γ ) =
SNR S − 1 γ .
The closed-form expression for the optimal causal regularization parameter λ ∗ C ( γ ) := arg inf λ ∈ (0 , ∞ )
R Cλ is a root of a 4 th order polynomial and as such considerably intricate.
For readability, we do not include it here.
We investigate the behavior of the optimal causal and statistical regularization in Section 4 and 5.
3.2 Basic Behavior of the Limiting Risk
We start to analyze the results by assessing the basic behavior of the limiting causal risk.
The causal risk of the null estimator ˆ β = 0 serves as a natural baseline to evaluate the performance of the the min-norm interpolator and the ridge regression estimators.
Regimes of the min-norm interpolator Theorem 3.1 characterizes the limiting causal risk of the min-norm interpolator.
Its behavior is controlled by the causal signal-to-noise ratio, which we deﬁned as SNR C = (
1 − ζ ) SNR S .
However, as we will later see, the causal risk of the min-norm interpolator can be lower than null risk when ζ < 0 .
To distinguish the regimes of the min-norm interpolator, it is therefore convenient to consider the closely related quantity S = (1 − 2 ζ ) SNR S .
It distinguishes between three different regimes (visualized in Figure
• For S > 1 , the causal signal dominates the noise and the min-norm interpolator can perform better than null risk in both under- and overparameterized regime.
• For 0 ≤ S ≤ 1 , the causal signal is weaker than the noise.
Only the underparameterized regime can beat the null risk, whereas the overparameterized regime is always worse.
• The previous two cases resemble the behavior of the statistical risk in the corresponding regimes of the statistical SNR.
Contrary to the statistical risk, however, the causal risk admits a third regime S < 0 .
In this case, the min-norm interpolator always performs worse than null risk.
Here, the causal signal (cid:104) β, ˜ β (cid:105) is dominated by the confounding signal (cid:104) Γ , ˜ β (cid:105) , and interpolating the observational data overﬁts to the confounding
Bias and variance
The bias-variance decomposition of the causal risk given in Theorem 3.1 is visualized in Figure 4 for the min-norm interpolator and the optimally ridge-regularized regressor.
The ﬁgure also shows the causal risk based on ﬁnite samples from the model, which is in high agreement with our asymptotic results.
We compare the causal risk to the corresponding statistical risk.
First note that the causal and statistical variance terms coincide exactly for both the min-norm interpolator and ridge regressors.
This is because the variance term of the squared loss depends only on the variance in the training data, but not on the target parameter β or ˜ β .
Since the training data are the same for both causal and statistical learning, the variance terms trivially coincide.
For the min-norm interpolator, as in the statistical case, the variance term is responsible for the double-descent behavior of the causal risk curve because it explodes at the interpolation threshold γ = 1 and decreases in the overparameterized regime γ > 1 .
In the statistical setting, the bias strictly increases in the overparameterized regime and, as a consequence, the best risk is always achieved in the underparameterized setting.
In contrast, the causal bias of the min-norm interpolator can be decreasing in the overparameterized regime and therefore the optimal causal risk can be achieved in the highly overparameterized regime γ → ∞ .
However, this only happens in the regime S <where the risk of the min-norm interpolator is always worse than null risk.
Figure 4 shows the causal risk of the optimally regularized ridge regression estimator which trivially is always below that of the min-norm risk.
Similar to the statistical setting, the corresponding generalization curve does not exhibit the double descent phenomenon.
There are qualitatively different reasons for why regularization helps in statistical and causal learning.
For both statistical and causal learning, regularization decreases the shared variance, which corresponds to the ﬁnitesample error.
However, while the statistical bias always increases with regularization, the causal bias can actually decrease.
This implies that regularization not only helps with the ﬁnite-sample error, but can also reduce the error due to confounding.
Higher confounding implies higher causal risk for all λ
So far, we have investigated the causal risk under a single causal model.
Now we can compare different causal models using the confounding strength measure ζ introduced in Section 2.1.
The next proposition shows that ζ governs the hardness of causal learning from observational data.
Speciﬁcally, the causal risk of the ridge regression for any λ ∈ (0 , ∞ ) increases as the causal model becomes more confounded.
A proof is given in Appendix D.
Proposition 3.2 (Causal Risk Increases with Confounding Strength).
Consider the family of causal models parameterized as in (1) that entail the same observational distribution.
Let C {+
+} and Cbe two such causal models with confounding strengths ζ 1 and ζ 2 and alignments η 1 and η 2 (deﬁned in Theorem 3.1), respectively.
Then for all λ, γ ∈ (0 , ∞ ) ,
[Equation]
In particular, for any ﬁxed η , the measure of confounding strength ζ establishes a strict ordering of causal models.
This includes the ICM under which η = 0 .
Benign Causal Overﬁtting
A large number of recent works suggest that minimum-norm interpolators can be optimal for statistical generalization (Belkin et al., 2018; Belkin et al., 2019a; Muthukumar et al., 2020).
This phenomenon is often referred to as benign overﬁtting.
Moreover, the optimal statistical generalization may even be achieved for negative regularization λ <
(Kobak et al., 2020; Bartlett et al., 2020; Tsigler et al., 2020).
It is unclear, however, if such interpolators, which have implicit small-norm biases, can also be optimal when there is a shift between the training and test distributions.
In particular, we ask: can the optimal causal regularization be 0 or even negative, that is, do we observe benign causal overﬁtting?
To show that the optimal regularization can be negative, we simply show that the derivative of the causal risk at 0 is positive.
We summarize our key ﬁndings in Theorem 4.1.
Theorem 4.1 (Optimal Regularization can be Negative).
For any causal model parameterized as in (1) , the following cases distinguish between whether the min-norm interpolator is optimal or not.
For negative confounding strength ζ < 0 the optimal causal regularization λ ∗ C can be 0 or even negative.
A necessary and sufﬁcient condition for λ ∗ C ≤ 0 depends on the difference in causal and statistical signal-to-noise ratios and is given by
[Equation]
For positive confounding strength ζ > 0 the optimal causal regularization is positive λ ∗ C >and R C
> R Cλ ∗ C , hence regularization is beneﬁcial.
This includes the ICM.
In the highly overparameterized regime ( γ → ∞ ) , the beneﬁt of explicit regularization vanishes and both the causal and statistical risks of the ridge regression estimator converge to their corresponding null risks, independently of the regularization.
We do not refer to this as benign overﬁtting.
However, we can observe benign causal overﬁtting when the causal SNR is larger than the statistical SNR ( ζ < 0 ), which happens when causal and statistical parameter are strongly aligned.
This implies that the norm of the statistical parameter is smaller than the norm of the causal parameter.
Consequentially, statistical regressors are implicitly biased towards solutions of smaller norm and causal learning exhibits self-induced regularization.
Compare this to benign statistical overﬁtting, which happens for certain alignments between the regression parameter ˜ β and the covariance matrix Σ .
In our isotropic setting Σ
= I d , we can therefore never observe benign statistical overﬁtting, but we can observe benign causal overﬁtting.
This phenomenon occurs in both the underparameterized as well as the overparameterized regime.
The range of γ for which the optimal causal regularization is negative increases with the dominance of the causal signal over the statistical signal.
As γ approaches the interpolation threshold, it becomes harder for the optimal causal regularization to be negative.
When the causal SNR is smaller than the statistical SNR ( ζ > 0 ) and in particular under the ICM ( 0 < ζ ≤ 1 ), the optimal causal regularization is strictly positive and the beneﬁt of explicit regularization does not vanish.
This can be the case even when the optimal statistical regularization vanishes.
To see this consider the statistical risk in the highly underparameterized regime γ → 0 .
In this regime, the beneﬁt of explicit regularization vanishes and the min-norm interpolator indeed achieves the optimal statistical risk.
The optimal causal regularization is given explicitly by λ ∗ C =
ζ/ (1 − ζ ) for 0 ≤ ζ ≤ 1 and λ ∗ C = ∞ for ζ > 1 .
This is strictly positive and increasing in the confounding strength ζ , and in fact diverges as ζ approaches 1 (see Theorem 5.2).
On Optimal Regularization
In this section, we investigate two key questions which are natural in the context of our work.
How does the optimal causal regularization λ ∗ C compare to the optimal statistical regularization λ ∗ S ?
What is the dependence of the optimal causal regularization λ ∗ C on the confounding strength ζ ?
Optimal statistical vs. causal regularization When the training and test distributions coincide, approaches such as cross-validation or information criteria (for example AIC or BIC) can be used to estimate the regularization parameter for optimal out-of-sample generalization.
However, choosing the correct regularization parameter for causal learning can be challenging without interventional data.
To understand the optimal causal regularization, it is natural to compare it to the optimal statistical regularization, which can usually be estimated from data.
Interestingly, our analysis reveals that when confounding strength is positive ζ > 0 and in particular under the ICM one needs to regularize more strongly for causal generalization than for statistical generalization.
However, when the confounding strength is negative, that is, when the causal signal dominates the statistical signal, the optimal causal regularization λ ∗ C can actually be smaller than the optimal statistical regularization λ ∗ S .
We formally present this result in Theorem 5.1.
Theorem 5.1 (Optimal Statistical vs. Causal Regularization).
For any causal model parameterized as in (1) , the condition ζ = 0 deﬁnes a phase transition for the optimal regularization via ζ [Equation]
[Equation]
In particular under the ICM, the optimal causal regularization λ ∗ C is always strictly larger than the optimal statistical regularization λ ∗ S , unless ζ = 0 , in which case they coincide.
Dependence on confounding strength ζ
The problem of causal learning from observational data is a distribution shift problem where the distribution of the training data is shifted from that of the test distribution.
As discussed earlier in Proposition 3.2, the confounding strength measure ζ quantiﬁes the strength of this distribution shift.
Therefore, we expect the optimal causal regularization to increase with confounding strength.
Theorem 5.2 indeed conﬁrms this intuition.
Theorem 5.2 (Increasing Confounding Strength Requires Stronger Regularization).
Consider the family of causal models parameterized as in (1) that entail the same observational distribution.
The optimal causal regularization λ ∗ C only depends on the confounding strength ζ and λ ∗
Summary and Extensions
We characterize the role of explicit regularization for causal learning from observational data by computing the asymptotic risk of ridge-regularized regressors and the min-norm interpolator (Theorem 3.1).
Under the principle of independent causal mechanisms (ICM), we ﬁnd that causal learning requires stronger regularization than statistical learning (Theorem 5.1).
A practical implication is that the regularization parameter for causal learning should be chosen larger than what is suggested by cross-validation.
We can precisely state how much larger based on an estimate of confounding strength (Janzing et al., 2017; Janzing et al., 2018).
Beyond ICM, we show that strong alignments between causal and statistical parameters can cause self-induced regularization and lead to benign causal overﬁtting (Theorem 4.1).
One could consider generalizing our assumptions: arbitrary covariances, shifts in the marginal distributions of covariates, soft interventions, more complex hypothesis classes, or non-linear causal relationships.
Since the linear model already exhibits rich behavior, we focus in this paper on understanding the simple setting. Below, we brieﬂy discuss extensions of our analysis to causal learning under soft interventions, non-linearity, and non-Gaussianity.
Soft interventions It is not always appropriate to consider causal learning under hard interventions.
Instead, it is often of interest to consider soft interventions .
In these settings, the qualitative statements derived from our analysis still hold. To illustrate this, we consider the class of shift interventions where the structural dependence of the covariates x is not destroyed as in the case of hard interventions but the observed covariates are merely perturbed (i.e., interventions of the form do ( x := x + ν ) ). Then it turns out that Causal risk soft = Causal risk hard + Statistical risk .
From our results, it then follows that under ICM, λ statistical ≤ λ causalsoft ≤ λ causalhard This also supports our intuition since under soft interventions, we typically aim to achieve a tradeoff between statistical and causal predictability.
We include a complete analysis under shift interventions in Appendix F.
Extensions to non-linear models It is feasible to extend the analysis to structural causal models that arise in a reproducing kernel Hilbert space corresponding to a positive deﬁnite kernel i.e, where the best statistical model ˜ f and the best causal model f are functions in some RKHS).
There are two major technical challenges to deriving the theoretical analysis in such non-linear settings.
Both are beyond what can be done in this paper and are left for future work, but we brieﬂy outline them below.
Extend the deﬁnition of confounding strength ζ beyond the linear setting.
Since such a deﬁnition is non-trivial already in the linear setting, it is challenging to meaningfully generalize this to the non-linear setting.
However, under non-linear causal models in the RKHS, we can naturally extend this deﬁnition by replacing the Euclidean norms with functional norms in the RKHS.
Generalizing the analysis beyond this setting would require further careful consideration.
Derive limiting expressions for causal risk of regularized regressors in a non-linear hypothesis class.
In the case of kernel regression, this would still be feasible via recent random matrix theory results [27]. By optimizing the limiting expressions with respect to the regularization parameter, one can obtain the parameter that achieves the optimal causal risk and subsequently identify the relationship between optimal causal regularization and confounding strength. Beyond Gaussianity The analysis can be extended beyond the Gaussian setting by considering random variables generated by ﬁnite mixtures of Gaussians. Due to the universality phenomenon in the high-dimensional limit, we believe that our limiting expressions (and the qualitative messages derived henceforth) would be rather robust to shifts in the marginal distribution as long as moments of order (4 + δ ) for some δ > 0 are bounded.
We conducted experiments to verify this claim and the corresponding results can be found in Appendix G. They show that for distributions with ﬁnite 4th moments, the ﬁnite-sample risks of the min-norm interpolator and causally optimally regularized ridge regressor closely match the theoretically derived asymptotic risks.
Acknowledgments and Disclosure of Funding
This work has been supported by the German Research Foundation through the Cluster of Excellence “Machine Learning – New Perspectives for Science” (EXC 2064/1:390727645) and the BMBF Checklist For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] While our analysis is exhaustive, we paid special attention to the additional ICM assumption described at the end of Section 2.
Further possible generalizations of our model are discussed in Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Right before Section 3.1, we cautioned against using our results on optimal causal regularization as an excuse to dismiss the hardness of causal learning.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] All proofs are deferred to the appendix and referenced correspondingly in the main paper.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Interpolation and Regularization for Causal Learning
Supplementary Materials
A Proof of Proposition 2.
For the statistical risk, we ﬁrst need one standard result about the distribution of a multivariate normal random variable conditioned on an afﬁne function:
Lemma A.1.
Consider a multivariate normal random variable X ∼ N ( µ, Σ) with mean µ ∈ R d and covariance Σ ∈ R d × d .
Then for any A ∈ R k × d , b ∈ R k , and y ∈ R k
it holds
[Equation]
In particular, if X is a standard normal random variable ( Σ = I d , µ = 0 ) and b = 0 , it is
[Equation]
Proof.
Let Y
=
AX + b .
The joint distribution of X and Y is again a multivariate normal, because it can be written as an afﬁne transformation of X :
[Equation]
which implies that
[Equation]
A T (cid:19) ) .
The claim then follows from the standard formula for conditionals of multivariate normal distributions, which states that if
[Equation]
then
[Equation]
Proposition 2.1 (Causal and Statistical Risk).
For any ˆ β ∈ R d , the risks deﬁned in Eq.
(2) satisfy
[Equation]
Proof.
The key step for this proof is to characterize the distribution of y under the do -intervention y | do ( x ) and the usual observational conditional y | x .
We start with the proof for the causal risk under the do -intervention.
Intervening on x under the causal model given by Eq.
(1) corresponds to removing all arrows to x , which corresponds to the structural equations
[Equation]
In this model, z acts as additional independent noise on y through z T α ∼ N (0 , (cid:107) α (cid:107) 2 ) , which implies that y | do ( x ) ∼ N ( x T β, (cid:107) α (cid:107) 2 + σ 2 ) .
Equivalently, y | do ( x ) has the same distribution as x T β + ε (cid:48) with ε (cid:48) ∼ N (0 , ˜ σ 2 + ω 2 ) because (cid:107) α (cid:107) 2 + σ 2 = ˜
σ 2 + ω 2 .
This lets us compute the causal risk of a linear predictor ˆ β ∈ R d as
[Equation]
which proves the claim for the causal risk.
The proof for the statistical risk is analogous once we have characterized the conditional distribution y | x under the causal model.
Recall that Σ = MM T , Γ = M + T α , and ω 2 = (cid:107) Γ (cid:107) 2Σ .
We ﬁrst observe that x
=
Mz is a linear map of the Gaussian distribution z ∼ N (0 , I l ) , for which Lemma A.1 yields
[Equation]
and therefore z
[Equation]
where the last equality used the identity
[Equation]
Since y = x T β + z T α + ε , it follows that
[Equation]
which concludes the proof.
B Proofs for Section 3.
The bias-variance decomposition of the causal risk is based on the following general lemma:
Lemma B.1 (Bias-Variance Decomposition for General Norm).
Consider a random variable Z on R d , a constant c ∈ R d , and the general norm (cid:107) x (cid:107)
A
= x T Ax for some positive-deﬁnite A ∈ R d × d .
Then we have the decomposition
[Equation]
An alternative form of the variance term is given by E
[Equation]
Proof.
Let E :=
E Z and µ := E Z .
It is
[Equation]
which proves the ﬁrst part of the statement.
For the second part, let Σ Z :=
E ZZ T and denote the
Hadamard product between matrices A, B ∈ R
d × d by ( A (cid:12) B ) i,j =
A i,j B i,j .
It is
[Equation]
ba
[Equation] =
a [Equation] ( Tr [Equation]
B
[Equation]
Tr( B T ) and linearity of trace) = Tr
[Equation]
Cov Z · A ] .
[Equation]
Proposition B.2 (Causal Bias-Variance Decomposition for the Ridge Estimator).
For any λ > 0 , the expectation over the causal risk of the ridge regression estimator ˆ β λ conditioned on X admits the bias-variance decomposition
[Equation]
where B CX (ˆ β λ )
= (cid:107) ( I − (ˆΣ+ λI d )ˆΣ)˜ β − Γ (cid:107) 2Σ and V CX
(ˆ β λ ) =
˜ σ 2 n Tr[ˆΣ(ˆΣ+ λI d ) − 2 Σ] .
The empirical covariance matrix of X is denoted by ˆΣ :=
X T X/n .
Proof.
Recall that R CX (ˆ β λ ) =
E Y | X (cid:13)(cid:13)(cid:13) ˆ β λ − β
(cid:13)(cid:13)(cid:13) 2 Σ .
The ﬁrst part of the statement follows directly from Lemma B.1 with ˆ β λ as a random variable in Y | X and β .
The remainder of the proof consists of computing expectation and covariance of the ridge regression solution ˆ β λ
= ˆ β λ
( X, Y ) under the distribution Y | X .
The samples ( X, Y ) are drawn from the observational distribution of the causal model deﬁned in Eq.
As shown in the proof of Proposition 2.1, the corresponding conditional distribution is y | x ∼ N ( x T ˜ β, ˜ σ 2 ) .
Since ( X, Y ) consist of independent draws, this implies Y | X ∼ N ( X ˜ β, ˜ σ 2 I n ) .
Together with ˆ β λ =
( X T X + nλI ) − 1 X T Y this yields
[Equation]
The characterizations of B CX (ˆ β λ ) and V CX (ˆ β λ ) then simply follow from plugging in expectation and covariance of ˆ β λ :
[Equation]
and, using the alternate form of the variance term from Lemma B.1,
[Equation]
where the last equality used that (cid:16) ˆΣ + λI d (cid:17) − 1 commutes with ˆΣ .
Theorem 2 (Limiting Causal Bias-Variance Decomposition for the Ridge Estimator).
Let (cid:107) β (cid:107) 2 = r 2 , (cid:107) Γ (cid:107) 2 = ω 2 , (cid:104) Γ , β (cid:105) =
η , and σ 2˜ (cid:15) =
˜ σ 2 .
Then as n, d → ∞ such that d/n → γ ∈ (0 , ∞ ) , it holds almost surely in X for every λ > 0 that
[Equation]
where m ( λ ) =
((1 − γ − λ ) − (cid:112) (1 − γ − λ ) 2 − 4 γλ ) / (2 γλ ) and ˜ r 2 = r
+ ω 2 + 2 η .
Therefore
R CX (ˆ β λ )
→ R Cλ := B Cλ + V Cλ + ˜ σ 2 + ω 2 .
The corresponding limiting quantities for the min-norm interpolator can be obtained by taking the limit λ
→ 0 + in equations (7) and (8) , which yields
[Equation]
Therefore R CX (ˆ β 0 )
→ R C
= B C
+ V C 0 + ˜ σ
+ ω 2 .
Proof.
From Proposition B.2, the causal risk R CX (ˆ β λ ) can be decomposed as a sum of the causal bias B CX (ˆ β λ ) , and causal variance V CX (ˆ β λ ) .
In what follows, we derive the limiting expressions for
B CX (ˆ β λ ) and V CX (ˆ β λ ) to obtain the limiting causal risk for any γ ∈ (0 , ∞ ) .
Limiting expressions for causal bias
[Equation]
First, let us consider the sequence of functions given by
[Equation]
Add and
[Equation]
To derive the limiting expression for this sequence, we utilize the “derivative trick”.
This technique has been employed in a similar context in Dobriban et al.
More generally similar terms (although not identical) often also arise in the analysis of the statistical of the ridge regression estimator and therefore one can ﬁnd similar approaches to deriving the limiting expressions for such terms in the statistical analysis for ridge regression (for example, Hastie et al.
Dobriban et al.
(2018), and Dicker (2016)).
Here, we include a self-contained proof of the result.
The idea relies on an application of Vitali’s convergence theorem (see Bai et al. (2010, Lemma 2.14)) to obtain the limit of derivatives of a sequence of functions analytic on some domain D ⊂ C by the derivative of the limit of the sequence of functions.
Observe that
[Equation]
By recognizing the quantity (ˆΣ + λI ) − 1 as the resolvent Q ( − λ ) , we can invoke the MarchenkoPastur Theorem due to Mar ˇ cenko et al.
and Silverstein (1995) which states that the Stieltjes transform of the empirical distribution ˆ m ( z ) of eigenvalues of ˆΣ converges almost surely to the
Stieltjes transform m ( z ) of the empirical spectral distribution given by the Marchenko-Pastur Law F for any z ∈ C / R + . [-
-]2 That is, we have for all λ > 0 ,
[Equation]
Rubio et al. (2011, Theorem 1) provide a generalization of this result which includes providing almost sure convergence of quadratic forms of resolvents of the form u T (ˆΣ − zI ) v for sequences of vectors { u } , { v } such that their outer product uv T has a bounded trace norm for any z ∈ C / R + .
By this result, it is easy to verify that for any λ > 0 ,
[Equation]
It is easy to see that the sequence of functions (cid:110) [Equation] d
[Equation]
λ ) =
Tr
[Equation]
β ˜ β
[Equation]
ˆΣ [Equation] λI )
[Equation]
1 (cid:105)(cid:111) is analytic for λ > 0 .
Furthermore, for any λ > 0 , the absolute value of the sequence of functions { f d ( λ ) } is uniformly bounded in d since
[Equation]
Therefore, by Vitali’s convergence theorem, it holds (almost surely) that for every λ > 0 , the derivatives of the sequence of functions f 1 , f 2 , · · · converges to the derivative of their limit and we have
[Equation]
where m (cid:48) F ( − λ ) denotes the derivative of the Stieltjes transform of the Marchenko-Pastur Law evaluated at − λ .
To obtain the limiting function of the sequence (cid:104) Γ , Π λ ˜ β (cid:105) , observe that
[Equation]
where the limit is obtained by invoking Rubio et al. (2011, Theorem 1).
Therefore, we have that as n, d → ∞ and d/n → γ ,
[Equation]
Limiting expressions for causal variance.
By recalling the expression for variance we have
[Equation]
By Marchenko-Pastur Theorem (Marˇcenko et al., 1967; Silverstein, 1995), we already know that for any λ >[-
-]
[Equation]
Further, recognizing that
[Equation]
and that | Tr[ 1 d (ˆΣ + λI ) − 1 ] | ≤ 1 λ , we can again invoke Vitali’s convergence theorem to obtain the limit of the derivatives by taking the derivative of the limit to obtain
[Equation]
Marchenko-Pastur Law admits an explicit form under our model assumptions (see for example, (Bai et al., 2010, Page 52)) for any z ∈ C
+ (which can be extended by analytic continuity arguments for any z ∈ C / R + ) and is given by
[Equation]
Following arguments similar to Dobriban et al. (2018) and Hastie et al.
for exchanging the limits n, d → ∞ and λ → 0 + , we can derive the limiting expressions for the causal bias and variance of the min-norm estimator.
C Asymptotics for the Statistical Risk
The following theorems describes the limiting expressions for the statistical risk analogue to the causal results from Theorem 3.1.
Theorem C.1 (Limiting Statistical Bias-Variance Decompositions).
Let ˆ β 0 be the min-norm interpolator.
Then as n, d → ∞ such that d/n → γ ∈ (0 , ∞ ) , it holds almost surely in X that
[Equation]
and therefore, R SX (ˆ β 0 )
→ R S
= B S
+ V S 0 + ˜ σ 2 .
For λ > 0 and the corresponding ridge regression estimator ˆ β λ , it holds almost surely in X that
[Equation]
where m ( λ ) =
(1 − γ − λ ) − √ (1 − γ − λ ) 2 − 4 γλ 2 γλ .
Therefore, R SX (ˆ β λ ) → R Sλ = B
Sλ + V Sλ + ˜
σ 2 .
Proof.
As stated in the main paper, this result for the statistical model was already proven in Hastie et al.
D Proof of Proposition 3.
Proposition 3.2 (Causal Risk Increases with Confounding Strength).
Consider the family of causal models parameterized as in (1) that entail the same observational distribution.
Let C {+
+} and Cbe two such causal models with confounding strengths ζ 1 and ζ 2 and alignments η 1 and η 2 (deﬁned in Theorem 3.1), respectively.
Then for all λ, γ ∈ (0 , ∞ ) ,
[Equation]
In particular, for any ﬁxed η , the measure of confounding strength ζ establishes a strict ordering of causal models.
This includes the ICM under which η = 0 .
Proof.
For any ﬁxed λ ∈ (0 , ∞ ) , the difference in limiting causal risks incurred by ˆ β λ on causal models C 1 and C 2 is given by
[Equation]
Since, as shown below, (1 − λm ( − λ )) > 0 for any λ, γ ∈ (0 , ∞ ) , it holds that ζ
[Equation]
> ζ 2 , η
[Equation]
≤ η
[Equation]
⇒ R C 1 ( γ, λ ) > R C 2 ( γ, λ ) .
[Equation]
since γ > 0)
E Proofs for Sections 4 and
We start with a technical lemma that we need in the proofs of the following theorems.
It controls a function that appears in the derivative of the limiting causal riks ∂ λ R Cλ .
Lemma E.1.
For λ ≥ 0 and γ, S > 0 consider the function
[Equation]
This function has the following properties
(i) f is increasing in λ , (ii) f ( λ, γ, S )
−−−−→ λ →∞ 1 , and (iii) f
( λ, γ, S ) −−−→ λ → [Equation] −
[Equation]
( γ −
[Equation]
[Equation]
γ
[Equation]
[Equation]
Proof.
For readability, we use the shorthand notations x = 1 + λ + γ and ϕ = x 2 − 4 γ , under which f is given by
[Equation]
(i)
The partial derivative of f in λ is given by
[Equation]
where the ﬁrst fraction is positive because ϕ > x 2 and x − √ ϕ > 0 .
It is therefore sufﬁcient to show g ( λ ) ≥ 0 for ∂
λ f ( λ, γ, S ) ≥ 0 .
We ﬁrst get rid of the S term via
[Equation]
Finally, we lower bound √ ϕ in two different ways depending on γ .
For γ ≤ 1 , it is ϕ
=
(1 + λ − γ ) 2 + 4 γλ and therefore √ ϕ ≥ 1 + λ − γ = x − 2 γ .
This yields
[Equation]
For γ > 1 , it is ϕ =
( − 1 + λ + γ ) 2 + 4 λ
and therefore √ ϕ ≥ − 1 + λ + γ = x − 2 .
This yields
[Equation]
In summary, we have shown ∂ λ f ( λ, γ, S ) ≥ g ( λ ) ≥ 0 . [Equation] ii)
With the ﬁrst order Taylor approximation 1 − √ 1 − h
= 1 / 2 h + O ( h 2 ) , we get ( x − √ ϕ ) ϕ
= (cid:32)
[Equation]
(cid:114)
[Equation]
(cid:33) [Equation] (cid:19) xϕ = 2 γx + O ( λ − 1 )
=
2 γλ + O (1) , which yields
[Equation]
(iii)
The denominator satisﬁes
[Equation]
Since λ − S − 1 γ −−−→ λ → 0 S − 1 γ < 0 , the claim follows.
Recall that the optimal causal regularization is deﬁned as the minimizer of the causal risk λ ∗ C ( γ ) =
arg inf λ ∈ (0 , ∞ ) R Cλ .
The following lemma distinguishes between three different regimes of the risk function R Cλ depending on the confounding strength ζ .
Lemma E.2 (Regimes of the Optimal Causal Regularization).
For any causal model parameterized as in (1), we can distinguish the following regimes of λ ∗ C ( γ ) :
The function λ (cid:55)→
R
Cλ is increasing (which implies λ ∗ C ( γ ) = 0 ),
if and only if γ (cid:54) = 1 and
[Equation]
For any γ > 0 , the function λ (cid:55)→ R
Cλ is decreasing (which implies λ ∗ C ( γ ) = ∞ )
if and only if ζ ≥ 1 . For any ζ ∈ R , γ ∈ (0 , ∞ ) which do not satisfy the conditions 1. or 2., it is λ ∗ C ( γ ) ∈ (0 , ∞ ) and it λ C ( γ ) satisﬁes the critical point condition
[Equation]
, or equivalently,
[Equation]
where ϕ ( λ ) =
(1 + λ + γ ) 2 − 4 γ .
Proof.
We use the shorthand notation ϕ ( λ ) =
(1 + λ + γ ) 2 − 4 γ .
Recall the confounding strength ζ =
( r 2 + η ) / ˜ r 2 and the statistical signal-to-noise ratio SNR S
=
˜
r 2 / ˜ σ 2 .
The derivative of the limiting causal risk R Cλ in λ is given by
[Equation]
The ﬁrst condition ∂ λ R Cλ ≥ 0 for all λ > 0 can be equivalently rearranged for the confounding strength as
[Equation]
where f is the function investigated in Lemma E.1.
This in turn is equivalent to taking the inﬁmum over λ , which is given by Lemma E.1 as
[Equation]
Note that for γ = 1 this inﬁmum is −∞ , so the condition cannot be satisﬁed for any ζ .
The proof of the second claim is analogue to the ﬁrst with the reverse inequality ∂ λ R Cλ ≤ 0 .
Rearranging for ζ and using Lemma E.1 yields the equivalent condition
[Equation]
For the third claim, assume that the pair of ζ and γ satisﬁes neither of the ﬁrst points.
We will use this to show that the derivative at 0 is negative ∂ λ R Cλ (0) < [-
-]0 and the derivative ∂ λ R Cλ for sufﬁciently large λ is positive.
This together then implies that the minimum λ ∗ C ( γ ) of the function
R Cλ is indeed attained at a ﬁnite value in (0 , ∞ ) , and R Cλ satisﬁes the critical point condition
[Equation]
For the derivative at 0, assume that the converse is true, that is, ∂ λ R Cλ (0) ≥ 0 .
Rearranging this condition for ζ yields similarly to the ﬁrst case of this lemma that ζ ≤ f (0 , γ, SNR S ) .
However
Lemma E.1 states that f is increasing in λ , which means that this condition already implies ζ ≤ f ( λ, γ, SNR S ) for all λ .
This means that the pair ζ, γ would satisfy the condition of the ﬁrst case, which contradicts our assumption.
For the behavior of large λ , observe that the sign of the derivative is determined by the sign of the term λ − SNR [Equation]
+ [Equation] − [Equation]
ϕ [Equation] λ ) [Equation]
ϕ ( λ ) .
As derived in the proof of Lemma E.1, we have the asymptotic behavior
[Equation]
which yields
[Equation]
Since the pair ζ, γ does by assumption not satisfy the conditions of the second case, we have ζ < 1 , which means that the above term is eventually positive.
Theorem 4.1 (Optimal Regularization can be Negative).
For any causal model parameterized as in (1) , the following cases distinguish between whether the min-norm interpolator is optimal or not.
For negative confounding strength ζ < 0 the optimal causal regularization λ ∗ C can be 0 or even negative.
A necessary and sufﬁcient condition for λ ∗ C ≤ 0 depends on the difference in causal and statistical signal-to-noise ratios and is given by
[Equation]
For positive confounding strength ζ > 0 the optimal causal regularization is positive λ ∗ C >and R C {+
+}
> R Cλ ∗ C , hence regularization is beneﬁcial.
This includes the ICM.
Proof.
The ﬁrst statement of the theorem is a special case of Theorem 5.2.
The necessary and sufﬁcient condition for λ ∗ C = 0 stated there is equivalently reformulated as
[Equation]
where the last part used the equality SNR C =
(1 − ζ ) SNR S .
The statement about negative λ ∗ C refers to the fact that the derivative of the risk at 0 can be positive, that is, ∂ R Cλ (0) > 0 .
This was shown in the proof of Lemma E.2 and suggests that without our restriction λ ∗ C ≥ 0 , a negative value of λ would yield an even smaller risk.
For the second statement, observe that the condition ζ > 0 implies the cases 2. or 3. from Lemma E.2.
In particular, this implies λ ∗ C > 0 .
The proof of Lemma E.2 showed that in both of these cases it holds ∂ λ R Cλ (0) < 0 , which means that the causal limiting risk B
Cλ is strictly decreasing in a small neighborhood around 0. In particular, this implies that the minimal risk is strictly smaller than the risk at 0, that is, R
[Equation] Theorem 5.1 (Optimal Statistical vs. Causal Regularization).
For any causal model parameterized as in (1) , the condition ζ = 0 deﬁnes a phase transition for the optimal regularization via
[Equation]
In particular under the ICM, the optimal causal regularization λ ∗ C is always strictly larger than the optimal statistical regularization λ ∗ S , unless ζ = 0 , in which case they coincide.
Proof.
Lemma E.2 distinguishes between three different regimes of ζ .
The ﬁrst two regimes yield
[Equation]
Combined with λ ∗ S = SNR S − 1 γ ∈ (0 , ∞ ) , these regimes agree with the claim in the theorem.
It remains to show that the theorem also holds for the last regime − SNR S − 1 γ max { 1 ,γ } (1 − γ )< ζ < 1 .
In this regime according to Lemma E.2, the optimal causal regularization λ ∗ C satisﬁes the critical point condition
[Equation]
⇔ [Equation] Since the term 1 / (2 γ ) (cid:0) 1 + λ ∗ C + γ − (cid:112) ϕ
( λ ∗ C ) (cid:1) ϕ
( λ ∗ C ) is positive, the sign of λ ∗ C − λ ∗ S is determined by the sign of ζ as claimed in the theorem.
Theorem 5.2 (Increasing Confounding Strength Requires Stronger Regularization).
Consider the family of causal models parameterized as in (1) that entail the same observational distribution.
The optimal causal regularization λ ∗ C only depends on the confounding strength ζ and λ ∗ C is an increasing function in ζ .
More speciﬁcally, using (cid:37) =
− SNR S − 1 γ max { 1 , γ } / (1 − γ ) 2 : (cid:37) <
ζ < 1 = ⇒ λ ∗ C ∈ (0 , ∞ ) with ∂ ζ λ ∗ C > 0 , λ ∗ C = {+
 if ζ ≤ (cid:37) and+} λ ∗ C = ∞ for ζ ≥ 1 .
Proof.
The theorem follows directly from Lemma E.2, except for the statement about λ ∗ C being strictly increasing in ζ .
In the corresponding regime, Lemma E.2 states that λ ∗ C satisﬁes the critical point condition ∂ λ R Cλ ( λ ∗ C ) = 0 , which we will use to show that the derivative of λ ∗ C in ζ is strictly positive.
For readability, we use the notation x ( ζ ) = 1 + λ ∗ C ( ζ ) + γ and ϕ ( ζ ) =
x ( ζ ) 2 − 4 γ .
The optimal causal regularization λ ∗ C ( ζ ) satisﬁes the critical point condition
[Equation]
Rearranging this equation yields
[Equation]
The partial derivatives of the function g = g ( x, ζ ) evaluated at ( x ( ζ ) , ζ ) are given by
[Equation]
and
[Equation]
Since ϕ ( ζ ) =
( x ( ζ ) − 2 √ γ )( x ( ζ ) + 2 √ γ ) <
( x ( ζ ) + 2 √ γ )
[Equation]
it further follows
[Equation]
With these results
[Equation]
we can take the derivative in ζ of the critical point condition
= g ( x ( ζ ) , ζ ) and obtain
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125) <
· 1 , which yields 0 < d x d ζ ( ζ ) =
d λ ∗ C d ζ ( ζ ) .
This implies that λ ∗ C is increasing in ζ and concludes the proof.
F Shift interventions. F.1 Causal risk under relative interventions.
Here, we characterize the causal risk of any linear predictor under relative or shift interventions.
Similar to the deﬁnition of causal risk under hard interventions, to isolate the effects of the choice of α on the risk, we draw perturbations from the marginal of x .
Formally, intervening on x under the causal model given by Eq.
(1) corresponds to the structural equations z
[Equation]
Similar to the proof of Proposition 2.1, the key step here is to characterize the distribution of y under the shift intervention y | do ( x (cid:48) :
= x + ν ) for some ν chosen independently of x .
This lets us compute the risk of a linear predictor ˆ β ∈ R d under a shift intervention as
[Equation]
To obtain the last equality, refer to the derivation of the statistical and causal risks in Proposition 2.1.
The expected risk under conditioning of X is then given by
[Equation]
[Equation]
F.2 Asymptotics and Optimal Ridge Regularization.
The limiting risk of any ridge estimator can then be directly derived from Theorems 3.1 and C.1.
Theorem F.1 (Limiting Causal Risk of the Ridge Estimator Under Shift Interventions).
Let (cid:107) β (cid:107) 2 = r 2 , (cid:107) Γ (cid:107) 2 = ω 2 , (cid:104) Γ , β (cid:105) =
η , and ﬁx ˜ σ 2 .
Then as n, d → ∞ such that d/n → γ ∈ (0 , ∞ ) , it holds almost surely in X for every λ > 0 that
[Equation]
− λm
[Equation]
λ ))
[Equation]
where m ( λ ) =
{+((1 − γ − λ ) − (cid:112) (1 − γ − λ ) 2 − 4 γλ ) / (2 γλ ) and ˜ r 2 = r 
+}
+ ω 2 + 2 η .
The corresponding limiting quantities for the min-norm interpolator can be obtained by taking the limit λ
→ 0 + .
Lemma F.2 (Regimes of the Optimal Causal Regularization Under Shift Interventions).
For any causal model parameterized as in (1), we can distinguish the following regimes of λ ∗ C ( γ ) :
The function λ (cid:55)→ R C soft λ is increasing (which implies λ ∗ C soft ( γ ) = 0 ),
if and only if γ (cid:54) = 1 and ζ
[Equation]
S −
[Equation]
For any γ > 0 , the function λ (cid:55)→ R C soft λ is decreasing (which implies λ ∗ C soft ( γ ) = ∞ )
if and only
if ζ ≥ 2 .
For any ζ ∈ R , γ ∈ (0 , ∞ ) which do not satisfy the conditions 1. or 2., it is λ ∗ C soft ( γ ) ∈ (0 , ∞ ) and it λ ∗ C soft ( γ ) satisﬁes the critical point condition ∂ λ R C soft λ ( λ ∗ C soft ( γ ))
= 0 , or equivalently,
[Equation]
where ϕ ( λ ) =
(1 + λ + γ ) 2 − 4 γ .
Proof.
We use the shorthand notation ϕ ( λ ) =
(1 + λ + γ ) 2 − 4 γ .
Recall the confounding strength ζ =
( r 2 + η ) / ˜ r 2 and the statistical signal-to-noise ratio SNR S
=
˜
r 2 / ˜ σ 2 .
The derivative of the limiting causal risk under shift interventions R C soft λ in λ is given by
[Equation]
The ﬁrst condition ∂ λ R C soft λ ≥ 0 for all λ > 0 can be equivalently rearranged for the confounding strength as
[Equation]
where f is the function investigated in Lemma E.1.
This in turn is equivalent to taking the inﬁmum over λ , which is given by Lemma E.1 as
[Equation]
Note that for γ = 1 this inﬁmum is −∞ , so the condition cannot be satisﬁed for any ζ .
The proof of the second claim is analogue to the ﬁrst with the reverse inequality ∂ λ R C soft λ ≤ 0 .
Rearranging for ζ and using Lemma E.1 yields the equivalent condition
[Equation]
above.
We will use this to show that the derivative at 0 is negative ∂ λ R C soft λ (0) < 0 and the derivative ∂ λ R C soft λ for sufﬁciently large λ is positive.
This together then implies that the minimum λ ∗ C soft ( γ ) of the function R C soft λ is indeed attained at a ﬁnite value in (0 , ∞ ) , and R C soft λ satisﬁes the critical point condition ∂ λ R C soft λ ( λ ∗ C soft ( γ ))
= 0 .
For the derivative at 0, assume that the converse is true, that is, ∂ λ R C soft λ (0) ≥ 0 .
Rearranging this condition for ζ yields similarly to the ﬁrst case of this lemma that 2 ζ ≤ f (0 , γ, SNR S ) .
However
Lemma E.1 states that f is increasing in λ , which means that this condition already implies ζ ≤ 2 f ( λ, γ, SNR S ) for all λ .
This means that the pair ζ, γ would satisfy the condition of the ﬁrst case, which contradicts our assumption.
For the behavior of large λ , observe that the sign of the derivative is determined by the sign of the term
λ − SNR S − 1 γ − ζ 4 γ (cid:16) 1 + λ + γ − (cid:112) ϕ ( λ ) (cid:17) ϕ ( λ ) .
As derived in the proof of Lemma E.1, we have the asymptotic behavior
[Equation]
which yields
[Equation]
Since the pair ζ, γ does by assumption not satisfy the conditions of the second case, we have ζ < 1 , which means that the above term is eventually positive.
Theorem F.3 (Optimal Causal Regularization Under Shift Interventions).
For any causal model parameterized as in (1) ,
If ζ ≥ 0 , then the optimal causal regularization under shift interventions λ ∗ C soft satisﬁes λ ∗ S ≤ λ ∗ C soft ≤ λ ∗ C .
If ζ < 0 , then λ ∗ C ≤ λ ∗ C soft ≤ λ ∗ S .
Indeed, the optimal causal regularization under shift interventions satisﬁes λ ∗ C soft = λ ∗ S
+( λ ∗ C − λ ∗ S ) / 2 .
Proof.
Lemma E.2 distinguishes between three different regimes of ζ .
The ﬁrst two regimes yield
[Equation]
and
[Equation]
Combined with λ ∗ S = SNR S − 1 γ ∈ (0 , ∞ ) , these regimes agree with the claim in the theorem.
It remains to show that the theorem also holds for the last regime − 2 SNR S − 1 γ max { 1 ,γ } (1 − γ )<
ζ < 2 .
In this regime according to Lemma E.2, the optimal causal regularization λ ∗ C satisﬁes the critical point condition
[Equation]
Similarly, we know from the proof of Theorem 5.1 λ ∗ C satisﬁes
[Equation]
Since the term 1 / (2 γ ) (cid:16) 1 + λ ∗ C soft + γ − (cid:113) ϕ
( λ ∗ C soft ) (cid:17) ϕ
( λ ∗ C soft ) is positive, the sign of λ ∗ C soft − λ ∗ S and λ ∗ C − λ ∗ C soft is determined by the sign of ζ as claimed in the theorem.
G Beyond Gaussianity
[Figure]
[Figure]
The analysis of this paper can be extended beyond the Gaussian setting by considering random variables generated by ﬁnite mixtures of Gaussians.
The analysis can get considerably more technical and is left as future work, but we include a brief discussion here.
Due to the Universality phenomenon in the high-dimensional limit, we believe that our limiting expressions (and the qualitative messages derived henceforth) would be rather robust to shifts in the marginal distribution as long as moments of order (4 + δ ) for some δ > 0 are bounded.
We conducted experiments to verify this claim and the corresponding results can be found in Figure 5.
These experiments compare our theoretically derived asymptotic risks with ﬁnite-sample risks of the min-norm interpolator and causally optimally regularized ridge regressor.
{+Instead of Gaussian confounders z ∼ N (0 , I l ) , we only ﬁx the ﬁrst two moments 
+}
and I l and generate z such that E
[ z ] = 0 ,
Cov[ z ] =
I from heavy-tailed multivariate t -distribution with different degrees of freedom, and ﬁnite mixture of Gaussians.
Each plot shows the causal risk of min-norm interpolator and optimally regularized ridge regressor based on ﬁnite samples along with our theoretical asymptotic predictions.
Our experiments show that, for distributions with ﬁnite 4th moments, the ﬁnite-sample risks closely match the theoretical results.