Interpolation and Regularization for Causal Learning
Abstract
Recent work shows that in complex model classes, interpolators can achieve statistical generalization and even be optimal for statistical learning.
However, despite increasing interest in learning models with good causal properties, there is no understanding of whether such interpolators can also achieve causal generalization .
To address this gap, we study causal learning from observational data through the lens of interpolation and its counterpart—regularization.
Under a simple linear causal model, we derive precise asymptotics for the causal risk of the min-norm interpolator and ridge regressors in the high-dimensional regime.
We ﬁnd a large range of behavior that can be precisely characterized by a new measure of confounding strength .
When confounding strength is positive, which holds under independent causal mechanisms—a standard assumption in causal learning—we ﬁnd that interpolators cannot be optimal.
Indeed, causal learning requires stronger regularization than statistical learning.
Beyond this assumption, when confounding is negative, we observe a phenomenon of self-induced regularization due to positive alignment between statistical and causal signals.
Here, causal learning requires weaker regularization than statistical learning, interpolators can be optimal, and optimal regularization can even be negative.
Introduction
We consider the problem of learning the causal inﬂuence of multivariate covariates x ∈ R d on a scalar target variable
y ∈
R purely from observational data and under the presence of hidden confounders.
Formally, given ﬁnite samples { ( x i , y i ) } n i =1 drawn independently and identically (i.i.d) from the joint observational distribution p ( x, y ) = p ( x ) p
( y | x ) , the goal of causal learning is to predict the effects on the target variable y under interventions on the covariates x .
In other words, the goal is to learn a predictive model that minimizes the expected loss on a random draw from the interventional distribution p do ( x, y )
= p ( x ) p ( y | do ( x )) , which can be different from the observational distribution.
Recently, Janzing (2019) established a close analogy between statistical and causal learning (albeit under a highly constructed confounded model).
As a consequence, Janzing (2019) suggested that standard statistical learning-theoretic techniques (such as norm-based regularization) may also help learn good causal models.
However, the classical statistical principles of bias-variance trade-off have been challenged in recent years by highly complex classes of models that are trained to interpolate the data and yet achieve remarkable generalization properties across a broad range of problem domains (Zhang et al., 2021).
A large volume of recent work suggests that interpolation can be compatible with and may even be necessary to achieve optimal statistical generalization in the high-dimensional regime (Belkin et al., 2018; Belkin et al., 2019a; Liang et al., 2020; Feldman, 2020).
Despite the surge in interest, causal properties of such interpolators have not yet been explored.
In this work, we consider a simple linear causal model in the high-dimensional regime ( n, d → ∞ , d/n ∈
O (1) ) and ask: can interpolators achieve good causal generalization?
1.1 Motivation and Related Work
Resemblance between statistical and causal generalization Causal learning can be regarded as an instance of the general problem of learning under distribution shifts, where the training (observational) distribution is shifted from the test (interventional) distribution.
In the framework of out-of-distribution generalization, an interesting proposition for causal learning arises from the following high-level idea.
Observing small sample sizes may induce a similar bias as distribution shifts.
Therefore, techniques for learning models with good out-of-sample generalization (such as regularization) may also help to learn models with good out-of-distribution generalization and vice-versa.
The literature provides plentiful evidence to support this general principle for different classes of distribution shifts.
For instance, under a broad class of distribution shifts, distributionally robust optimization is equivalent to norm-based regularization (Xu et al., 2009; Shaﬁeezadeh Abadeh et al., 2015; Gao et al., 2017; Shaﬁeezadeh-Abadeh et al., 2019; Blanchet et al., 2019; Kuhn et al., 2019).
Analogously, distributionally robust optimization techniques are also employed for statistical learning under limited samples (Zhu et al., 2020).
Particularly relevant to our work is Janzing (2019), which formally establishes a close analogy between “generalizing from empirical to observational distributions ” and “generalizing from observational to interventional distributions ” under a highly constructed confounding model.
This analogy suggests that standard norm-based regularization such as lasso or ridge, typically used for statistical learning, may also help learn better causal models.
Interpolation can be compatible with statistical learning Explicit norm-based regularization techniques were initially motivated by the classical learning theory principle of bias-variance tradeoff, which is characterized by a U-shaped generalization curve.
This principle recommends to avoid interpolation and instead suggests to balance data ﬁtting with the complexity of the hypothesis class.
Recently, however, these classical principles have been challenged by deep learning models.
Despite being highly complex with the ability to ﬁt even random labels and often trained to interpolate the training data, they achieve state-of-the-art out-of-sample generalization across many domains (Zhang et al., 2021).
A partial explanation is provided by the double-descent phenomenon (Belkin et al., 2019b; Belkin, 2021).
Extending the generalization curve beyond the interpolation threshold reveals two regimes: the classical U-curve in the underparameterized regime and a decreasing curve in the overparameterized regime.
This behaviour is not limited to deep neural networks, but extends to other settings such as random feature models and random forests (Belkin et al., 2019b; Hastie et al., 2022; Mei et al., 2021).
Follow-up work suggests that in the overparameterized regime, interpolators can indeed achieve low statistical risk (Belkin et al., 2019a;
Liang et al., 2020; Bartlett et al., 2020;
Tsigler et al., 2020; Muthukumar et al., 2020).
Is interpolation compatible with causal learning?
On account of the parallels between statistical (out-of-sample) and causal (out-of-distribution) learning, it is therefore natural to ask: can interpolators also learn good causal models?
One line of empirical work suggests that naively applying distributionally robust learning techniques such as importance reweighting or distributionally robust optimization approaches (which are equivalent to certain forms of regularization) offers vanishing beneﬁts over empirical risk minimization in overparameterized model classes (Byrd et al., 2019;
Sagawa et al., 2020; Gulrajani et al., 2021).
However, there is also empirical evidence that suggests that augmenting such techniques with additional explicit norm-based regularization may help to learn distributionally robust models in the overparameterized regime (Sagawa et al., 2020; Donhauser et al., 2021).
In the context of causal learning, it has been suggested that explicit regularization can be beneﬁcial and might even need to be stronger than for statistical learning (Janzing, 2019; Vankadara et al., 2021).
Existing work, therefore, remains unclear about the role of explicit regularization in causal learning, or correspondingly, whether interpolation is compatible with causal learning.
In this work, we take a theoretical approach to systematically address these questions.
1.2 Our Contributions
We provide a ﬁrst analysis of causal generalization from observational data in the modern, overparameterized and interpolating regime under a simple linear causal model.
Speciﬁcally, we consider the interpolating minimum l 2 norm least-squares estimator and the family of regularized ridge regression estimators
in
the
proportional asymptotic
regime.
We seek answers to the following
questions
is there a regime where the optimal causal regularization parameter is 0, that is, can we observe benign causal
overﬁtting
Furthermore, if the optimal causal regularization parameter is positive, how strongly do we need to regularize?
How does the optimal causal regularization compare to the optimal statistical regularization
While our analysis is exhaustive, we emphasize the results under the assumption of independent causal mechanisms
(Janzing et al
), a standard assumption in causal learning
• Precise asymptotics of the causal risk (Section 3).
We provide precise asymptotics of the causal risk of the ridge regression estimator as well as the minimum l 2 norm interpolator in the high-dimensional setting: n, d → ∞ , d/n → γ ∈ (0 , ∞ ) .
Our results conﬁrm that, similar to the statistical setting, the causal generalization curve of the min-norm estimator exhibits the double-descent phenomenon.
This is because the variance term diverges at the interpolation threshold and is decreasing in the overparameterized regime ( γ > 1 ).
• A measure of confounding strength ζ
(Section 2.1).
We introduce a new measure of confounding strength ζ that quantiﬁes the relative contribution of the “confounding signal” to the “causal signal”.
It can be interpreted as the strength of the distribution shift between the observational and interventional distributions.
While ζ can take any real value in general, it is restricted to [0 , 1] under the assumption of independent causal mechanisms.
There, it induces a strict, model-independent ordering of all causal models that entail the same observational distribution.
• Benign causal overﬁtting (Section 4).
When the causal signal dominates the statistical signal ( ζ < 0 ), we observe a phenomenon of self-induced regularization due to the confounding signal.
As a consequence, the optimal causal regularization can be 0 or negative even if the optimal statistical regularization is strictly positive.
Under the assumption of independent causal mechanisms, however, we show that there is no benign causal overﬁtting.
This is in contrast to benign statistical overﬁtting, which can occur in the highly underparameterized regime ( γ → 0 ).
• Optimal causal vs. statistical regularization (Section 5).
We show that causal learning requires weaker regularization than statistical learning when the confounding strength ζ is negative.
However, when ζ > 0 and in particular under the principle of independent causal mechanisms, we show that causal learning requires stronger regularization than statistical learning.
More speciﬁcally, the optimal causal regularization is strictly increasing in confounding strength.
Problem Setup
[Equation]
[Figure]
We consider a linear causal model with parameters M ∈ R d × l , α ∈ R l , β ∈ R d with l ≥ d and σ 2 > 0 described via the structural equations
[Equation]
The covariates x ∈ R d and the target y ∈ R are confounded through z ∈ R l , which follows a standard normal distribution.
This structure implies that E x = 0 and the covariance of x is Σ :=
Cov x = MM T .
A graphical representation of this causal model is given in Figure 1a.
The observational joint distribution of this causal model is given by p ( x, y )
= p ( x ) p ( y | x ) , where x ∼ N (0 , Σ) and y | x ∼ N ( x T ˜ β, ˜ σ 2 ) .
Here, the statistical parameter ˜ β := β + Γ consists of the causal parameter β and a confounding parameter Γ :=
M + T α , where M + T is shorthand for ( M + ) T and M + denotes the Moore-Penrose inverse of M .
The statistical noise is given by ˜ σ 2 := σ 2 + (cid:107) α (cid:107) 2 − (cid:107) Γ (cid:107) 2Σ , where (cid:107) x
(cid:107) 2Σ :
= x T Σ x denotes the generalized norm.
Note that the observational distribution alone cannot distinguish the causal model from the one in Figure 1b
The goal of statistical learning is to predict y after observing x , which is captured by the conditional distribution p ( y | x ) .
In contrast, the goal of causal learning is to predict y after manipulating or intervening on x .
This is formally captured by Pearl’s do -calculus (Pearl, 2009), which describes interventions on random variables as a shift in the joint distribution.
Intervening on x with the value x 0 , denoted as do ( x = x 0 ) , removes all arrows to x and sets x = x 0 .
In our causal model (1), the intervention do ( x = x 0 ) removes the arrow z
→ x and yields the updated structural causal equations z ∼ N (0 , I l ) ,
[Equation]
The corresponding distribution of y after intervening on x is therefore given by y | do ( x = x 0 ) ∼
N ( x T 0 β, ˜ σ 2 + (cid:107) Γ (cid:107) 2Σ ) .
Since arbitrary interventions can introduce arbitrary distribution shifts, we consider the natural class of interventions drawn from the observational marginal distribution on x .
This yields the interventional joint distribution p do ( x, y )
= p ( x ) p ( y | do ( x )) with the slight abuse of notation do ( x ) in which the random variable x and its value coincide.
Causal learning from observational data Assume we are given i.i.d.
samples { ( x i , y i ) } ni =1 from the observational joint distribution p ( x, y ) , which we collect in X ∈ R n × d and Y ∈ R n .
The usual statistical learning aims for the observational conditional p ( y | x ) , which means that train and test distributions coincide.
Causal learning aims for the interventional conditional p ( y | do ( x )) , a distribution shift problem for which train and test distributions differ.
We deﬁne the corresponding causal risk R C and statistical risk R S of any linear regressor ˆ β ∈ R d under the squared loss as
R
[Equation]
do
[Equation]
x
[Equation]
x T ˆ
[Equation]
and R S (ˆ β ) :=
E x E y | x ( x T
[Equation]
The following proposition (proven in Appendix A) characterizes the risks under the model (1).
Proposition 2.1 (Causal and Statistical Risk).
For any ˆ β ∈ R d , the risks deﬁned in Eq.
(2) satisfy
[Equation]
Therefore, β is the optimal causal parameter and ˜ β is the optimal statistical parameter.
In the following, we simply refer to them as causal and statistical parameters.
2.1 A New Measure of Confounding Strength
Since the interventional distribution generally differs from the observational distribution, we require a measure that quantiﬁes how this shift inﬂuences causal learning from observational data.
Signal-to-noise ratios (SNRs) Before we deﬁne our measure of confounding strength, we ﬁrst deﬁne the statistical and causal signal-to-noise ratios, which help to intuitively understand our confounding strength measure.
Recall that every causal model entails a statistical model since the causal parameter β and the confounding parameter Γ jointly specify the statistical parameter ˜ β
= β + Γ .
The statistical SNR is deﬁned as usual by SNR S :
= (cid:107) ˜
β (cid:107) 2 / ˜ σ 2 .
For the causal SNR, a natural notion would be (cid:107) β (cid:107) 2 / (˜ σ 2 + (cid:107) Γ (cid:107) 2Σ ) if the learning algorithm had access to data from the interventional distribution y | do ( x ) ∼ N ( x T β, ˜ σ 2 + (cid:107) Γ (cid:107) 2 Σ ) ; but since we are constrained to data from the observational conditional y | x ∼ N ( x T ˜ β, ˜ σ 2 ) , the corresponding causal SNR, which quantiﬁes the hardness of the learning problem, needs to take this into consideration.
Accordingly, we consider the causal SNR as the ratio of the alignment between the statistical and causal parameters and the variance of the observational conditional.
Formally, we deﬁne it as SNR C := (cid:104) β, ˜ β (cid:105) /
˜ σ 2 .
In what follows, we therefore often refer to (cid:104) β, ˜ β (cid:105) as the causal signal and (cid:107)
˜ β (cid:107) 2 as the statistical signal .
Correspondingly, we refer to (cid:104) ˜ β − β, ˜ β (cid:105) =
(cid:104) Γ , ˜ β (cid:105) as the confounding signal , which is the alignment between the confounding parameter Γ and the statistical parameter ˜ β .
Confounding strength Regression on observational data implicitly assumes that the interventional distribution coincides with the observational distribution, while it can be shifted in general.
To quantify the impact of this distribution shift on the corresponding causal risk, we introduce a new confounding strength measure ζ .
It measures the relative contribution of the confounding signal to the statistical signal and is deﬁned by
[Equation]
Other notions of confounding strength are possible, but we will see later that this deﬁnition is well-suited to capture the shift strength for causal learning from observational data.
Without further restrictions, ζ can take any value in R .
This measure divides the causal models into the following three regimes, depending on the relationship between causal and statistical signal:
• ζ ≥ 1 : the causal signal (cid:104) β, ˜ β (cid:105) is non-positive, which implies that causal and statistical parameters are orthogonal or negatively aligned.
Statistical learning is adversarial to causal learning.
• 0 < ζ < 1 : causal and statistical parameters are positively aligned but the causal signal is weaker than the statistical signal (cid:107) ˜
β (cid:107) 2 , for example β = ˜
β/ 2 .
• ζ ≤ 0 : the causal signal dominates the statistical signal, for example β = 2˜ β .
The SNRs are related to the confounding strength measure via SNR C
= (1 − ζ ) SNR S .
In particular, the causal signal decreases as the confounding strength increases.
The regime 0 ≤ ζ ≤ 1 is practically most relevant Causal learning often requires strong assumptions because causal models cannot be uniquely identiﬁed by their observational distribution.
A standard assumption is the principle of independent causal mechanisms (ICM)
(Janzing et al., 2010; Lemeire et al., 2013; Peters et al., 2017), which informally asserts that causal mechanisms share no information.
In our causal model (1), a corresponding assumption could be that the causal mechanisms α and β are drawn from rotationally invariant distributions.
This implies that (cid:104) β, Γ (cid:105) →as
d → ∞ , which in turn falls in the regime 0 ≤ ζ ≤ 1 .
While our following analysis covers all possible causal models, we pay special attention to this regime because it might be of highest practical relevance.
Note that for (cid:104) β, Γ (cid:105) =
0 , our measure of confounding strength coincides with the measure ζ (cid:48)
= (cid:107) Γ (cid:107) 2 / ( (cid:107) Γ (cid:107) 2 + (cid:107) β
(cid:107) 2 ) introduced by Janzing et al.
It measures the relative contribution of causal and confounding signal in terms of lengths rather than inner products.
Causal and Statistical Risk of High-Dimensional Regression Models
Causal learning is extremely challenging, because it requires scarcely available interventional data or has to rely on other information such as exogenous (Rothenhäusler et al., 2021) or instrumental variables (Angrist et al., 1991).
In our setting where only observational data are available, causal learning requires additional model assumptions.
One such approach has been followed by the
Concorr method (Janzing, 2019) which leverages the ICM assumption to make an improved choice of regularization parameter under a linear regression model.
To fully characterize the effect of regularization on causal generalization, we consider two estimators for learning causal models from observational data ( X, Y ) ∈ R n × d × R n : the min-norm interpolator and ridge regressors.
The min-norm interpolator is the minimum l 2 norm solution to the least squares regression problem
[Equation]
A closed form is given by ˆ β 0 ( X, Y ) =
( X T X ) + X T Y .
For λ > 0 , the ridge regressor solves
[Equation]
which has the explicit solution ˆ β λ ( X, Y )
=
( X T X + nλI d ) − 1 X T Y .
The min-norm interpolator can be obtained as a limiting case from the ridge regression solution via ˆ β 0 ( X, Y ) =
lim λ
→ 0 + ˆ β λ ( X, Y ) .
Whenever it is clear from the context, we drop the dependence of the predictors on X and Y .
Before proceeding with the analysis, we motivate the idea that appropriate regularization can help to learn causal models from purely observational data.
To this end, we compare regularization chosen by statistical cross validation to regularization based on an interventional validation set in Figure 2.
Since cross validation implicitly assumes that there is no confounding, it is close to Bayes optimal for ζ = 0 when n (cid:29) d .
However, as confounding increases, it falls behind regularization based on the interventional validation set.
The latter even yields Bayes optimal risk again in the purely confounded setting ζ = 1 , where the lack of causal signal ( β = 0) is encoded by inﬁnite regularization.
While we might not have access to an interventional validation set in practice, our theory will show that knowledge of confounding strength is sufﬁcient for choosing appropriate regularization.
Finally, we want to caution that even though regularization can help, it does not remove the hardness of causal learning.
Reliable causal inference still requires stronger assumptions or additional data.
Confounding strength ζ
C a u s a l e x ce ss r
i s k
||
ˆ
β − β
||
[Figure]
3.1 Precise Asymptotics of the Causal and Statistical Risks
In this section, we provide precise asymptotics for the causal and statistical risks of the min-norm interpolator and ridge regression solutions in the high-dimensional regime.
This regime is characterized by both n, d → ∞ such that d/n → γ ∈ (0 , ∞ ) , where γ is called the overparameterization ratio .
We distinguish between the underparameterized regime ( γ < 1 ) and the overparameterized regime ( γ > 1 ).
All proofs for this section are deferred to Appendix B. Since the predictors
ˆ β
=
ˆ β ( X, Y ) are random variables in the training data X and Y , so is their corresponding causal risk.
We consider the expectation of this risk under Y conditioned on X .
According to Proposition 2.1, it is given by
R CX (ˆ β ) :=
E Y | X R C (ˆ β ) =
E Y | X (cid:107) ˆ β − β (cid:107) 2Σ + ˜ σ 2 + (cid:107) Γ (cid:107) 2Σ .
Due to its simple form, similar to the usual statistical risk, the causal excess risk can be decomposed into bias and variance:
[Equation]
The next theorem is one of our main results.
It gives a closed-form expression for the limiting causal bias and variance of the min-norm interpolator and ridge regression estimators.
We make the simplifying assumption of isotropic covariance Σ
= I d .
The proof relies on recent techniques from random matrix theory.
It employs arguments similar to Dicker (2016), Dobriban et al.
(2018), and
Hastie et al.
and can correspondingly be extended to arbitrary covariances under boundedness assumptions on the spectrum.
We leave such extensions for future work and focus on thoroughly understanding the isotropic causal model, because it already exhibits rather rich behavior.
Theorem 3.1 (Limiting Causal Bias-Variance Decomposition for the Ridge Estimator).
Let (cid:107) β (cid:107) 2 = r 2 , (cid:107) Γ (cid:107) 2 = ω 2 , (cid:104) Γ , β (cid:105) =
η , and ﬁx ˜ σ 2 .
Then as n, d → ∞ such that d/n → γ ∈ (0 , ∞ ) , it holds almost surely in X for every λ > 0 that
[Equation]
where m ( λ ) =
((1 − γ − λ ) − (cid:112) (1 − γ − λ ) 2 − 4 γλ ) / (2 γλ ) and ˜ r 2 = r
+ ω 2 + 2 η .
Therefore
R CX (ˆ β λ ) → R Cλ = B Cλ + V Cλ + ˜ σ 2 + ω 2 .
The corresponding limiting quantities for the min-norm interpolator can be obtained by taking the limit λ
→ 0 + in (7) and (8) .
From these limiting expressions we can see that the causal risk curve of the min-norm interpolator exhibits the double descent phenomenon: it diverges at the interpolation threshold γ = 1 due to the variance term and decreases again for γ > 1 .
A corresponding visualization is given in Figure 4.
Explicit regularization dampens the divergence of the variance term.
While we are primarily interested in the causal risk, the corresponding statistical risk serves as a natural baseline.
An analogue set of results for the statistical risk is given in Appendix C. These results have already been derived by Hastie et al.
and can also be recovered as a special case of our causal results: for ﬁxed statistical parameters ˜ β and ˜ σ 2 , the statistical risk coincides with the causal risk of an unconfounded causal model deﬁned with β = ˜ β , σ 2 = ˜ σ 2 , and α = 0 .
In particular, the corresponding statistical limiting expressions are the same as in Theorem 3.1 after setting η = ω 2 = 0 .
Optimal statistical and causal regularization By directly optimizing the closed form expressions for limiting causal and statistical risks we can ﬁnd the optimal causal and statistical regularization.
Overparameterization ratio γ
C a u s a l r i s k
R C
S =
S = 0 .
S = −
Overparameterization ratio γ
Risk
Bias
Variance
For any γ ∈ (0 , ∞ ) , the optimal statistical regularization λ ∗ S ( γ ) := arg inf λ ∈ (0 , ∞ ) R Sλ can be expressed in closed-form as λ ∗ S ( γ ) =
SNR S − 1 γ .
The closed-form expression for the optimal causal regularization parameter λ ∗ C ( γ ) := arg inf λ ∈ (0 , ∞ )
R Cλ is a root of a 4 th order polynomial and as such considerably intricate.
For readability, we do not include it here.
We investigate the behavior of the optimal causal and statistical regularization in Section 4 and 5.
3.2 Basic Behavior of the Limiting Risk
We start to analyze the results by assessing the basic behavior of the limiting causal risk.
The causal risk of the null estimator ˆ β = 0 serves as a natural baseline to evaluate the performance of the the min-norm interpolator and the ridge regression estimators.
Regimes of the min-norm interpolator Theorem 3.1 characterizes the limiting causal risk of the min-norm interpolator.
Its behavior is controlled by the causal signal-to-noise ratio, which we deﬁned as SNR C = (
1 − ζ ) SNR S .
However, as we will later see, the causal risk of the min-norm interpolator can be lower than null risk when ζ < 0 .
To distinguish the regimes of the min-norm interpolator, it is therefore convenient to consider the closely related quantity S =
(1 − 2 ζ ) SNR S .
It distinguishes between three different regimes (visualized in Figure
• For S > 1 , the causal signal dominates the noise and the min-norm interpolator can perform better than null risk in both under- and overparameterized regime.
• For 0 ≤ S ≤ 1 , the causal signal is weaker than the noise.
Only the underparameterized regime can beat the null risk, whereas the overparameterized regime is always worse.
• The previous two cases resemble the behavior of the statistical risk in the corresponding regimes of the statistical SNR.
Contrary to the statistical risk, however, the causal risk admits a third regime S < 0 .
In this case, the min-norm interpolator always performs worse than null risk.
Here, the causal signal (cid:104) β, ˜ β (cid:105) is dominated by the confounding signal (cid:104) Γ , ˜ β (cid:105) , and interpolating the observational data overﬁts to the confounding
Bias and variance
The bias-variance decomposition of the causal risk given in Theorem 3.1 is visualized in Figure 4 for the min-norm interpolator and the optimally ridge-regularized regressor.
The ﬁgure also shows the causal risk based on ﬁnite samples from the model, which is in high agreement with our asymptotic results.
We compare the causal risk to the corresponding statistical risk.
First note that the causal and statistical variance terms coincide exactly for both the min-norm interpolator and ridge regressors.
This is because the variance term of the squared loss depends only on the variance in the training data, but not on the target parameter β or ˜ β .
Since the training data are the same for both causal and statistical learning, the variance terms trivially coincide.
For the min-norm interpolator, as in the statistical case, the variance term is responsible for the double-descent behavior of the causal risk curve because it explodes at the interpolation threshold γ = 1 and decreases in the overparameterized regime γ > 1 .
In the statistical setting, the bias strictly increases in the overparameterized regime and, as a consequence, the best risk is always achieved in the underparameterized setting.
In contrast, the causal bias of the min-norm interpolator can be decreasing in the overparameterized regime and therefore the optimal causal risk can be achieved in the highly overparameterized regime γ → ∞ .
However, this only happens in the regime S <where the risk of the min-norm interpolator is always worse than null risk.
Figure 4 shows the causal risk of the optimally regularized ridge regression estimator which trivially is always below that of the min-norm risk.
Similar to the statistical setting, the corresponding generalization curve does not exhibit the double descent phenomenon.
There are qualitatively different reasons for why regularization helps in statistical and causal learning.
For both statistical and causal learning, regularization decreases the shared variance, which corresponds to the ﬁnitesample error.
However, while the statistical bias always increases with regularization, the causal bias can actually decrease.
This implies that regularization not only helps with the ﬁnite-sample error, but can also reduce the error due to confounding.
Higher confounding implies higher causal risk for all λ
So far, we have investigated the causal risk under a single causal model.
Now we can compare different causal models using the confounding strength measure ζ introduced in Section 2.1.
The next proposition shows that ζ governs the hardness of causal learning from observational data.
Speciﬁcally, the causal risk of the ridge regression for any λ ∈ (0 , ∞ ) increases as the causal model becomes more confounded.
A proof is given in Appendix D.
Proposition 3.2 (Causal Risk Increases with Confounding Strength).
Consider the family of causal models parameterized as in (1) that entail the same observational distribution.
Let C
and Cbe two such causal models with confounding strengths ζ 1 and ζ 2 and alignments η 1 and η 2 (deﬁned in Theorem 3.1), respectively.
Then for all λ, γ ∈ (0 , ∞ ) ,
[Equation]
In particular, for any ﬁxed η , the measure of confounding strength ζ establishes a strict ordering of causal models.
This includes the ICM under which η = 0 .
Benign Causal Overﬁtting
A large number of recent works suggest that minimum-norm interpolators can be optimal for statistical generalization (Belkin et al., 2018; Belkin et al., 2019a; Muthukumar et al., 2020).
This phenomenon is often referred to as benign overﬁtting.
Moreover, the optimal statistical generalization may even be achieved for negative regularization λ <
(Kobak et al., 2020; Bartlett et al., 2020; Tsigler et al., 2020).
It is unclear, however, if such interpolators, which have implicit small-norm biases, can also be optimal when there is a shift between the training and test distributions.
In particular, we ask: can the optimal causal regularization be 0 or even negative, that is, do we observe benign causal overﬁtting?
To show that the optimal regularization can be negative, we simply show that the derivative of the causal risk at 0 is positive.
We summarize our key ﬁndings in Theorem 4.1.
Theorem 4.1 (Optimal Regularization can be Negative).
For any causal model parameterized as in (1) , the following cases distinguish between whether the min-norm interpolator is optimal or not.
For negative confounding strength ζ < 0 the optimal causal regularization λ ∗ C can be 0 or even negative.
A necessary and sufﬁcient condition for λ ∗ C ≤ 0 depends on the difference in causal and statistical signal-to-noise ratios and is given by
[Equation]
For positive confounding strength ζ > 0 the optimal causal regularization is positive λ ∗ C >and R C
> R Cλ ∗ C , hence regularization is beneﬁcial.
This includes the ICM.
In the highly overparameterized regime ( γ → ∞ ) , the beneﬁt of explicit regularization vanishes and both the causal and statistical risks of the ridge regression estimator converge to their corresponding null risks, independently of the regularization.
We do not refer to this as benign overﬁtting.
However, we can observe benign causal overﬁtting when the causal SNR is larger than the statistical SNR ( ζ < 0 ), which happens when causal and statistical parameter are strongly aligned.
This implies that the norm of the statistical parameter is smaller than the norm of the causal parameter.
Consequentially, statistical regressors are implicitly biased towards solutions of smaller norm and causal learning exhibits self-induced regularization.
Compare this to benign statistical overﬁtting, which happens for certain alignments between the regression parameter ˜ β and the covariance matrix Σ .
In our isotropic setting Σ
= I d , we can therefore never observe benign statistical overﬁtting, but we can observe benign causal overﬁtting.
This phenomenon occurs in both the underparameterized as well as the overparameterized regime.
The range of γ for which the optimal causal regularization is negative increases with the dominance of the causal signal over the statistical signal.
As γ approaches the interpolation threshold, it becomes harder for the optimal causal regularization to be negative.
When the causal SNR is smaller than the statistical SNR ( ζ > 0 ) and in particular under the ICM ( 0 < ζ ≤ 1 ), the optimal causal regularization is strictly positive and the beneﬁt of explicit regularization does not vanish.
This can be the case even when the optimal statistical regularization vanishes.
To see this consider the statistical risk in the highly underparameterized regime γ → 0 .
In this regime, the beneﬁt of explicit regularization vanishes and the min-norm interpolator indeed achieves the optimal statistical risk.
The optimal causal regularization is given explicitly by λ ∗ C =
ζ/ (1 − ζ ) for 0 ≤ ζ ≤ 1 and λ ∗ C = ∞ for ζ > 1 .
This is strictly positive and increasing in the confounding strength ζ , and in fact diverges as ζ approaches 1 (see Theorem 5.2).
On Optimal Regularization
In this section, we investigate two key questions which are natural in the context of our work.
How does the optimal causal regularization λ ∗ C compare to the optimal statistical regularization λ ∗ S ?
What is the dependence of the optimal causal regularization λ ∗ C on the confounding strength ζ ?
Optimal statistical vs. causal regularization When the training and test distributions coincide, approaches such as cross-validation or information criteria (for example AIC or BIC) can be used to estimate the regularization parameter for optimal out-of-sample generalization.
However, choosing the correct regularization parameter for causal learning can be challenging without interventional data.
To understand the optimal causal regularization, it is natural to compare it to the optimal statistical regularization, which can usually be estimated from data.
Interestingly, our analysis reveals that when confounding strength is positive ζ > 0 and in particular under the ICM one needs to regularize more strongly for causal generalization than for statistical generalization.
However, when the confounding strength is negative, that is, when the causal signal dominates the statistical signal, the optimal causal regularization λ ∗ C can actually be smaller than the optimal statistical regularization λ ∗ S .
We formally present this result in Theorem 5.1.
Theorem 5.1 (Optimal Statistical vs. Causal Regularization).
For any causal model parameterized as in (1) , the condition ζ = 0 deﬁnes a phase transition for the optimal regularization via ζ
[Equation] [Equation]
In particular under the ICM, the optimal causal regularization λ ∗ C is always strictly larger than the optimal statistical regularization λ ∗ S , unless ζ = 0 , in which case they coincide.
Dependence on confounding strength ζ
The problem of causal learning from observational data is a distribution shift problem where the distribution of the training data is shifted from that of the test distribution.
As discussed earlier in Proposition 3.2, the confounding strength measure ζ quantiﬁes the strength of this distribution shift.
Therefore, we expect the optimal causal regularization to increase with confounding strength.
Theorem 5.2 indeed conﬁrms this intuition.
Theorem 5.2 (Increasing Confounding Strength Requires Stronger Regularization).
Consider the family of causal models parameterized as in (1) that entail the same observational distribution.
The optimal causal regularization λ ∗ C only depends on the confounding strength ζ and λ ∗
Summary and Extensions
We characterize the role of explicit regularization for causal learning from observational data by computing the asymptotic risk of ridge-regularized regressors and the min-norm interpolator (Theorem 3.1).
Under the principle of independent causal mechanisms (ICM), we ﬁnd that causal learning requires stronger regularization than statistical learning (Theorem 5.1).
A practical implication is that the regularization parameter for causal learning should be chosen larger than what is suggested by cross-validation.
We can precisely state how much larger based on an estimate of confounding strength (Janzing et al., 2017; Janzing et al., 2018).
Beyond ICM, we show that strong alignments between causal and statistical parameters can cause self-induced regularization and lead to benign causal overﬁtting (Theorem 4.1).
One could consider generalizing our assumptions: arbitrary covariances, shifts in the marginal distributions of covariates, soft interventions, more complex hypothesis classes, or non-linear causal relationships.
Since the linear model already exhibits rich behavior, we focus in this paper on understanding the simple setting.
Below, we brieﬂy discuss extensions of our analysis to causal learning under soft interventions, non-linearity, and non-Gaussianity.
Soft interventions It is not always appropriate to consider causal learning under hard interventions.
Instead, it is often of interest to consider soft interventions .
In these settings, the qualitative statements derived from our analysis still hold.
To illustrate this, we consider the class of shift interventions where the structural dependence of the covariates x is not destroyed as in the case of hard interventions but the observed covariates are merely perturbed (i.e., interventions of the form do ( x := x + ν ) ).
Then it turns out that Causal risk soft =
Causal risk hard + Statistical risk .
From our results, it then follows that under ICM, λ statistical ≤ λ causalsoft ≤ λ causalhard This also supports our intuition since under soft interventions, we typically aim to achieve a tradeoff between statistical and causal predictability.
We include a complete analysis under shift interventions in Appendix F.
Extensions to non-linear models It is feasible to extend the analysis to structural causal models that arise in a reproducing kernel Hilbert space corresponding to a positive deﬁnite kernel i.e, where the best statistical model ˜ f and the best causal model f are functions in some RKHS).
There are two major technical challenges to deriving the theoretical analysis in such non-linear settings.
Both are beyond what can be done in this paper and are left for future work, but we brieﬂy outline them below.
Extend the deﬁnition of confounding strength ζ beyond the linear setting.
Since such a deﬁnition is non-trivial already in the linear setting, it is challenging to meaningfully generalize this to the non-linear setting.
However, under non-linear causal models in the RKHS, we can naturally extend this deﬁnition by replacing the Euclidean norms with functional norms in the RKHS.
Generalizing the analysis beyond this setting would require further careful consideration.
Derive limiting expressions for causal risk of regularized regressors in a non-linear hypothesis class.
In the case of kernel regression, this would still be feasible via recent random matrix theory results [27].
By optimizing the limiting expressions with respect to the regularization parameter, one can obtain the parameter that achieves the optimal causal risk and subsequently identify the relationship between optimal causal regularization and confounding strength.
Beyond Gaussianity The analysis can be extended beyond the Gaussian setting by considering random variables generated by ﬁnite mixtures of Gaussians.
Due to the universality phenomenon in the high-dimensional limit, we believe that our limiting expressions (and the qualitative messages derived henceforth) would be rather robust to shifts in the marginal distribution as long as moments of order (4 + δ ) for some δ > 0 are bounded.
We conducted experiments to verify this claim and the corresponding results can be found in Appendix G.
They show that for distributions with ﬁnite 4th moments, the ﬁnite-sample risks of the min-norm interpolator and causally optimally regularized ridge regressor closely match the theoretically derived asymptotic risks.
Acknowledgments and Disclosure of Funding
This work has been supported by the German Research Foundation through the Cluster of Excellence “Machine Learning – New Perspectives for Science” (EXC 2064/1:390727645) and the BMBF
[-Consider a random variable Z on R d , a constant c ∈ R d , and the general norm (cid:107) x (cid:107) 
-]
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] While our analysis is exhaustive, we paid special attention to the additional ICM assumption described at the end of Section 2.
Further possible generalizations of our model are discussed in Section 6.
[-((1 − γ − λ ) − (cid:112) (1 − γ − λ ) 2 − 4 γλ ) / (2 γλ ) and ˜ r 2 = r 
-]
(c) Did you discuss any potential negative societal impacts of your work?
[-→ R C 
-]
[-= B C 
-]
[-+ V C 0 + ˜ σ 
-]
[Yes] Right before Section 3.1, we cautioned against using our results on optimal causal regularization as an excuse to dismiss the hardness of causal learning.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[-→ R S 
-]
[-= B S 
-]
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] All proofs are deferred to the appendix and referenced correspondingly in the main paper.
If you ran experiments... (a) Did you include the [-same observational distribution. Let C 
 and Cbe two such causal models with confounding strengths ζ 1-]code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
[-For positive confounding strength ζ > 0 the optimal causal regularization is positive λ ∗ C >and R C 
-]
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[-ζ < 1 = ⇒ λ ∗ C ∈ (0 , ∞ ) with ∂ ζ λ ∗ C > 0 , λ ∗ C = 
-]
[N/A] 5.
[-we can take the derivative in ζ of the critical point condition 
-]
[-where m ( λ ) = ((1 − γ − λ ) − (cid:112) (1 − γ − λ ) 2 − 4 γλ ) / (2 γλ ) and ˜ r 2 = r 
 + ω 2 + 2 η . The corresponding limiting quantities for the min-norm interpolator can be obtained by taking-]If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[-Instead of Gaussian confounders z ∼ N (0 , I l ) , we only ﬁx the ﬁrst two moments 
-]