DeepMed: Semiparametric Causal Mediation Analysis with Debiased Deep Learning
Abstract
Causal mediation analysis can unpack the black box of causality and is therefore a powerful tool for disentangling causal pathways in biomedical and social sciences, and also for evaluating machine learning fairness.
To reduce bias for estimating
Natural Direct and Indirect Effects in mediation analysis, we propose a new method called DeepMed that uses deep neural networks (DNNs) to cross-fit the infinitedimensional nuisance functions in the efficient influence functions.
We obtain novel theoretical results that our DeepMed method (1) can achieve semiparametric efficiency bound without imposing sparsity constraints on the DNN architecture and (2) can adapt to certain low-dimensional structures of the nuisance functions, significantly advancing the existing literature on DNN-based semiparametric causal inference.
Extensive synthetic experiments are conducted to support our findings and also expose the gap between theory and practice.
As a proof of concept, we apply DeepMed to analyze two real datasets on machine learning fairness and reach conclusions consistent with previous findings.
Introduction
Tremendous progress has been made in this decade on deploying deep neural networks (DNNs) in real-world data science problems (Krizhevsky et al., 2012;
Wolf et al., 2019; Jumper et al., 2021).
Causal inference is no exception.
In semiparametric causal inference, several important works (Chen et al., 2020; Chernozhukov et al., 2020; Farrell et al., 2021) initiated the investigation of statistical properties of causal effect estimators when the nuisance functions (the outcome regressions and propensity scores) are estimated by DNNs.
However, there are a few limitations in the current literature that need to be addressed before the theoretical results can be used to guide practice: (1) Most recent works focus on total (treatment) effect (Chen et al., 2020; Farrell et al., 2021).
In many settings, however, more intricate causal parameters are often of greater interests.
In biomedical and social sciences, one is often interested in “mediation analysis” to decompose the total effect into direct and indirect effect to unpack the underlying black-box causal mechanism (Baron and Kenny, 1986).
More recently, mediation analysis also percolated into machine learning fairness.
For instance, in the context of predicting the recidivism risk, Nabi and Shpitser (2018) argued that, for a “fair” algorithm, sensitive features such as race should have no direct effect on the predicted recidivism risk.
If such direct effects can be accurately estimated, one can detect the potential unfairness of a machine learning algorithm.
We will revisit such applications in Section 5 and Appendix G. (2) Statistical properties of DNN-based causal estimators in recent works mostly follow from several (recent) results on the convergence rates of DNN-based nonparametric regression estimators (Suzuki, 2019; Schmidt-Hieber, 2020; Tsuji and Suzuki, 2021), with the limitation of relying on sparse DNN architectures.
The theoretical properties are in turn evaluated by relatively simple synthetic experiments not designed to generate nearly infinite-dimensional nuisance functions, a setting considered by almost all the above related works.
The above limitations raise the tantalizing question whether the available statistical guarantees for
DNN-based causal inference have practical relevance.
In this work, we plan to partially fill these gaps by developing a new method called DeepMed for semiparametric mediation analysis with DNNs.
We focus on the Natural Direct/Indirect Effects (NDE/NIE) (Robins and Greenland, 1992; Pearl, 2001) (defined in Section 2.1), but our results can also be applied to more general settings; see Remark 2.
The DeepMed estimators leverage the “multiply-robust” property of the efficient influence function (EIF) of NDE/NIE (Tchetgen Tchetgen and Shpitser, 2012; Farbmacher et al., 2022)
(see Proposition 1 in Section 2.2), together with the flexibility and superior predictive power of DNNs (see Section 3.1 and Algorithm 1).
In particular, we also make the following novel contributions to deepen our understanding of DNN-based semiparametric causal inference:
• On the theoretical side, we obtain novel results that our DeepMed method can achieve semiparametric efficiency bound without imposing sparsity constraints on the DNN architecture and can adapt to certain low-dimensional structures of the nuisance functions (see Section 3.2), thus significantly advancing the existing literature on DNN-based semiparametric causal inference.
Non-sparse DNN architecture is more commonly employed in practice (Farrell et al., 2021), and the low-dimensional structures of nuisance functions can help avoid curse-of-dimensionality.
These two points, taken together, significantly advance our understanding of the statistical guarantee of DNN-based causal inference.
• More importantly, on the empirical side, in Section 4, we designed sophisticated synthetic experiments to simulate nearly infinite-dimensional functions, which are much more complex than those in previous related works (Chen et al., 2020; Farrell et al., 2021; Adcock and Dexter, 2021).
We emphasize that these nontrivial experiments could be of independent interest to the theory of deep learning beyond causal inference, to further expose the gap between deep learning theory and practice (Adcock and Dexter, 2021; Gottschling et al., 2020); see Remark 9 for an extended discussion.
As a proof of concept, in Section 5 and Appendix G, we also apply DeepMed to re-analyze two real-world datasets on algorithmic fairness and reach similar conclusions to related works.
Finally, a user-friendly R package can be found at https://github.com/siqixu/DeepMed.
Making such resources available helps enhance reproducibility, a highly recognized problem in all scientific disciplines, including (causal) machine learning (Pineau et al., 2021; Kaddour et al., 2022).
Definition, identification, and estimation of NDE and
NIE 2.1 Definition of NDE and NIE
Throughout this paper, we denote Y as the primary outcome of interest, D as a binary treatment variable, M as the mediator on the causal pathway from D to Y , and X ∈ [0 , 1] p (or more generally, compactly supported in R p ) as baseline covariates including all potential confounders.
We denote the observed data vector as O ≡ ( X, D, M, Y ) .
Let M ( d ) denote the potential outcome for the mediator when setting D
= d and Y ( d, m ) be the potential outcome of Y under D
= d and M = m , where d ∈ { 0 , 1 } and m is in the support M of M .
We define the average total (treatment) effect as τ tot :=
E [ Y (1 , M (1)) − Y (0 , M (0))] , the average NDE of the treatment D on the outcome Y when the mediator takes the natural potential outcome when D = d as τ NDE ( d ) :
=
E [ Y (1 , M ( d )) −
Y (0 , M ( d ))] , and the average NIE of the treatment D on the outcome Y via the mediator M as τ NIE ( d ) :=
E [ Y ( d, M (1)) − Y ( d, M (0))] .
We have the trivial decomposition τ tot ≡ τ NDE ( d ) + τ NIE ( d ′ ) for d ̸ = d ′ .
In causal mediation analysis, the parameters of interest are τ NDE ( d ) and τ NIE ( d ) .
2.2 Semiparametric multiply-robust estimators of NDE/NIE
Estimating τ NDE ( d ) and τ NIE ( d ) can be reduced to estimating ϕ ( d, d ′ ) :
= E [ Y ( d, M ( d ′ ))] for d, d ′ ∈ { 0 , 1 } .
We make the following standard identification assumptions:
i. Consistency : if D = d , then M = M ( d ) for all d ∈ { 0 , 1 } ; while if D = d and M = m , then Y =
Y ( d, m ) for all d ∈ { 0 , 1 } and all m in the support of M . ii.
Ignorability : Y ( d, m ) ⊥ D | X , Y ( d, m ) ⊥ M | X, D , M ( d ) ⊥ D | X , and Y ( d, m ) ⊥ M
( d ′ ) | X , almost surely for all d, ∈ { 0 , 1 } and all m ∈ M .
The first three conditions are, respectively, no unmeasured treatment-outcome, mediator-outcome and treatment-mediator confounding, whereas the fourth condition is often referred to as the “cross-world” condition.
We provide more detailed comments on these four conditions in Appendix A.
iii.
Positivity : The propensity score a ( d | X ) ≡ Pr ( D = d | X ) ∈ ( c, C ) for some constants 0 < c ≤ C < 1 , almost surely for all d ∈ { 0 , 1 } ; f ( m | X, d ) , the conditional density (mass) function of M = m (when M is discrete) given X and D = d , is strictly bounded between [¯ ρ, ¯ ρ ] for some constants 0 < ¯ ρ ≤ ¯ ρ < ∞ almost surely for all m in M and all d ∈ { 0 , 1 } .
Under the above assumptions, the causal parameter ϕ
( d, d ′ ) for d, d ′ ∈ { 0 , 1 } can be identified as either of the following three observed-data functionals:
[Equation]
where 1 {·} denotes the indicator function, p ( x ) denotes the marginal density of X , and µ ( x, d, m ) :
=
E [ Y | X = x, D = d, M = m ] is the outcome regression model, for which we also make the following standard boundedness assumption: iv.
µ ( x, d, m ) is also strictly bounded between [ − R, R ] for some constant R > 0 .
Following the convention in the semiparametric causal inference literature, we call a, f, µ “nuisance functions”.
Tchetgen Tchetgen and Shpitser (2012) derived the EIF of ϕ ( d, d ′ ) :
EIF d,d ′ ≡ ψ d,d ′ ( O ) − ϕ ( d, d ′ ) , where
[Equation]
The nuisance functions µ ( x, d, m ) , a ( d | x ) and f ( m | x, d ) appeared in ψ d,d ′ ( o ) are unknown and generally high-dimensional.
But with a sample D ≡ { O j } Nj =1 of the observed data, based on ψ d,d ′ ( o ) , one can construct the following generic sample-splitting multiply-robust estimator of ϕ ( d, d ′ ) :
[Equation]
where D n ≡ { O i } ni =1 is a subset of all N data, and (cid:101) ψ d,d ′ ( o ) replaces the unknown nuisance functions a, f, µ in ψ d,d ′ ( o ) by some generic estimators (cid:101) a, (cid:101) f, (cid:101) µ computed using the remaining N − n nuisance sample data , denoted as D ν .
Cross-fit is then needed to recover the information lost due to sample splitting; see Algorithm 1.
It is clear from (2) that (cid:101) ϕ ( d, d ′ ) is a consistent estimator of ϕ ( d, d ′ ) as long as any two of (cid:101) a, (cid:101) f, (cid:101) µ are consistent estimators of the corresponding true nuisance functions, hence the name “multiply-robust”.
Throughout this paper, we take n ≍ N − n and assume: v. Any nuisance function estimators are strictly bounded within the respective lower and upper bounds of a, f, µ .
To further ease notation, we define: for any d ∈ { 0 , 1 } , r a,d := (cid:8)(cid:82) δ
a,d ( x ) 2 d F ( x ) (cid:9) 1 / 2 , r f,d := (cid:8)(cid:82) δ f,d ( x, m ) 2 d F ( x, m | d = 0) (cid:9) 1 / 2 , and r µ,d := (cid:8)(cid:82) δ µ,d ( x, m )
2 d F ( x, m | d = 0) (cid:9) 1 / 2 , where δ a,d ( x ) :
= (cid:101) a ( d | x ) − a ( d | x ) , δ f,d ( x, m ) :
= (cid:101) f
( m | x, d ) − f ( m | x, d ) and δ µ,d ( x, m ) :
= (cid:101) µ ( x, d, m ) − µ ( x, d, m ) are point-wise estimation errors of the estimated nuisance functions.
In defining the above
L 2 -estimation errors, we choose to take expectation with respect to (w.r.t.)
the law F ( m, x | d = 0) only for convenience, with no loss of generality by Assumptions iii and v.
To show the cross-fit version of (cid:101) ϕ ( d, d ′ ) is semiparametric efficient for ϕ ( d, d ′ ) , we shall demonstrate under what conditions √ n ( (cid:101) ϕ ( d, d ′ )
− ϕ
( d, d ′ )) L → N (0 , E [ EIF 2 d,d ′ ]) (Newey, 1990).
The following proposition on the statistical properties of (cid:101) ϕ ( d, d ′ ) is a key step towards this objective.
Proposition 1.
Denote Bias( (cid:101) ϕ ( d, d ′ )) :
= E [ (cid:101) ϕ ( d, d ′ ) − ϕ
( d, d ′ ) |D ν ] as the bias of (cid:101) ϕ ( d, d ′ ) conditional on the nuisance sample D ν .
Under Assumptions i – v, Bias( (cid:101) ϕ ( d, d ′ )) is of second-order:
[Equation]
Furthermore, if the RHS of (4) is o ( n − 1 / 2 ) , then
[Equation]
Although the above result is a direct consequence of the EIF ψ d,d ′ ( O ) , we prove Proposition 1 in
Appendix B for completeness.
Remark 2.
The total effect τ tot = ϕ (1 , 1) − ϕ (0 , 0) can be viewed as a special case, for which d = d ′ for ϕ ( d, d ′ ) .
Then EIF d,d ≡ EIF d corresponds to the nonparametric EIF of ϕ ( d, d ) ≡ ϕ ( d ) ≡
E [ Y ( d, M ( d ))] :
EIF d = ψ d ( O ) − ϕ
( d ) with ψ d ( O ) =
1 { D = d } a ( d | X ) Y + (cid:18) 1 − 1 { D = d } a ( d | X ) (cid:19) µ ( X, d ) , where µ ( x, d ) :=
E [ Y | X = x, D = d ] .
Hence all the theoretical results in this paper are applicable to total effect estimation. Our framework can also be applied to all the statistical functionals that satisfy a so-called “mixed-bias” property, characterized recently in Rotnitzky et al.
This class includes the quadratic functional, which is important for uncertainty quantification in machine learning.
Estimation and inference of NDE/NIE using DeepMed
We now introduce DeepMed , a method for mediation analysis with nuisance functions estimated by
DNNs.
By leveraging the second-order bias property of the multiply-robust estimators of NDE/NIE (Proposition 1), we will derive statistical properties of DeepMed in this section.
The nuisance function estimators by DNNs are denoted as (cid:98) a, (cid:98) f, (cid:98) µ .
3.1 Details on DeepMed
First, we introduce the fully-connected feed-forward neural network with the rectified linear units (ReLU) as the activation function for the hidden layer neurons (FNN-ReLU), which will be used to estimate the nuisance functions.
Then, we will introduce an estimation procedure using a V -fold cross-fitting with sample-splitting to avoid the Donsker-type empirical-process assumption on the nuisance functions, which, in general, is violated in high-dimensional setup.
Finally, we provide the asymptotic statistical properties of the DNN-based estimators of τ tot , τ NDE ( d ) and τ NIE ( d ) .
We denote the ReLU activation function as σ ( u ) :
=
max( u, 0) for any u ∈ R .
Given vectors x, b , we denote σ b ( x ) :
= σ ( x − b ) , with σ acting on the vector x − b component-wise.
Let F nn denote the class of the FNN-ReLU functions
[Equation]
where ◦ is the composition operator, L is the number of layers (i.e. depth) of the network, and for l = 1 , · · · , L , W ( l ) is a K l +1 × K l -dimensional weight matrix with K l being the number of neurons in the l -th layer (i.e. width) of the network, with K 1 = p and K L +1 = 1 , and b ( l ) is a
K l -dimensional vector.
To avoid notation clutter, we concatenate all the network parameters as Θ = ( W ( l ) , b ( l ) , l = 1 , · · · , L ) and simply take K 2 = · · · = K L = K .
We also assume Θ to be bounded: ∥ Θ ∥ ∞ ≤ B for some universal constant B > 0 .
We may let the dependence on L , K , B explicit by writing F nn as F nn ( L, K, B ) .
τ tot , τ NDE ( d ) , τ NIE ( d ) by (3), with the nuisance functions a, f, µ estimated using
F nn with the V -fold cross-fitting strategy, summarized in Algorithm 1 below; also see Farbmacher et al.
DeepMed inputs the observed data D ≡ { O i } Ni =1 and outputs the estimated total effect (cid:98) τ tot , NDE (cid:98) τ NDE ( d ) and NIE (cid:98) τ NIE ( d ) , together with their variance estimators (cid:98) σ 2 tot , (cid:98) σ 2 NDE ( d ) and (cid:98) σ 2 NIE ( d ) .
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
ψ
[Table]
− [Table]
ψ
[Table] − [Table]
N
[Table]
τ
[Table] [Table]
σ
[Table]
d
[Table] = [Table]
[-d, 
-]ψ
[Table] [Table] − [Table]
[Table]
[Table]
=
[Table]
[Table]
ψ [Table] [Table] 2 − 1 N (cid:98) τ
[Table]
( d ) Output: (cid:98) τ tot , (cid:98) τ NDE ( d ) , (cid:98) τ NIE ( d ) , (cid:98) σ 2 tot , (cid:98) σ 2 NDE ( d ) and (cid:98) σ 2 NIE ( d )
Remark 3 (Continuous or multi-dimensional mediators) .
For binary treatment D and continuous or multi-dimensional M , to avoid nonparametric/high-dimensional conditional density estimation, we can rewrite f ( m | x,d ′ ) a ( d | x ) f
( m | x,d ) as 1 − a ( d | x,m ) a ( d | x,m )(1 − a ( d | x )) by the Bayes’ rule and the integral w.r.t.
f ( m | x, d ′ ) in (2) as E [ µ ( X, d, M ) | X
= x, D = d ′ ] .
Then we can first estimate µ ( x, d, m ) by (cid:98) µ ( x, d, m ) and in turn estimate E [ µ ( X, d, M ) | X
= x, D = d ′ ] by regressing (cid:98) µ ( X, d, M ) against ( X, D ) using the FNN-ReLU class.
We mainly consider binary M to avoid unnecessary complications; but see
Appendix G for an example in which this strategy is used.
Finally, the potential incompatibility between models posited for a ( d | x ) and a ( d | x, m ) and the joint distribution of ( X, A, M, Y ) is not of great concern under the semiparametric framework because all nuisance functions are estimated nonparametrically; again, see Appendix G for an extended discussion.
3.2 Statistical properties of DeepMed: Non-sparse DNN architecture and low-dimensional structures of the nuisance functions
According to Proposition 1, to analyze the statistical properties DeepMed , it is sufficient to control the L 2 -estimation errors of nuisance function estimates (cid:98) a, (cid:98) f, (cid:98) µ fit by DNNs.
To ease presentation, we first study the theoretical guarantees on the L 2 -estimation error for a generic nuisance function g : W ∈ [0 , 1] p → Z ∈ R , for which we assume:
vi.
Z = g ( W ) + ξ , with ξ sub-Gaussian with mean zero and independent of W
Note that when g corresponds to a, f, µ , ( W, Z ) corresponds to ( X, 1 ( D = 1)) , (( X, D ) , 1 ( M = 1)) and (( X, D, M ) , Y ) , respectively.
We denote the DNN output from the nuisance sample D ν as (cid:98) g .
For theoretical results, we consider (cid:98) g as the following empirical risk minimizer (ERM):
[Equation]
To avoid model misspecification, one often assumes g ∈ G , where G is some infinite-dimensional function space.
A common choice is G
= H p ( α ; C ) , the H¨older ball on the input domain [0 , 1] p , with smoothness exponent α and radius C .
H¨older space is one of the most well-studied function spaces in statistics and it is convenient to quantify its complexity by a single smoothness parameter α ; see Appendix C for a review.
It is well-known that estimating H¨older functions suffers from curse-of-dimensionality (Stone, 1982).
One remedy is to consider the following generalized H¨older space, by imposing certain low-dimensional structures on g :
[Equation]
Remark 4.
The above definition contains g ( w )
=
h ( w I ) , where I ⊂ { 1 , · · · , p } , as a special case, in which g is assumed to only depend on a subset of the feature vector w .
One can easily generalize the above definition to additive models g ( w ) = (cid:80) pj [-=
-]=1 h j ( w j ) where h j ∈ H k j ( α j ; C j ) , allowing even more modeling flexibility.
To avoid complications, we only consider the above simpler model.
We can show that the ERM estimator (cid:98) g (6) from the FNN-ReLU class F nn ( L, K, B ) attains the optimal estimation rate over H † k ( α ; C ) up to log factors, by choosing the depth and width appropriately without assuming sparse neural nets.
Lemma 5.
Under Assumptions iii – vi, if g ∈ H † k ( α ; C ) for k ≤ p , with LK ≍ n k 2( k +2 α ) , we have
[Equation]
Lemma 5, together with Proposition 1, implies the main theoretical result of the paper.
Theorem 6.
Under Assumptions i – vi and the following condition on a, f, µ : a ∈ H † k ( α a ; C ) , f [-
-]∈
H † k ( α f ; C ) , µ ∈ H † k ( α µ ; C ) , with
[Equation]
for k ≤ p and some arbitrarily small ϵ > 0 , if (cid:98) a , (cid:98) f , (cid:98) µ are respectively the ERM (6) from FNN-ReLU classes F nn ( L a , K a , B ) , F nn ( L f , K f , B ) , F nn ( L µ , K µ , B ) , of which the product of the depth and width satisfies L g K g ≍ n k 2( k +2 αg ) for g ∈ { a, f, µ } , then the DeepMed estimators (cid:98) τ tot , (cid:98) τ NDE ( d ) and (cid:98) τ NIE ( d ) computed by Algorithm 1 are semiparametric efficient: (cid:98) σ − 1 tot ( (cid:98) τ tot − τ tot ) , (cid:98) σ − 1 NDE ( d )( (cid:98)
τ
NDE ( d ) − τ NDE ( d )) , (cid:98) σ − 1 NIE ( d )( (cid:98) τ NIE ( d ) − τ NIE ( d ))
L −→ N (0 , 1) , with
N (cid:98) σ 2 tot p →
E [( EIF 1 − EIF 0 ) 2 ] , N (cid:98) σ 2 NDE ( d ) p → E [( EIF 1 ,d − EIF 0 ,d ) 2 ] , and N (cid:98) σ 2 NIE ( d ) p
→ E [( EIF d, 1 −
EIF d, 0 ) 2 ] , i.e. (cid:98) σ 2 tot , (cid:98) σ 2 NDE and (cid:98) σ 2 NIE are consistent variance estimators.
Remark 7.
To unload notation in the above theorem, consider the special case where the smoothness of all the nuisance functions coincides, i.e. α a = α f = α µ = α .
Then Condition (7) reduces to α > k/ 2 + ϵ for some arbitrarily small ϵ > 0 .
For example, if the covariates X have dimension p = 2 and no low-dimensional structures are imposed on the nuisance functions (i.e. k ≡ p ), one needs α > 1 to ensure semiparametric efficiency of the DeepMed estimators.
We emphasize that Lemma 5 and Theorem 6 do not constrain the network sparsity S , better reflecting how DNNs are usually used in practice.
Theorem 6 improves existing results on total effect estimation with non-sparse DNNs (Farrell et al., 2021, Theorem 1) in terms of (1) weaker smoothness conditions and (2) adapting to certain low-dimensional structures of the nuisance functions.
The proof of Lemma 5 follows from a combination of the improved DNN approximation rate obtained in Lu et al.
Jiao et al.
(2021) and standard DNN metric entropy bound (Suzuki, 2019).
We prove Lemma 5 and
Theorem 6 in Appendix C for completeness.
One weakness of Lemma 5 and Theorem 6, as well as in other contemporary works (Chen et al., 2020; Farrell et al., 2021), is the lack of algorithmic/training process considerations (Chen et al., 2022); see Remark 10 and Appendix E for extended discussions.
Remark 8 (Explicit input-layer regularization) .
Training DNNs in practice involves hyperparameter tuning, including the depth L and width K in Theorem 6 and others like epochs.
In the synthetic experiments, we consider the nuisance functions only depending on a k -subset of p -dimensional input.
A reasonable heuristic is to add L 1 -regularization in the input-layer of the DNN.
Then the regularization weight λ is also a hyperparameter.
In practice, we simply use cross-validation to select the hyperparameters that minimize the validation loss.
We leave its theoretical justification and the performance of other alternative approaches such as the minimax criterion (Robins et al., 2020; Cui and Tchetgen Tchetgen, 2019) to future works.
Synthetic experiments
In this section and Appendix E, we showcase five synthetic experiments.
Since ground truth is rarely known in real data, we believe synthetic experiments play an equally, if not more, important role as real data.
Before describing the experimental setups, we garner the following key take-home message:
(a) Compared with the other competing methods, DeepMed exhibits better finite-sample performance in most of our experiments; (b) Cross-validation for DNN hyperparameter tuning works reasonably well in our experiments; (c) We find DeepMed with explicit regularization in the input layer improves performance (see Table A2) when the true nuisance functions have certain low-dimensional structures in their dependence on the covariates.
Farrell et al.
(2021) warned against blind explicit regularization in DNNs for total effect estimation.
Our observation does not contradict Farrell et al.
(2021) as (1) the purpose of the input-layer regularization is not to control the sparsity of the DNN architecture and (2) we do not further regularize hidden layers; (d) Experimental setups for Cases 3 to 5 generate nuisance functions that are nearly infinitedimensional and close to the boundary of a H¨older ball with a given smoothness exponent (Liu et al., 2020; Li et al., 2005).
Thus these synthetic experiments should be better benchmarks than Cases 1 and 2 or settings in other related works such as Farrell et al.
We hope that these highly nontrivial synthetic experiments are helpful to researchers beyond mediation analysis or causal inference.
We share the code for generating these functions as a part of the DeepMed package.
We consider a sample with 10,000 i.i.d.
observations.
The covariates X =
( X 1 , ..., X p ) ⊤ are independently drawn from uniform distribution Uniform([ − 1 , 1]) .
The outcome Y , treatment D and mediator M are generated as follows:
[Equation]
where s ( x ) :
= (1 + e − x ) − 1 , and we consider the following three cases to generate the nonlinear functions d ( x ) , m ( x ) and y ( x ) in the main text:
• Case 1 (simple functions):
[Equation]
• Case 2 (composition of simple functions): we simulate more complex interactions among covariates by composing simple functions as follow:
[Equation] ◦ d 0 ( x 1 , · · · , x 5 ) , with d 0 ( x 1 , · · · , x 5 ) =
(cid:32) 2 (cid:89) i =
x i , 5 (cid:89) i = x i , 2 (cid:89) i = sin x i , 5 (cid:89) i = sin x i (cid:33) , d 1 ( a 1 , · · · , a 4 ) =
(sin( a 1 + a 2 ) , sin a 2 , a 3 , a 4 ) , and d 2 ( b 1 , · · · , b 4 ) = 0 .
5 sin( b 1 + b 2 ) + 0 .
5( b 3 + b 4 ) , m ( x ) =
m 1 ◦ m 0 ( x 1 , . .
, x 5 ) , with m 0 ( x 1 , · · · , x 5 ) =
(sin x 1 , · · · , sin x 5 ) , m 1 ( a 1 , · · · , a 5 )
= 5 sin 5 (cid:88) i = a i and y ( x ) = y 2 ◦ y 1 ◦
{+y 
+}
( x 1 , · · · , x 5 ) , with y 0 ( x 1 , · · · , x 5 ) =
(cid:32) sin 2 (cid:88)
i =
x i , sin 5 (cid:88) i = x i , sin 5 (cid:88) i =
x i (cid:33) , y 1 ( a 1 , a 2 , a 3 ) =
(sin( a 1 + a 2 ) , a 3 ) , and y 2 ( b 1 , b 2 ) = 10 sin( b 1 + b 2 ) .
• Case 3 (H¨older functions): we consider more complex nonlinear functions as follows:
d ( x ) = x 1 x 2 + x 3 x 4 x 5 + 0 . 5 η (0 . 2 x 1 ; α ) , m ( x ) = 5 (cid:88) i = η (0 .
5 x i ; α ) , y ( x ) = x {+1 x 2 + 3 η (cid:32) 
+}
(cid:88) i =
x i ; α (cid:33) where η ( x ; α ) =
(cid:80) j ∈ J,l ∈ Z 2 − j ( α +0 . 25) w j,l ( x ) with J = { 0 , 3 , 6 , 9 , 10 , 16 } and w j,l ( · ) is the D father wavelet functions dilated at resolution j shifted by l .
By construction, η ( x ; α ) ∈ H 1 ( α ; B ) for some known constant B > 0 following H¨ardle
et al.
(1998, Theorem 9.6).
Here we set α = 1 . 2 and the intrinsic dimension k = 1 .
Thus we expect the DeepMed estimators are semiparametric efficient.
It is indeed the case based on the columns corresponding to Case 3 in Table 1, suggesting that DNNs can be adaptive to certain low-dimensional structures.
Remark 9.
The nuisance functions in Cases 3 – 5 (see Appendix E) are less smooth than what have been considered elsewhere, including Farrell et al.
(2021), Chen et al.
(2020), and even Adcock and Dexter (2021), a paper dedicated to exposing the gap between theoretical approximation rates and DNN practice.
These nuisance functions are designed to be near the boundary of a H¨older ball with a given smoothness exponent as we add wavelets at very high resolution in η ( x ; α ) .
This is the assumption under which most of the known statistical properties of DNNs are developed.
[Table]
In all the above cases, τ tot = 0 . 4 and τ NDE ( d ) =
τ NIE ( d ) = 0 . 2 for d ∈ { 0 , 1 } .
We also consider the cases where the total number of covariates p = 20 and 100 but only the first five covariates are relevant to Y , M and D .
All simulation results are based on 200 replicates.
The sigmoid function is used in the final layer when the response variable is binary.
For comparison, we also use the Lasso, random forest (RF) and gradient boosted machine (GBM) to estimate the nuisance functions, and use the true nuisance functions (Oracle) as the benchmark.
The Lasso is implemented using the R package “hdm” with a data-driven penalty.
The DNN, RF and GBM are implemented using the
R packages “keras”, “randomForest” and “gbm”, respectively.
We adopt a 3-fold cross-validation to choose the hyperparameters for DNNs (depth L , width K , L 1 -regularization parameter λ and epochs), RF (number of trees and maximum number of nodes) and GBM (numbers of trees and depth).
We use a completely independent sample for the hyperparameter selection.
In this paper, we only use one extra dataset to conduct the cross-validation for hyperparameter selection, so our simulation results are conditional on this extra dataset.
We use the cross-entropy loss for the binary response and the mean-squared loss for the continuous response.
We fix the batch-size as 100 and the other hyperparameters for the other methods are set to the default values in their R packages.
See
Appendix E for more details.
We compare the performances of different methods in terms of the biases, empirical standard errors (SE) and root mean squared errors (RMSE) of the estimates as well as the coverage probabilities (CP) of their 95% confidence intervals.
When p = k = 5 (all covariates are relevant or no low-dimensional structures), DeepMed has smaller bias and RMSE than the other competing methods, and is only slightly worse than Oracle.
Lasso has the largest bias and poor CP as expected since it does not capture the nonlinearity of the nuisance functions.
RF and GBM also have substantial biases, especially in
Case 2 with compositions of simple functions.
Overall, DeepMed performs better than the competing methods (Table 1).
From the empirical distributions, we can also see that they are nearly unbiased and normally distributed in Cases 1-3 (Figures A1-A3).
When p = 20 or 100 but only the first five covariates are relevant ( k = 5 ), L 1 -regularization in the input-layer drastically improves the performance of DeepMed (Table A2).
DeepMed with L 1 -regularization in the input-layer also has smaller bias and RMSE than the other competing methods (Tables A3 and A4).
As expected, more precise nuisance function estimates (i.e., smaller validation loss) generally lead to more precise causal effect estimates.
The validation losses of nuisance function estimates from
DeepMed are generally much smaller than those using Lasso, RF and GBM (Tables A5-A7).
Remark 10.
Due to space limitations, we defer Cases 4, 5 to Appendix E, in which DeepMed fails to be semiparametric efficient, compared to the Oracle; see an extended discussion in Appendix E. We conjecture this may be due to the implicit regularization of gradient-based training algorithm such as SGD (Table A11) or adam (Kingma and Ba, 2015) (all simulation results except Table A11), which is used to train the DNNs to estimate the nuisance parameters, instead of actually solving the ERM (6) .
Most previous works focus on the benefit of implicit regularization (Neyshabur, 2017; Bartlett et al., 2020) on generalization.
Yet, implicit regularization might inject implicit bias into causal effect estimates, which could make statistical inference invalid.
Such a potential curse of implicit regularization has not been documented in the DNN-based causal inference literature before and exemplify the value of our synthetic experiments.
We believe this is an important open research direction for theoretical results to better capture the empirical performance of DNN-based causal inference methods such as DeepMed .
Real data analysis on fairness
As a proof of concept, we use DeepMed and other competing methods to re-analyze the COMPAS algorithm (Dressel and Farid, 2018).
In particular, we are interested in the NDE of race D on the recidivism risk (or the COMPAS score)
Y with the number of prior convictions as the mediator
M .
For race, we mainly focus on the Caucasians population ( D = 0 ) and the African-Americans population ( D = 1 ), and exclude the individuals of other ethnicity groups.
The COMPAS score ( Y ) is ordinal, ranging from 1 to 10 (1: lowest risk; 10: highest risk).
We also include the demographic information (age and gender) as covariates X .
All the methods find significant positive NDE of race on the COMPAS score at α -level 0.005 (Table 2; all p-values < 10 − 7 ), consistent with previous findings (Nabi and Shpitser, 2018).
Thus the
COMPAS algorithm tends to assign higher recidivism risks to African-Americans than to Caucasians, even when they have the same number of prior convictions.
The validation losses of nuisance function estimates by DeepMed are smaller than the other competing methods (Table A8), possibly suggesting smaller biases of the corresponding NDE/NIE estimators.
We emphasize that research in machine learning fairness should be held accountable (Bao et al., 2021).
Our data analysis is merely a proof-of-concept that DeepMed works in practice and the conclusion from our data analysis should not be treated as definitive.
We defer the comments on potential issues of unmeasured confounding to Appendix F and another real data analysis to Appendix G.
[Table]
Conclusion and Discussion
In this paper
[Table]
we proposed DeepMed for semiparametric mediation analysis with DNNs.
We established novel statistical properties for DNN-based causal effect estimation that can (1) circumvent sparse DNN architectures and (2) leverage certain low-dimensional structures of the nuisance functions.
These results significantly advance our current understanding of DNN-based causal inference including mediation analysis.
Evaluated by our extensive synthetic experiments, DeepMed mostly exhibits improved finite-sample performance over the other competing machine learning methods.
But as mentioned in Remark 10, there is still a large gap between statistical guarantees and empirical observations. Therefore an important future direction is to incorporate the training process while investigating the statistical properties to have a deeper theoretical understanding of DNN-based causal inference.
It is also of future research interests to enable DeepMed to handle unmeasured confounding and more complex path-specific effects (Malinsky et al., 2019; Miles et al., 2020), and incorporate other hyperparameter tuning strategies that leverage the multiply-robustness property, such as the minimax criterion (Robins et al., 2020; Cui and Tchetgen Tchetgen, 2019).
Finally, we warn readers that all causal inference methods, including DeepMed , may have negative societal impact if they are used without carefully checking their working assumptions. Acknowledgement and Disclosure of Funding
The authors thank four anonymous reviewers and one anonymous area chair for helpful comments and
Fengnan Gao (Fudan University) for some initial discussion on how to incorporate low-dimensional manifold assumptions using DNNs.
The authors would also like to thank Department of Statistics and Actuarial Sciences at The University of Hong Kong for providing high-performance computing servers that supported the numerical experiments in this paper.
L. Liu gratefully acknowledges funding support by Shanghai Pujiang Program Research Grant No.20PJ140890, Shanghai Municipal Science and Technology Major Project No.2021SHZDZX0102, Natural Science Foundation of Shanghai
Grant No.21ZR1431000, Shanghai Science and Technology Commission Grant No.21JC1402900, and Natural Science Foundation of China Grant No.12090024 and No.12101397.