DeepMed: Semiparametric Causal Mediation Analysis with Debiased Deep Learning
s)
AfﬁliationAddress
Abstract
Causal mediation analysis can unpack the black box of causality and is therefore apowerful tool for disentangling causal pathways in biomedical and social sciences,and also for evaluating machine learning fairness.
To reduce bias for estimating
Natural Direct and Indirect Effects in mediation analysis, we propose a new methodcalled DeepMed that uses deep neural networks (DNNs) to cross-ﬁt the inﬁnitedimensional nuisance functions in the efﬁcient inﬂuence functions.
We obtainnovel theoretical results that our DeepMed method (1) can achieve semiparametricefﬁciency bound without imposing sparsity constraints on the DNN architectureand (2) can adapt to certain low-dimensional structures of the nuisance functions,signiﬁcantly advancing the existing literature on DNN-based semiparametric causalinference.
Extensive synthetic experiments are conducted to support our noveltheoretical ﬁndings.
As a proof of concept, we apply DeepMed to analyze tworeal datasets on machine learning fairness and reach conclusions consistent withprevious ﬁndings.
Introduction
Tremendous progress has been made in this decade on deploying deep neural networks (DNNs) inreal-world data science problems (Krizhevsky et al., 2012;
Wolf et al., 2019; Jumper et al., 2021).
Causal inference is no exception.
In semiparametric causal inference, several recent works (Chenet al., 2020; Chernozhukov et al., 2020; Farrell et al., 2021) initiated the investigation of statisticalproperties of causal effect estimators when the nuisance functions (the outcome regressions andpropensity scores) are estimated by DNNs.
However, there are several important limitations in therecent literature that need to be addressed before the theoretical results can be used to guide practice:(1)
Most recent works focus on total (treatment) effect (Chen et al., 2020; Farrell et al., 2021).
Inmany settings, however, more intricate causal parameters are often of greater interests.
In biomedicaland social sciences, one is often interested in “mediation analysis” to decompose the total effect intodirect and indirect effect to unpack the underlying black-box causal mechanism (Baron and Kenny,1986).
More recently, mediation analysis also percolated into machine learning fairness.
For instance,in the context of predicting the recidivism risk, Nabi and Shpitser (2018) argued that, for a “fair”algorithm, sensitive features such as race should have no direct effect on the predicted recidivismrisk.
If such direct effects can be accurately estimated, one can detect the potential unfairness of amachine learning algorithm.
We will revisit such applications in Section 5 and Appendix G.(2) Statistical properties of DNN-based causal estimators in recent works mostly follow fromseveral (recent) results on the convergence rates of DNN-based nonparametric regression estimators(Chen and White, 1999; Suzuki, 2019; Schmidt-Hieber, 2020; Tsuji and Suzuki, 2021), with thelimitation of relying on sparse DNN architectures.
The theoretical properties are in turn evaluated by relatively simple synthetic experiments not designed to generate nearly inﬁnite-dimensional nuisancefunctions, a setting considered by almost all the above related works.
The above limitations raise the tantalizing question whether the available statistical guarantees for
DNN-based causal inference have practical relevance.
In this work, we plan to partially ﬁll these gapsby developing a new method called DeepMed for semiparametric mediation analysis with DNNs.
Wefocus on the Natural Direct/Indirect Effects (NDE/NIE) (Robins and Greenland, 1992; Pearl, 2001)(deﬁned in Section 2.1).
The DeepMed estimators leverage the “multiply-robust” property of theefﬁcient inﬂuence function (EIF) of NDE/NIE (Tchetgen Tchetgen and Shpitser, 2012; Farbmacheret al., 2022) (see Proposition 1 in Section 2.2), together with the ﬂexibility and superior predictivepower of DNNs (see Section 3.1 and Algorithm 1).
In particular, we also make the following novelcontributions to deepen our understanding of DNN-based semiparametric causal inference:
• On the theoretical side, we obtain novel results that our DeepMed method can achieve semi- parametric efﬁciency bound without imposing sparsity constraints on the DNN architecture
and can adapt to certain low-dimensional structures of the nuisance functions (see Section 3.2), thus signiﬁcantly advancing the existing literature on DNN-based semiparametric
causal inference.
Non-sparse DNN architecture is more commonly employed in practice, and the low-dimensional structures of nuisance functions can help avoid curse-of-dimensionality.
These two points, taken together, make our theoretical results the state-of-the-art over those
of Farrell et al.
• On the empirical side, in Section 4, we designed sophisticated synthetic experiments to
simulate nearly inﬁnite-dimensional functions, which are much more complex than those in
previous related works (Chen et al., 2020; Farrell et al., 2021; Adcock and Dexter, 2021).
We emphasize that these novel experiments
could be of
independent interest beyond causalmediation analysis or even causal inference; see Remark 9 for an extended discussion.
As aproof of concept, in Section 5 and Appendix G, we also apply DeepMed to re-analyze tworeal-world datasets on algorithmic fairness and reach similar conclusions to related works.
Smaller validation losses of DeepMed suggest smaller bias than competing methods.• Finally, a user-friendly R package will be freely available after the review process.
Makingsuch resources available helps enhance reproducibility, a highly recognized problem in allscientiﬁc disciplines, including machine learning (Pineau et al., 2021).
Deﬁnition, identiﬁcation, and estimation of NDE and NIE2.1 Deﬁnition of NDE and NIE
Throughout this paper, we denote Y as the primary outcome of interest, D as a binary treatmentvariable, M as the mediator on the causal pathway from D to Y , and X 2 [0 , 1] p (or more generally,compactly supported in R p ) as baseline covariates including all potential confounders.
We denotethe observed data vector as O ⌘
( X, D, M, Y ) .
Let M ( d ) denote the potential outcome for themediator when setting D =
d and Y ( d, m ) be the potential outcome of Y under D
= d and M = m ,where d 2 { 0 , 1 } and m is in the support M of M .
We deﬁne the average total (treatment) effect 73 as ⌧ tot :=
E [ Y (1 , M (1))  Y (0 , M (0))] , the average NDE of the treatment D on the outcome
Ywhen the mediator takes the natural potential outcome when D = d as ⌧ NDE ( d ) :
=
E [ Y (1 , M ( d ))  Y (0 , M ( d ))] , and the average NIE of the treatment D on the outcome Y via the mediator M as⌧ NIE ( d ) :=
E [ Y ( d, M (1))  Y ( d, M (0))] .
We have the trivial decomposition ⌧ tot ⌘ ⌧ NDE ( d ) +⌧ NIE ( d 0 ) for d 6 = d 0 .
In causal mediation analysis, the parameters of interest are ⌧ NDE ( d ) and ⌧ NIE ( d ) .
2.2 Semiparametric multiply-robust estimators of NDE/NIE
Estimating ⌧ NDE ( d ) and ⌧ NIE ( d ) can be reduced to estimating  ( d, d 0 ) :
=
E [ Y ( d, M ( d 0 ))] for d, d 0 2{ 0 , 1 } .
We make the following standard identiﬁcation assumptions:
i. Consistency : if D = d , then M = M ( d ) for all d 2 { 0 , 1 } ; while if D = d and M = m ,then Y =
Y ( d, m ) for all d 2 { 0 , 1 } and all m in the support of M .ii.
Ignorability : Y ( d, m ) ? D | X , Y ( d, m ) ?
M | X, D , M ( d ) ? D | X , and Y ( d, m ) ?
M ( d 0 ) | X , almost surely for all d, 2 { 0 , 1 } and all m 2 M . The ﬁrst three conditions are, respectively, no unmeasured treatment-outcome, mediator-outcome and treatment-mediatorconfounding, whereas the fourth condition is often referred to as the “cross-world” condition. We provide more detailed comments on these four conditions in Appendix A. iii.
Positivity : The propensity score a ( d | X ) ⌘ Pr
( D = d | X ) 2 ( c, C ) for some constants 0 < c  C < 1 , almost
surely
for all d 2 { 0 , 1 } ; f ( m | X, d ) , the conditional density (mass) function of M = m (when M is discrete) given X and D = d , is strictly bounded between [¯ ⇢ , ¯ ⇢
] for some constants 0 < ¯ ⇢  ¯ ⇢ < 1 almost surely for all m
in
M
and all
d 2 { 0 , 1 } .
Under the above assumptions, the causal parameter  ( d, d 0 ) for d, d 0 2 { 0 , 1 } can be identiﬁed as 93 either of the following three observed-data functionals:
[Equation]
1) where {·} denotes the indicator function and µ ( x, d, m ) :
=
E [ Y | X = x, D = d, M = m ] is theoutcome regression model, for which we also make the following standard boundedness assumption:iv.
µ ( x, d, m ) is also strictly bounded between [  R, R ] for some constant R > 0 .
Following the convention in the semiparametric causal inference literature, we call a, f, µ “nuisance 98 functions”.
Tchetgen Tchetgen and Shpitser (2012) derived the EIF of  ( d, d 0 ) :
EIF d,d [Equation]  99  ( d, d 0 ) , where
[Equation]
The nuisance functions µ ( x, d, m ) , a ( d | x ) and f ( m | x, d ) appeared in d,d 0 ( o ) are unknown andgenerally high-dimensional.
But with a sample D ⌘ { O j } Nj =1 of the observed data, based on d,d 0 ( o ) ,one can construct the following generic sample-splitting multiply-robust estimator of  ( d, d 0 ) :
[Equation]
where D n ⌘ { O i } ni =1 is a subset of all N data, and e d,d 0 ( o ) replaces the unknown nuisance functionsa, f, µ in d,d 0 ( o ) by some generic estimators e a, e f, e µ computed using the remaining N  n nuisancesample data , denoted as D ⌫ .
Cross-ﬁt is then needed to recover the information lost due to samplesplitting; see Algorithm 1.
It is clear from (2) that e  ( d, d 0 ) is a consistent estimator of  ( d, d 0 ) aslong as any two of e a, e f, e µ are consistent estimators of the corresponding true nuisance functions,hence the name “multiply-robust”.
Throughout this paper, we take n ⇣ N  n and assume: v. Any nuisance function estimators are strictly bounded within the respective lower and upper bounds of a, f, µ
To further ease notation, we deﬁne: for any d 2 { 0 , 1 } , r a,d := R  a,d ( x ) 2 d F ( x ) 1 / 2 , r f,d := R  f,d ( x, m ) 2 d F ( x, m | d = 0) 1 / 2 , and r µ,d :=
R  µ,d ( x, m ) 2 d F ( x, m | d = 0) 1 / 2 , where a,d ( x ) :
=
e a ( d | x )  a ( d | x ) ,  f,d ( x, m ) :=
e f ( m | x, d )  f
( m | x, d ) and  µ,d ( x, m ) :=
e µ ( x, d, m ) µ
( x, d, m ) are point-wise estimation errors of the estimated nuisance functions.
In deﬁning the above
L 2 -estimation errors, we choose to take expectation with respect to (w.r.t.)
the law F ( m, x | d = 0)only for convenience, with no loss of generality by Assumptions iii and v.
To show the cross-ﬁt version of e  ( d, d 0 ) is semiparametric efﬁcient for  ( d, d 0 ) , we shall demonstrateunder what conditions p n ( e  ( d, d 0 )   ( d, d 0 ))
L !
N (0 , E [ EIF 2 d,d 0 ]) (Newey, 1990).
The followingproposition on the statistical properties of e  ( d, d 0 ) is a key step towards this objective.
Proposition 1.
Denote Bias( e  ( d, d [Equation]
=
E [ e  ( d, d 0 )   ( d, d 0 ) |D ⌫ ] as the bias of e  ( d, d 0 ) conditionalon the nuisance sample D ⌫ .
Under Assumptions i – v, Bias( e  ( d, d 0 )) is of second-order:
[Equation]
Furthermore, if the RHS of (4) is o ( n  1 / 2 ) , then
[Equation]
Although the above result is a direct consequence of the EIF d,d 0 ( O ) , we prove Proposition 1 in Appendix B for completeness.
Remark 2.
The total effect ⌧ tot =  (1 , 1)   (0 , 0) can be viewed as a special case, for whichd =
d 0 for  ( d, d 0 ) .
Then EIF d,d ⌘ EIF d corresponds to the nonparametric EIF of  ( d, d ) ⌘  ( d ) ⌘ 127 E [ Y ( d, M ( d ))] : EIF d
[Equation] d ( O )   ( d ) with d ( O ) [Equation]
[Equation]
d
[Equation]
where µ ( x, d ) :
=
E [ Y | X = x, D = d ] .
Hence all the theoretical results in this paper are applicableto total effect estimation, improving the results of Farrell et al.
Estimation and inference of NDE/NIE using DeepMed
We now introduce DeepMed , a method for mediation analysis with nuisance functions estimated by
DNNs.
By leveraging the second-order bias property of the multiply-robust estimators of NDE/NIE(Proposition 1), we will derive statistical properties of DeepMed in this section.
The nuisance functionestimators by DNNs are denoted as
[Equation]
3.1 Details on DeepMed
First, we introduce the fully-connected feed-forward neural network with the rectiﬁed linear units(ReLU) as the activation function for the hidden layer neurons (FNN-ReLU), which will be used toestimate the nuisance functions.
Then, we will introduce an estimation procedure using a V -foldcross-ﬁtting with sample-splitting to avoid the Donsker-type empirical-process assumption on thenuisance functions, which, in general, is violated in high-dimensional setup.
Finally, we provide theasymptotic statistical properties of the DNN-based estimators of ⌧ tot , ⌧ NDE ( d ) and ⌧ NIE ( d ) .
We denote the ReLU activation function as  ( u ) :
=
max( u, 0) for any u 2 R .
Given vectors x, b , wedenote  b ( x ) :
=
( x  b ) , with  acting on the vector x  b component-wise.
Let F nn denote the class of the FNN-ReLU functions
[Equation]
where  is the composition operator, L is the number of layers (i.e. depth) of the network, andfor l = 1 , · · · , L , W ( l ) is a K l +1 ⇥ K l -dimensional weight matrix with K l being the number ofneurons in the l -th layer (i.e. width) of the network, with K 1 = p and K L +1 = 1 , and b ( l ) is
a
K l -dimensional vector.
To avoid notation clutter, we concatenate all the network parameters as⇥ =
( W ( l ) , b ( l ) , l = 1 , · · · , L ) and simply take K 2 = · · · = K L = K .
We also assume ⇥ to bebounded: k ⇥ k 1  B for some universal constant B > 0 .
We may let the dependence on L , K , B 151 explicit by writing
F nn as F nn ( L, K, B ) .
DeepMed estimates ⌧ tot , ⌧ NDE ( d ) , ⌧ NIE ( d ) by (3), with the nuisance functions a, f, µ estimated using
F nn with the V -fold cross-ﬁtting strategy, summarized in Algorithm 1 below.
DeepMed inputsthe observed data D ⌘ { O i } Ni =1 and outputs the estimated total effect b ⌧ tot , NDE b ⌧ NDE ( d ) and NIEb ⌧ NIE ( d ) , together with their variance estimators b  2 tot , b  2 NDE ( d ) and b  2 NIE ( d ) . [Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation] [Equation] [Equation]
[Equation] [Equation] [Equation]
[Equation]
[Equation]
[Equation]
[Equation]
{+d, 
+}
( O i ))
2  1 N b ⌧ 2 NIE ( d )
Output: b ⌧ tot , b ⌧ NDE ( d ) , b ⌧ NIE ( d ) , b  2 tot , b  2 NDE ( d ) and b  2 NIE ( d )
Remark 3 (Continuous or multi-dimensional mediators) .
For binary treatment D and continuous ormulti-dimensional M , to avoid nonparametric/high-dimensional conditional density estimation, wecan rewrite f ( m | x,d 0 ) a ( d | x ) f ( m | x,d ) as [Equation]
[Equation] 1  [Equation] ( d | x )) by the Bayes’ rule and the integral w.r.t.
f ( m | x, d 0 )in (2) as E [ µ ( X, d, M ) | X
= x, D = d 0 ] .
Then we can ﬁrst estimate µ ( x, d, m ) by b µ ( x, d, m ) andin turn estimate E [ µ ( X, d, M ) | X = x, D
=
d 0 ] by regressing b µ ( X, d, M ) against ( X, D ) usingthe FNN-ReLU class.
We mainly consider binary M to avoid unnecessary complications; but see
Appendix G for an example in which this strategy is used.
Finally, the potential incompatibilitybetween models posited for a ( d | x ) and a ( d | x, m ) and the joint distribution of ( X, A, M, Y ) is not ofgreat concern under the semiparametric framework because all nuisance functions are estimatednonparametrically; again, see Appendix G for an extended discussion.
3.2 Statistical properties of DeepMed: Non-sparse DNN architecture and low-dimensional
structures of the nuisance functions
According to Proposition 1, to analyze the statistical properties DeepMed , it is sufﬁcient to controlthe L 2 -estimation errors of nuisance function estimates [Equation] by DNNs.
To ease presentation,we ﬁrst study the theoretical guarantees on the L 2 -estimation error for a generic nuisance functiong :
W 2 [0 , 1] p !
Z 2 R , for which we assume: vi.
Z = g ( W ) + ⇠ , with ⇠ sub-Gaussian with mean zero and independent of W .
Note that when g corresponds to a, f, µ , ( W, Z ) corresponds to ( X, ( D = 1)) , (( X, D ) , ( M = 1))and (( X, D, M ) , Y ) , respectively.
We denote the DNN output from the nuisance sample D ⌫ as b g .
For theoretical results, we consider b gas the following empirical risk minimizer (ERM):
[Equation]
To avoid model misspeciﬁcation, one often assumes g 2 G , where G is some inﬁnite-dimensional 193 function space.
A common choice is
G = H p ( ↵ ; C ) , the H¨older ball on the input domain [0 , 1] p ,with smoothness exponent ↵ and radius C .
H¨older space is one of the most well-studied functionspaces in statistics and it is convenient to quantify its complexity by a single smoothness parameter↵ ; see Appendix C for a review.
It is well-known that estimating H¨older functions suffers fromcurse-of-dimensionality (Stone, 1982).
One remedy is to consider the following generalized H¨olderspace, by imposing certain low-dimensional structures on g :
[Equation]
Remark 4.
The above deﬁnition contains g ( w )
=
h ( w I ) , where I ⇢ { 1 , · · · , p } , as a special case, 200 in which g is assumed to only depend on a subset of the feature vector w .
One can easily generalizethe above deﬁnition to additive models g ( w ) =
P pj =
h j ( w j ) where h j 2 H k j ( ↵ j ; C j ) , allowingeven more modeling ﬂexibility.
To avoid complications, we only consider the above simpler model.
We can show that the ERM estimator b g (6) from the FNN-ReLU class F nn ( L, K, B ) attains the optimal estimation rate over H † k ( ↵ ; C ) up to log factors, by choosing the depth and width appropriately,and unlike Farrell et al. (2021), without assuming sparse neural nets.
Lemma 5.
Under Assumptions iii – vi, if g 2 H † k ( ↵ ; C ) for k  p , with LK ⇣ n k
[Equation]
k +
[Equation]
) , we havesup
[Equation]
Lemma 5, together with Proposition 1, implies the main theoretical result of the paper.
Theorem 6.
Under Assumptions i – vi and the following condition on a, f, µ : a 2 H † k ( ↵ a ; C ) , f
H † k ( ↵ f ; C ) , µ 2 H † k ( ↵ µ ; C ) , with
[Equation]
for k  p and some arbitrarily small ✏ > 0 , if b a , b f , b µ are respectively the ERM (6) from FNN-ReLUclasses F nn ( L a , K a , B ) , F nn ( L f , K f , B ) , F nn ( L µ , K µ , B ) , of which the product of the depth andwidth satisﬁes L g K g ⇣ n k 2( k +2 ↵ g ) for g 2 { a, f, µ } , then the DeepMed estimators b ⌧ tot , b ⌧ NDE ( d ) andb ⌧ NIE ( d ) computed by Algorithm 1 are semiparametric efﬁcient:b   1 tot ( b ⌧ tot  ⌧ tot ) , b  [Equation] NDE
[Equation]
d )( b [Equation] NDE ( d )  ⌧ NDE ( d )) , b   1 NIE ( d )( b ⌧ NIE ( d )  ⌧ NIE ( d ))
L !
N (0 , 1) , with N b  2 tot p !
E [( EIF 1  EIF 0 ) 2 ] , N b  2 NDE ( d ) p ! E [( EIF 1 ,d  EIF 0 ,d ) 2 ] , and N b  2 NIE ( d ) p !
E [( EIF d, 1  EIF d, 0 ) 2 ] , i.e. b  2 tot , b  2 NDE and b  2 NIE are consistent variance estimators.
Remark 7.
To unload notation in the above theorem, consider the special case where the smoothnessof all the nuisance functions coincides, i.e. ↵ a = ↵ f = ↵ µ = ↵ .
Then Condition (7) reduces to↵ > k/ 2 + ✏ for some arbitrarily small ✏ > 0 .
For example, if the covariates X have dimensionp = 2 and no low-dimensional structures are imposed on the nuisance functions (i.e. k ⌘ p ), oneneeds ↵ > 1 to ensure semiparametric efﬁciency of the DeepMed estimators.
We emphasize that Lemma 5 and Theorem 6 do not constrain the network sparsity S , better reﬂectinghow DNNs are usually used in practice.
Theorem 6 improves existing results on total effect estimationwith non-sparse DNNs (Farrell et al., 2021, Theorem 1) in terms of (1) weaker smoothness conditionsand (2) adapting to certain low-dimensional structures of the nuisance functions.
The proof of Lemma5 follows from a combination of the improved DNN approximate rate obtained in Lu et al.
Jiao et al.
(2021) and standard DNN metric entropy bound (Suzuki, 2019).
We prove Lemma 5 and Theorem 6 in Appendix C for completeness.
One weakness of Lemma 5 and Theorem 6, as well as inother contemporary works (Chen et al., 2020; Farrell et al., 2021), is the lack of algorithmic/trainingprocess considerations; see Remark 10 and Appendix E for extended discussions.
Remark 8 (Explicit input-layer regularization) .
Training DNNs in practice involves hyperparameters,including the depth L and width K in Theorem 6 and others like epochs.
In the synthetic experiments,we consider the nuisance functions only depending on a k -subset of p -dimensional input.
A reasonableheuristic is to add L 1 -regularization in the input-layer of the DNN.
Then the regularization weight is also a hyperparameter.
In practice, we simply use cross-validation to select the hyperparametersthat minimize the validation loss.
We leave the theoretical justiﬁcation to future works.
Synthetic experiments
In this section and Appendix E, we showcase ﬁve synthetic experiments.
Since ground truth is rarelyknown in real data, we believe synthetic experiments play an equally, if not more, important role asreal data.
Before describing the experimental setups, we garner the following key take-home message:
(a) Compared with the other competing methods, DeepMed exhibits better ﬁnite-sample perfor-
mance in most of our experiments
(b) Cross-validation for DNN hyperparameter tuning works reasonably well in our experiments;(c)
We ﬁnd DeepMed with explicit regularization in the input layer improves performance(see Table A2) when the true nuisance functions have certain low-dimensional structuresin their dependence on the covariates.
Farrell et al.
(2021) warned against blind explicitregularization in DNNs for total effect estimation.
Our observation does not contradict
Farrell et al.
(2021) as (1) the purpose of the input-layer regularization is not to control thesparsity of the DNN architecture and (2) we do not further regularize hidden layers;(d)
Experimental setups for Cases 3 to 5 generate nuisance functions that are nearly inﬁnitedimensional and close to the boundary of a H¨older ball with a given smoothness exponent.
Thus these synthetic experiments should be better benchmarks than Cases 1 and 2 or settingsin other related works such as Farrell et al.
We hope that these novel syntheticexperiments are helpful to researchers beyond mediation analysis or causal inference.
Weshare the code for generating these functions as a part of the DeepMed package.
We consider a sample with 10,000 i.i.d.
observations.
The covariates X =
( X 1 , ..., X p ) > areindependently drawn from uniform distribution Uniform([  1 , 1]) .
The outcome Y , treatment D andmediator M are generated as follows:
[Equation]
where s ( x ) :
= (1 + e  x )  1 , and we consider the following three cases to generate the nonlinearfunctions d ( x ) , m ( x ) and y ( x ) in the main text:
• Case 1 (simple functions):
[Equation]
• Case 2 (composition of simple functions): we simulate more complex interactions among covariates 264 by composing simple functions as follow:d ( x ) =
d 2  d 1  d 0 ( x 1 , · · · , x 5 ) , with d 0 ( x 1 , · · · , x 5 ) =
[Equation]
1 ( a 1 , · · · , a 4 ) = (sin( a 1 + a 2 ) , sin a 2 , a 3 , a 4 ) , and d 2 ( b 1 , · · · , b 4 ) = 0 . 5 sin( b 1 + b 2 ) + 0 . 5( b 3 + b 4 ) , m ( x ) =
m 1  m 0 ( x 1 , . . , x 5 ) , with m 0 ( x 1 , · · · , x 5 ) = (sin x 1 , · · · , sin x 5 ) , m 1 ( a 1 , · · · , a 5 ) = 5 sin
X i =
a i and y ( x ) = y 2  y 1  y 0 ( x 1 , · · · , x 5 ) , with y 0 ( x 1 , · · · , x 5 ) = sin
X i = x i , sin
[Equation] i = [Equation]
[Equation] y 1 ( a 1 , a 2 , a 3 ) = (sin( a 1 + a 2 ) , a 3 ) , and y 2 ( b 1 , b 2 ) = 10 sin( b 1 + b 2 ) . • Case 3 (H¨older functions): we consider more complex nonlinear functions as follows:
[Equation]
X i = x i ; ↵ ! where ⌘ ( x ; ↵ ) =
P j 2 J,l 2 Z 2  j ( ↵ +0 . 25) w j,l ( x ) with J = { 0 , 3 , 6 , 9 , 10 , 16 } and w j,l ( · ) is the D6father wavelet functions dilated at resolution j shifted by l .
By construction, ⌘ ( x ; ↵ ) 2 H 1 ( ↵ ; B ) forsome known constant B > 0 following H¨ardle
et al.
(1998, Theorem 9.6).
Here we set ↵ = 1 .
2 andthe intrinsic dimension k = 1 .
Thus we expect the DeepMed estimators are semiparametric efﬁcient.
It is indeed the case based on the columns corresponding to Case 3 in Table 1, suggesting that DNNscan be adaptive to certain low-dimensional structures.
Remark 9.
The nuisance functions in Cases 3 – 5 (see Appendix E) are less smooth than what havebeen considered elsewhere, including Farrell et al.
(2021), Chen et al.
(2020), and even Adcockand Dexter (2021), a paper dedicated to exposing the gap between theoretical approximation ratesand DNN practice.
These nuisance functions are designed to be near the boundary of a H¨older ballwith a given smoothness exponent as we add wavelets at very high resolution in ⌘ ( x ; ↵ ) .
This is theassumption under which most of the known statistical properties of DNNs are developed. [Table]
In all the above cases, ⌧ tot = 0 . 4 and ⌧ NDE ( d ) =
⌧ NIE ( d ) = 0 . 2 for d 2 { 0 , 1 } .
We also considerthe cases where the total number of covariates p = 20 and 100 but only the ﬁrst ﬁve covariates arerelevant to Y , M and D .
All simulation results are based on 200 replicates.
The sigmoid function isused in the ﬁnal layer when the response variable is binary.
For comparison, we also use the Lasso,random forest (RF) and gradient boosted machine (GBM) to estimate the nuisance functions, anduse the true nuisance functions (Oracle) as the benchmark.
The Lasso is implemented using the Rpackage “hdm” with a data-driven penalty.
The DNN, RF and GBM are implemented using the
R packages “keras”, “randomForest” and “gbm”, respectively.
We adopt a 3-fold cross-validationto choose the hyperparameters for DNNs (depth L , width K , L 1 -regularization parameter  andepochs), RF (number of trees and maximum number of nodes) and GBM (numbers of trees anddepth).
We use a completely independent sample for the hyperparameter selection.
In this paper,we only use one extra dataset to conduct the cross-validation for hyperparameter selection, so oursimulation results are conditional on this extra dataset.
We use the cross-entropy loss for the binaryresponse and the mean-squared loss for the continuous response.
We ﬁx the batch-size as 100 and theother hyperparameters for the other methods are set to the default values in their R packages.
See
Appendix E for more details.
We compare the performances of different methods in terms of the biases, empirical standard errors(SE) and root mean squared errors (RMSE) of the estimates as well as the coverage probabilities (CP)of their 95% conﬁdence intervals.
When p = k = 5 (all covariates are relevant or no low-dimensionalstructures), DeepMed has smaller bias and RMSE than the other competing methods, and is onlyslightly worse than Oracle.
Lasso has the largest bias and poor CP as expected since it does not capturethe nonlinearity of the nuisance functions.
RF and GBM also have substantial biases, especially in
Case 2 with compositions of simple functions.
Overall, DeepMed performs better than the competingmethods (Table 1).
From the empirical distributions, we can also see that they are nearly unbiasedand normally distributed in Cases 1-3 (Figures A1-A3).
When p = 20 or 100 but only the ﬁrstﬁve covariates are relevant ( k = 5 ), L 1 -regularization in the input-layer drastically improves theperformance of DeepMed (Table A2).
DeepMed with L 1 -regularization in the input-layer also hassmaller bias and RMSE than the other competing methods (Tables A3 and A4).
As expected, more precise nuisance function estimates (i.e., smaller validation loss) generally leadto more precise causal effect estimates.
The validation losses of nuisance function estimates from DeepMed are generally much smaller than those using Lasso, RF and GBM (Tables A5-A7).
Remark 10.
Due to space limitations, we defer Cases 4, 5 to Appendix E, in which DeepMedfails to be semiparametric efﬁcient; see an extended discussion in Appendix E. We conjecture thismay be due to the implicit regularization of gradient-based training algorithm such as SGD (Table
A11) or adam (Kingma and Ba, 2015) (all simulation results except Table A11), which is used to train the DNNs to estimate the nuisance parameters, instead of actually solving the ERM (6) . Mostprevious works focus on the beneﬁt of implicit regularization (Neyshabur, 2017) on generalization. Yet, implicit regularization might inject implicit bias into causal effect estimates, which could makestatistical inference invalid.
Such a potential peril of implicit regularization has not been documentedin the DNN-based causal inference literature before and exemplify the value of our novel syntheticexperiments.
We believe this is an important open research direction for theoretical results to bettercapture the empirical performance of DNN-based causal inference methods such as DeepMed .
Real data analysis on fairness
As a proof of concept, we use DeepMed and other competing methods to re-analyze the COMPASalgorithm (Dressel and Farid, 2018).
In particular, we are interested in the NDE of race D on therecidivism risk (or the COMPAS score)
Y with the number of prior convictions as the mediator
M .
For race, we mainly focus on the Caucasians population ( D = 0 ) and the African-Americanspopulation ( D = 1 ), and exclude the individuals of other ethnicity groups.
The COMPAS score ( Y )is ordinal, ranging from 1 to 10 (1: lowest risk; 10: highest risk).
We also include the demographicinformation (age and gender) as covariates X .
All the methods ﬁnd signiﬁcant positive NDE of race on the COMPAS score at ↵ -level 0.005 (Table2; all p-values < 10  7 ), consistent with previous ﬁndings (Nabi and Shpitser, 2018).
Thus the
COMPAS algorithm tends to assign higher recidivism risks to African-Americans than to Caucasians,even when they have the same number of prior convictions.
The validation losses of nuisance functionestimates by DeepMed are smaller than the other competing methods (Table A8), possibly suggestingsmaller biases of the corresponding NDE/NIE estimators.
We emphasize that research in machine learning fairness should be held accountable (Bao et al., 2021).
Our data analysis is merely a proof-of-concept that DeepMed works in practice and the conclusionfrom our data analysis should not be treated as deﬁnitive.
We defer the comments on potential issuesof unmeasured confounding to Appendix F and another real data analysis to Appendix G
[Table]
[Table]
Conclusion and Discussion
In this paper [Table] we proposed DeepMed for semiparametric mediation analysis with DNNs.
Weestablished the state-of-the-art statistical properties for DNN-based causal effect estimation thatcan (1) circumvent sparse DNN architectures and (2) leverage certain low-dimensional structures ofthe nuisance functions.
These results signiﬁcantly advance our current understanding of DNN-basedcausal inference including mediation analysis.
Evaluated by our extensive synthetic experiments, DeepMed exhibits improved ﬁnite-sample performance over the other competing machine learning methods.
But as mentioned in Remark 10, animportant future direction is to incorporate the training process while investigating the statisticalproperties to have a deeper theoretical understanding of DNN-based causal inference.
It is also offuture research interests to enable DeepMed to handle unmeasured confounding.
Finally, we warn readers that all causal inference methods, including DeepMed , may have negativesocietal impact if they are used without carefully checking their working assumptions.