Empirical Gateaux Derivatives for Causal Inference
Abstract
We study a constructive algorithm that approximates Gateaux derivatives for statistical functionals by ﬁnite differencing, with a focus on functionals that arise in causal inference.
We study the setting where probability distributions are not known a priori but need to be estimated from data.
These estimated distributions lead to empirical Gateaux derivatives, and we study the relationships between empirical, numerical, and analytical Gateaux derivatives.
Starting with a case study of the interventional mean (average potential outcome), we delineate the relationship between the empirical Gateaux derivative (via ﬁnite differencing) and the analytical Gateaux derivative.
We then derive requirements on the rates of numerical approximation in perturbation and smoothing that preserve the statistical beneﬁts of one-step adjustments, such as rate double robustness.
We further study more complicated functionals such as dynamic treatment regimes and the linearprogramming formulation for policy optimization in inﬁnite-horizon Markov decision processes.
The ability to approximate bias adjustments in the presence of arbitrary constraints in these more complicated settings illustrates the usefulness of constructive approaches for Gateaux derivatives.
We also ﬁnd that the statistical structure of the functional (rate double robustness) can permit less conservative rates for ﬁnite-difference approximation.
This property, however, can be speciﬁc to particular functionals; e.g., it occurs for the average potential outcome (hence average treatment effect) but not the inﬁnite-horizon MDP policy value.
Introduction
Inferential targets in causal machine learning often take the form of statistical functionals of the data distribution.
Examples include average treatment effects or average policy values for inﬁnitehorizon ofﬂine reinforcement learning.
Estimation of these statistical functionals can be vulnerable to the ﬁrst-stage bias introduced by the estimation of nuisance functions such as outcome regressions or transition probabilities.
However, by leveraging the causal structure, it is possible to derive biasadjusted estimators that weaken the need for accurate estimation of these nuisance functions.
The celebrated doubly robust estimator is one such bias adjustment, and there are general frameworks for deriving such estimators, including semiparametric efﬁciency, double robustness, Neyman orthogonality, and “debiased/double” machine learning [7, 12, 13, 35, 52, 53, 56, 62].
These general frameworks can also be applied to more complicated functionals in longitudinal causal inference and ofﬂine reinforcement learning (also known as ofﬂine policy evaluation/learning)
A drawback of these general frameworks is that signiﬁcant analytic effort is often necessary to obtain concrete estimators that are appropriate for particular situations.
Indeed, a practitioner may be interested in novel variants of an estimand, or be working within a constrained class of functionals that are appropriate for a particular class of problems.
For example, in constrained Markov Decision Processes (MDPs) [3], or optimization-based estimators more broadly, a practitioner can easily avail themselves of custom convex-optimization solvers that are computationally efﬁcient and easy to deploy to add additional constraints.
Unfortunately, such choices may require case-speciﬁc reanalysis to establish the desired statistical properties, and deriving the actual estimator that yields bias adjustment may require signiﬁcant analytical effort.
It is therefore important to develop a suite of constructive, numerical or algorithmic methods that yield desired forms of bias adjustment.
Such a suite would be complementary to analytic derivations.
To this end, we develop off-the-shelf procedures for estimating the statistical functionals for bias adjustment in causal inference.
We do so via numerical approximation of the Gateaux derivatives that underlie the general analytic frameworks, building on prior work in this area.
We focus on interventional effects.
[57] suggested numerical differentiation to estimate Gateaux derivatives of functionals unavailable in closed form but approximated by computational procedures, such as the solution to a system of differential equations; this approach is later used in [34].
Recent pedagogical surveys of [35, 53] also emphasize the usefulness of Gateaux derivatives for deriving inﬂuence functions, an approach which is directly applicable for discrete data and generalizes to continuous distributions via a smoothing argument as discussed in [10, 39].
In this spirit, we build on the line of work of Carone et al.
[10], Frangakis et al.
[24], Ichimura and Newey [39], Newey [57], which proposes a constructive procedure by numerical approximation of a Gateaux derivative (in the sense of Hampel [32], Huber [38]) by ﬁnite differences.
Evaluating inﬂuence functions by numerical differentiation can be useful because it only requires black-box evaluation of the functional.
In other areas such as optimization, numerical differentiation underpins common subroutines that enable analysts to use optimization algorithms without specialized training in a particular modeling language or paradigm.
Although more advanced paradigms compute exact gradients, they can require specialized training to use.
In addition, the numerical differentiation viewpoint connects the conceptual interpretation of inﬂuence functions as qualitative sensitivity analysis (i.e. sensitivity of a functional to perturbations in the distribution) to the use of inﬂuence functions in causal inference.
Typical presentations of the latter framework are not concretely connected to the former notion.
We provide a concrete connection, which can be interesting independent of computerized estimation for algorithmic, pedagogical, or conceptual reasons.
In this paper, we build on the previously mentioned line of work on the use of numerical derivatives to approximate inﬂuence functions, focusing on the implications for statistical estimation.
We provide exact characterizations of empirical Gateaux derivatives, in contrast to numerical derivatives computed based on oracle knowledge of probability distributions.
This work is focused on exact characterizations that can inform statistical and computational trade-offs.
Although we expect that some of these characterizations may be apparent to experts, we also hope that this level of concreteness can be helpful to non-experts.
These trade-offs could eventually be practically relevant because the ﬁner-scaled discretization required for lower approximation error can incur fundamental issues with ﬂoating-point computation that prohibit further gains in accuracy [23, 64].
Our contributions are as follows.
We begin with a case study of the average potential outcome.
The average treatment effect is the difference of the average potential outcomes, and our analysis applies also to the average treatment effect.
We discuss the treated mean for simplicity; by symmetry the same results hold for the control mean.
In this setting, we characterize the exact relationship between ﬁnite-differences and the analytical Gateaux derivative.
This characterization helps concretize the proposal of empirical Gateaux derivatives by allowing us to study the rates of numerical approximation that can preserve statistical performance.
Finally, with these concretizations in hand, we turn to more complicated motivating examples including a ﬁnite-horizon dynamic treatment regime and inﬁnite-horizon reinforcement learning.
We illustrate with the case of policy learning in ﬁnite-state, ﬁnite-action inﬁnite-horizon Markov decision processes how this constructive approach to Gateaux derivatives can directly extend also to the case of custom constraints.
Problem Setup: From Numerical to Empirical Gateaux Derivatives
We begin by introducing our ﬁrst example, that of the mean potential outcome, and the canonical doubly-robust estimator.
We then brieﬂy introduce the key objects in the more general framework of orthogonalizing or debiasing statistical functionals by Gateaux derivative adjustments via the onestep estimator.
We also describe the numerical approximation of these adjustments via perturbed black-box evaluations of the statistical functional and plug-in estimation.
After introducing these frameworks, we clarify what we mean by empirical rather than numerical Gateaux derivatives and delineate our speciﬁc research questions.
We let O ⇠ P denote a draw of observations following the distribution P belonging to a statistical model M .
We will focus on the estimation of statistical functionals ( P ) , where : M !
R q is pathwise differentiable.
Throughout, we take as given the assumptions that grant causal identiﬁability (i.e., overlap/positivity, unconfoundedness/ignorability).
The mean potential outcome and augmented inverse propensity weighting.
We very brieﬂy overview the celebrated doubly robust estimate of the mean potential outcome.
The average treatment effect is simply the difference of these doubly robust estimators for the treated and control means; for simplicity, we discuss the treated mean only.
Example 1 (Mean potential outcome) .
The observation O =
( X, A, Y ) includes covariate X , treatment A 2 { 0 , 1 } , and outcome Y 2 R .
The statistical functional corresponding to the mean potential outcome is:
[Equation]
Plug-in estimation of the conditional expectation is termed regression adjustment .
The overall functional is also referred to as inverse propensity weighting,
[Equation]
i , where e
[Equation] = [Equation]
x ) is the propensity score .
(We omit explicit discussion of assumptions such as consistency and ignorability, under which these identiﬁcation results hold).
Given two identifying functionals for the mean potential outcome, a natural question is: can these identiﬁcation approaches be combined in some way to improve estimation?
The doubly-robust estimator of Robins et al.
[62] achieves this as follows:
[Equation]
i Appealing properties such as the mixed-bias property and the rate double robustness properoty can be veriﬁed readily from its functional form.
This provides a canonical example of one-step adjustment by inﬂuence functions that applies more broadly.
Inﬂuence function and Gateaux derivative.
A functional is Gateaux differentiable at P if 0 ( H ; P ) , d ( P + ✏ H )
d ✏ | ✏ =0 exists and is linear and continuous in H for all H .
Hampel [32] and Huber [38] deﬁne the inﬂuence function  ( O ; P ) as a Gateaux derivative with respect to perturbations H =
o  P , where  o is a degenerate Dirac measure satisfying the identity R o  o {+
+}
( o ) = o 0 .
That is, we have:
[Equation]
Numerical Gateaux derivatives.
Although this deﬁnition of inﬂuence function via perturbation is not quite enough for semiparametric inference with continuous distributions, Carone et al.
[10] and Ichimura and Newey [39] show how to obtain absolute continuity in semiparametric models by taking an additional limit as a smoothed perturbation converges to the point mass.
They replace the
Dirac delta measure  o with a smoothed distribution ˜
{+  o 
+} ( o ) , a function satisfying
[Equation]
and dominated by P .
A common choice for ˜   o 0 has the form ˜ {+  o 
+}
( o ) =
K  ( o  o 0 ) , where K  is a bandwidth-normalized kernel function and product kernel.
We then deﬁne a perturbation in the direction of a generic observation o i as follows:
P o i ✏ ,  = P i ✏ ,  = (1  ✏ ) P + ✏ ˜   o i .
Carone et al.
[10], Frangakis et al.
[24] discuss numerical approximation by ﬁnite differences: for a ﬁxed ✏ ,  with ✏ ⌧  , evaluate the numerical derivative ˆ  ( o ; P ) =
✏  1 ( ( P ✏ ,  )  ( P )) .
Further, [35, 53] highlight inﬂuence function derivations based on the analytic Gateaux derivative.
Carone et al.
[10] establish that the limit of P o i ✏ ,  when ✏ and  tend to zero is precisely the Gateaux derivative.
and they provided generic approximation rates in ( ✏ ,  ) for a multi-point ﬁnite difference scheme.
These rates establish sufﬁcient conditions for approximation error that ensure validity of the numerical approximation, but can be conservative.
For a two-point approximation with a uniform kernel, the rate from Carone et al.
[10, Thm. 5] is [Equation]
We later strengthen this result.
Empirical Gateaux derivatives.
In the context of statistical estimation, the underlying probability distributions also need to be estimated.
We consider plug-in estimation of the statistical functional via density estimation of the collection of joint probability estimates ˜
p
( o ) that comprise ˜ P .
For example, a plug-in representation of the mean potential outcome is as follows:
[Equation]
It is important to distinguish between the empirical Gateaux derivative computed by numerical approximation with estimated probability densities (e.g., ˜ P ( Y, A = 1 , X ) ), the numerical Gateaux derivative obtained by ﬁnite differencing based on the true probability densities, and the analytic
Gateaux derivative:
[Equation]
smoothed and estimated distributions
[Equation]
distributions [Equation]
It is the ﬁrst of these representations that is appropriate for serving as a statistical estimator of inﬂuence functions.
In particular, the one-step estimator is justiﬁed by the following asymptotically linear representation [60]:
[Equation]
See Appendix A for further discussion of this expansion.
Given these deﬁnitions, we now turn to the major mathematical and statistical questions that we address in this paper:
Q1 :
How exactly does the empirical Gateaux derivative approximate the analytical derivative?
Q2 :
What are the required rates of numerical approximation in perturbation and smoothing based on ( ✏ ,  ) that preserve the beneﬁcial statistical properties of the constructed estimator?
In Section 4, we study these questions in the simple case of our running example: estimation of the mean potential outcome.
In that setting we can provide a ﬁne-grain comparison of the ﬁnitedifference estimator and the classical doubly robust estimator.
In Section 5, we study more complicated functionals to highlight more general applicability.
Example 2 (Example 1, continued) .
The inﬂuence function [31, 62] for a mean potential outcome is:
[Equation]
Hence the canonical doubly-robust estimator (augmented inverse propensity weighting, or AIPW) can be viewed as a one-step adjustment.
We conclude our introductory remarks by noting some limitations of our study.
We omit discussion of semiparametric efﬁciency since we focus on computing inﬂuence functions without the consideration of model restrictions and tangent spaces.
That is, we focus on Gateaux derivatives under a nonparametric model.
Throughout, we assume the functional is pathwise differentiable.
This assumption does warrant caution, because this condition can pose a fundamental barrier to efforts for
“automation”; hence empirical Gateaux derivatives are intended to augment but not supplant or replace analytical expertise.
See Appendix D.2 for extended discussion on how stronger smoothness conditions admit sufﬁcient conditions for asymptotic linearity.
Related Work
In this section, we discuss the most closely related directions but do not attempt to provide a survey or overview of inﬂuence functions or semiparametric statistics.
See Bickel et al.
[7], Fisher and
Kennedy [21], Kennedy [53], Tsiatis [69] or the appendix for a broader overview and further details.
In Appendix D.1, for additional context, we also overview work on numerical derivatives in optimization and statistical machine learning more broadly.
Numerical approximation of Gateaux derivatives.
Recent pedagogical reviews [21, 35, 53] emphasize the Gateaux derivative and its use as a heuristic tool for deriving inﬂuence functions.
While our point of departure is a ﬁnite-difference approach to the former, we also consider more complicated optimization functionals, with a focus on how dual variables characterize the Gateaux derivatives therein.
[10] derive high-level sufﬁcient conditions for approximation error via general results for ﬁnite-difference approximations.
Relative to those general sufﬁcient conditions, we focus on speciﬁc improvements in approximation error for various functionals.
For the speciﬁc case of the mean potential outcome/average treatment effect, Frangakis et al.
[24] interchange integration and differentiation as a result of analytical insight hence their approach is less “automatic.”
In general, it may be difﬁcult to deduce the form of the nuisance functions without deriving the inﬂuence function analytically.
The approach in Appendix B of [9] considers the speciﬁc case of debiasing moment conditions with respect to ﬁrst stages but also implicitly leverages this change in order of integration.
Other work on automating causal inference.
Recent work in causal inference aims at “automatic” or “computerized” semiparametric inference via a variety of methods [4, 14–16, 20, 45, 51].
To summarize brieﬂy the difference with our approach, our approach focuses on the use of numerical derivatives, and requires only a zeroth-order computation oracle of perturbed functional evaluations, without additional nuisance estimation.
This approach can be particularly helpful for incorporating debiased causal inference in the design and analysis of algorithms.
On the other hand, it may not achieve as strong improvements in estimation performance as other approaches that estimate additional nuisance functions.
Concurrently with the initial submission of this work, [66] studied automatic debiased machine learning for dynamic treatment effects.
Our development is most distinctive in the inﬁnite-horizon setting, which is not a generalized regression residual.
We discuss differences to other works in the line of work of “automatic” semiparametric estimation that are less directly related in Appendix A.1.
Empirical Gateaux Derivative of the Mean Potential Outcome
To illustrate the numerical approach to Gateaux derivatives, we work through the example of the mean potential outcome and indicate precisely how the numerical differentiation incurs error relative to the analytical derivative, which here is the doubly robust or augmented IPW (AIPW) estimator.
Relating the empirical and analytical Gateaux derivatives
Setup.
We target the estimation of a mean potential outcome as in Example 1.
For the smoothed perturbation, we specialize to kernel functions K  ( u ) from kernel density estimation satisfying
R 1 1 K ( u ) du = 1 (normalization to a probability density) and K (  u ) =
K ( u ) , for all u (symmetry).
The mean potential outcome is identiﬁed as follows,
[Equation]
We obtain ( ˜ P ✏ ) by plugging in the ˜ p probability density estimates.
Note this is distinct from the plug-in estimator in empirical process theory, which plugs in the empirical CDF; we plug in statistical estimates.
See Algorithm 1 for a summary of the procedure
Comparison of empirical Gateaux derivative and augmented inverse propensity weighting.
We have argued that, in the context of statistical estimation, approximating Gateaux derivatives requires plug-in estimation of the statistical functional with estimated probabilities.
Towards resolving
Q1, we observe that the perturbed probability densities induce the functional form of the nuisance functions in the analytical derivative.
This is in contrast to the analytical derivative, which describes the estimand in terms of, e.g., conditional expectations, and allows the analyst to choose the functional form for estimating the nuisance function.
We will denote the ˜ P -induced conditional expectation as
[Equation]
For example, the density-induced conditional expectation of kernel density estimates is exactly
Nadaraya-Watson regression.
Given this, we study empirical derivatives with kernel density estimation because they provide a classic textbook example of density estimates that induce conditional expectation estimators.
Other density estimation approaches can be used and their rates of convergence would simply be substituted in Lemma 1.
In Appendix D we discuss other choices of density estimates.
Of course, all such nonparametric estimation approaches suffer from the curse of dimensionality, so additional structure is required to satisfy the product-rate conditions.
Our characterization will show exactly how plug-in evaluation with ﬁnite differences is nearly equivalent to evaluating the doubly-robust estimator with a Nadaraya-Watson regressor as the conditional outcome regression for E [ Y | A = 1 , X ] and a kernel density estimate of the propensity score.
It is equivalent up to an additive bias that arises due to the smoothed perturbation distribution, and what we will term a smoothed nuisance evaluation .
We discuss what this smoothed nuisance function is, and how we study its approximation error relative to estimation error of the induced nuisance.
Integrating the smoothed perturbation induces smoothed nuisance functions.
Integrating with respect to the smoothed perturbation distribution can be interpreted as a smoothed evaluation of the resulting nuisance functions.
We deﬁne the ˜   x 0 -smoothed conditional expectation which smooths the evaluation point, i.e., smooths evaluation around x 0 rather than precisely at x 0 .
Deﬁnition 1 ( ˜   x 0 -smoothed conditional expectation) .
[Equation]
A similar deﬁnition appears in the literature on nonregular inference or the localization of a global functional [57].
The following lemma, similar to van der Laan et al.
[72, Lemma 25.1], summarizes convergence rate implications of these smoothed nuisances; these are direct consequences of typical analyses of kernel density estimation.
Lemma 1.
Let ˜ µ ✏ ( X ) =
˜ E ˜ P ✏ [ Y | A = 1 , X ] and ˜ e ✏ ( X ) = ˜
p ✏ ( A = 1 , X ) / ˜ p
( X ) denote the nuisances induced by plug-in estimation of probability distributions ˜ p .
For any bounded function g ( x ) in a H¨older class of degree  , (
R ˜   x ( u ) g ( u ) d u  g ( x ))
d x = O (   ) .
Assume that ˜   is a product kernel in a H¨older class of degree  , and
[Equation]
Assume Y is bounded.
Then the perturbed nuisances satisfy the following rates:
[Equation]
The perturbed nuisances are asymptotically consistent, so long as ˜ p induces asymptotically consistent nuisances and we ﬁx ✏ as a sequence vanishing in n at a rate appropriate to counteract the growth in  due to “roughness”; i.e., the variability of ˜   .
The difference between the function evaluation and its smoothed nuisance arises from the bias analysis of kernel density estimators and the dimension-dependence of ensuring absolute continuity, but it is a non-stochastic, entirely deterministic argument.
While the analysis improves mildly in the dimension dependence on  compared to the general analysis of Carone et al.
[10, Thm. 5], this still highlights the generally unfavorable dependence of the numerical approach on the dimension due to smoothing.
1: Inputs: ˜ P probability density estimates, data { O i } 1: n , perturbation and smoothing ( ✏ ,  ) parameters, functional 2: for i = 1 , . . .
n do 3:
[Equation]
4: ˜
[Equation] O [Equation]
5: end for 6: Output: one-step adjusted estimate from empirical Gateaux derivatives, [Equation] 1 n [Equation]
o i )
Our ﬁrst proposition establishes the exact approximation error induced by ﬁnite differencing.
Proposition 1 (Gateaux derivative of probability density representation) .
Consider perturbations in the direction of o i =
( x i , a i , y i ) .
Let ˜ p
( x ) ,
˜ p ( A = 1 , x ) , ˜ p ( y, A = 1 , x ) denote kernel density estimates.
Let ( P ) be as in Equation (2) :
[Equation]
We compare the expansion of Proposition 1 to the canonical AIPW estimator in Example 2.
We specialize to a uniform kernel for ˜   y i ( y )
so that R y ˜
y i
( y )
d y = y i .
We next observe that the smoothing kernel introduces additive bias relative to evaluating terms in the expression at the observation o i .
Corollary 1.
When the perturbation observation is the observation datapoint O i =
( X i , A
i , Y i ) , ˜
[Equation]
O i
[Equation]
=
[Equation]
Lemma 1 allows us to further simplify and conclude that the decomposition of Proposition 1 is close to the canonical AIPW estimator up to the O (   ) bias induced by smoothed evaluation.
Implications for estimation Next, we use our characterization in Proposition 1, Lemma 1, and
Corollary 1 to study the statistical properties of our computational/numerical approximation.
Corollary 1 arises from deterministic equivalences and allows us to study the relationship to the typical “oracle” AIPW estimator; albeit with induced ✏ -perturbed nuisances.
An appealing property of bias-adjusted treatment effect estimation is rate double robustness : because we incur the product of convergence rates of the nuisances, we may enjoy parametric n  12 rate convergence of the target functional while nuisances converge at slower rates, for example at n  14 ; see [13, 63].
We use the rate double robustness property of the target and Lemma 1 to infer the required rate conditions on ( ✏ ,  ) that retain the beneﬁcial statistical properties of the canonical AIPW estimator.
The next result combines our previous characterizations with the standard analysis of AIPW.
Assumption 1 (Regularity conditions) .
Assume the following regularity conditions hold:
(i) Y is bounded.
(ii) p ( y |
A = 1 , x ) , p ( A = 1 | x ) are in H¨older classes of minimum degree  .
(iii)
The estimates p ( y, a, x ) , p ( a, x ) , p ( x ) belong to a Donsker function class.
(iv) Assume ˜ µ, ˜
e satisfy the product-rate condition: k ˜ µ ( X )  µ ( X ) k ⇥ k
˜ e ( X )  e ( X ) k
= O p ( n  12 ) .
Assume ˜ µ, ˜ e are RMSE-consistent with r µ , r e , respectively, so that r µ + r e  12 .
Theorem 1 (Rate double robustness) .
Consider the one step-estimator with the empirical Gateaux derivative.
Under Assumption 1, when ✏  d/ 2 = o ( n  max( r µ ,r e ) ) , and   = o ( n  12 ) :
[Equation]
We now interpret the qualitative estimation implications of the analysis.
Theorem 1 states the rate conditions required of numerical approximation in ✏ ,  to preserve the statistical property of rate[-0.01 0.001 0.0001 1e0. 1e1e+00 9e-01 9e-01 9e-01 9e5e-01 3e-01 3e-01 3e-01 3e4e-01 4e-02 4e-02 4e-02 4e4e-01 7e-03 4e-
-]
[Figure]
double robustness.
Assuming that the nuisances induced by the probability density estimates satisfy a product-rate condition that the product of their RMSEs is faster than the parametric O ( n  12 ) rate, the form of Proposition 1 suggests that the rate of RMSE convergence of perturbed nuisances is required to be faster than the rates of the unperturbed nuisance functions, max( r µ , r e ) , which can be a slower rate than the ✏ = o ( n  12 ) implied by the generic analysis of ﬁnite differences.
For a direct comparison, the rate of Carone et al.
[10, Thm. 5] suggests a rate requirement of  ✏  d s +
= o ( n  12 ) .
In general, the dimension dependence on  in Lemma 1 will prevent setting a slower rate for ✏ for nontrivial dimensionality.
Note that without the smoothing requirement, the potential improvement in the rate of ✏ could be on the order of generic rate improvements implied by a central difference scheme.
That is, we would have attained the approximation beneﬁts of generically approximationimproved ﬁnite-difference schemes via the specialized statistical structure of the target adjustments.
Applications: Mean Potential Outcome and Dynamic Settings
As empirical illustrations, we ﬁrst show a plot of numerical approximation error ranging over ( ✏ ,  ) in the simple case of the mean potential outcome.
We then apply the tools that we derived in the previous section to analyze more complicated settings.
5.1 Mean potential outcome
We ﬁrst conduct a validation of our decomposition and theoretical characterization.
Our datagenerating process includes a piecewise linear outcome model (such that kernel regression is misspeciﬁed).
We conduct plug-in evaluation of the mean potential outcome with kernel density estimates.
In Figure 1a we include an ( ✏ ,  ) -plot for the simple case of AIPW (see [10] for more discussion and examples).
We consider a one-dimensional case with uniformly distributed X , piecewiselinear Y , and smooth propensity scores that are logistic in sin( X ) .
We use n = 500 and ﬁx the bandwidth h = 0 . 05 .
Colors denote magnitude of the mean absolute error, included in text on the heatmap.
Without loss of generality, we study the estimation of a mean under missingness, E [ Y (1)] .
Figure 1b illustrates the estimation error of various strategies with the comparable kernel-based estimates (DM is regression adjustment).
Note that even in this simple setting AIPW offers beneﬁts and sample efﬁciency relative to DM.
We include further numerical experiments in the Appendix.
5.2 Dynamic treatment regimes
We study the estimation properties of the empirical Gateaux derivative for multi-stage DTR from the probability density representation.
In T -stage dynamic treatment regimes, the causal quantity of interest is the mean potential outcome E
[ Y (¯ a )] , where ¯ a =
( a 0 , . .
. , a T ) is the (deterministic) treatment strategy.
Assuming Y (¯ a ) is sequentially ignorable given the treatment and covariate history, ( ¯ A t , ¯ X t ) , at each time t , this causal quantity can be identiﬁed by the g -formula,
[Equation]
where ¯ A =
( A 0 , . . .
, A t ) ,
¯ X =
( X 0 , . . .
, X t ) .
To derive its inﬂuence function, we take the empirical
Gateaux derivative by considering the perturbation in the direction of o i =
(¯ x i , ¯ a i , y i ) , where ¯ x i =
( x i 0 , . . .
, x it ) and ¯ a i = ( a i 0 , . . .
, a it ) , ( ˜ P i ✏
[Equation]
Below we characterize how this empirical Gateaux derivative at the smoothed distribution differs from the one at the (unsmoothed) estimated distribution.
The following result is analogous to Proposition 1 and Corollary 1 but is extended to the dynamic treatment regime. Proposition 2.
[Equation]
¯ a i,T  s , ¯ x i,T [Equation] s ⇤ + O (
[Equation]
+ O ( [Equation]  ) The expansion veriﬁes that the requirements for numerical approximation are similar in ✏ as in the case of a single-timestep mean potential outcome, despite the additional complexity of the dynamic treatment regime function as a nested expectation.
Again, smoothing incurs overall unfavorable dependence in the dimension, as the extension of Lemma 1 to E ˜ P ✏
[ Y | ¯ A, ¯ X ] also incurs unfavorable dependence on the dimension.
The history-dependent nuisances imply that the dimension additionally grows in the time horizon.
5.3 Perturbations for (constrained) inﬁnite-horizon off-policy evaluation
We show how this framework can be applied to derive generic bias corrections for inﬁnite-horizon off-policy optimization in ofﬂine reinforcement learning, via the canonical linear programming characterization [17, 61].
Our application is relevant to a recent line of work on off-policy evaluation and learning (OPE/L) [5, 43, 47–49, 67, 68].
Setup.
We recall the classical linear programming formulation of ﬁnding an optimal policy in a tabular (ﬁnite-state, ﬁnite-action) Markov decision process, due to De Farias and Van Roy [17], Puterman [61].
Recent work has revisited this formulation with interest in developing primal-dual algorithms [1, 55, 65, 74].
The dynamics follow an inﬁnite-horizon Markov decision process with discount factor  .
The ofﬂine dataset is comprised of trajectories of state, action, next state observations: { ( s 0 i , a 0 i , . . .
, s ti , . . .
, s Ti ) } 1: N .
We derive our bias adjustment under a statistical model where the stationary distribution factorizes as p ( s, a, s 0 ) =
d ( s, a ) P ( s | s, a ) .
From the joint stateaction-state occupancy distribution p ( s, a, s 0 ) , we estimate the marginalization d ( s, a ) , the stationary state-action occupancy distribution, which we use to estimate the transition probability
P ( s 0 | a, s ) = p ( s,a,s 0 )
d ( s,a ) .
Let µ 0 ( s ) denote the initial state distribution, estimated from ofﬂine data and assumed to be equivalent in the original confounded and evaluation setting.
Because we focus on the tabular case with discrete distributions, we do not require smoothing.
The optimal policy linear program.
The optimal value function solves the following linear program in a ﬁnite-state and ﬁnite-action setting; the objective value at optimality is the policy value.
Here, P a 2 R | S | ⇥ | S | is transition matrix at action a , with P a ( s, s 0 )
=
P ( s 0 | s, a ) , µ 0 ( s ) is the marginal occupancy distribution of the initial state.
We assume, as is fairly common in theoretical analyses of reinforcement learning, that the reward function is a known function of state and action, r ( s, a ) .
Let r a
=
r
( s, a ) } s 2 S be the vector of reward values per state, for ﬁxed action a .
We have:
[Equation]
The dual formulation is well known to parametrize the stationary occupancy probabilities of the optimal policy (at optimality):
[Equation]
We verify the bias correction from the Gateaux derivative dd ✏ D ( P ✏ )  ✏ =0 .
Note that evaluating this functional at the perturbed distribution evaluates the linear program with perturbed left-hand side coefﬁcients; i.e., perturbing the constraints of the coefﬁcient matrix as well as the objective (marginalizing over a perturbed initial state distribution.
We evaluate the Gateaux derivative in the direction of (˜ s, ˜ a, ˜ s 0 ) .
We assume nondegeneracy (unique optimal solution); this can be relaxed.
Assumption 2.
The optimal solution is unique; there is no degeneracy.
Proposition 3.
Perturb in the direction of a generic observation (˜ s, ˜ a, ˜ s 0 ) :
[Equation]
Sketch of argument.
The statement of Proposition 3 is not surprising since we can immediately verify the mean-zero sample version of the Bellman residual, and so one might plausibly derive this form from double robustness.
Rather, we provide an alternative argument based on the sensitivity analysis characterization of [25].For ✏ small enough, the active basis remains the same.
When the perturbation matrix is P + ✏ G for some matrix G , [25] notes that the derivative of the optimal value is d 0 ( ✏ ) =
⇡ Gx (where ( ⇡ , x ) are the dual and primal optimal solutions, respectively) can be obtained from the derivative of a matrix inverse, which yields higher-order expansions.
We next compute the approximation error that arises from evaluating ﬁnite differences, so long as ✏ is small enough to maintain the same active basis, which is empirically veriﬁable.
Proposition 4 (Error analysis from ﬁnite differencing) .
Perturb in the direction of a generic observation (˜ s, ˜ a, ˜ s 0 ) .
Then:
[Equation]
In this setting, unlike the earlier case of the treatment mean in Corollary 1 and Theorem 1, even in a favorable discrete-state case without smoothing requirements, rate double robustness does not admit weaker numerical requirements on ✏ .
In the analysis, eq. (45) shows that V ✏ ( s )  V ( s )
=
O ( ✏ ) , so that Proposition 4 holds with the unperturbed nuisances and the same order of approximation error.
Generalization to constraints.
We argue the main beneﬁt of this approach is that by computing these ﬁnite differences, we can obtain generic bias corrections via a more broadly applicable argument that does not require interpretation of the closed-form solution to an optimization problem.
For example, consider a relevant subclass of constrained Markov decision processes comprised of additional constraints on the policy variables: the linear programming formulation is particularly appealing because linear constraints can be directly added [3].
The additional constraint µ 2 P reﬂects linearly representable constraints on state-occupancy under the optimal policy.
For example, relevant constraints could require the occupancy in certain risky states to be bounded.
[59] develops a framework with a “caution function,” a convex function on the state-action occupancy measures.
Example 3 (Constrained policy optimization) .
[Equation]
Conclusions
We have presented a constructive algorithm that approximates Gateaux derivatives ﬁnite differencing, with a focus on the statistical functionals used in causal inference.
There are several directions which are natural follow-ups to this work.
First, the smoothing perturbation, although it restores absolute continuity, introduces technical barriers in the implied approximation rates with regards to dimension dependence.
Another useful direction would be to explore methods that are fully adaptive, not requiring any prior knowledge of beneﬁcial statistical properties.
Finally, our work has been mainly theoretical, and it would be useful to conduct further empirical comparisons.
Acknowledgements.
Angela Zhou gratefully acknowledges support from the Foundations of Data
Science Institute and the Simons Institute’s Program on Causality.
Part of this work was done while the author was visiting the Simons Institute for the Theory of Computing.
Yixin Wang acknowledges
National Science Foundation Grant NSF-CHE-2231174.
Michael Jordan acknowledges support from the Vannevar Bush Faculty Fellowship program under grant number N00014-21-1-2941.
[-d(cid:15) (cid:12)(cid:12)(cid:12) (cid:15) =
-]