Original: Off-Policy Evaluation with Policy-Dependent Optimization Response
Revised: Off-Policy Evaluation with Policy-Dependent Optimization Response

Original: Anonymous Author(s)

Original: AfﬁliationAddress

Original: Abstract
Revised: Abstract

Original: The intersection of causal inference and machine learning for decision-makingis rapidly expanding, but the default decision criterion remains an average ofindividual causal outcomes across a population.
Revised: The intersection of causal inference and machine learning for decision-making is rapidly expanding, but the default decision criterion remains an average of individual causal outcomes across a population.

Original: In practice, various operationalrestrictions ensure that a decision-maker’s utility is not realized as an average butrather as an output of a downstream decision-making problem (such as matching, assignment, network ﬂow, minimizing predictive risk).
Revised: In practice, various operational restrictions ensure that a decision-maker’s utility is not realized as an average but rather as an output of a downstream decision-making problem (such as matching, assignment, network ﬂow, minimizing predictive risk).

Original: In this work, wedevelop a new framework for off-policy evaluation with policy-dependent linear optimization responses: causal outcomes introduce stochasticity in objectivefunction coefﬁcients.
Revised: In this work, we develop a new framework for off-policy evaluation with policy-dependent linear optimization responses: causal outcomes introduce stochasticity in objective function coefﬁcients.

Original: Under this framework, a decision-maker’s utility dependson the policy-dependent optimization, which introduces a fundamental challengeof optimization bias even for the case of policy evaluation.
Revised: Under this framework, a decision-maker’s utility depends on the policy-dependent optimization, which introduces a fundamental challenge of optimization bias even for the case of policy evaluation.

Original: We construct unbiased estimators for the policy-dependent estimand by a perturbation method, anddiscuss asymptotic variance properties for a set of adjusted plug-in estimators.
Revised: We construct unbiased estimators for the policy-dependent estimand by a perturbation method, and discuss asymptotic variance properties for a set of adjusted plug-in estimators.

Original: Lastly, attaining unbiased policy evaluation allows for policy optimization: weprovide a general algorithm for optimizing causal interventions.
Revised: Lastly, attaining unbiased policy evaluation allows for policy optimization: we provide a general algorithm for optimizing causal interventions.

Original: We corroborateour theoretical results with numerical simulations.
Revised: We corroborate our theoretical results with numerical simulations.

Original:  Introduction
Revised:  Introduction

Original: The interface of causal inference and machine learning offers to “deliver the right intervention,at the right time, to the right person”.
Revised: The interface of causal inference and machine learning offers to “deliver the right intervention, at the right time, to the right person”.

Original: An extensive line of research studies off-policy evaluation(OPE) and learning—evaluating the average causal outcomes under alternative personalized treatmentassignment policies that differ from the treatment assignment which generated the data (and may haveintroduced confounding) so that one may optimize over the best such treatment rule [Manski, 2004,
Revised: An extensive line of research studies off-policy evaluation (OPE) and learning—evaluating the average causal outcomes under alternative personalized treatment assignment policies that differ from the treatment assignment which generated the data (and may have introduced confounding), so that one may optimize over the best such treatment rule [Manski, 2004,

Original: Dudík et al., 2011, Zhao et al., 2012, Thomas et al., 2015, Athey et al., 2017, Kitagawa and Tetenov,2018, Kallus and Zhou, 2018b].
Revised: Dudík et al., 2011, Zhao et al., 2012, Thomas et al., 2015, Athey et al., 2017, Kitagawa and Tetenov, 2018, Kallus and Zhou, 2018b].

Original: Most of this work is based on the assumption that the appropriatedecision criterion is an average of individuals across a population.
Revised: Most of this work is based on the assumption that the appropriate decision criterion is an average of individuals across a population.

Original: But various operational restrictionsor settings imply that a decision-maker’s utility is often not realized as an average but rather as anoutput of a downstream planning or decision-making problem.
Revised: But various operational restrictions or settings imply that a decision-maker’s utility is often not realized as an average but rather as an output of a downstream planning or decision-making problem.

Original: For example, in studying the effects of price incentives in a matching market (e.g., on a rideshare platform), a ﬁrm’s revenue is not realized until it matches riders to drivers under certainconstraints [Mejia and Parker, 2021, Ma et al., 2021].
Revised: For example, in studying the effects of price incentives in a matching market (e.g., on a rideshare platform), a ﬁrm’s revenue is not realized until it matches riders to drivers under certain constraints [Mejia and Parker, 2021, Ma et al., 2021].

Original: While the marketplace may offer incentives todrive or accept rides and induce causal effects on individuals, the ﬁnal utility is determined by thenew matches, taking into account operational constraints and structure.
Revised: While the marketplace may offer incentives to drive or accept rides and induce causal effects on individuals, the ﬁnal utility is determined by the new matches, taking into account operational constraints and structure.

Original: As another example, although job training (and personalized provision thereof) is commonly toutedin causal inference and machine learning papers as a promising example for personalized treatmentpolicy assignment [Athey et al., 2017, Kitagawa and Tetenov, 2018, Knaus et al., 2020], labor  that  possible existence of equilibrium effects on the efﬁciency of the programs seems quite real” [Crépon and Van Den Berg, 2016, p.541].
Revised: As another example, although job training (and personalized provision thereof) is commonly touted in causal inference and machine learning papers as a promising example for personalized treatment policy assignment [Athey et al., 2017, Kitagawa and Tetenov, 2018, Knaus et al., 2020], labor economists voice a general concern that “the possible existence of equilibrium effects on the efﬁciency of the programs seems quite real” [Crépon and Van Den Berg, 2016, p.541].

Original: The equilibrium concernis that personalized provision of job training may not lead to actual beneﬁcial gains at the populationlevel due to externalities (substitution effects/congestion in matching) of the labor search processin a ﬁnite market.
Revised: The equilibrium concern is that personalized provision of job training may not lead to actual beneﬁcial gains at the population level due to externalities (substitution effects/congestion in matching) of the labor search process in a ﬁnite market.

Original: While impressive cluster-randomized trials have been deployed to assess theseeffects [Crépon et al., 2013], it would be useful if there exists a framework that can model theequilibrium effect and evaluate treatment policies directly based on available data of individual-levelcausal effects.
Revised: While impressive cluster-randomized trials have been deployed to assess these effects [Crépon et al., 2013], it would be useful if there exists a framework that can model the equilibrium effect and evaluate treatment policies directly based on available data of individual-level causal effects.

Original: In some settings, population-level impacts may be well-modeled as a downstreamoptimization response.
Revised: In some settings, population-level impacts may be well-modeled as a downstream optimization response.

Original: The development of such a framework is our focus in the current paper.
Revised: The development of such a framework is our focus in the current paper.

Original: We study a new framework for policy evaluation and optimization where there is a personalizedtreatment policy on individual-level outcomes, and a policy-dependent optimization response.
Revised: We study a new framework for policy evaluation and optimization where there is a personalized treatment policy on individual-level outcomes, and a policy-dependent optimization response.

Original: Thekey difference between this model and previous work on off-policy evaluation and optimization isthat: although treatments realize causal effects on individuals , a treatment policy’s value depends ona further downstream policy-dependent optimization.
Revised: The key difference between this model and previous work on off-policy evaluation and optimization is that: although treatments realize causal effects on individuals , a treatment policy’s value depends on a further downstream policy-dependent optimization.

Original: We study how to evaluate different policieswithout bias (off-policy evaluation) and how to optimize for the optimal policy under this framework(policy optimization).
Revised: We study how to evaluate different policies without bias (off-policy evaluation) and how to optimize for the optimal policy under this framework (policy optimization).

Original: Our contributions are as follows: we ﬁrst introduce the model of policy-dependent optimizationresponse, which we formulate as a nonconvex stochastic optimization problem.
Revised: Our contributions are as follows: we ﬁrst introduce the model of policy-dependent optimization response 2 , which we formulate as a nonconvex stochastic optimization problem.

Original: For off-policyevaluation, we develop a framework of policy-dependent optimization response , decompose the biasthat arises in this framework (“optimization bias”) and show how to control it via the design ofestimators for the policy-dependent estimand.
Revised: For off-policy evaluation, we develop a framework of policy-dependent optimization response , decompose the bias that arises in this framework (“optimization bias”) and show how to control it via the design of estimators for the policy-dependent estimand.

Original: Finally, we provide a general algorithm for optimizingcausal interventions.
Revised: Finally, we provide a general algorithm for optimizing causal interventions.

Original: We corroborate the theoretical results with experimental comparisons.
Revised: We corroborate the theoretical results with experimental comparisons.

Original: 1.1 Related work
Revised: 1.1 Related work

Original: We highlight the most relevant work from causal inference, off-policy evaluation, and optimizationunder uncertainty in the main text.
Revised: We highlight the most relevant work from causal inference, off-policy evaluation, and optimization under uncertainty in the main text.

Original: We include additional or tangential discussion in Appendix A.
Revised: We include additional or tangential discussion in Appendix A.

Original: There is an extensive literature on off-policy evaluation and optimization [see, e.g., Manski, 2004,
Revised: There is an extensive literature on off-policy evaluation and optimization [see, e.g., Manski, 2004,

Original: Dudík et al., 2011, Zhao et al., 2012, Swaminathan and Joachims, 2015].
Revised: Dudík et al., 2011, Zhao et al., 2012, Swaminathan and Joachims, 2015].

Original: Relative to this line ofwork, we focus on the introduction of a downstream decision response, arising for example fromoperational constraints.
Revised: Relative to this line of work, we focus on the introduction of a downstream decision response, arising for example from operational constraints.

Original: The case of constrained policies has been considered in the OPE literature.
Revised: The case of constrained policies has been considered in the OPE literature.

Original: Our setting is conceptuallydifferent but overlaps in some application contexts.
Revised: Our setting is conceptually different but overlaps in some application contexts.

Original: Speciﬁcally, we decouple the downstreampolicy-dependent response , i.e. over a similar constraint space, from treatment decisions that havecausal effects.
Revised: Speciﬁcally, we decouple the downstream policy-dependent response , i.e. over a similar constraint space, from treatment decisions that have causal effects.

Original: For example, Bhattacharya [2009] studies the setting of “roommate assignment”with discrete types; i.e., perfect bipartite matching.
Revised: For example, Bhattacharya [2009] studies the setting of “roommate assignment” with discrete types; i.e., perfect bipartite matching.

Original: A crucial difference is that in their setting, thecausal treatment of interest is the assignment decision to the other individual type; while in oursetting the causal treatment only affects certain parameters of the assignment decision, such as edgecosts.
Revised: A crucial difference is that in their setting, the causal treatment of interest is the assignment decision to the other individual type; while in our setting the causal treatment only affects certain parameters of the assignment decision, such as edge costs.

Original: We instantiate an analogous example in our framework to highlight our decoupled causalintervention and prediction decisions.
Revised: We instantiate an analogous example in our framework to highlight our decoupled causal intervention and prediction decisions.

Original: Consider a setting with a causal intervention, such as a diversityinformation intervention affecting a student’s probability of getting along with various types.
Revised: Consider a setting with a causal intervention, such as a diversity information intervention affecting a student’s probability of getting along with various types.

Original: Here thepolicy performs treatments on individual’s diversity information, and the ﬁnal assignment decision ispolicy-dependent response.
Revised: Here the policy performs treatments on individual’s diversity information, and the ﬁnal assignment decision is policy-dependent response.

Original: Other work considers resource-budgeted allocation Kube et al.
Revised: Other work considers resource-budgeted allocation Kube et al.

Original: [2019],which is structured because of reformulation of thresholds Lopez et al.
Revised: [2019], which is structured because of reformulation of thresholds Lopez et al.

Original: Sun [2021] studiessharp asymptotics for the additional challenge of stochasticity in the budget.
Revised: Sun [2021] studies sharp asymptotics for the additional challenge of stochasticity in the budget.

Revised: Some works that illustrate the embedding of causal effect estimates in optimization-based decision problems include Rahmattalabi et al.

Revised: [2022], although their formulation is ultimately a mixed-integer optimization.

Original: In contrast to an extensive line of work on heterogeneous causal effect estimation [Shalit et al., 2017,
Revised: In contrast to an extensive line of work on heterogeneous causal effect estimation [Shalit et al., 2017,

Original: Wager and Athey, 2018, Künzel et al., 2019], often crucially leveraging simpler structure of thetreatment contrast rather than the conditional outcomes, in this work we require estimation of the latterdue to the downstream optimization and distributional convergence for the perturbation method.
Revised: Wager and Athey, 2018, Künzel et al., 2019], often crucially leveraging simpler structure of the treatment contrast rather than the conditional outcomes, in this work we require estimation of the latter due to the downstream optimization and distributional convergence for the perturbation method.

Original: Inturn, combining causal outcome estimation with adjustments for optimization bias requires differentproperties of the estimation strategy, namely plug-in estimation of a modiﬁed regression model;we focus on estimators that modify the ﬁrst-order conditions of a regression model to algebraicallyachieve an AIPW-type adjustment as discussed in Bang and Robins [2005].
Revised: In turn, combining causal outcome estimation with adjustments for optimization bias requires different properties of the estimation strategy, namely plug-in estimation of a modiﬁed regression model; we focus on estimators that modify the ﬁrst-order conditions of a regression model to algebraically achieve an AIPW-type adjustment as discussed in Bang and Robins [2005].

Original: See also Scharfstein et al.[1999], Tran et al.
Revised: See also Scharfstein et al. [1999], Tran et al.

Original: [2019], Shi et al.
Revised: [2019], Shi et al.

Original: [2019], Chernozhukov et al.
Revised: [2019], Chernozhukov et al.

Original: This work focuses on the challenge of optimization bias for policy evaluation introduced in our setting,for generic linear optimization problems.
Revised: This work focuses on the challenge of optimization bias for policy evaluation introduced in our setting, for generic linear optimization problems.

Original: This well-known challenge of in-sample optimization bias(“sample average approximation bias”) fundamentally demarcates the statistical regime of optimiza- tion under uncertainty from sample mean estimation [Bayraksan and Morton, 2006, Shapiro et al.,2021].
Revised: This well-known challenge of in-sample optimization bias (“sample average approximation bias”) fundamentally demarcates the statistical regime of optimization under uncertainty from sample mean estimation [Bayraksan and Morton, 2006, Shapiro et al., 2021].

Original: Recent work develops bagging, jackknife, perturbation and variance-corrected perturbationapproaches for bias adjustment [Lam and Qian, 2018, Ito et al., 2018, Kannan et al., 2020, Gupta et al.,2021].
Revised: Recent work develops bagging, jackknife, perturbation and variance-corrected perturbation approaches for bias adjustment [Lam and Qian, 2018, Ito et al., 2018, Kannan et al., 2020, Gupta et al., 2021].

Original: We extend a perturbation method of Ito et al.
Revised: We extend a perturbation method of Ito et al.

Original: [2018] to the setting of nonlinear predictions.
Revised: [2018] to the setting of nonlinear predictions.

Original:  Preliminaries
Revised:  Preliminaries

Original: We ﬁrst deﬁne the setting for off-policy evaluation with policy-dependent responses.
Revised: We ﬁrst deﬁne the setting for off-policy evaluation with policy-dependent responses.

Original: We distinguishbetween the causal decision policy π and the downstream optimization response x .
Revised: We distinguish between the causal decision policy ⇡ and the downstream optimization response x .

Original: The causaldecision policy π intervenes on individual units, while the policy-dependent responses are solutionsto a downstream optimization problem on the causal responses of all the units.1.
Revised: The causal decision policy ⇡ intervenes on individual units, while the policy-dependent responses are solutions to a downstream optimization problem on the causal responses of all the units.

Original: Off-policy evaluation.
Revised: Off-policy evaluation.

Original: We ﬁrst describe the single time step off-policy policy evaluation andoptimization problem [see Dudík et al., 2014, Hirano and Porter, 2020, for further context].
Revised: We ﬁrst describe the single time step off-policy policy evaluation and optimization problem [see Dudík et al., 2014, Hirano and Porter, 2020, for further context].

Original: Letcovariates be W ∈ W ⊆
Revised: Let covariates be W 2 W ✓

Original: R d , binary treatment be T ∈ { 0 , 1 } 1 , and potential outcomes be c ( T ) .
Revised: R d , binary treatment be T 2 { 0 , 1 } 3 , and potential outcomes be c ( T ) .

Original: Denote the covariates’ distribution as P .
Revised: Denote the covariates’ distribution as P .

Original: Without loss of generality we consider lower is better forc ; e.g. we minimize costs.
Revised: Without loss of generality we consider lower is better for c ; e.g. we minimize costs.

Original: We consider a setting of learning causal responses from a dataset oftuples D 1 = { ( W i , T i , c i ) } n i =1 where treatment is assigned randomly or in an observational setting;henceforth we call this the observational / experimental dataset .
Revised: We consider a setting of learning causal responses from a dataset of tuples D 1 = { ( W i , T i , c i ) } ni =1 where treatment is assigned randomly or in an observational setting; henceforth we call this the observational / experimental dataset .

Original: We let π t : W (cid:55)→ [0 , 1] denote a personalized policy mapping from covariates to a (probability of)treatment t .
Revised: We let ⇡ t : W 7! [0 , 1] denote a personalized policy mapping from covariates to a (probability of) treatment t .

Original: Later we will focus on parameterized policies, such as π t ( w )
Revised: Later we will focus on parameterized policies, such as ⇡ t ( w )

Original: = sigmoid ( ϕ (cid:62) w ) orpolicies that admit global enumeration.
Revised: = sigmoid ( ' > w ) or policies that admit global enumeration.

Original: The goal of off-policy learning is to optimize the causalinterventions (aka policies) by estimating average outcomes induced by any given policy.
Revised: The goal of off-policy learning is to optimize the causal interventions (aka policies) by estimating average outcomes induced by any given policy.

Original: Throughoutwe will follow the convention that, a random variable c ( π t ) denotes c ( π t ) =
Revised: Throughout we will follow the convention that, a random variable c ( ⇡ t ) denotes c ( ⇡ t ) =

Original: c ( t ) I ( Z t = t ) , where
Revised: c ( t ) I ( Z t = t ) , where

Original: Z π ∈ { 0 , 1 } is a Bernoulli random variable of policy assignment: Z π ∼ Bern ( π 1 ) .
Revised: Z ⇡ 2 { 0 , 1 } is a Bernoulli random variable of policy assignment: Z ⇡ ⇠ Bern ( ⇡ 1 ) .

Original: Then, the(random) outcome for a given covariate with policy π is:
Revised: Then, the (random) outcome for a given covariate with policy ⇡ is:

Original: [Equation]
Revised: [Equation]

Original: The average treatment effect (ATE) of a policy π ( · ) is then E [ c ( π )] , where the expectation is takenover the randomness of the covariates W ∼ P , assignments induced by π , and c conditional onrealized treatment t and covariates.2.
Revised: The average treatment effect (ATE) of a policy ⇡ ( · ) is then E [ c ( ⇡ )] , where the expectation is taken over the randomness of the covariates W ⇠ P , assignments induced by ⇡ , and c conditional on realized treatment t and covariates.

Original: Policy-dependent responses.
Revised: Policy-dependent responses.

Original: Policy-dependent optimization solves a downstream stochasticlinear optimization problem over a decision problem x ∈ X ⊆ R m on the m units given a causalintervention policy. 
Revised: Policy-dependent optimization solves a downstream stochastic linear optimization problem over a decision problem x 2 X ✓ R m on the m units given a causal intervention policy. In particular, m represents the dimension of the downstream decision problem.

Original: Relative to the downstream decision problem, causal outcomes may enter eitheras uncertain objective coefﬁcients (in c ) or constraint capacities (in b ).
Revised: Relative to the downstream decision problem, causal outcomes may enter either as uncertain objective coefﬁcients (in c ) or constraint capacities (in b ).

Original: Dimensionality of the responses.
Revised: Dimensionality of the responses.

Original: We consider two different asymptotic regimes: an out-of-sample,ﬁxed-dimension, ﬁxed- m regime and an in-sample, growing-dimension, growing- n regime.
Revised: We consider two different asymptotic regimes: an out-of-sample, ﬁxed-dimension, ﬁxed- m regime and an in-sample, growing-dimension, growing- n regime.

Original: Weformalize the former regime, the main focus of the paper, in the following assumption.
Revised: We formalize the former regime, the main focus of the paper, in the following assumption.

Original: Assumption 2.1 (Out-of-sample, ﬁxed-dimension regime) .
Revised: Assumption 2.1 (Out-of-sample, ﬁxed-dimension regime) .

Original: As n → ∞ , the dimension of theoptimization problem m , given by a new draw of contexts D 2 = { W i } 1: m remains ﬁnite.
Revised: As n ! 1 , the dimension of the optimization problem m , given by a new draw of contexts D 2 = { W i } 1: m remains ﬁnite.

Original: Thedecision-dependent response on m units is measurable with respect to D 2 .
Revised: The decisiondependent response on m units is measurable with respect to D 2 .

Original: The policy value v ∗ π is:
Revised: Let c i ( ⇡ ) def = E [ c ( ⇡ ( w ) | w = W i ] , we have that the policy value v ⇤ ⇡ is:

Original: [Equation]
Revised: [Equation]

Original: Assumption 2.1 deﬁnes our policy-dependent estimand in this regime.
Revised: Assumption 2.1 deﬁnes our policy-dependent estimand in this regime.

Original: The expectation is taken overthe randomness of the policy π and the randomness of the ﬁnite samples { w i } mi =1 .
Revised: The expectation is taken over the randomness of the policy ⇡ and the randomness of the ﬁnite samples { w i } mi =1 .

Original: The main textfocuses on statements in the regime of Assumption 2.1.
Revised: The main text focuses on statements in the regime of Assumption 2.1.

Original: Evaluating regret with respect to a ﬁxeddimension is standard or implicit in the predictive optimization literature.
Revised: Evaluating regret with respect to a ﬁxed dimension is standard or implicit in the predictive optimization literature.

Original: Assumption 2.2 (In-sample, growing-dimension regime) .
Revised: Assumption 2.2 (In-sample, growing-dimension regime) .

Original: As n → ∞ , the limit of the objectivefunction is an expectation over contexts.
Revised: As n ! 1 , the limit of the objective function is an expectation over contexts.

Original: The estimand is:
Revised: The estimand is:

Original: [Equation]
Revised: [Equation]

Revised: Recall that a policy maps from covariates to a (probability of) treatment.

Revised: Assumption 2.2 precisely takes an expectation over the two sources of randomness: the outer expectation is taken over the randomness of the policy, and the inner expectation is taken over the randomness of the covariates w .

Original: The limiting object in the growing-dimension regime is a “ﬂuid limit” or asymptotic regime:
Revised: The limiting object in the growing-dimension regime is a “ﬂuid limit” or asymptotic regime:

Original: informally we assume a meaningfully constrained optimization in the limit.
Revised: informally we assume a meaningfully constrained optimization in the limit.

Original: We instantiate our frameworkin the following example.
Revised: We instantiate our framework in the following example.

Original: Example 2.3 (Min-cost bipartite matching) .
Revised: Example 2.3 (Min-cost bipartite matching) .

Original: Our framework is precisely motivated by the practicalchallenges in causal inference tasks, where the problem of “policy dependent" optimizations pops uprepeatedly.
Revised: Our framework is precisely motivated by the practical challenges in causal inference tasks, where the problem of “policy dependent" optimizations pops up repeatedly.

Original: For instance, for price incentives in a matching market (such as a rideshare platform), therevenue/welfare outcome is not realized until the riders and drivers are matched under constraints.
Revised: For instance, for price incentives in a matching market (such as a rideshare platform), the revenue/welfare outcome is not realized until the riders and drivers are matched under constraints.

Original: Asanother example, consider a manager wants to assign agents to different jobs, and assigning an agentto a job is associated with some cost.
Revised: As another example, consider a manager wants to assign agents to different jobs, and assigning an agent to a job is associated with some cost.

Original: Our goal is to assign each agent to at most one job such thatthe overall cost is minimized.
Revised: Our goal is to assign each agent to at most one job such that the overall cost is minimized.

Original: To incentivize the workers to complete the jobs, the company mightwant to provide some bonus to the agents.
Revised: To incentivize the workers to complete the jobs, the company might want to provide some bonus to the agents.

Original: However, the overall efﬁciency and total payments are notrealized until all the assignments are determined.
Revised: However, the overall efﬁciency and total payments are not realized until all the assignments are determined.

Original: The above type of application can be modeled as a min-cost bipartite matching problem, which iswell known to have a totally unimodular linear relaxation.
Revised: The above type of application can be modeled as a min-cost bipartite matching problem, which is well known to have a totally unimodular linear relaxation.

Original: Clearly, the agents (or riders) and the jobs(or passenger requests) form the two sides of nodes for the matching.
Revised: Clearly, the agents (or riders) and the jobs (or passenger requests) form the two sides of nodes for the matching.

Original: The edge costs in the matchingstand for the cost or payment for an agent to complete that job.
Revised: The edge costs in the matching stand for the cost or payment for an agent to complete that job.

Original: A treatment ( T = 1 ) serves asintervention on the edge costs for that agent, and the covariates W could be any observable featuresof the agents, such as preferences, demographic information, etc.
Revised: A treatment ( T = 1 ) serves as intervention on the edge costs for that agent, and the covariates W could be any observable features of the agents, such as preferences, demographic information, etc.

Original: Given any allocation rule of thebonuses, the manager faces a downstream min-cost bipartite matching:
Revised: Given any allocation rule of the bonuses, the manager faces a downstream min-cost bipartite matching:

Original: [Equation]
Revised: [Equation]

Original: Here N ( i ) is the set of all edges contains node i , the c e are the edge costs, and x = { x e } e ∈Erepresents the matching where x e = 1 means that edge e is selected 5 .
Revised: Here N ( i ) is the set of all edges contains node i , the c e are the edge costs, and x = { x e } e 2 E represents the matching where x e = 1 means that edge e is selected 7 .

Original: In Appendix D we include an additional example of predictive risk optimization, beyond linearoptimization, which requires a different estimation strategy.3.
Revised: In Appendix E we include an additional example of predictive risk optimization, beyond linear optimization, which requires a different estimation strategy.

Revised: Out of sample, ﬁxed m (Assumption 2.1)

Revised: In-sample, growing n (Assumption 2.2, Appendix D) Evaluation

Original: Policy optimization with policy-dependent responses.
Revised: Policy optimization Evaluation Policy optimization

Original: Putting together the pieces of theprevious subsections, the off-policy optimization over candidate policies π ∈ Π is:
Revised: AIPW N/A [Table]

Original: [Equation]
Revised: [Equation]

Original: where m represents the dimension of the decision problem (e.g., the number of edges in Example 2.3),and x denotes the whole response vector { x i } i ∈ [ m ] .
Revised: where m represents the dimension of the decision problem (e.g., the number of edges in Example 2.3), and x denotes the whole response vector { x i } i 2 [ m ] .

Original: We illustrate this framework by revisiting our examples.
Revised: We illustrate this framework by revisiting our examples.

Original: Example 2.4 (Policy optimization for Example 2.3, min-cost matching) .
Revised: Example 2.4 (Policy optimization for Example 2.3, min-cost matching) .

Original: In the min-cost bipartitematching example, the optimal assignments with a given policy π can be solved via the linear programin Equation (3).
Revised: In the min-cost bipartite matching example, the optimal assignments with a given policy ⇡ can be solved via the linear program in Equation (3).

Original: Suppose that we want to ﬁnd the best intervention policy which gives the lowestmatching cost.
Revised: Suppose that we want to ﬁnd the best intervention policy which gives the lowest matching cost.

Original: Then, the policy optimization problem is:
Revised: Then, the policy optimization problem is:

Original: [Equation]
Revised: [Equation]

Original: where Π denotes the set of all policies that are of interest.
Revised: where ⇧ denotes the set of all policies that are of interest.

Original:  Problem Description: Optimization Bias
Revised:  Problem Description: Optimization Bias

Original: We focus on off-policy evaluation in view of the downstream optimization over the decision variablesx = { x i } i ∈ [ m ] .
Revised: We focus on off-policy evaluation in view of the downstream optimization over the decision variables x = { x i } i 2 [ m ] .

Original: We ﬁrst discuss plug-in estimation approaches without causal adjustment to introducethe challenge of optimization bias in this regime.
Revised: We ﬁrst discuss plug-in estimation approaches without causal adjustment to introduce the challenge of optimization bias in this regime.

Original: We then discuss causal estimation in Section 4.
Revised: We then discuss causal estimation in Section 4.

Original: Out of sample, ﬁxed m (Assumption 2.1) In-sample, growing n (Assumption 2.2, Appendix C) Evaluation Policy optimization 
Revised: From estimation bias to optimization bias. Denote µ

Original: Evaluation
Revised: [Equation]

Original: Policy optimization AIPW N/A Sample splitting (ﬁnite VC-dim x ) Uniform generalization requires problem-dependent structure (ﬁnite VC-dim x ) WDM
Revised: =

Original: Perturbation method Uniform generalization from out-of-sample risk bounds Perturbation GRDR Perturbation method PerturbationDoubly-robust estimation
Revised: E

Original: Denote µ t ( w ) = E [ c ( t ) | W = w ] as the conditionaloutcome mean of the population with treatment t and covariates w .
Revised: [Equation] c ( t ) | W = w ] as the conditional outcome mean of the population with treatment t and covariates w .

Original: We consider “predict-thenoptimize” approaches which learn some ˆ µ t ( w ) =
Revised: We consider “predict-thenoptimize” approaches which learn some ˆ µ t ( w ) =

Original: E [ c | W = w, T = t ] and optimize with respect toit, so that our estimator is:
Revised: E [ c | W = w, T = t ] and optimize with respect to it, so that our estimator is:

Original: [Equation]
Revised: [Equation]

Revised: Note that due to the estimation and minimization step, ˆ v ⇡ is not an unbiased estimator for v ⇤ ⇡ .

Original: Deﬁne the  error of ˆ v π with respect to the target estimand of Equation (1) as: err = v ∗ π − E
Revised: Deﬁne the overall error of ˆ v ⇡ with respect to the target estimand of Equation (1) as: err = v ⇤ ⇡  E

Original: [ˆ v π ] .
Revised: [ˆ v ⇡ ] .

Original: Theoverall errordecomposes into two parts: the estimation bias of the plug-in estimator, and the optimization bias.
Revised: We decompose the overall error into two parts: the estimation bias of the plug-in estimator, and the optimization bias.

Original: Denote ˜ v π , the best-in-class feasible estimate using the true conditional expectations µ ∗ t :
Revised: Denote ˜ v ⇡ , the best-in-class feasible estimate using the true conditional expectations µ ⇤ t :

Original: [Equation]
Revised: [Equation]

Original: (cid:110)(cid:80)

Original: [Equation]

Original: [Equation]

Original: Then, the estimation and optimization biases are: (by triangle inequality, | err | ≤ | bias est | + | bias opt | )bias
Revised: Then, the estimation and optimization biases are: (by triangle inequality, | err |  | bias est | + | bias opt | )

Original: [Equation]
Revised: [Equation]

Original: − E [˜ v π ] , bias opt = v ∗ π − E [˜ v π ] .

Original: In-sample estimation bias due to optimization.
Revised: In-sample estimation bias due to optimization.

Original: It is well known that in-sample estimation of thevalue of optimization problems is biased; e.g., ˆ v is a biased estimate for the true objective valuev ∗ π due to optimization.
Revised: It is well known that in-sample estimation of the value of optimization problems is biased; e.g., ˆ v is a biased estimate for the true objective value v ⇤ ⇡ due to optimization.

Original: Ito et al.
Revised: Ito et al.

Original: [2018] studies a bias correction for afﬁne linear objectives withan unbiased estimate of a parameter θ .
Revised: [2018] studies a bias correction for afﬁne linear objectives with an unbiased estimate of a parameter ✓ .

Original: To understand the source of the bias due to optimization,observe that clearly(cid:80) mi =1 µ t ( w i ) x i ≥ min x (cid:80) mi =1 µ t ( w i ) x i .
Revised: To understand the source of the bias due to optimization, observe that clearly

Original:  The inequality remains valid whenevaluating expectations over training datasets so that E [(cid:80) mi =1 µ t ( w i ) x i ] ≥ E [min x (cid:80) mi =
Revised: [Equation] The inequality remains valid when evaluating expectations over training datasets so that E [

Original: 1 µ t ( w i ) x i ] .
Revised: [Equation]

Original: Noting that the RHS is the true objective v ∗ π , we obtain in general the well-known optimistic bias,that E [˜ v π ] ≥ v ∗ π .
Revised: Noting that the RHS is the true objective v ⇤ ⇡ , we obtain in general the well-known optimistic bias, that E [˜ v ⇡ ]  v ⇤ ⇡ .

Original: In the policy evaluation setting, our estimates converge to the LHS, ˜ v π , so that ourestimator ˆ v π is in general a biased estimate of the decision-dependent policy value even if we obtainunbiased estimates of the cost coefﬁcient.
Revised: In the policy evaluation setting, our estimates converge to the LHS, ˜ v ⇡ , so that our estimator ˆ v ⇡ is in general a biased estimate of the decision-dependent policy value even if we obtain unbiased estimates of the cost coefﬁcient.

Original: Causal Estimation withPolicy-Dependent Responses
Revised: with

Original: In this section we present an estimation approach building upon a perturbation method that adjustsfor the aforementioned optimization bias.
Revised: In this section we present an estimation approach building upon a perturbation method that adjusts for the aforementioned optimization bias.

Original: We summarize tradeoffs among estimation strategies indifferent regimes in Table 1 and possible extensions and additional structure in Appendix C.
Revised: We summarize tradeoffs among estimation strategies in different regimes in Table 1 and possible extensions and additional structure in Appendix D.

Original: 4.1 Estimating causal effects: estimation bias
Revised: 4.1 Estimating causal effects: estimation bias

Original: Assumption 4.1 (Ignorability, overlap, SUTVA) .
Revised: Assumption 4.1 (Ignorability, overlap, SUTVA) .

Original: For all t , c ( t ) ⊥⊥ T | W .
Revised: For all t , c ( t ) ?? T | W .

Original: The evaluation policyis absolutely continuous with respect to treatment probabilities in the training dataset.
Revised: The evaluation policy is absolutely continuous with respect to treatment probabilities in the training dataset.

Original: Assume thestable unit treatment value assumption.
Revised: Assume the stable unit treatment value assumption.

Original: Confounding-adjusted plug-in estimators.
Revised: Confounding-adjusted plug-in estimators.

Original: In general, plug-in estimation of ˆ µ t ( W ) does not admit unbiased predictions because of selection bias and model misspeciﬁcation.
Revised: In general, plug-in estimation of ˆ µ t ( W ) does not admit unbiased predictions because of selection bias and model misspeciﬁcation.

Original: Existing importancesampling based estimators, e.g. the inverse propensity weighting (IPW) estimator and the doublyrobust augmented inverse probability weighting (AIPW) uses the propensity score to adjust confounding, under Assumption 4.1.
Revised: Existing importancesampling based estimators, e.g. the inverse propensity weighting (IPW) estimator and the doublyrobust augmented inverse probability weighting (AIPW) uses the propensity score to adjust confounding, under Assumption 4.1.

Original: Note importance sampling cannot directly be applied in our main regime ofinterest with out-of-sample evaluation as in Assumption 2.1, see Appendix B for a detailed overview.
Revised: Note importance sampling cannot directly be applied in our main regime of interest with out-of-sample evaluation as in Assumption 2.1, see Appendix B for a detailed overview.

Original: We depart from previous work in off-policy evaluation, in view of the optimization bias adjustment(detailed in the next section), and study estimation methods that are plug-in estimates for OPE:206 E [
Revised: We depart from previous work in off-policy evaluation, in view of the optimization bias adjustment (detailed in the next section), and study estimation methods that are plug-in estimates for OPE:

Original: [Equation]
Revised: E [ c ( ⇡ )] =

Original: ] , for some outcome model ˆ µ t that is confounding-adjusted. 
Revised: P t E [ ⇡ t ( W )ˆ µ t ( W )] , for some outcome model ˆ µ t that is confounding-adjusted. Note that IPW/AIPW-type estimators cannot be applied in the out-of-sample regime of Assumption 2.1. However, we may obtain out-of-sample risk bounds on the decision regret in this regime by virtue of out-of-sample generalization risk bounds on the generated regressors. We include more detailed discussion in Appendix D.3.

Original: Weighted direct method (WDM).
Revised: Weighted direct method (WDM).

Original: Outcome regression, learning ˆ µ t ( W ) =
Revised: Outcome regression, learning ˆ µ t ( W ) =

Original: E [ c | T
Revised: E [ c | T

Original: = t, W ]directly from D 1 , is sometimes called the direct method .
Revised: = t, W ] directly from D 1 , is sometimes called the direct method .

Original: However, when ˆ µ is a misspeciﬁedregression model such a method incurs bias.
Revised: However, when ˆ µ is a misspeciﬁed regression model such a method incurs bias.

Original: Nonetheless, re-weighting the estimation ˆ µ (maximumlikelihood, empirical risk minimization) by the inverse probability weights 1 /e is known to adjust forthe covariate shift; by a similar argument as that of Shimodaira [2000], Cao et al.
Revised: Nonetheless, re-weighting the estimation ˆ µ (maximum likelihood, empirical risk minimization) by the inverse probability weights 1 /e is known to adjust for the covariate shift; by a similar argument as that of Shimodaira [2000], Cao et al.

Original: [2009], Wang et al.[2019].
Revised: [2009], Wang et al.

Original: We call this approach weighted direct method ( WDM ), which solves:
Revised: We call this approach weighted direct method ( WDM ), which solves:

Original: [Equation]
Revised: [Equation]

Original: Doubly-robust direct method (GRDR).
Revised: Doubly-robust direct method (GRDR).

Original: We also consider an approach that achieves doubly-robustestimation of the treatment-effect due to Bang and Robins [2005].
Revised: We also consider an approach that achieves doubly-robust estimation of the treatment-effect due to Bang and Robins [2005].

Original: [See also Scharfstein et al., 1999,
Revised: [See also Scharfstein et al., 1999,

Original: Tran et al., 2019].
Revised: Tran et al., 2019].

Original: This approach has been used for CATE estimation Shi et al.
Revised: This approach has been used for CATE estimation Shi et al.

Original: [2019], Chernozhukovet al.
Revised: [2019], Chernozhukov et al.

Original: The inverse propensity score reweighted treatment indicator is added as a covariate inthe model, inducing coefﬁcients (cid:15) 0 , (cid:15) 1 .
Revised: The inverse propensity score reweighted treatment indicator is added as a covariate in the model, inducing coefﬁcients ✏ 0 , ✏ 1 .

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Optimizing over ˆ µ by (nonlinear) least-squares yields the following ﬁrst-order optimality conditionsfor θ GRDR =
Revised: Optimizing over ˆ µ by (nonlinear) least-squares yields the following ﬁrst-order optimality conditions for ✓ GRDR =

Original: [¯ θ, (cid:15) 1 , (cid:15) 0 ] :
Revised: [¯ ✓ , ✏ 1 , ✏ 0 ] :

Original: [Equation]
Revised: [Equation]

Original: = 0 .
Revised: [Equation]

Original: Bang and Robins [2005] show that the ﬁrst-order optimality conditions ensure that plug-in estimationof an average treatment effect with the model is equivalent to AIPW, hence doubly-robust.
Revised: Bang and Robins [2005] show that the ﬁrst-order optimality conditions ensure that plug-in estimation of an average treatment effect with the model is equivalent to AIPW, hence doubly-robust.

Revised: Because it is designed primarily for estimation of the ATE, its use as an outcome predictor is more speculative.

Revised: Although one can verify that its output is covariate-conditionally equivalent to CATE in expectation, and one can use this fact to again regress upon the pseudooutcomes, this ﬁnal procedure would require re-verifying asymptotic convergence; we don’t outline those arguments here.

Revised: We include further discussion on the different estimation interpretations of GRDR in the two regimes in Appendix D.3.

Original: 4.2 Estimating the decision-dependent estimand
Revised: 4.2 Estimating the decision-dependent estimand

Original: Our procedure is adapted from the perturbation method of Ito et al.
Revised: Our procedure is adapted from the perturbation method of Ito et al.

Original: [2018] which we describe herefor completeness; we extend it from linear to nonlinear predictors.
Revised: [2018] which we describe here for completeness; we extend it from linear to nonlinear predictors.

Original: The method of Ito et al.
Revised: The method of Ito et al.

Original: [2018]focuses on one parameter that we denote ξ =
Revised: Otherwise if using ˆ ⌃ , estimator of asymptotic variance of ⇠ , approximate the distribution of ⇠ ⇤ + (1 + h )  . Add ˆ ⇠ to ˆ ✓ where ˆ  ⇠ N (0 , (1+ h ) 2  1 N ˆ ⌃ ) . Then set ˆ ⇠ ( j ) ⇧ = ˆ ⇠ + ˆ  j . focuses on one parameter that we denote ⇠ =

Original: [ θ, γ ] , where we assume as outlined in Assumption 4.4that it encompasses parameters of the outcome and propensity model (respectively).
Revised: [ ✓ ,  ] , where we assume as outlined in Assumption 4. that it encompasses parameters of the outcome and propensity model (respectively).

Original: Deﬁne thepolicy-induced outcome model, µ π ( w ) =
Revised: Deﬁne the policy-induced outcome model, µ ⇡ ( w ) =

Original: (cid:80) t π t ( w ) µ t ( w ) , the estimation error 
Revised: P t ⇡ t ( w ) µ t ( w ) , the estimation error  =

Original: δ = ˆ ξ − ξ ∗ , andthe (parametrized) optimal solution at a given predictive model x ( ξ ) .
Revised: ˆ ⇠  ⇠ ⇤ , and the (parametrized) optimal solution at a given predictive model x ( ⇠ ) .

Original: The perturbation method ismotivated by a ﬁnite-difference approximation to the optimization bias induced by estimation error δ .
Revised: The perturbation method is motivated by a ﬁnite-difference approximation to the optimization bias induced by estimation error  .

Original: Deﬁne the auxiliary functions given a scalar (cid:15) parametrizing the direction of δ :
Revised: Deﬁne the auxiliary functions given a scalar ✏ parametrizing the direction of  :

Original: [Equation]
Revised: [Equation]

Original: We require regularity conditions for derivatives of these functions to exist:
Revised: We require regularity conditions for derivatives of these functions to exist:

Original: Assumption 4.2 (Perturbation method assumptions) .
Revised: Assumption 4.2 (Perturbation method assumptions) .

Original: (i)
Revised: (i)

Original: The optimal solution x ( ξ ) is unique.
Revised: The optimal solution x ( ⇠ ) is unique.

Original: (ii) ˆ ξis an unbiased estimator of ξ ∗ .
Revised: (ii) ˆ ⇠ is an unbiased estimator of ⇠ ⇤ .

Original: We generalize Prop. 3 of Ito et al.
Revised: We generalize Prop. 3 of Ito et al.

Original: [2018] for nonlinear models.
Revised: [2018] for nonlinear models.

Original: Proposition 4.3.
Revised: Proposition 4.3.

Original: We have η ( (cid:15) ) =
Revised: We have ⌘ ( ✏ ) =

Original: φ ( (cid:15) ) − (cid:15)φ (cid:48) ( (cid:15) ) + O ( (cid:15) 2 ) .
Revised:  ( ✏ )  ✏ 0 ( ✏ ) + O ( ✏ 2 ) .

Original: The plug-in estimated optimal value ˆ v π unbiasedly estimates φ (1) .
Revised: The plug-in estimated optimal value ˆ v ⇡ unbiasedly estimates  (1) .

Original: Note φ (cid:48) (1) is equivalent to thevalue of the bias.
Revised: Note  0 (1) is equivalent to the value of the bias.

Original: The perturbation method estimates φ (cid:48) (1) by ( φ (1 + h ) − φ (1)) /h for some small h .
Revised: The perturbation method estimates  0 (1) by (  (1 + h )   (1)) /h for some small h .

Original: It remains to estimate φ (1 + h ) .
Revised: It remains to estimate  (1 + h ) .

Original: First we obtain s samples of the perturbed parameterˆ ξ h = ξ ∗ + (1 + h ) δ , denoted as { ˆ ξ ( j ) h } sj =1 .
Revised: First we obtain s samples of the perturbed parameter ˆ ⇠ h = ⇠ ⇤ + (1 + h )  , denoted as { ˆ ⇠ ( j ) h } sj =1 .

Original: Each replicate of ˆ ξ ( j ) leads to an optimization estimateˆ v ( j ) =
Revised: Each replicate of ˆ ⇠ ( j ) leads to an optimization estimate ˆ v ( j ) =

Original: min x(cid:80) mi
Revised: min x

Original:  =1 x i ˆ µ π ( w i , ˆ ξ ( j ) h ) .
Revised: P mi =1 x i ˆ µ ⇡ ( w i , ˆ ⇠ ( j ) h ) .

Original: The debiased estimator is:
Revised: The debiased estimator is:

Original: [Equation]
Revised: [Equation]

Original: Our Proposition 4.3 then implies asymptotic unbiasedness (cf.
Revised: Our Proposition 4.3 then implies asymptotic unbiasedness (cf.

Original: Prop. 4 of Ito et al.
Revised: Prop. 4 of Ito et al.

Original:  so thatlim h {+! 
+}
Revised: [2018]) so that lim h {+! 
+}

Original: → 0 E [ ρ h ] =
Revised: E [ ⇢ h ] =

Original: E [min x (cid:80) mi
Revised: E [min x P m i =

Original: =1 x i ˆ µ π ( w i , ξ ∗ )] .
Revised: 1 x i ˆ µ ⇡ ( w i , ⇠ ⇤ )] .

Original: We summarize the method in Algorithm 1.
Revised: We summarize the method in Algorithm 1.

Original: Asymptotic variance of estimation methods.
Revised: Asymptotic variance of estimation methods.

Original: We discuss the asymptotic variance of the weighteddirect method and GRDR via classical asymptotic analysis of generated regressors (speciﬁcally,stacked estimation equations of GMM)
Revised: We discuss the asymptotic variance of the weighted direct method and GRDR via classical asymptotic analysis of generated regressors (speciﬁcally, stacked estimation equations of GMM)

Original: [Newey and McFadden, 1994].
Revised: [Newey and McFadden, 1994].

Original: We summarize this frameworkin Appendix B.1 for completeness and include the main result here that we invoke.
Revised: We summarize this framework in Appendix B.1 for completeness and include the main result here that we invoke.

Original: Otherwise if using ˆΣ , estimator of asymptotic variance of ξ , approximate the distribution of ξ ∗ + (1 + h ) δ .

Original: Add ˆ ξ to ˆ θ where ˆ δ ∼ N (0 , (1+ h ) 2 − 1 N ˆΣ) .

Original: Then set ˆ ξ ( j

Original: (cid:5) = ˆ ξ + ˆ δ j Assumption 4.4 (Estimators via GMM with generated regressors) .
Revised: Assumption 4.4 (Estimators via GMM with generated regressors) .

Original: Suppose the propensity score eand outcome model µ are indexed by true parameters γ ∗ , θ ∗ that solve the respective estimating equations E [ h ( W, γ ∗ )]
Revised: Suppose the propensity score e and outcome model µ are indexed by true parameters  ⇤ , ✓ ⇤ that solve the respective estimating equations E [ h ( W,  ⇤ )]

Original: = 0 ,
Revised: = 0 ,

Original: E [ g ( W, θ ∗ , γ ∗ )] = 0 .
Revised: E [ g ( W, ✓ ⇤ ,  ⇤ )] = 0 .

Original: The functions e t ( w ) , µ t ( w ) are in a Donsker class.
Revised: The functions e t ( w ) , µ t ( w ) are in a Donsker class.

Original: Remark 4.5 (Strength of assumptions) .
Revised: Remark 4.5 (Strength of assumptions) .

Original: Algorithm 1 requires both unbiased and asymptoticallynormal predictions—stronger conditions than merely inference on the ATE.
Revised: Algorithm 1 requires both unbiased and asymptotically normal predictions—stronger conditions than merely inference on the ATE.

Original: The Donsker assumptionpreserves asymptotic normality with generated regressors.
Revised: The Donsker assumption preserves asymptotic normality with generated regressors.

Original: The framework allows for nonparametricestimation via linear sieves (but not some high-dimensional regimes; see Ackerberg et al.
Revised: The framework allows for nonparametric estimation via linear sieves (but not some high-dimensional regimes; see Ackerberg et al.

Revised: 1: Input: step size ⌘ , linear objective function f .

Revised: 2: for j = 1 , 2 , · · · do 3:

Revised: At ' k , obtain a subgradient in subdifferential S ⇤ ( ⇡ k ' )

Original: Theorem 4.6 (Thm. 6.1, eq. 6.12 of Newey and McFadden [1994]) .
Revised: = { x ⇤ : f ( x ⇤ ; ⇡ k ' ) = min x f ( x ; ⇡ k ' ) } 4:

Revised: Compute subgradient r ' (min x f ( x ; ⇡ k ' ))

Revised: r ' f ( x ⇤ ; ⇡ ' ) 5:

Revised: Update subgradient step: ' k + ' k  ⌘ r '  min x f ( x ; ⇡ k ' )

Revised:  6: end for

Revised: Theorem 4.6 (Thm. 6.1, eq. 6.12 of Newey and McFadden [1994]) .

Original: Suppose Assumption 4.4holds.
Revised: Suppose Assumption 4. holds.

Original: Let ˆ G α , ˆ G θ , ˆ H denote the Jacobian matrices of partial derivatives of the moment conditions g, h with respect to the respective parameters, i.e. ˆ G γ = n − 1 (cid:80) ni
Revised: Let ˆ G ↵ , ˆ G ✓ , ˆ H denote the Jacobian matrices of partial derivatives of the moment conditions g, h with respect to the respective parameters, i.e. ˆ G  = n

Original:  =1 ∇ γ g ( w i , ˆ θ, ˆ γ ) .
Revised: P ni =1 r  g ( w i , ˆ ✓ , ˆ  ) .

Original: Letˆ V γ =
Revised: Let ˆ V

Original: [Equation]

Original:  ˆ [Equation] 1 ˆ h i )( ˆ H − 1 ˆ h i ) [Equation]
Revised: = ( ˆ H  1 ˆ h i )( ˆ H  1 ˆ h i ) > .

Original: Then an estimator of the asymptotic variance is:
Revised: Then an estimator of the asymptotic variance is:

Original: [Equation]
Revised: [Equation]

Original: Since ˆ V γ depends only on the speciﬁcation of the propensity score, to completely specify theasymptotic variance for the above formula we state the mixed terms ˆ G γ , ˆ G θ .
Revised: Since ˆ V  depends only on the speciﬁcation of the propensity score, to completely specify the asymptotic variance for the above formula we state the mixed terms ˆ G  , ˆ G ✓ .

Original: Proposition 4.7 (Asymptotic normality of WDM ) .
Revised: Proposition 4.7 (Asymptotic normality of WDM ) .

Original: Let e t ( w ) , µ t ( w ) satisfy Assumption 4.4 withthe moment condition g t ( W, θ, γ ) =
Revised: Let e t ( w ) , µ t ( w ) satisfy Assumption 4.4 with the moment condition g t ( W, ✓ ,  ) =

Original: e t ( W ; γ ) − 1 ( c − µ t ( W ; θ )) [-
-] and g =
Revised: e t ( W ;  )  1 ( c  µ t ( W ; ✓ )) [-
-]2 and g =

Original: [ g 0 , g 1 ] .
Revised: [ g 0 , g 1 ] .

Original: Then [Equation] Proposition 4.8 (Asymptotic normality of GRDR ) . Let e t ( w ) , µ ( w ) satisfy Assumption 4.4 with the moment condition for µ given by Equation (6) .
Revised: Then ˆ G  =

Original: Then

Original: [Equation]
Revised: [Equation]

Original: These formulas are generally computable from standard output of optimization solvers for nonlinearleast squares: gradients and Hessians.
Revised: These formulas are generally computable from standard output of optimization solvers for nonlinear least squares: gradients and Hessians.

Original: In practice, using the parametric bootstrap may be simpler at ahigher computational cost.
Revised: In practice, using the parametric bootstrap may be simpler at a higher computational cost.

Original: 4.3 Optimizing Causal Interventions
Revised: 4.3 Optimizing Causal Interventions

Original: Algorithm 1 provides estimation for a ﬁxed policy.
Revised: Algorithm 1 provides estimation for a ﬁxed policy.

Original: We now discuss how to optimize over policies;e.g. , implementing the outer optimization over policies min π ∈ Π in Equation (4).
Revised: We now discuss how to optimize over policies; e.g., implementing the outer optimization over policies min ⇡ 2 ⇧ in Equation (4).

Original: We focus on the casewhere the policy π t ( w ) is parametrized by and differentiable in a parameter ϕ ∈ Ψ .
Revised: We focus on the case where the policy ⇡ t ( w ) is parametrized by and differentiable in a parameter '.

Original: For example,for the logistic policy parameterization, π t ( w ) = sigmoid ( ϕ (cid:62) t w ) .
Revised: For example, for the logistic policy parameterization, ⇡ t ( w ) = sigmoid ( ' > t w ) .

Original: We consider a robust subgradientmethod, based on Danskin’s theorem, detailed in Algorithm 2.
Revised: We consider a robust subgradient method, based on Danskin’s theorem, detailed in Algorithm 2.

Original: Such an approach is a commonheuristic used in adversarial machine learning.
Revised: Such an approach is a common heuristic used in adversarial machine learning.

Original: Algorithm

Original: 1: Input: step size η , linear objective function f .

Original: 2: for j = 1 , 2 , · · · do 3:

Original: At ϕ k , obtain a subgradient in subdifferential S ∗ ( π kϕ )

Original: = { x ∗ : f ( x ∗ ; π kϕ ) =

Original: min x f ( x ; π kϕ ) } 4:

Original: Compute subgradient ∇ ϕ (min x f ( x ; π kϕ ))

Original: ← ∇ ϕ f ( x ∗ ; π ϕ ) 5:

Original: Update subgradient step: ϕ k +1 ← ϕ k − η ∇ ϕ (cid:0) min x f ( x ; π kϕ ) (cid:1) 6: end for

Original: We solve the inner optimization problem to full optimality in line 3 and take (sub)gradient stepsfor the outer optimization.
Revised: We solve the inner optimization problem to full optimality in line 3 and take (sub)gradient steps for the outer optimization.

Original: We evaluate (sub)gradients of the inner optimization solution in line 3by evaluating the gradient of the objective with respect to ϕ , ﬁxing the inner optimization variablex ∗ .
Revised: We evaluate (sub)gradients of the inner optimization solution in lineby evaluating the gradient of the objective with respect to ' , ﬁxing the inner optimization variable x ⇤ .

Original: Danskin’s theorem implies that ∇ ϕ is a subgradient [Danskin, 1966].
Revised: Danskin’s theorem implies that r ' is a subgradient [Danskin, 1966].

Original: The inner minimizationcan be solved via a linear optimization oracle for any ﬁxed choice of policy.
Revised: The inner minimization can be solved via a linear optimization oracle for any ﬁxed choice of policy.

Original: This use of the linearoptimization oracle can be beneﬁcial when special problem structures, such as matching and networkﬂows, may also admit readily-available algorithmic solutions to full optimization.
Revised: This use of the linear optimization oracle can be beneﬁcial when special problem structures, such as matching and network ﬂows, may also admit readily-available algorithmic solutions to full optimization.

Original: The perturbation method is compatible with our optimization procedure because the bias-adjustedperturbation estimated from Algorithm 1 is afﬁne in the optimization problems corresponding to eachparameter replicate.
Revised: The perturbation method is compatible with our optimization procedure because the bias-adjusted perturbation estimated from Algorithm 1 is afﬁne in the optimization problems corresponding to each parameter replicate.

Original: Hence, run Algorithm 1 with an expanded linear objective over the s -productspace x 
Revised: Hence, run Algorithm 1 with an expanded linear objective over the s -product space x 0 2 X s where f (˜ x, ⇡ )

Original: (cid:48) ∈
Revised: =

Original: X s where f (˜ x, π ) = ˆ v (0) π (˜ x 0 ) − 1 h (ˆ v (0) 
Revised: ˆ v (0) ⇡ (˜ x 0 )  1 h (ˆ v (0) ⇡ (˜ x 0 )  1 s

Original: π (˜ x 0 ) − 1 s (cid:80) sj =1 ˆ v ( j ) π
Revised: P sj =1 ˆ v ( j ) ⇡

Original: (˜ x j )) .
Revised: (˜ x j )) .

Original: So, re-optimize ˜ x ∗ j ∈ arg min x ∈X (cid:80) mi
Revised: So, re-optimize ˜ x ⇤ j 2 arg min x 2 X

Original:  =1 x i ˆ µ (cid:5) π ( w i ; ˆ ξ ( j ) (cid:5) ) and apply Danskin’s theorem to each optimization problem in the sum over ˆ v ( j )
Revised: P mi =1 x i ˆ µ ⇧ ⇡ ( w i ; ˆ ⇠ ( j ) ⇧ ) and apply Danskin’s theorem to each optimization problem in the sum over ˆ v ( j )

Original: π comprising f ( x (cid:48) , π ) .
Revised: ⇡ comprising f ( x 0 , ⇡ ) .

Original: In fact, though adversarial machinelearning focuses on min-max rather than our min-min optimization problem, this particular approachis simply subgradient descent on a nonconvex function (the solution to the inner optimization).
Revised: In fact, though adversarial machine learning focuses on min-max rather than our min-min optimization problem, this particular approach is simply subgradient descent on a nonconvex function (the solution to the inner optimization).

Original: Experimental Evaluation
Revised: [Table] [Table] ) Effect of W [Table]

Original: [Figure]
Revised: [Figure]

Revised:  Experimental Evaluation

Original: Since real data suitable for both policy evaluation anddownstream optimization is unavailable, we focus onsynthetic data and downstream bipartite matching.
Revised: Since real data suitable for both policy evaluation and downstream optimization is unavailable, we focus on synthetic data and downstream bipartite matching.

Original: Weﬁrst illustrate estimation properties of the different approaches before showing the improvement obtainedvia policy optimization.
Revised: We ﬁrst illustrate estimation properties of the different approaches before showing the improvement obtained via policy optimization.

Original: Though we are not aware ofprior approaches that are directly comparable for optimizing causal policy with a downstream optimizationdependent response, we include more comparisons tononparametric estimators (e.g. causal forests [Wagerand Athey, 2018]), and full implementation details.
Revised: Though we are not aware of prior approaches that are directly comparable for optimizing causal policy with a downstream optimization-dependent response, we include more comparisons to nonparametric estimators (e.g. causal forests [Wager and Athey, 2018]), and full implementation details.

Original: All code will be published.1.
Revised: All code will be published.

Original: Causal effect estimation.
Revised: Causal effect estimation.

Original: First, we investigate andillustrate the properties of different estimators.
Revised: First, we investigate and illustrate the properties of different estimators.

Original: Wegenerated dataset D 1 = { ( W, T, c ) } with covariate W ∼ N (0 , 1) , confounded treatment T , and outcome c .
Revised: We generated dataset D 1 = { ( W, T, c ) } with covariate W ⇠ N (0 , 1) , confounded treatment T , and outcome c .

Original: Treatment is drawn with probabilityπ bt ( W ) =
Revised: Treatment is drawn with probability ⇡ bt ( W ) =

Original: (1 + e − ϕ 1 W + ϕ 2 ) − 1 , ϕ 1 = ϕ 2 = 0 . 
Revised: (1 + e  ' 1 W + ' 2 )  1 , ' 1 = ' 2 = 0 . 5 .

Original: The true outcome model is given by a degree-2polynomial, 7 c t ( w ) =
Revised: The true outcome model is given by a degree-2 polynomial, 9 c t ( w ) =

Original: poly θ ( t, w ) + (cid:15) , where (cid:15) ∼ N (0 , 1) . .
Revised: poly ✓ ( t, w ) + ✏ , where ✏ ⇠ N (0 , 1) . .

Original: In Figure 1a and 1b, we illustratethe (covariate-conditional) estimation error of the three estimators.
Revised: In Figure 1a and 1b, we illustrate the (covariate-conditional) estimation error of the three estimators.

Original: In the mis-speciﬁed settingthat induces confounding, the outcome model is a vanilla linear regression over W without thepolynomial expansion.
Revised: In the mis-speciﬁed setting that induces confounding, the outcome model is a vanilla linear regression over W without the polynomial expansion.

Original: The direct method results in more bias under mis-speciﬁcation, while WDMand GRDR are robust as expected.2.
Revised: The direct method results in more bias under mis-speciﬁcation, while WDM and GRDR are robust as expected.

Original: Policy evaluation.
Revised: Policy evaluation.

Original: We compare the perturbation method (Algorithm 1) with three differentestimators (direct, WDM , and GRDR ).
Revised: We compare the perturbation method (Algorithm 1) with three different estimators (direct, WDM , and GRDR ).

Original: In both the well-speciﬁed / mis-speciﬁed model setting,[Figure] [Figure] we evaluate the mean-squared-error (MSE) of the estimated policy value with the three estimators,where the MSE is computed with regard to the ground-truth outcome model.
Revised: In both the well-speciﬁed / mis-speciﬁed model setting, we evaluate the mean-squared-error (MSE) of the estimated policy value with the three estimators, where the MSE is computed with regard to the ground-truth outcome model.

Original: Training data size nincreases from 500 to 2000 samples.
Revised: Training data size n increases from 500 to 2000 samples.

Original: We scaled the MSE down by the number of edges (a constant)and computed the MSE in terms of the averaged cost per edge in the matching.
Revised: We scaled the MSE down by the number of edges (a constant) and computed the MSE in terms of the averaged cost per edge in the matching.

Original: For the policy-dependent optimization, we evaluate a min-cost bipartite matching problem, wherethe causal policy intervene on the edge costs (as detailed in Example 2.3).
Revised: For the policy-dependent optimization, we evaluate a min-cost bipartite matching problem, where the causal policy intervene on the edge costs (as detailed in Example 2.3).

Original: Speciﬁcally, the bipartitegraph contains m = 500 left side nodes W 1 , · · · , W m , and m (cid:48){+
+}
Revised: Speciﬁcally, the bipartite graph contains m = 500 left side nodes W 1 , · · · , W m , and m {+
+}

Original: = 300 right side nodes.
Revised: = 300 right side nodes.

Original: The policyπ t applies treatments to the left side nodes and the outcome is the edge cost of edges with that node.
Revised: The policy ⇡ t applies treatments to the left side nodes and the outcome is the edge cost of edges with that node.

Original: While we grow the training data size, we ﬁxed m, m (cid:48) (with m > m (cid:48) ) and evaluate over ten randomdraw of train/test data for each value of n .
Revised: While we grow the training data size, we ﬁxed m, m 0 (with m > m 0 ) and evaluate over ten random draw of train/test data for each value of n .

Original: Figure 2 plots the results.
Revised: Figure 2 plots the results.

Original: When there is mis-speciﬁcation,even a large training dataset cannot bring bias correction for the direct method, where both WDMand GRDR enjoy smaller and decreasing MSE.
Revised: When there is mis-speciﬁcation, even a large training dataset cannot bring bias correction for the direct method, where both WDM and GRDR enjoy smaller and decreasing MSE.

Original: We also conduct an ablation study for the corresponding performance in the mis-speciﬁed setting(i.e. , no bootstrapping in Alg. 1).
Revised: We also conduct an ablation study for the corresponding performance in the mis-speciﬁed setting (i.e., no bootstrapping in Alg. 1).

Original: Results indicate that the perturbation method is helpful for MSEreduction for both WDM and GRDR .
Revised: Results indicate that the perturbation method is helpful for MSE reduction for both WDM and GRDR .

Original: We further conduct evaluations with different bootstrapreplicates’ sizes, and the above conclusions remain robust for different replicate sizes (additionalresults in Appendix E).3.
Revised: We further conduct evaluations with different bootstrap replicates’ sizes, and the above conclusions remain robust for different replicate sizes (additional results in Appendix F).

Original: Policy optimization.
Revised: Policy optimization.

Original: Lastly, we integrate the policy evaluation and the sub-gradient method (Alg 2)to conduct policy optimization.
Revised: Lastly, we integrate the policy evaluation and the sub-gradient method (Alg 2) to conduct policy optimization.

Original: At each iteration of Alg 2, the perturbation algorithm (Alg 1) and oneof the three different estimation methods are applied to evaluate the policy objective. 
Revised: At each iteration of Alg 2, the perturbation algorithm (Alg 1) and one of the three different estimation methods are applied to evaluate the policy objective. We consider [Table] [Figure]

Original: We considera logistic policy π t ( W )
Revised: [Figure] [Table] [Figure] a logistic policy ⇡ t ( W )

Original: = sigmoid ( ϕ 1 · W + ϕ 2 ) .
Revised: = sigmoid ( ' 1 · W + ' 2 ) .

Original: To study the convergence and the effectivenessof the subgradient algorithm for minimization, we ﬁx a test set and perform subgradient descent over60 iterations for each run.
Revised: To study the convergence and the effectiveness of the subgradient algorithm for minimization, we ﬁx a test set and perform subgradient descent over 60 iterations for each run.

Original: We average the policy values at each iteration over ten runs, where in eachrun we generate a random set of training data and a random initialization of the starting policy.
Revised: We average the policy values at each iteration over ten runs, where in each run we generate a random set of training data and a random initialization of the starting policy.

Original: We compare to the oracle estimator using the ground truth outcome model (Oracle OPT).
Revised: We compare to the oracle estimator using the ground truth outcome model (Oracle OPT).

Original: Results arepresented in Figure 3.
Revised: Results are presented in Figure 3.

Original: Again, WDM and GRDR quickly converge to the oracle estimation, while thelarge bias of the direct method leads to poor policy optimization.
Revised: Again, WDM and GRDR quickly converge to the oracle estimation, while the large bias of the direct method leads to poor policy optimization.

Original: We further evaluate the impact ofaverage random selected initial policies to the performance, and compared Figure 3 with the resultsusing a ﬁxed initial policy.
Revised: We further evaluate the impact of average random selected initial policies to the performance, and compared Figure 3 with the results using a ﬁxed initial policy.

Original: We observe that in this relatively low-dimensional example, the policyvalue converges to estimation-oracle-optimal after a few iterations (additional results and full trainingdetails in Appendix E).
Revised: We observe that in this relatively low-dimensional example, the policy value converges to estimation-oracle-optimal after a few iterations (additional results and full training details in Appendix F).

Original:  Conclusion
Revised:  Conclusion

Original: We studied a new framework for causal policy optimization with a policy-dependent optimizationresponse.
Revised: We studied a new framework for causal policy optimization with a policy-dependent optimization response.

Original: We proposed evaluation algorithms and analysis to address the fundamental challengeof an additional optimization bias.
Revised: We proposed evaluation algorithms and analysis to address the fundamental challenge of an additional optimization bias.

Original: Simulations for both the policy evaluation and optimizationalgorithms demonstrate the effectiveness of this approach.
Revised: Simulations for both the policy evaluation and optimization algorithms demonstrate the effectiveness of this approach.

Original: Interesting further directions includestudying individual fairness of optimal allocations in applications such as school assignments or jobmatching, and/or computational improvements to the policy optimization algorithms. 
Revised: Interesting further directions include studying individual fairness of optimal allocations in applications such as school assignments or job matching, and/or computational improvements to the policy optimization algorithms. Acknowledgements We support N

Revised: - edges