Original: Counterfactual Decision Support Under Treatment-Conditional Outcome Measurement Error
Revised: Counterfactual Decision Support Under Treatment-Conditional Outcome Measurement Error

Original: Abstract
Revised: Abstract

Original: Growing work in algorithmic decision support proposes methods for combining predictive models with human judgment to improve decision quality.
Revised: Growing work in algorithmic decision support proposes methods for combining predictive models with human judgment to improve decision quality.

Original: A challenge that arises in this setting is predicting the risk of a decision-relevant target outcome under multiple candidate actions.
Revised: A challenge that arises in this setting is predicting the risk of a decision-relevant target outcome under multiple candidate actions.

Original: While counterfactual prediction techniques have been developed for these tasks, current approaches do not account for measurement error in observed labels.
Revised: While counterfactual prediction techniques have been developed for these tasks, current approaches do not account for measurement error in observed labels.

Original: This is a key limitation because in many domains, observed labels (e.g., medical diagnoses, test scores) serve as a proxy for the target outcome of interest (e.g., biological medical outcomes, student learning).
Revised: This is a key limitation because in many domains, observed labels (e.g., medical diagnoses, test scores) serve as a proxy for the target outcome of interest (e.g., biological medical outcomes, student learning).

Original: We develop a method for counterfactual prediction of target outcomes observed under treatment-conditional outcome measurement error (TC-OME).
Revised: We develop a method for counterfactual prediction of target outcomes observed under treatment-conditional outcome measurement error (TC-OME).

Original: Our method minimizes risk with respect to target potential outcomes given access to observational data and estimates of measurement error parameters.
Revised: Our method minimizes risk with respect to target potential outcomes given access to observational data and estimates of measurement error parameters.

Original: We also develop a method for estimating error parameters in cases where these are unknown in advance.
Revised: We also develop a method for estimating error parameters in cases where these are unknown in advance.

Original: Through a synthetic evaluation, we show that our approach achieves performance parity with an oracle model when measurement error parameters are known and retains performance given moderate bias in error parameter estimates.
Revised: Through a synthetic evaluation, we show that our approach achieves performance parity with an oracle model when measurement error parameters are known and retains performance given moderate bias in error parameter estimates.

Original:  Introduction
Revised:  Introduction

Original: Predictive models are increasingly being introduced to support expert decision-making in real-world tasks.
Revised: Predictive models are increasingly being introduced to support expert decision-making in real-world tasks.

Original: In the medical domain, clinical models have been developed to inform patient treatment decisions by predicting the likelihood of adverse health outcomes (e.g, heart attack, stroke)
Revised: In the medical domain, clinical models have been developed to inform patient treatment decisions by predicting the likelihood of adverse health outcomes (e g, heart attack, stroke)

Original: [Mullainathan and Obermeyer, 2022].
Revised: [Mullainathan and Obermeyer, 2022].

Original: In the educational domain, learning analytics tools have been introduced to allocate additional tutoring resources to at-risk students [Livieris et al., 2016].
Revised: In the educational domain, learning analytics tools have been introduced to allocate additional tutoring resources to at-risk students [Livieris et al., 2016].

Original: In these settings, policy makers often wish to estimate the risk of a downstream target outcome under multiple alternative actions [Schulam and Saria, 2017].
Revised: In these settings, policy makers often wish to estimate the risk of a downstream target outcome under multiple alternative actions [Schulam and Saria, 2017].

Original: However, because outcomes were observed under the past decision-making policy, we do not observe the counterfactual outcome that would under a different decision [Pearl, 2009].
Revised: However, because outcomes were observed under the past decision-making policy, we do not observe the counterfactual outcome that would under a different decision [Pearl, 2009].

Original: This makes it challenging to learn a counterfactual model given observational data.
Revised: This makes it challenging to learn a counterfactual model given observational data.

Original: Recent work proposes counterfactual modeling and evaluation approaches designed for decision-support settings Coston et al.
Revised: Recent work proposes counterfactual modeling and evaluation approaches designed for decision-support settings Coston et al.

Original: [2020b,a].
Revised: [2020b,a].

Original: This work builds upon causal inference methods for conditional average treatment effect (CATE) estimation [Abrevaya et al., 2015], which have received growing interest within the machine learning community [Johansson et al., 2018].
Revised: This work builds upon causal inference methods for conditional average treatment effect (CATE) estimation [Abrevaya et al., 2015], which have received growing interest within the machine learning community [Johansson et al., 2018].

Original: We consider counterfactual modeling in two real-world decision support tasks.
Revised: We consider counterfactual modeling in two real-world decision support tasks.

Original: In selective intervention tasks, a model estimates baseline risk under no intervention (e.g., risk of child neglect given no welfare services [Chouldechova et al., 2018], or poor learning outcomes given no tutoring).
Revised: In selective intervention tasks, a model estimates baseline risk under no intervention (e.g., risk of child neglect given no welfare services [Chouldechova et al., 2018], or poor learning outcomes given no tutoring).

Original: In selective opportunity settings, a model estimates risk of a target outcome under a proposed opportunity (e.g., likelihood default given receiving a loan, or performance ratings given a new job).
Revised: In selective opportunity settings, a model estimates risk of a target outcome under a proposed opportunity (e.g., likelihood default given receiving a loan, or performance ratings given a new job).

Original: We study these counterfactual prediction settings where the target outcome is subject to measurement error.
Revised: We study these counterfactual prediction settings where the target outcome is subject to measurement error.

Original: Outcome measurement error occurs when policy-relevant target outcomes (e.g., heart attack, student learning) are imperfectly approximated by proxies (e.g., heart attack diagnosis , test scores) in data [Jacobs and Wallach, 2021].
Revised: Outcome measurement error occurs when policy-relevant target outcomes (e.g., heart attack, student learning) are imperfectly approximated by proxies (e.g., heart attack diagnosis , test scores) in data [Jacobs and Wallach, 2021].

Original: While measurement error and label noise have been studied in previous literature [Menon et al., 2015, Natarajan et al., 2013, Wang et al., 2021, Fogliato et al., 2020, 2021], this challenge has not been addressed in counterfactual prediction settings.
Revised: While measurement error and label noise have been studied in previous literature [Menon et al., 2015, Natarajan et al., 2013, Wang et al., 2021, Fogliato et al., 2020, 2021], this challenge has not been addressed in counterfactual prediction settings.

Original: Therefore, in this work, we study the novel setting of counterfactual prediction in the presence of treatment-conditional outcome measurement error (TC-OME).
Revised: Therefore, in this work, we study the novel setting of counterfactual prediction in the presence of treatment-conditional outcome measurement error (TC-OME).

Original: We offer the following contributions:
Revised: We offer the following contributions:

Original: • We formalize the problem of counterfactual prediction under treatment-conditional outcome measurement error.
Revised: • We formalize the problem of counterfactual prediction under treatment-conditional outcome measurement error.

Original: • We develop a risk minimization approach (FRM-SL; Algorithm 1) for estimating target potential outcomes given observed covariates, past decisions, and proxy outcomes observed under treatment-conditional outcome measurement error.
Revised: • We develop a risk minimization approach (FRM-SL; Algorithm 1) for estimating target potential outcomes given observed covariates, past decisions, and proxy outcomes observed under treatment-conditional outcome measurement error.

Original: Our method combines covariate adjustment techniques designed for CATE inference [Johansson et al., 2020] with a surrogate loss developed by Natarajan et al.
Revised: Our method combines covariate adjustment techniques designed for CATE inference [Johansson et al., 2020] with a surrogate loss developed by Natarajan et al.

Original: [2013] for label noise correction.
Revised: [2013] for label noise correction.

Original: • We develop a method for estimating treatment-conditional measurement error parameters (CCPE; Algorithm 2).
Revised: • We develop a method for estimating treatment-conditional measurement error parameters (CCPE; Algorithm 2).

Original: Our approach builds on class probability estimation (CPE) techniques designed for label noise settings [Menon et al., 2015, Scott et al., 2013].
Revised: Our approach builds on class probability estimation (CPE) techniques designed for label noise settings [Menon et al., 2015, Scott et al., 2013].

Original: • We evaluate FRM-SL and CCPE via synthetic experiments and show that FRM-SL achieves performance parity with an oracle model given access to ground-truth error parameters.
Revised: • We evaluate FRM-SL and CCPE via synthetic experiments and show that FRM-SL achieves performance parity with an oracle model given access to ground-truth error parameters.

Original: We also show that FRM-SL performance is tolerant to moderate bias in CCPE parameter estimates, and show that bias in estimates decreases as a function of sample size.
Revised: We also show that FRM-SL performance is tolerant to moderate bias in CCPE parameter estimates, and show that bias in estimates decreases as a function of sample size.

Original:  Problem setup
Revised:  Problem setup

Original: We consider a counterfactual distribution defined over p ∗ ( X, D, Y ∗ 0 , Y ∗ 1 ) ,
Revised: We consider a counterfactual distribution defined over p ∗ ( X, D, Y ∗ 0 , Y ∗ 1 ) ,

Original: where X ∈ X ⊆
Revised: where X ∈ X ⊆

Original: R
Revised: R

Original: d are covariates, D ∈ { 0 , 1 } are past decisions, and Y ∗ 0 , Y ∗ 1 ∈ { 0 , 1 } are target binary potential outcomes of interest to human decision-makers.
Revised: d are covariates, D ∈ { 0 , 1 } are past decisions, and Y ∗ 0 , Y ∗ 1 ∈ { 0 , 1 } are target binary potential outcomes of interest to human decision-makers.

Original: Under potential outcomes [Rubin, 2005], Y ∗ 0 is the hypothetical outcome we would see under the baseline condition when d = 0 , while Y ∗ 1 is the outcome we would observe under the proposed intervention (selective intervention) or opportunity (selective opportunity) when d = 1 .
Revised: Under potential outcomes [Rubin, 2005], Y ∗ 0 is the hypothetical outcome we would see under the baseline condition when d = 0 , while Y ∗ 1 is the outcome we would observe under the proposed intervention (selective intervention) or opportunity (selective opportunity) when d = 1 .

Original: Following the standard setup in causal inference, we only observe Y ∗ 0 or Y ∗ 1 for a given instance such that Y ∗ =
Revised: Following the standard setup in causal inference, we only observe Y ∗ 0 or Y ∗ 1 for a given instance such that Y ∗ =

Original: D · Y ∗ 1 + (1 − D ) · Y ∗ 

Revised: D · Y ∗ 1 + (1 − D ) · Y ∗

Original: [Pearl, 2009].
Revised: [Pearl, 2009].

Original: Given the counterfactual joint p ∗ , we would like to estimate the target quantity
Revised: Given the counterfactual joint p ∗ , we would like to estimate the target quantity

Original: [Equation]
Revised: [Equation]

Original: where d = 0 in selective intervention settings, and d = 1 in selective opportunity settings.
Revised: where d = 0 in selective intervention settings, and d = 1 in selective opportunity settings.

Original: However, rather than sampling directly from p ∗ , we draw samples i.i.d.
Revised: However, rather than sampling directly from p ∗ , we draw samples i.i.d.

Original: from p ( X, D, Y ) , where
Revised: from p ( X, D, Y ) , where

Original: Y ∈ { 0 , 1 } is a binary proxy outcome.
Revised: Y ∈ { 0 , 1 } is a binary proxy outcome.

Original: Two challenges complicate estimation of η ∗ d ( x ) given samples from p .
Revised: Two challenges complicate estimation of η ∗ d ( x ) given samples from p .

Original: First, observed proxies Y arise from potential outcomes such that Y
Revised: First, observed proxies Y arise from potential outcomes such that Y

Original: = D · Y 1 + (1 − D ) · Y 0 .
Revised: = D · Y 1 + (1 − D ) · Y 0 .

Original: Therefore, we do not know the outcome that would have occurred had the counterfactual decision been made in the past.
Revised: Therefore, we do not know the outcome that would have occurred had the counterfactual decision been made in the past.

Original: Second, proxy potential outcomes Y d are subject to outcome measurement error.
Revised: Second, proxy potential outcomes Y d are subject to outcome measurement error.

Original: In our treatment-conditional outcome measurement error (TC-OME) model, the proxy potential outcome is observed under a false positive rate α d and false negative rate β d given by
Revised: In our treatment-conditional outcome measurement error (TC-OME) model, the proxy potential outcome is observed under a false positive rate α d and false negative rate β d given by

Original: [Equation]
Revised: [Equation]

Original: where α d + β d < 1 .
Revised: where α d + β d < 1 .

Original: Figure 1 shows an example of TC-OME in a heart attack predictioncontext, including factual and counterfactual target potential outcomes that can be observed under different error rates for d = { 0 , 1 } .
Revised: Figure 1 shows an example of TC-OME in a heart attack prediction

Original: [Figure]
Revised: [Figure]

Original: Y
Revised: Y

Original: Under this model, the class probability function of the proxy potential outcome Y d can be given by
Revised: Under this model, the class probability function of the proxy potential outcome Y d can be given by

Original: [Equation]
Revised: [Equation]

Original: which gives an expression for the observed class probability η d ( x ) in terms of the target class probability η ∗ d ( x )
Revised: which gives an expression for the observed class probability η d ( x ) in terms of the target class probability η ∗ d ( x )

Original: Modeling measurement error requires making assumptions on the relationship between target and proxy outcomes [Jacobs and Wallach, 2021].
Revised: Modeling measurement error requires making assumptions on the relationship between target and proxy outcomes [Jacobs and Wallach, 2021].

Original: We make the following assumptions in our model:
Revised: We make the following assumptions in our model:

Original: Assumption 2.1 (Measurement Error Model) .
Revised: Assumption 2.1 (Measurement Error Model) .

Original: Error parameters not depend on covariates X or unmeasured confounders
Revised: Error parameters not depend on covariates X or unmeasured confounders

Original: Z : P ( Y d | Y ∗ d ) =
Revised: Z : P ( Y d | Y ∗ d ) =

Original: P ( Y d | Y ∗ d , X = x ) =
Revised: P ( Y d | Y ∗ d , X = x ) =

Original: P ( Y d | Y ∗ d , Z = z ) .
Revised: P ( Y d | Y ∗ d , Z = z ) .

Original: While this assumption follows from the class-conditional model studied in past literature [Menon et al., 2015, Scott et al., 2013], our methods can be readily extended to settings where P ( Y d |
Revised: While this assumption follows from the class-conditional model studied in past literature [Menon et al., 2015, Scott et al., 2013], our methods can be readily extended to settings where P ( Y d |

Original: Y ∗ d )
Revised: Y ∗ d )

Original: ̸ = P ( Y d | Y ∗ d , X = x ) by applying a group-dependent error model [Wang et al., 2021].
Revised: ̸ = P ( Y d | Y ∗ d , X = x ) by applying a group-dependent error model [Wang et al., 2021].

Original: It also follows from e.q. 3 that ∀ d ∈ D , η d ( x ) is a strictly monotone increasing transform of η ∗ d ( x ) .
Revised: It also follows from e.q. 3 that ∀ d ∈ D , η d ( x ) is a strictly monotone increasing transform of η ∗ d ( x ) .

Original: 2.1 Identifiability conditions
Revised: 2.1 Identifiability conditions

Original: Figure 2 shows a causal diagram specifying the assumptions we make on the data generating process in a TC-OME setting.
Revised: Figure 2 shows a causal diagram specifying the assumptions we make on the data generating process in a TC-OME setting.

Original: Class probability functions η ∗ d and η d are identifiable if they can be computed uniquely from the observed distribution p ( X, D, Y ) .
Revised: Class probability functions η ∗ d and η d are identifiable if they can be computed uniquely from the observed distribution p ( X, D, Y ) .

Original: Our target causal estimand
Revised: Our target causal estimand

Original: η ∗ d ( x ) is not identifiable directly because potential outcomes { Y ∗ 0 , Y ∗ 1 } are unobserved.
Revised: η ∗ d ( x ) is not identifiable directly because potential outcomes { Y ∗ 0 , Y ∗ 1 } are unobserved.

Original: However, { Y 0 , Y 1 } are identifiable under a standard set of causal identifiability assumptions:
Revised: However, { Y 0 , Y 1 } are identifiable under a standard set of causal identifiability assumptions:

Original: Assumption 2.2 (Consistency) .
Revised: Assumption 2.2 (Consistency) .

Original: An instance receiving decision d ∈ { 0 , 1 } has outcome Y =
Revised: An instance receiving decision d ∈ { 0 , 1 } has outcome Y =

Original: Y d :
Revised: Y d :

Original: Y = D · Y 1 + (1 − D ) · Y 0 .
Revised: Y = D · Y 1 + (1 − D ) · Y 0 .

Original: Assumption 2.3 (Ignorability) .
Revised: Assumption 2.3 (Ignorability) .

Original: Potential outcomes and decisions are conditionally independent given X : { Y 0 , Y 1 } ⊥⊥ D | X .
Revised: Potential outcomes and decisions are conditionally independent given X : { Y 0 , Y 1 } ⊥⊥ D | X .

Original: Assumption 2.4 (Positivity) .
Revised: Assumption 2.4 (Positivity) .

Original: For any set of covariates x ∈ X , both decisions have non-zero probability of observation in the data: ∀ x ∈ X, d ∈ { 0 , 1 } : p ( D = d | X = x ) > 0 .
Revised: For any set of covariates x ∈ X , both decisions have non-zero probability of observation in the data: ∀ x ∈ X, d ∈ { 0 , 1 } : p ( D = d | X = x ) > 0 .

Original:  Methodology
Revised:  Methodology

Original: First, we develop an estimator for η ∗ d ( x ) given i.i.d.
Revised: First, we develop an estimator for η ∗ d ( x ) given i.i.d.

Original: samples drawn from the observational joint p ( X, D, Y ) assuming a priori knowledge of error terms (Section 3.1).
Revised: samples drawn from the observational joint p ( X, D, Y ) assuming a priori knowledge of error terms (Section 3.1).

Original: In practice, α d and β d are unknown in advance.
Revised: In practice, α d and β d are unknown in advance.

Original: Therefore, in Section 3.2, we develop an approach for error parameter estimation (Algorithm 2).
Revised: Therefore, in Section 3.2, we develop an approach for error parameter estimation (Algorithm 2).

Original: This approach requires access to observational data from p and an additional weak separability assumption commonly applied in class-conditional error settings [Menon et al., 2015].
Revised: This approach requires access to observational data from p and an additional weak separability assumption commonly applied in class-conditional error settings [Menon et al., 2015].

Original: 3.1 Risk minimization
Revised: 3.1 Risk minimization

Original: Let f ∈ H be a probabilistic decision function belonging to H ⊂ { h : X → [0 , 1] } and let ℓ : R × {± 1 } → R + be a bounded loss function.
Revised: Let f ∈ H be a probabilistic decision function belonging to H ⊂ { h : X → [0 , 1] } and let ℓ : R × {± 1 } → R + be a bounded loss function.

Original: The ℓ -risk of f over Y ∗ d can be given by
Revised: The ℓ -risk of f over Y ∗ d can be given by

Original: [Equation]
Revised: [Equation]

Original: where R ∗ ℓ ( f ) is the marginal risk over the full population.
Revised: where R ∗ ℓ ( f ) is the marginal risk over the full population.

Original: Because we wish to recover a class probability η ∗ d ( x ) , we restrict ℓ to be strictly proper composite such that arg min f R ∗ ℓ ( f ) is a monotone transform ϕ of η ∗ d (e.g., the logistic and exponential loss)
Revised: Because we wish to recover a class probability η ∗ d ( x ) , we restrict ℓ to be strictly proper composite such that arg min f R ∗ ℓ ( f ) is a monotone transform ϕ of η ∗ d (e.g., the logistic and exponential loss)

Original: [Agarwal, 2014, Menon et al., 2015].
Revised: [Agarwal, 2014, Menon et al., 2015].

Original: This allows for recovering class probabilities from the optimal prediction via the link function ϕ .
Revised: This allows for recovering class probabilities from the optimal prediction via the link function ϕ .

Original: Johansson et al.
Revised: Johansson et al.

Original: [2020] show that the marginal risk over the full population can be decomposed into factual and counterfactual components scaled by π = p ( D = 1) via
Revised: [2020] show that the marginal risk over the full population can be decomposed into factual and counterfactual components scaled by π = p ( D = 1) via

Original: [Equation]
Revised: [Equation]

Original: for d ∈ { 0 , 1 } .
Revised: for d ∈ { 0 , 1 } .

Original: The factual risk, denoted by subscript d , can be directly estimated over the sample that received treatment D = d .
Revised: The factual risk, denoted by subscript d , can be directly estimated over the sample that received treatment D = d .

Original: Under ignorability, this factual risk is identifiable as
Revised: Under ignorability, this factual risk is identifiable as

Original: [Equation]
Revised: [Equation]

Original: The factual risk R ∗ d,ℓ ( f ) is a biased estimator for population risk R ∗ ℓ ( f ) in observational settings (i.e., when X ̸⊥⊥ D )
Revised: The factual risk R ∗ d,ℓ ( f ) is a biased estimator for population risk R ∗ ℓ ( f ) in observational settings (i.e., when X ̸⊥⊥ D )

Original: [Johansson et al., 2020].
Revised: [Johansson et al., 2020].

Original: However, previous work has demonstrated empirically that bias correction techniques such as propensity re-weighting offer limited performance improvements in counterfactual risk assessment settings given sufficient sample size and an expressive model class [Coston et al., 2020b].
Revised: However, previous work has demonstrated empirically that bias correction techniques such as propensity re-weighting offer limited performance improvements in counterfactual risk assessment settings given sufficient sample size and an expressive model class [Coston et al., 2020b].

Original: Therefore, in this work, we develop a minimizer for the factual risk R ∗ d,ℓ ( f ) and leave a comparison with re-weighting based approaches for future work.
Revised: Therefore, in this work, we develop a minimizer for the factual risk R ∗ d,ℓ ( f ) and leave a comparison with re-weighting based approaches for future work.

Original: The factual risk R ∗ d,ℓ ( f ) cannot be estimated directly because target potential outcomes are unobserved.
Revised: The factual risk R ∗ d,ℓ ( f ) cannot be estimated directly because target potential outcomes are unobserved.

Original: Instead, we construct a surrogate loss ˜ ℓ such that minimizing factual ˜ ℓ -risk w.r.t.
Revised: Instead, we construct a surrogate loss ˜ ℓ such that minimizing factual ˜ ℓ -risk w.r.t.

Original: proxy outcomes Y is equivalent to minimizing factual ℓ -risk w.r.t.
Revised: proxy outcomes Y is equivalent to minimizing factual ℓ -risk w.r.t.

Original: target outcomes
Revised: target outcomes

Original: Y ∗ in expectation [Natarajan et al., 2013].
Revised: Y ∗ in expectation [Natarajan et al., 2013].

Original: Under ignorability, the factual risk over the proxy potential outcome Y d is identifiable by conditioning on D = d via
Revised: Under ignorability, the factual risk over the proxy potential outcome Y d is identifiable by conditioning on D = d via

Original: [Equation]
Revised: [Equation]

Original: Given knowledge of error parameters, we wish to construct a surrogate loss ˜ ℓ s.t.
Revised: Given knowledge of error parameters, we wish to construct a surrogate loss ˜ ℓ s.t.

Original: R d, ˜ ℓ ( f ) gives an unbiased estimator for R ∗ d,ℓ ( f )
Revised: R d, ˜ ℓ ( f ) gives an unbiased estimator for R ∗ d,ℓ ( f )

Original: [Equation]
Revised: [Equation]

Original: ∀ x ∈ X s.t.
Revised: ∀ x ∈ X s.t.

Original: D =
Revised: D =

Original: d , where ℓ is computed on the target y ∗ and ˜ ℓ is computed over proxy outcomes Y .
Revised: d , where ℓ is computed on the target y ∗ and ˜ ℓ is computed over proxy outcomes Y .

Original: By Lemma 1 in Natarajan et al.
Revised: By Lemma 1 in Natarajan et al.

Original: [2013] , such an ˜ ℓ can be constructed via
Revised: [2013] , such an ˜ ℓ can be constructed via

Original: [Equation]
Revised: [Equation]

Original: ∀ x ∈ X
Revised: ∀ x ∈ X

Original: [Equation]
Revised: [Equation]

Original: Input: Data W ∼ p
Revised: Input: Data W ∼ p

Original: Output: Learned for
Revised: Output: Learned for

Original: ∗ x ) 1 Compute parameter estimates
Revised: ∗ x ) 1 Compute parameter estimates

Original: [Equation]
Revised: [Equation]

Original: 2 Construct ˜ ℓ parameterized by ˆ α d , ˆ
Revised: 2 Construct ˜ ℓ parameterized by ˆ α d , ˆ

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: Input: Data W = { X i , D i , Y i } i ∈ N ∼ p
Revised: Input: Data W = { X i , D i , Y i } i ∈ N ∼ p

Original: Output: Parameter estimates ˆ α d , ˆ β d 1 Partition W into subsets W 1 , W2 Learn ˆ η d ( x ) :
Revised: Output: Parameter estimates ˆ α d , ˆ β d 1 Partition W into subsets W 1 , W2 Learn ˆ η d ( x ) :

Original: = arg min f ∈H ˆ R d,ℓ ( f ) on W3 Use ˆ η d ( x ) to estimate error terms on W 2 : ˆ α
Revised: = arg min f ∈H ˆ R d,ℓ ( f ) on W3 Use ˆ η d ( x ) to estimate error terms on W 2 : ˆ α

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Because R d, ˜ ℓ ( f ) is over the observed joint p ( X, D, Y ) , we can compute the empirical ˜ ℓ -risk on samples ( X 1 , D 1 , Y 1 ) , ..., ( X n , D n , Y n ) from p
Revised: Because R d, ˜ ℓ ( f ) is over the observed joint p ( X, D, Y ) , we can compute the empirical ˜ ℓ -risk on samples ( X 1 , D 1 , Y 1 ) , ..., ( X n , D n , Y n ) from p

Original: [Equation]
Revised: [Equation]

Original: where D ′ =
Revised: where D ′ =

Original: { i ∈ 1 , ..., N : D i = d } .
Revised: { i ∈ 1 , ..., N : D i = d } .

Original: We can then learn a predictor from observed data by minimizing the empirical risk
Revised: We can then learn a predictor from observed data by minimizing the empirical risk

Original: [Equation]
Revised: [Equation]

Original: By Lemma 1 in Natarajan et al.
Revised: By Lemma 1 in Natarajan et al.

Original: [2013], e.q. 11 converges to the factual risk of the target potential outcome R ∗ d,ℓ ( f ) in expectation.
Revised: [2013], e.q. 11 converges to the factual risk of the target potential outcome R ∗ d,ℓ ( f ) in expectation.

Original: Under the condition that ℓ is strictly proper composite, ˆ R ∗ d ( f ) can be used to recover desired class probabilities η ∗ d ( x ) .
Revised: Under the condition that ℓ is strictly proper composite, ˆ R ∗ d ( f ) can be used to recover desired class probabilities η ∗ d ( x ) .

Original: Therefore, given a priori knowledge of α d , β d , we can learn an estimator for η ∗ d ( x ) given samples from p ( X, D, Y ) .
Revised: Therefore, given a priori knowledge of α d , β d , we can learn an estimator for η ∗ d ( x ) given samples from p ( X, D, Y ) .

Original: 3.2 Error parameter estimation
Revised: 3.2 Error parameter estimation

Original: Directly minimizing ˆ R d, ˜ ℓ ( f ) is challenging because error parameters are often unknown in advance.
Revised: Directly minimizing ˆ R d, ˜ ℓ ( f ) is challenging because error parameters are often unknown in advance.

Original: Therefore, we develop a procedure for estimating error terms, then use estimates ˆ α d , ˆ β d to construct the surrogate loss.
Revised: Therefore, we develop a procedure for estimating error terms, then use estimates ˆ α d , ˆ β d to construct the surrogate loss.

Original: Our parameter estimation approach places a weak separability assumption on p ∗ .
Revised: Our parameter estimation approach places a weak separability assumption on p ∗ .

Original: Assumption 3.1 (Weak Separability) .
Revised: Assumption 3.1 (Weak Separability) .

Original: For fixed d ∈ { 0 , 1 } , there exist target potential outcomes Y ∗ d such that inf x ∈ X { η ∗ d ( x ) } = 0 and sup x ∈ X { η ∗ d ( x ) } = 1 .
Revised: For fixed d ∈ { 0 , 1 } , there exist target potential outcomes Y ∗ d such that inf x ∈ X { η ∗ d ( x ) } = 0 and sup x ∈ X { η ∗ d ( x ) } = 1 .

Original: This assumption has been widely applied in observational label noise settings [Menon et al., 2015,
Revised: This assumption has been widely applied in observational label noise settings [Menon et al., 2015,

Original: Xia et al., 2019, Wang et al., 2021].
Revised: Xia et al., 2019, Wang et al., 2021].

Original: In our counterfactual setting, this assumption stipulates that there exists an individual at no risk ( Y ∗ d = 0 ) and another individual at certain risk ( Y ∗ d = 1 ) under d =in selective intervention settings or d = 1 in selective opportunity settings.
Revised: In our counterfactual setting, this assumption stipulates that there exists an individual at no risk ( Y ∗ d = 0 ) and another individual at certain risk ( Y ∗ d = 1 ) under d =in selective intervention settings or d = 1 in selective opportunity settings.

Original: This assumption need only hold for d = 0 or d = 1 depending on the target quantity of interest in a given setting.
Revised: This assumption need only hold for d = 0 or d = 1 depending on the target quantity of interest in a given setting.

Original: Given weak separability, error parameters can be estimated by substituting η ∗ d ( x ) = 0 , η ∗ d ( x ) =into the TC-OME model (e.q. 3) and solving
Revised: Given weak separability, error parameters can be estimated by substituting η ∗ d ( x ) = 0 , η ∗ d ( x ) =into the TC-OME model (e.q. 3) and solving

Original: [Equation]
Revised: [Equation]

Original: (12) where we can take the infimum and supremum of η d ( x ) because it is a monotone transform of η ∗ d ( x ) by e.q. 3.
Revised: (12) where we can take the infimum and supremum of η d ( x ) because it is a monotone transform of η ∗ d ( x ) by e.q. 3.

Original: Therefore, we estimate α d , β d by learning ˆ η d ( x ) , then computing the minimum and maximum over class probabilities predicted on a held-out sample (Algorithm 2).
Revised: Therefore, we estimate α d , β d by learning ˆ η d ( x ) , then computing the minimum and maximum over class probabilities predicted on a held-out sample (Algorithm 2).

Original: For statistical purposes, we perform each step on disjoint data folds [Menon et al., 2015].
Revised: For statistical purposes, we perform each step on disjoint data folds [Menon et al., 2015].

Original: Estimates ˆ α , ˆ β can then be used to construct ˜ ℓ and minimize q
Revised: Estimates ˆ α , ˆ β can then be used to construct ˜ ℓ and minimize q

Original: ˆ η
Revised: ˆ η

Original: [Table]
Revised: [Table]

Original:  Experiments
Revised:  Experiments

Original: Setup .
Revised: Setup .

Original: TC-OME evaluation on real-world data is challenging due to compounding uncertainty from (1) unobserved potential outcomes [Pearl, 2009, Coston et al., 2020b] and
Revised: TC-OME evaluation on real-world data is challenging due to compounding uncertainty from (1) unobserved potential outcomes [Pearl, 2009, Coston et al., 2020b] and

Original: (2) measurement error
Revised: (2) measurement error

Original: [De-Arteaga et al., 2021, Fogliato et al., 2021].
Revised: [De-Arteaga et al., 2021, Fogliato et al., 2021].

Original: Therefore, we conduct a controlled synthetic evaluation to validate our approach.
Revised: Therefore, we conduct a controlled synthetic evaluation to validate our approach.

Original: Our evaluation emulates a selective intervention setting with target quantity η ∗ 0 ( x ) .
Revised: Our evaluation emulates a selective intervention setting with target quantity η ∗ 0 ( x ) .

Original: We use a unidimensional covariate X ∼ U ( − 1 , 1) and sinusoidal functions η ∗ 0 ( x ) , η ∗ 1 ( x ) satisfying weak separability.
Revised: We use a unidimensional covariate X ∼ U ( − 1 , 1) and sinusoidal functions η ∗ 0 ( x ) , η ∗ 1 ( x ) satisfying weak separability.

Original: We sample Y ∗ 0 , Y ∗ 1 ∼ Bern ( η ∗ d ( x )) , ∀ d ∈ D and generate proxy outcomes by flipping Y ∗ 0 , Y ∗ 1 with probability given by α 0 , β 0 .
Revised: We sample Y ∗ 0 , Y ∗ 1 ∼ Bern ( η ∗ d ( x )) , ∀ d ∈ D and generate proxy outcomes by flipping Y ∗ 0 , Y ∗ 1 with probability given by α 0 , β 0 .

Original: We observe outcomes Y by sampling from a propensity function π ( x ) =
Revised: We observe outcomes Y by sampling from a propensity function π ( x ) =

Original: P ( D = 1 | X = x ) that is linear in X .
Revised: P ( D = 1 | X = x ) that is linear in X .

Original: We use an MLP trained via binary cross-entropy loss in all experiments.
Revised: We use an MLP trained via binary cross-entropy loss in all experiments.

Original: Appendix A.1 contains additional details.
Revised: Appendix A.1 contains additional details.

Original: Experiments .
Revised: Experiments .

Original: Experiment one (Table 1) compares (i) an observational model targeting Y (OBS); (ii) factual risk minimization with unmodified loss (FRM); and (iii) factual risk minimization with a surrogate loss parameterized by α 0 , β d (FRM+SL; Algorithm 1).
Revised: Experiment one (Table 1) compares (i) an observational model targeting Y (OBS); (ii) factual risk minimization with unmodified loss (FRM); and (iii) factual risk minimization with a surrogate loss parameterized by α 0 , β d (FRM+SL; Algorithm 1).

Original: We compare against oracles predicting Y ∗ 0 (TARGET ∗ ) and Y 0 (PROXY ∗ ).
Revised: We compare against oracles predicting Y ∗ 0 (TARGET ∗ ) and Y 0 (PROXY ∗ ).

Original: We report accuracy with respect to Y ∗ 0 on a heldout sample in line with a selective intervention setting targeting η ∗ 0 ( x ) .
Revised: We report accuracy with respect to Y ∗ 0 on a heldout sample in line with a selective intervention setting targeting η ∗ 0 ( x ) .

Original: As shown in Table 1,
Revised: As shown in Table 1,

Original: OBS performs poorly across all configurations.
Revised: OBS performs poorly across all configurations.

Original: FRM and oracle methods perform comparably in configurations with (1) no measurement error ( α 0 = 0 , β 0 = 0) and (2) with symmetric error terms (0 . 2 , 0 . 2) .
Revised: FRM and oracle methods perform comparably in configurations with (1) no measurement error ( α 0 = 0 , β 0 = 0) and (2) with symmetric error terms (0 . 2 , 0 . 2) .

Original: In the former case, a surrogate loss is not needed to correct for measurement error.
Revised: In the former case, a surrogate loss is not needed to correct for measurement error.

Original: In the latter, FRM and PROXY ∗ perform well because the optimal threshold minimizing misclassification risk can be computed directly from observed labels given symmetric error terms (see Menon et al.
Revised: In the latter, FRM and PROXY ∗ perform well because the optimal threshold minimizing misclassification risk can be computed directly from observed labels given symmetric error terms (see Menon et al.

Original: [2015], Appendix F.1).
Revised: [2015], Appendix F.1).

Original: In more realistic asymmetric error configurations ( . 1 , . 3) , ( . 3 , . 1) , FRM+SL outperforms FRM and performs at parity with TARGET ∗ .
Revised: In more realistic asymmetric error configurations ( . 1 , . 3) , ( . 3 , . 1) , FRM+SL outperforms FRM and performs at parity with TARGET ∗ .

Original: While this experiment assumes oracle access to α 0 , β d , we conduct an additional experiment (Figure 4) showing that FRM+SL performance is robust to bias in ˆ α 0 , ˆ β 0 in the neighborhood of ± .
Revised: While this experiment assumes oracle access to α 0 , β d , we conduct an additional experiment (Figure 4) showing that FRM+SL performance is robust to bias in ˆ α 0 , ˆ β 0 in the neighborhood of ± .

Original: 05 to ± .
Revised: 05 to ± .

Original: 1 depending magnitude of α 0 , β 0 .
Revised: 1 depending magnitude of α 0 , β 0 .

Original: Experiment two (Figure 3) evaluates CCPE as a function of sample size.
Revised: Experiment two (Figure 3) evaluates CCPE as a function of sample size.

Original: We construct π ( x ) and η ∗ 0 ( x ) such that π ( x ) ∝ η ∗ 0 ( x ) .
Revised: We construct π ( x ) and η ∗ 0 ( x ) such that π ( x ) ∝ η ∗ 0 ( x ) .

Original: This mirrors the real-world setting in which individuals at high baseline risk are more likely to receive a risk-reducing intervention.
Revised: This mirrors the real-world setting in which individuals at high baseline risk are more likely to receive a risk-reducing intervention.

Original: Because this results in lower sample density over the high risk region of η ∗ 0 ( x ) , the learned approximation ˆ η 0 ( x ) is likely to be worse near the supremum of η ∗ 0 ( x ) than at its infimum.
Revised: Because this results in lower sample density over the high risk region of η ∗ 0 ( x ) , the learned approximation ˆ η 0 ( x ) is likely to be worse near the supremum of η ∗ 0 ( x ) than at its infimum.

Original: As a result, we expect more bias in ˆ β d than ˆ α d .
Revised: As a result, we expect more bias in ˆ β d than ˆ α d .

Original: Figureshows that estimates improve as N increases, and confirms slower convergence for ˆ β d than ˆ α d .
Revised: Figureshows that estimates improve as N increases, and confirms slower convergence for ˆ β d than ˆ α d .

Original:  Discussion
Revised:  Discussion

Original: We introduce a novel treatment-conditional outcome measurement error model that formalizes key challenges in counterfactual prediction settings.
Revised: We introduce a novel treatment-conditional outcome measurement error model that formalizes key challenges in counterfactual prediction settings.

Original: We provide risk minimization (Algorithm 1) and parameter estimation (Algorithm 2) techniques designed for this setting.
Revised: We provide risk minimization (Algorithm 1) and parameter estimation (Algorithm 2) techniques designed for this setting.

Original: Synthetic results demonstrate that FRM+SL provides a strong improvement over FRM alone in asymmetric error settings.
Revised: Synthetic results demonstrate that FRM+SL provides a strong improvement over FRM alone in asymmetric error settings.

Original: Results also show that FRM+SL performance remains robust given bias in ˆ α d , ˆ β d , and that CCPE recovers reasonable estimates for α d , β d .
Revised: Results also show that FRM+SL performance remains robust given bias in ˆ α d , ˆ β d , and that CCPE recovers reasonable estimates for α d , β d .

Original: Our counterfactual identification and measurement model assumptions can be violated in some realworld settings.
Revised: Our counterfactual identification and measurement model assumptions can be violated in some realworld settings.

Original: For instance, ignorability (Assumption 2.3) can be violated if humans make use of predictive contextual factors that are not recorded as covariates Kleinberg et al.
Revised: For instance, ignorability (Assumption 2.3) can be violated if humans make use of predictive contextual factors that are not recorded as covariates Kleinberg et al.

Original: [2018], Lakkaraju et al.
Revised: [2018], Lakkaraju et al.

Original: This issue can be partially circumvented if unobservables are only un-available at runtime
Revised: This issue can be partially circumvented if unobservables are only un-available at runtime

Original: Coston et al.
Revised: Coston et al.

Original: [2020b].
Revised: [2020b].

Original: In decision support settings, positivity violations are not of major concern.
Revised: In decision support settings, positivity violations are not of major concern.

Original: This is because the instances that require support from predictive models often have uncertain risk profiles and could receive either decision.
Revised: This is because the instances that require support from predictive models often have uncertain risk profiles and could receive either decision.

Original: In contrast, the “cut-and-dry” cases potentially violating positivity (Assumption 2.4) are normally routed through other administrative decision-making procedures.
Revised: In contrast, the “cut-and-dry” cases potentially violating positivity (Assumption 2.4) are normally routed through other administrative decision-making procedures.

Original: It is also important to carefully consider, based on domain expertise, whether weak separability (Assumption 2) is likely to hold in a given modeling context.
Revised: It is also important to carefully consider, based on domain expertise, whether weak separability (Assumption 2) is likely to hold in a given modeling context.

Original: This assumption has shown to be unreasonable in some settings (e.g., criminal justice [Fogliato et al., 2020, 2021]), and should not be relied upon without careful cross-checking with existing domain knowledge.
Revised: This assumption has shown to be unreasonable in some settings (e.g., criminal justice [Fogliato et al., 2020, 2021]), and should not be relied upon without careful cross-checking with existing domain knowledge.

Original:  Related work
Revised:  Related work

Original: To our knowledge, the treatment-conditional error setting we study is novel in this work.
Revised: To our knowledge, the treatment-conditional error setting we study is novel in this work.

Original: Techniques have been developed for addressing class-conditional
Revised: Techniques have been developed for addressing class-conditional

Original: [Menon et al., 2015, Scott et al., 2013], groupdependent [Wang et al., 2021], and instance-dependent [Xia et al., 2020] label noise models.
Revised: [Menon et al., 2015, Scott et al., 2013], groupdependent [Wang et al., 2021], and instance-dependent [Xia et al., 2020] label noise models.

Original: Menon et al.
Revised: Menon et al.

Original: [2015], Scott et al.
Revised: [2015], Scott et al.

Original: [2013] and Northcutt et al.
Revised: [2013] and Northcutt et al.

Original: [2021] develop noise rate estimation approaches commonly used in label noise settings.
Revised: [2021] develop noise rate estimation approaches commonly used in label noise settings.

Original: Recently, De-Arteaga et al.
Revised: Recently, De-Arteaga et al.

Original: [2021] propose a method leveraging inter-expert consistency to adjust for measurement error in proxy outcomes.
Revised: [2021] propose a method leveraging inter-expert consistency to adjust for measurement error in proxy outcomes.

Original: Yet this approach is designed for observational, rather than counterfactual, prediction settings.
Revised: Yet this approach is designed for observational, rather than counterfactual, prediction settings.

Original: Coston et al.
Revised: Coston et al.

Original: [2020b] develops counterfactual modeling and evaluation approaches for decision-support settings, which target proxy outcomes.
Revised: [2020b] develops counterfactual modeling and evaluation approaches for decision-support settings, which target proxy outcomes.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] We highlight opportunities for future work in(c)
Revised: [Yes] We highlight opportunities for future work in(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] https://anonymous.4open.science/r/counterfactual-decision-support-under-TCEF53E/ (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] https://anonymous.4open.science/r/counterfactual-decision-support-under-TCEF53E/ (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Appendix A.1 Details of experimental setup
Revised: A Appendix A.1 Details of experimental setup

Original: Synthetic setup : Conducting evaluations for settings in which measurement error and treatment effects impact observed outcomes is challenging because often there is no way of recovering the target counterfactual outcome or the measurement error parameters.
Revised: Synthetic setup : Conducting evaluations for settings in which measurement error and treatment effects impact observed outcomes is challenging because often there is no way of recovering the target counterfactual outcome or the measurement error parameters.

Original: Therefore, we design an initial empirical evaluation via synthetic data to validate our proposed approach.
Revised: Therefore, we design an initial empirical evaluation via synthetic data to validate our proposed approach.

Original: Previous approaches have also used a uni-variate feature and protected attribute Coston et al.
Revised: Previous approaches have also used a uni-variate feature and protected attribute Coston et al.

Original: [2020b] or 2D synthetic data
Revised: [2020b] or 2D synthetic data

Original: Natarajan et al.
Revised: Natarajan et al.

Original: [2013], De-Arteaga et al.
Revised: [2013], De-Arteaga et al.

Original: We plan to extend to these settings and others with semi-synthetic and real-world data in later evaluations.
Revised: We plan to extend to these settings and others with semi-synthetic and real-world data in later evaluations.

Original: In all experiments, we use sinusoidal target class probability functions η ∗ 0 ( x ) =
Revised: In all experiments, we use sinusoidal target class probability functions η ∗ 0 ( x ) =

Original:   .
Revised:   .

Original: 4 + . 4 cos(9 x + 5 . 5) x ∈ [ − 1 , − . 61] .
Revised: 4 + . 4 cos(9 x + 5 . 5) x ∈ [ − 1 , − . 61] .

Original: 5 + .
Revised: 5 + .

Original: 3 sin(8 x + . 9)
Revised: 3 sin(8 x + . 9)

Original: + .
Revised: + .

Original: 15 sin(10 x + . 2) + . 

Revised: 15 sin(10 x + . 2) + .

Original: sin(30 x + . 2) x ∈ ( − . 61 , .
Revised: sin(30 x + . 2) x ∈ ( − . 61 , .

Original: 921] xx ∈ ( . 921 , 1] (13) and η ∗ 1 ( x ) = .
Revised: 921] xx ∈ ( . 921 , 1] (13) and η ∗ 1 ( x ) = .

Original: 5+ . 5 sin(2 .
Revised: 5+ . 5 sin(2 .

Original: 9 x + . 1)
Revised: 9 x + . 1)

Original: ∀ x ∈
Revised: ∀ x ∈

Original: [ − 1 , 1] .
Revised: [ − 1 , 1] .

Original: We design η ∗ 0 ( x ) to be more challenging to estimate because we use it as our target quantity in this selective intervention setting.
Revised: We design η ∗ 0 ( x ) to be more challenging to estimate because we use it as our target quantity in this selective intervention setting.

Original: In all experiments, we use a linear propensity function π ( x ) = .
Revised: In all experiments, we use a linear propensity function π ( x ) = .

Original: 35 x + . 5 , ∀ x ∈ [ − 1 , 1] .
Revised: 35 x + . 5 , ∀ x ∈ [ − 1 , 1] .

Original: Model and hyperparameters : All experiments use a MLP implemented via PyTorch with layer sizes (1 , 40 , 20 , 4 , 1) and a binary cross-entropy loss satisfying our strictly proper composite criteria for class probability estimation.
Revised: Model and hyperparameters : All experiments use a MLP implemented via PyTorch with layer sizes (1 , 40 , 20 , 4 , 1) and a binary cross-entropy loss satisfying our strictly proper composite criteria for class probability estimation.

Original: We run experiments with α = .
Revised: We run experiments with α = .

Original: Evaluation .
Revised: Evaluation .

Original: In all experiments, we split data 70/30 into training and validation folds.
Revised: In all experiments, we split data 70/30 into training and validation folds.

Original: We evaluate accuracy with respect to Y ∗ 0 on a held-out validation fold.
Revised: We evaluate accuracy with respect to Y ∗ 0 on a held-out validation fold.

Original: We use accuracy as a performance measure rather than a metric such as AU-ROC because AU-ROC is immune to corruption from our error model (see Menon et al.
Revised: We use accuracy as a performance measure rather than a metric such as AU-ROC because AU-ROC is immune to corruption from our error model (see Menon et al.

Original: [2015] for additional details).
Revised: [2015] for additional details).

Original: Computing
Revised: Computing

Original: Environment Experiments were run on a MacBook Pro with 2.6 GHz 6-Core processor with 32 GB of RAM and Google Colab environment with standard runtime configuration.
Revised: Environment Experiments were run on a MacBook Pro with 2.6 GHz 6-Core processor with 32 GB of RAM and Google Colab environment with standard runtime configuration.

Original: A.2 Details of experiment configurations
Revised: A.2 Details of experiment configurations

Original: • Experiment 1 .
Revised: • Experiment 1 .

Original: Reported in Table 1.
Revised: Reported in Table 1.

Original: We run each setting with N = 10000 and average performance over 10 runs with 40 epochs of training per run.
Revised: We run each setting with N = 10000 and average performance over 10 runs with 40 epochs of training per run.

Original: • Experiment 2 .
Revised: • Experiment 2 .

Original: Reported in Figure 3.
Revised: Reported in Figure 3.

Original: We run each setting with N = { 1000 , 2000 , 3000 , 4000 , 5000 , 10000 , 20000 } with 150 runs per setting and 30 epochs of training per round.
Revised: We run each setting with N = { 1000 , 2000 , 3000 , 4000 , 5000 , 10000 , 20000 } with 150 runs per setting and 30 epochs of training per round.

Original: Each round, we sample α 0 , β 0 ∼ U (0 , . 3) .
Revised: Each round, we sample α 0 , β 0 ∼ U (0 , . 3) .

Original: • Experiment 3 .
Revised: • Experiment 3 .

Original: Reported in Figure 4.
Revised: Reported in Figure 4.

Original: We run each setting with N = 10000 with 15 runs per setting and 30 epochs of training per round.
Revised: We run each setting with N = 10000 with 15 runs per setting and 30 epochs of training per round.

Original: We vary α d ( β d ) from .1, .2, .3 and hold β( α 0 ) fixed at 0.
Revised: We vary α d ( β d ) from .1, .2, .3 and hold β( α 0 ) fixed at 0.

Original: We then construct the surrogate loss ˜ ℓ with biased parameter estimates ˆ α 0 , ˆ β 0 parameter estimates.
Revised: We then construct the surrogate loss ˜ ℓ with biased parameter estimates ˆ α 0 , ˆ β 0 parameter estimates.

Original: A.3 Baselines
Revised: A.3 Baselines

Original: • ˆ P ℓ [ Y = 1 | X = x ] (OBS) • ˆ P ℓ
Revised: • ˆ P ℓ [ Y = 1 | X = x ] (OBS) • ˆ P ℓ

Original: [ Y = 1 | D =
Revised: [ Y = 1 | D =

Original: d, X = x ] (FRM) •
Revised: d, X = x ] (FRM) •

Original: ˆ P ˜ ℓ
Revised: ˆ P ˜ ℓ

Original: [ Y = 1 | D =
Revised: [ Y = 1 | D =

Original: d, X = x ] (FRM-SL) • ˆ P ℓ
Revised: d, X = x ] (FRM-SL) • ˆ P ℓ

Original: [ Y d = 1 | X = x ] (PROXY ∗ )
Revised: [ Y d = 1 | X = x ] (PROXY ∗ )

Original: ˆ P ℓ
Revised: ˆ P ℓ

Original: [ Y ∗ d = 1 | X = x ] (TARGET ∗ )
Revised: [ Y ∗ d = 1 | X = x ] (TARGET ∗ )

Original: In causal inference settings, FRM is also referred to as a backdoor covariate adjustment or plug-in estimator Coston et al.
Revised: In causal inference settings, FRM is also referred to as a backdoor covariate adjustment or plug-in estimator Coston et al.

Original: [2020b,a], Pearl [2009].
Revised: [2020b,a], Pearl [2009].