Original: Counterfactual harm
Revised: Counterfactual harm

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: To act safely and ethically in the real world, agents must be able to reason aboutharm and avoid harmful actions.
Revised: To act safely and ethically in the real world, agents must be able to reason aboutharm and avoid harmful actions.

Original: In this paper we develop the ﬁrst statistical deﬁnition of harm and a framework for incorporating harm into algorithmic decisions.
Revised: In this paper we develop the ﬁrst statistical deﬁnition of harm and a framework for incorporating harm into algorithmic decisions.

Original: We argue that harm is fundamentally a counterfactual quantity, and show thatstandard machine learning algorithms that cannot perform counterfactual reasoningare guaranteed to pursue harmful policies following certain distributional shifts.
Revised: We argue that harm is fundamentally a counterfactual quantity, and show thatstandard machine learning algorithms that cannot perform counterfactual reasoningare guaranteed to pursue harmful policies following certain distributional shifts.

Original: Toresolve this we derive a family of counterfactual objective functions that robustlymitigate for harm.
Revised: Toresolve this we derive a family of counterfactual objective functions that robustlymitigate for harm.

Original: We demonstrate our framework on a real-world problem ofidentifying optimal doses for the drug Ariprazole, using a dose-response modellearned form a meta-analysis for randomized control trial data.
Revised: We demonstrate our framework for harm-averse policy learningon a real-world problem of identifying optimal doses for the drug Ariprazole, usinga dose-response model learned form a meta-analysis for randomized control trialdata.

Original:  Introduction
Revised:  Introduction

Original: Machine learning algorithms are increasingly used for making highly consequential decisions inareas such as medicine [21, 63, 84], criminal justice [3, 45, 90], ﬁnance [2, 9, 42], and autonomousvehicles [76].
Revised: Machine learning algorithms are increasingly used for making highly consequential decisions inareas such as medicine [21, 63, 85], criminal justice [3, 45, 90], ﬁnance [2, 9, 42], and autonomousvehicles [77].

Original: These algorithms optimize for outcomes such as survival time, recidivism rate, andﬁnancial cost.
Revised: These algorithms optimize for outcomes such as survival time, recidivism rate, andﬁnancial cost.

Original: However, these are purely factual objectives, depending only on what outcome occurs.
Revised: However, these are purely factual objectives, depending only on what outcome occurs.

Original: On the other hand, human decision making incorporates preferences that depend not only on theoutcome, but why it occurred.
Revised: On the other hand, human decision making incorporates preferences that depend not only on theoutcome, but why it occurred.

Original: For example, consider two treatments for a disease which, when left untreated, has a 50% mortalityrate.
Revised: For example, consider two treatments for a disease which, when left untreated, has a 50% mortalityrate.

Original: Treatment 1 has a 60% chance of curing a patient, and a 40% chance of having no effect, withthe disease progressing as if untreated.
Revised: Treatment 1 has a 60% chance of curing a patient, and a 40% chance of having no effect, withthe disease progressing as if untreated.

Original: Treatment 2 has an 80% chance of curing a patient and a20% chance of killing them.
Revised: Treatment 2 has an 80% chance of curing a patient and a20% chance of killing them.

Original: Treatments 1 and 2 have identical recovery rates, hence any agent thatselects actions based solely on the factual outcome statistics (e.g. the effect of treatment on the treated[39]) will not be able to distinguish between these two treatments.
Revised: Treatments 1 and 2 have identical recovery rates, hence any agent thatselects actions based solely on the factual outcome statistics (e.g. the effect of treatment on the treated[39]) will not be able to distinguish between these two treatments.

Original: However, doctors systematicallyfavour treatment 1 as it achieves the same recovery rate but never harms the patient—there areno patients that would have survived had they not been treated.
Revised: However, doctors systematicallyfavour treatment 1 as it achieves the same recovery rate but never harms the patient—there areno patients that would have survived had they not been treated.

Original: On the other hand, doctors whoadminister treatment 2 risk harming their patients—there are patients who die following treatmentwho would have lived had they not been treated.
Revised: On the other hand, doctors whoadminister treatment 2 risk harming their patients—there are patients who die following treatmentwho would have lived had they not been treated.

Original: While treatments 1 and 2 have the same mortalityrates, the resulting deaths have different causes (the doctor or the disease) which have different ethicalimplications.
Revised: While treatments 1 and 2 have the same mortalityrates, the resulting deaths have different causes (the doctor or the disease) which have different ethicalimplications.

Original: Determining what would have happened to a patient if the doctor hadn’t treated them is a counterfactual inference, which cannot be identiﬁed from outcome statistics alone but requires knowledge of thefunctional causal relation between actions and outcomes [6].
Revised: Determining what would have happened to a patient if the doctor hadn’t treated them is a counterfactual inference, which cannot be identiﬁed from outcome statistics alone but requires knowledge of thefunctional causal relation between actions and outcomes [6].

Original: However, standard machine learningalgorithms are limited to making factual inferences—e.g. learning correlations between actions andoutcomes.
Revised: However, standard machine learningalgorithms are limited to making factual inferences—e.g. learning correlations between actions andoutcomes.

Original: How then can we express these ethical preferences as objective functions and ultimatelyincorporate them into machine learning algorithms?
Revised: How then can we express these ethical preferences as objective functions and ultimatelyincorporate them into machine learning algorithms?

Original: This question has some urgency, given thatmachine learning algorithms currently under development or in deployment will be unable to reasonabout harm and factor it into their decisions.
Revised: This question has some urgency, given thatmachine learning algorithms currently under development or in deployment will be unable to reasonabout harm and factor it into their decisions.

Original: The concept of harm is foundational to many ethical codes and principles; from the Hippocraticoath [81] to environmental policy [53], and from the foundations of classical liberalism [58], to
Revised: The concept of harm is foundational to many ethical codes and principles; from the Hippocraticoath [82] to environmental policy [53], and from the foundations of classical liberalism [58], to

Original: Asimov’s laws of robotics [4].
Revised: Asimov’s laws of robotics [4].

Original: Despite this ubiquity, there is no formal statistical deﬁnition ofharm.
Revised: Despite this ubiquity, there is no formal statistical deﬁnition ofharm.

Original: We address this by translating the predominant account of harm [29, 47] into a statisticalmeasure— counterfactual harm .
Revised: We address this by translating the predominant account of harm [29, 47] into a statisticalmeasure— counterfactual harm .

Original: Our deﬁnition of harm enables us for the ﬁrst time to preciselyanswer the following basic questions,
Revised: Our deﬁnition of harm enables us for the ﬁrst time to preciselyanswer the following basic questions,

Original: Question 1 Did the actions of an agent cause harm, and if so, how much?
Revised: Question 1 Did the actions of an agent cause harm, and if so, how much?

Original: Question 2 How much harm can we expect an action to cause prior to taking it?
Revised: Question 2 How much harm can we expect an action to cause prior to taking it?

Original: Question 

Revised: Question

Original: How can we identify actions that balance the expected harms and beneﬁts?
Revised: How can we identify actions that balance the expected harms and beneﬁts?

Original: In section 2 we introduce concepts from causality, ethics, and expected utility theory used to deriveour results.
Revised: In section 2 we introduce concepts from causality, ethics, and expected utility theory used to deriveour results.

Original: In section 3 we present our deﬁnitions of harm and beneﬁt, and in section 4 wedescribe how harm-aversion can be factored into algorithmic decisions.
Revised: In section 3 we present our deﬁnitions of harm and beneﬁt, and in section 4 wedescribe how harm-aversion can be factored into algorithmic decisions.

Original: In section 5 we prove thatalgorithms based on standard statistical learning theory are incapable of reasoning about harm andare guaranteed to pursue harmful policies following certain distributional shifts, while our approachusing counterfactual objective functions overcomes these problems.
Revised: In section 5 we prove thatalgorithms based on standard statistical learning theory are incapable of reasoning about harm andare guaranteed to pursue harmful policies following certain distributional shifts, while our approachusing counterfactual objective functions overcomes these problems.

Original: Finally, in section 6 we applyour methods to an interpretable machine learning model for determining optimal drug doses, andcompare this to standard algorithms that maximize treatment effects.
Revised: Finally, in section 6 we applyour methods to an interpretable machine learning model for determining optimal drug doses, andcompare this to standard algorithms that maximize treatment effects.

Original:  Prelimiaries
Revised:  Prelimiaries

Original: In this section we introduce some basic principles from causal modelling, expected utility theory andethics that we use to derive our results.
Revised: In this section we introduce some basic principles from causal modelling, expected utility theory andethics that we use to derive our results.

Original: 2.1 Structural causal models
Revised: 2.1 Structural causal models

Original: We provide a brief overview of structural causal models (SCMs) (see Chapter 7 of [66] and [7] forreviews).
Revised: We provide a brief overview of structural causal models (SCMs) (see Chapter 7 of [66] and [7] forreviews).

Original: SCMs represent observable variables
Revised: SCMs represent observable variables

Original: X i as vertices of a directed acyclic graph (DAG),with directed edges representing causal relations between X i and their direct causes (parents), whichincludes observable (endogenous) parents pa i and a single exogenous noise variable E i .
Revised: X i as vertices of a directed acyclic graph (DAG),with directed edges representing causal relations between X i and their direct causes (parents), whichincludes observable (endogenous) parents pa i and a single exogenous noise variable E i .

Original: The value ofeach variable is assigned by a deterministic function
Revised: The value ofeach variable is assigned by a deterministic function

Original: f i of these parents.
Revised: f i of these parents.

Original: Deﬁnition 1 (Structural Causal Model (SCM)) .
Revised: Deﬁnition 1 (Structural Causal Model (SCM)) .

Original: A structural causal model M = h G , E, X, F, P i 64 speciﬁes:
Revised: A structural causal model M = h G , E, X, F, P i 65 speciﬁes:

Original: a set of observable (endogenous) random variables X
Revised: a set of observable (endogenous) random variables X

Original: = { X 1 , . . .
Revised: = { X 1 , . . .

Original: , X N } ,
Revised: , X N } ,

Original: a set of mutually independent noise (exogenous) variables E
Revised: a set of mutually independent noise (exogenous) variables E

Original: = { E 1 , . . .
Revised: = { E 1 , . . .

Original: , E N } with joint
Revised: , E N } with joint

Original: distribution P ( E = e ) =
Revised: distribution P ( E = e ) =

Original: Q
Revised: Q

Original: Ni =1 P
Revised: Ni =1 P

Original: [Equation]
Revised: [Equation]

Original: E i =
Revised: E i =

Original: e i )
Revised: e i )

Original: a directed acyclic graph G , whose nodes are the variables E
Revised: a directed acyclic graph G , whose nodes are the variables E

Original: [ X with directed edges from
Revised: [ X with directed edges from

Original: pa i , E i to X i , where pa i ✓ X denotes the endogenous parents of X i .
Revised: pa i , E i to X i , where pa i ✓ X denotes the endogenous parents of X i .

Original: a set of functions (mechanisms) F = { f 1 , . .
Revised: a set of functions (mechanisms) F = { f 1 , . .

Original: . , f N } , where the collection F forms a mapping
Revised: . , f N } , where the collection F forms a mappingfrom E to X , x i :=

Original: from E to X , x i := f i ( pa i , e i ) , for i = 1 , . . .
Revised: f i ( pa i , e i ) , for i = 1 , . . .

Original: , N,
Revised: , N,

Original: By recursively applying condition 4 every observable variable can be expressed as a function of thenoise variables alone, X i ( e ) =
Revised: By recursively applying condition 4 every observable variable can be expressed as a function of thenoise variables alone, X i ( e ) =

Original: x i where E = e is the joint state of the noise variables, and thedistribution over unobserved noise variables induces a distribution over the observable variables,
Revised: x i where E = e is the joint state of the noise variables, and thedistribution over unobserved noise variables induces a distribution over the observable variables,

Original: [Equation]
Revised: [Equation]

Original: where X = x is the joint state over X .
Revised: where X = x is the joint state over X .

Original: The power of SCMs is that they specify not only the joint distribution P ( X = x ) but also thedistribution of X under all interventions, including incompatible interventions (counterfactuals).
Revised: The power of SCMs is that they specify not only the joint distribution P ( X = x ) but also thedistribution of X under all interventions, including incompatible interventions (counterfactuals).

Original: Interventions describe an external agent changing the underlying causal mechanisms F .
Revised: Interventions describe an external agent changing the underlying causal mechanisms F .

Original: For example,do ( X 1 = x ) denotes intervening to force X 1 = x regardless of the state of its parents, replacingx 1 := f 1 ( pa 1 , e 1 ) !
Revised: For example,do ( X 1 = x ) denotes intervening to force X 1 = x regardless of the state of its parents, replacingx 1 := f 1 ( pa 1 , e 1 ) !

Original: x 1 := x .
Revised: x 1 := x .

Original: The variables in this post-intervention SCM are denoted X ix or
Revised: The variables in this post-intervention SCM are denoted X ix or

Original: X iX 1 =1 and their interventional distribution is given by (1) under the updated mechanisms.
Revised: X iX 1 =1 and their interventional distribution is given by (1) under the updated mechanisms.

Original: The trivial
Revised: The trivial

Original: These interventional distributions quantify the effects of actions and are used to inform decisions.
Revised: These interventional distributions quantify the effects of actions and are used to inform decisions.

Original: Forexample, the conditional average treatment effect (CATE) [78],
Revised: Forexample, the conditional average treatment effect (CATE) [79],

Original: [Equation]
Revised: [Equation]

Original: measures the expected value of outcome Y for treatment T = 1 compared to control T = 0conditional on X , and is used in decision making from precision medicine to economics [1, 77].
Revised: measures the expected value of outcome Y for treatment T = 1 compared to control T = 0conditional on X , and is used in decision making from precision medicine to economics [1, 78].

Original: Counterfactual distributions determine the probability that other outcomes would have occurred hadsome precondition been different.
Revised: Counterfactual distributions determine the probability that other outcomes would have occurred hadsome precondition been different.

Original: For example, for X, Y 2 { T, F } , the causal effect P ( Y X = T = T )is insufﬁcient to determine if do ( X = T ) was a necessary cause of Y = T [34].
Revised: For example, for X, Y 2 { T, F } , the causal effect P ( Y X = T = T )is insufﬁcient to determine if do ( X = T ) was a necessary cause of Y = T [34].

Original: Instead this iscaptured by the counterfactual probability of necessity [83],
Revised: Instead this iscaptured by the counterfactual probability of necessity [84],

Original: [Equation]
Revised: [Equation]

Original: which is the probability that Y would be false if X had been false (the counterfactual proposition),given that X and Y were both true (the factual data).
Revised: which is the probability that Y would be false if X had been false (the counterfactual proposition),given that X and Y were both true (the factual data).

Original: For example, if PN = 0 then Y
Revised: For example, if PN = 0 then Y

Original: =
Revised: =

Original: T wouldhave occured regardless of X = T , whereas if PN = 1 then Y =
Revised: T wouldhave occured regardless of X = T , whereas if PN = 1 then Y =

Original: T could not have occurred without
Revised: T could not have occurred without

Original: X = T .
Revised: X = T .

Original: Equation (3) involves the joint distribution over incompatible states of Y under differentinterventions, Y = T and Y X
Revised: Equation (3) involves the joint distribution over incompatible states of Y under differentinterventions, Y = T and Y X

Original: =
Revised: =

Original: F = F , which cannot be jointly measured and hence (3) cannot beidentiﬁed from data alone.
Revised: F = F , which cannot be jointly measured and hence (3) cannot beidentiﬁed from data alone.

Original: However, (3) can be calculated from the SCM using (1) for the combinedfactual and counterfactual propositions,
Revised: However, (3) can be calculated from the SCM using (1) for the combinedfactual and counterfactual propositions,

Original: [Equation]
Revised: [Equation]

Original: This ability to ascribe causal explanations for data has made SCMs and counterfactual inference akey ingredient in statistical deﬁnitions of blame, intent and responsibility [33, 46, 50], and has seenapplications in AI research ranging from explainability [55, 86], safety [24, 25], and fairness [48], toreinforcement learning [14, 31] and medical diagnosis [72].
Revised: This ability to ascribe causal explanations for data has made SCMs and counterfactual inference akey ingredient in statistical deﬁnitions of blame, intent and responsibility [33, 46, 50], and has seenapplications in AI research ranging from explainability [55, 87], safety [24, 25], and fairness [48], toreinforcement learning [14, 31] and medical diagnosis [73].

Original: 2.2 The counterfactual comparative account of harm
Revised: 2.2 The counterfactual comparative account of harm

Original: The concept of harm is deeply embedded in ethical principles, codes and law.
Revised: The concept of harm is deeply embedded in ethical principles, codes and law.

Original: One famous exampleis the bioethical principle “ﬁrst, do no harm” [80], asserting that the moral responsibility for doctorsto beneﬁt patients is superseded by their responsibility not to harm them [73].
Revised: One famous exampleis the bioethical principle “ﬁrst, do no harm” [81], asserting that the moral responsibility for doctorsto beneﬁt patients is superseded by their responsibility not to harm them [74].

Original: Another example is
Revised: Another example is

Original: John Stuart Mill’s harm principle which forms the basis of classical liberalism [58] and inspired the“zeroth law” of Asimov’s ﬁctional robot governed society, which states that “A robot may not harmhumanity or, by inaction, allow humanity to come to harm” [4].
Revised: John Stuart Mill’s harm principle which forms the basis of classical liberalism [58] and inspired the“zeroth law” of Asimov’s ﬁctional robot governed society, which states that “A robot may not harmhumanity or, by inaction, allow humanity to come to harm” [4].

Original: Despite these attempts to codify harm into rules for governing human and AI behaviour, it is notimmediately clear what we mean when we talk about harm.
Revised: Despite these attempts to codify harm into rules for governing human and AI behaviour, it is notimmediately clear what we mean when we talk about harm.

Original: This can lead to confusion—for example,does “allow humanity to come to harm” describe an agent causing harm by inaction [13] or failing tobeneﬁt?
Revised: This can lead to confusion—for example,does “allow humanity to come to harm” describe an agent causing harm by inaction [13] or failing tobeneﬁt?

Original: The meaning and measure of harm also has important practical ramiﬁcations.
Revised: The meaning and measure of harm also has important practical ramiﬁcations.

Original: For example,establishing negligence in tort law requires establishing that signiﬁcant harm has occurred [88].
Revised: For example,establishing negligence in tort law requires establishing that signiﬁcant harm has occurred [88].

Original: This has motivated work in philosophy, ethics and law to rigorously deﬁne harm, with the mostwidely accepted deﬁnition being the counterfactual comparative account (CCA) [28, 35, 47],
Revised: This has motivated work in philosophy, ethics and law to rigorously deﬁne harm, with the mostwidely accepted deﬁnition being the counterfactual comparative account (CCA) [28, 35, 47],

Original: Deﬁnition 2 (CCA) .
Revised: Deﬁnition 2 (CCA) .

Original: The counterfactual comparative account of harm and beneﬁt states,“An event e or action a harms (beneﬁts) a person overall if and only if she wouldhave been on balance better (worse) off if e had not occurred, or a had not beenperformed.”
Revised: The counterfactual comparative account of harm and beneﬁt states,“An event e or action a harms (beneﬁts) a person overall if and only if she wouldhave been on balance better (worse) off if e had not occurred, or a had not beenperformed.”

Original: Causation is known plays a key role in moral and legal reasoning [49].
Revised: Causation is known plays a key role in moral and legal reasoning [49].

Original: Simply stated, the CCAdeﬁnes harm as causing someone to be worse off [71].
Revised: Simply stated, the CCAdeﬁnes harm as causing someone to be worse off [72].

Original: The judgement of what would have happened‘had the agent had not acted’ is both causal and normative: it requires a causal model to compute thecounterfactual, and normative assumptions to determine what counterfactual comparison should beused.
Revised: The judgement of what would have happened‘had the agent had not acted’ is both causal and normative: it requires a causal model to compute thecounterfactual, and normative assumptions to determine what counterfactual comparison should beused.

Original: We discuss these issues further in Appendix C.
Revised: We discuss these issues further in Appendix C.

Original: The second aspect of the CCA is determining ifthe person would be “better (worse) off”, which we will address using expected utility theory [59].
Revised: The second aspect of the CCA is determining ifthe person would be “better (worse) off”, which we will address using expected utility theory [59].

Original: While the CCA is the predominant deﬁnition of harm and forms the basis of our results, alternativedeﬁnitions have been proposed [36, 62], motivated by scenarios where the CCA appears to give counter-intuitive results [13].
Revised: While the CCA is the predominant deﬁnition of harm and forms the basis of our results, alternativedeﬁnitions have been proposed [36, 62], motivated by scenarios where the CCA appears to give counter-intuitive results [13].

Original: Surprisingly, we ﬁnd these problematic cases are identical to thoseraised in the study of actual causality [34], and can be resolved with a formal causal analysis.
Revised: Surprisingly, we ﬁnd these problematic cases are identical to thoseraised in the study of actual causality [34], and can be resolved with a formal causal analysis.

Original: To ourknowledge, this connection with actual causality has not been noted before, and we present a detailedaccount of the arguments against CCA and how they can be addressed in Appendix B in the hope thiswill stimulate discussion between these two ﬁelds.
Revised: To ourknowledge, this connection with actual causality has not been noted before, and we present a detailedaccount of the arguments against CCA and how they can be addressed in Appendix B in the hope thiswill stimulate discussion between these two ﬁelds.

Original: 2.3 Decision theoretic setup
Revised: 2.3 Decision theoretic setup

Original: In this section we present our setup for calculating the CCA (Deﬁnition 2).
Revised: In this section we present our setup for calculating the CCA (Deﬁnition 2).

Original: The CCA describes aperson (referred to as the user ) being made better or worse off due to the actions of an agent.
Revised: The CCA describes aperson (referred to as the user ) being made better or worse off due to the actions of an agent.

Original: Wefocus on the simplest case involving single agents performing single actions, though ultimately ourdeﬁnitions could be extended to multi-agent or sequential decision problems.
Revised: Wefocus on the simplest case involving single agents performing single actions, though ultimately ourdeﬁnitions could be extended to multi-agent or sequential decision problems.

Original: For causal modelling inthese scenarios we refer the reader to [20, 25, 38].
Revised: For causal modelling inthese scenarios we refer the reader to [20, 25, 38].

Original: [Figure]
Revised: [Equation]

Original: We describe the environment with a structural causal model M which speciﬁes the distributionover environment states W as endogenous variables, P ( w ; M ) .
Revised: We describe the environment with a structural causal model M which speciﬁes the distributionover environment states W as endogenous variables, P ( w ; M ) .

Original: For single actions the environmentvariables can be divided into three disjoint sets W =
Revised: For single actions the environmentvariables can be divided into three disjoint sets W =

Original: A [ X [ Y (Figure 1 a))
Revised: A [ X [ Y (Figure 1 a))

Original: ; the agent’s action A ,outcome variables Y that are descendants of A and so can be inﬂuenced by the agent’s action, andcontext variables X that are non-descendants of nor in A .
Revised: ; the agent’s action A ,outcome variables Y that are descendants of A and so can be inﬂuenced by the agent’s action, andcontext variables X that are non-descendants of nor in A .

Original: In the following, M denotes our SCM model for the environment (Figure 1).
Revised: In the following, M denotes our SCM model for the environment (Figure 1).

Original: The user’s preferencesover environment states are speciﬁed by their utility function U ( w ) =
Revised: The user’s preferencesover environment states are speciﬁed by their utility function U ( w ) =

Original: U ( a, x, y ) , and for a givenaction the user’s expected utility is,
Revised: U ( a, x, y ) , and for a givenaction the user’s expected utility is,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In sections 3-4 we focus on the case where the agent acts to maximize the user’s expected utility, withoptimal actions a max = arg max a E
Revised: In sections 3-4 we focus on the case where the agent acts to maximize the user’s expected utility, withoptimal actions a max = arg max a E

Original: [ U | a, x ] .
Revised: [ U | a, x ] .

Original: While this is similar to the standard setup in expectedutility theory [30, 59, 74], note that U describes the preferences of the user, and in general the agentcan pursue any objective and could be entirely unaware of the user’s preferences.
Revised: While this is similar to the standard setup in expectedutility theory [30, 59, 75], note that U describes the preferences of the user, and in general the agentcan pursue any objective and could be entirely unaware of the user’s preferences.

Original: In section 5 weextend our analysis to agents pursuing arbitrary objectives.
Revised: In section 5 weextend our analysis to agents pursuing arbitrary objectives.

Original: No factual measure can robustly identify harmful actions.
Revised: Example 1: conditional average treatment effects.

Original: Before giving a formal deﬁnition ofharm, we return to our example from section 1 and show that any deﬁnition satisfying basic intuitionsmust be counterfactual. Treatment 1 has a 60% chance of curing the patient and 40% chance of havingno effect, with the disease progressing as if untreated, giving a total recovery rate of 80% .Treatment2 has 80% chance of curing the patient and 20% chance of killing them, likewise an 80% recoveryrate. Therefore the treatments have identical factual outcome statistics P ( y
Revised: Returning to our example from section 1,consider an agent choosing between a placebo A = 0 , or treatments 1 and 2 .

Original: | A = 1) = P ( y | A = 2)where Y =0 indicates mortality and Y =1 recovery.
Revised: Y = 1 indicatesrecovery and Y = 0 mortality.

Original: However, treatment 2 is intuitively more harmful,as there are patients killed by the treatment who would have recovered if we hadn’t treated them,whereas for treatment 1 any patients who die would have died if left untreated.
Revised: In the trial, harm is measured by comparison to ‘no treatment’ ¯ a = 0 . X = x describes any variables that can potentially inﬂuence the agent’s choice of intervention and/or Y , e.g. the patient’s medical history.

Original: Consider some factualmeasure of harm H ( a, P )which is a function of the treatment choice A and the factual outcomestatistics P =P ( y = 1 | 
Revised: Consider patients (users) whose preferences are fully determined by survival, e.g. U ( a, x, y ) = y . The expected utility is E [ U | a, x ] =

Original: A =1) =
Revised: [Equation]

Original: P  ( y = 1 | A =2) .
Revised: P y yP ( y a | x ) =

Original: Note there are no confounders so we can treat
Revised: E [ Y a | x ] , where inthe last line we have use the fact that there are no confounders between A and Y . The agent’s policy istherefore to maximize the treatment effect a max = arg max a E

Original: P ( y |  a ) and P ( y a ) interchangeably.
Revised: [ Y a | x ] , which is equivalent to a max =arg max a { E [ Y a | x ]  E

Original: To match our intuitions it must be that H (2 , P ) > H (1 , P ) . Consider a distributional shift where now treatment 1 has an 80% chance of curing and a 20% chanceof killing, whereas treatment 2 has a 60% chance of curing and a 40% chance of having no effect.
Revised: [ Y 0 | x ] } i.e. maximizing the CATE (2)—a standard objective for identifying 163 optimal treatments [11, 71].

Original: Essentially, we swap the causal mechanisms generating Y from T (see Appendix F  for the SCMdescribing this example). Following this shift, treatment 1 is harmful and treatment 2 is not. However,
Revised: In Appendix F we derive an SCM model for the treatments described inthe introduction, and show that E [ Y A =1 ] =

Original: P is not changed by this distributional shift. Therefore it must be that H (2 , P ) > H (1 , P ) followingthe shift, and the harm measure incorrectly identiﬁes treatment 1 as being less harmful than treatment2. In section 5 we extend this to a formal proof for all decision tasks.
Revised: E [ Y A =2 ] = 0 . 8 , i.e. the expected utility and CATE is thesame for treatments 1 and 2 .

Original:  Counterfactual harm
Revised:  Counterfactual harm

Original: We now present our deﬁnitions for harm and beneﬁt based on Deﬁnition 2.
Revised: We now present our deﬁnitions for harm and beneﬁt based on Deﬁnition 2.

Original: For the examples describedin this paper we focus on simple environments with single outcome variables, for which harm canbe determined by comparing the factual utility received by the user to the utility they would havereceived had the agent taken some default action
Revised: For the examples describedin this paper we focus on simple environments with single outcome variables, for which harm canbe determined by comparing the factual utility received by the user to the utility they would havereceived had the agent taken some default action

Original: A
Revised: A

Original: = ¯ a (Deﬁnition 3).
Revised: = ¯ a (Deﬁnition 3).

Original: In Appendix C we describehow this default action can be determined.
Revised: In Appendix C we describehow this default action can be determined.

Original: However, some environments studied in the harm literatureinvolve multiple outcome variables and require a path-dependent measure of harm, which we providein Appendix A along with examples in Appendices B-C. In the following, counterfactual states areidentiﬁed with an ⇤ , e.g. Z =
Revised: However, some environments studied in the harm literatureinvolve multiple outcome variables and require a path-dependent measure of harm, which we providein Appendix A along with examples in Appendices B-C. In the following, counterfactual states areidentiﬁed with an ⇤ , e.g. Z =

Original: z ⇤ .
Revised: z ⇤ .

Original: First we deﬁne the harm caused by an action given we observe theoutcome (Question 1).
Revised: First we deﬁne the harm caused by an action given we observe theoutcome (Question 1).

Original: Deﬁnition 3 (Counterfactual harm & beneﬁt) .
Revised: Deﬁnition 3 (Counterfactual harm & beneﬁt) .

Original: The harm caused by action A = a given context
Revised: The harm caused by action A = a given context

Original: X = x and outcome Y =
Revised: X = x and outcome Y =

Original: y compared to the default action A
Revised: y compared to the default action A

Original: = ¯ a is,
Revised: = ¯ a is,

Original: [Equation]
Revised: [Equation]

Original: and the expected beneﬁt is
Revised: and the expected beneﬁt is

Original: [Equation]
Revised: [Equation]

Original: where M is the SCM describing the environment and U is the user’s utility.
Revised: where M is the SCM describing the environment and U is the user’s utility.

Original: The counterfactual harm (beneﬁt) is the expected increase (decrease) in utility had the agent taken thedefault action
Revised: The counterfactual harm (beneﬁt) is the expected increase (decrease) in utility had the agent taken thedefault action

Original: A = ¯ a , given they took action A
Revised: A = ¯ a , given they took action A

Original: = a in context X =
Revised: = a in context X =

Original: x resulting in outcome Y =
Revised: x resulting in outcome Y =

Original: y .
Revised: y .

Original: Taking the max in (6) and (7) ensures that we only include counterfactual outcomes where the utilityis higher (lower) in the counterfactual world.
Revised: Taking the max in (6) and (7) ensures that we only include counterfactual outcomes where the utilityis higher (lower) in the counterfactual world.

Original: Note that X ¯ a = X as X is not a descendant of A , sothe factual and counterfactual contexts are identical [79].
Revised: Note that X ¯ a = X as X is not a descendant of A , sothe factual and counterfactual contexts are identical [80].

Original: To determine the expected harm or beneﬁt of an action prior to taking it (Question 2) we calculate the expected value of (6) and (7) over the outcome distribution, e.g. E
Revised: To determine the expected harm or beneﬁt of an action prior to taking it (Question 2) we calculate the expected value of (6) and (7) over the outcome distribution, e.g. E

Original: [ h | a, x ; M ] =
Revised: [ h | a, x ; M ] =

Original: R y P ( y | a, x ; M ) h ( a, x, y ; M ) .
Revised: R y P ( y | a, x ; M ) h ( a, x, y ; M ) .

Original: Note these form a counterfactual decomposition of the expectedutility.
Revised: Note these form a counterfactual decomposition of the expectedutility.

Original: Theorem 1 (harm-beneﬁt trade-off) .
Revised: Theorem 1 (harm-beneﬁt trade-off) .

Original: The difference in expected utility for action A = a and thedefault action
Revised: The difference in expected utility for action A = a and thedefault action

Original: A
Revised: A

Original: = ¯ a is given by,
Revised: = ¯ a is given by,

Original: [Equation]
Revised: [Equation]

Original: (Proof: see Appendix E).
Revised: (Proof: see Appendix E).

Original: Example 1: Returning to our treatment example, the expected counterfactual harm for default action¯ a = 0 is, E
Revised: Example 2: Returning to Example 1, the expected counterfactual harm for default action ¯ a = 0is, E

Original: [ h | a ; M 
Revised: [ h | a ; M ] =

Original: [Equation]
Revised: P y ⇤ ,y P ( Y ¯ a = y ⇤ , y | a ; M ) max { 0 , y ⇤  y }

Original: =
Revised: =

Original: [Equation] where we have used U ( a, y ) =
Revised: P ( Y 0 = 1 , Y a = 0; M ) wherewe have used U ( a, y ) =

Original: y and the fact that Y ¯ a ?
Revised: y and the fact that Y ¯ a ?

Original: A and there are no latent confounders togive P ( Y 0 =
Revised: A and there are no latent confounders to give P ( Y 0 =

Original: y ⇤ , Y =
Revised: y ⇤ , Y =

Original: y | a ) =
Revised: y | a ) =

Original: P ( Y 0 =
Revised: P ( Y 0 =

Original: y ⇤ , Y a ) .
Revised: y ⇤ , Y a ) .

Original: This is the probability that a patient dies followingtreatment A = a and would have recovered had they received no treatment.
Revised: This is the probability that a patient dies followingtreatment A = a and would have recovered had they received no treatment.

Original: In Appendix F weshow that P ( Y 0 = 1 , Y 1 = 0)
Revised: In Appendix F weshow that P ( Y 0 = 1 , Y 1 = 0)

Original: = 

Revised: =

Original: (treatment 1 causes zero harm) and P ( Y 0 = 1 , Y 2 = 0)
Revised: (treatment 1 causes zero harm) and P ( Y 0 = 1 , Y 2 = 0)

Original: = 0 .
Revised: = 0 .

Original: 1(treatment 2 causes non-zero harm), reﬂecting our intuition that treatment 2 is more harmful thantreatment 1, despite having the same causal effect / CATE.
Revised: 1(treatment 2 causes non-zero harm), reﬂecting our intuition that treatment 2 is more harmful thantreatment 1, despite having the same causal effect / CATE.

Original: Hence, the counterfactual harm allows usto differentiate between these two treatments.
Revised: Hence, the counterfactual harm allows usto differentiate between these two treatments.

Original:  Harm in decision making
Revised:  Harm in decision making

Original: Now we have expressions for the expected harm and beneﬁt, how can we incorporate these intothe agent’s decisions?
Revised: Now we have expressions for the expected harm and beneﬁt, how can we incorporate these intothe agent’s decisions?

Original: Consider two actions a, a 0 such that E [ b | a 0 , x ; M ] =
Revised: Consider two actions a, a 0 such that E [ b | a 0 , x ; M ] =

Original: E [ b | a, x ; M ] + Kand E
Revised: E [ b | a, x ; M ] + Kand E

Original: [ h | a 0 , x ; M ] =
Revised: [ h | a 0 , x ; M ] =

Original: E [ h | a, x ; M ] + K where K 2 R .
Revised: E [ h | a, x ; M ] + K where K 2 R .

Original: By Theorem 1 they must have the sameexpected utility, as E
Revised: By Theorem 1 they must have the sameexpected utility, as E

Original: [ U | a 0 , x ] =
Revised: [ U | a 0 , x ] =

Original: E [ b | a 0 , x ; M ]  E [ h | a 0 , x ; M ] + E
Revised: E [ b | a 0 , x ; M ]  E [ h | a 0 , x ; M ] + E

Original: [ U | ¯ a
Revised: [ U | ¯ a

Original: , x ] = E [ b | a, x ; M ] + K 
Revised: , x ] = E [ b | a, x ; M ] + K

Original: E [ h | a, x ; M ]  K + E
Revised: E [ h | a, x ; M ]  K + E

Original: [ U | ¯ a
Revised: [ U | ¯ a

Original: , x ] = E
Revised: , x ] = E

Original: [ U | a, x ] .
Revised: [ U | a, x ] .

Original: Therefore expected utility maximizers are indifferentto harm, and will are willing to increase harm for an equal increase in beneﬁt.
Revised: Therefore expected utility maximizers are indifferentto harm, and will are willing to increase harm for an equal increase in beneﬁt.

Original: We say that an agent is harm averse if they are willing to risk causing harm only if they expect acomparatively greater beneﬁt.
Revised: We say that an agent is harm averse if they are willing to risk causing harm only if they expect acomparatively greater beneﬁt.

Original: Theorem 1 suggests a simple way to overcome harm indifferencein the expected utility by assigning a higher weight to the harm component of the decomposition(8).
Revised: Theorem 1 suggests a simple way to overcome harm indifferencein the expected utility by assigning a higher weight to the harm component of the decomposition(8).

Original: If instead we maximize an adjusted expected utility E [ V | a, x ; M ] :=
Revised: If instead we maximize an adjusted expected utility E [ V | a, x ; M ] :=

Original: E [ U | x ] + E [ b | a, x ; M ]  217 (1 +  )
Revised: E [ U | x ] + E [ b | a, x ; M ]  212 (1 +  )

Original: E [ h | a, x ; M ] where  2 R , then if E [ V | a, x ; M ] =
Revised: E [ h | a, x ; M ] where  2 R , then if E [ V | a, x ; M ] =

Original: E [ V | a 0 , x ; M ] and E [ h | a 0 , x ; M ] =
Revised: E [ V | a 0 , x ; M ] and E [ h | a 0 , x ; M ] =

Original: E [ h | a, x ; M ] + K then E [ b | a 0 , x ; M ] =
Revised: E [ h | a, x ; M ] + K then E [ b | a 0 , x ; M ] =

Original: E [ b | a, x ; M ] + (1 +  ) K , and the agent is willing to tradeharm for beneﬁt at a 1 : (1 +  ) ratio.
Revised: E [ b | a, x ; M ] + (1 +  ) K , and the agent is willing to tradeharm for beneﬁt at a 1 : (1 +  ) ratio.

Original: Taking inspiration from risk aversion [40], we refer to  asthe agent’s harm aversion .
Revised: Taking inspiration from risk aversion [40], we refer to  asthe agent’s harm aversion .

Original: To achieve the desired trade-off we replace the utility function with theharm-penalized utility (HPU),
Revised: To achieve the desired trade-off we replace the utility function with theharm-penalized utility (HPU),

Original: Deﬁnition 4 (Harm penalized utility (HPU)) .
Revised: Deﬁnition 4 (Harm penalized utility (HPU)) .

Original: For utility U and an environment described by SCM
Revised: For utility U and an environment described by SCM

Original: M , the harm-penalized utility (HPU) is,
Revised: M , the harm-penalized utility (HPU) is,

Original: [Equation]
Revised: [Equation]

Original: where h ( a, x, y ; M ) is the harm (Deﬁnition 3) and  is the harm aversion.
Revised: where h ( a, x, y ; M ) is the harm (Deﬁnition 3) and  is the harm aversion.

Original: For  = 

Revised: For  =

Original: the agent is harm indifferent and optimizes the standard expected utility.
Revised: the agent is harm indifferent and optimizes the standard expected utility.

Original: For  > 0 theagent is harm averse , and is willing to reduce the expected utility to achieve a greater reduction to theexpected harm.
Revised: For  > 0 theagent is harm averse , and is willing to reduce the expected utility to achieve a greater reduction to theexpected harm.

Original: For  < 0 the agent is harm seeking , and is willing to reduce the expected utility inorder to increase the expected harm.
Revised: For  < 0 the agent is harm seeking , and is willing to reduce the expected utility inorder to increase the expected harm.

Original: Maximizing the expected HPU allows agents to choose actions that balance the expected beneﬁtsand harms (Quesiton 2 section 1).
Revised: Maximizing the expected HPU allows agents to choose actions that balance the expected beneﬁtsand harms (Quesiton 2 section 1).

Original: We refer to the expected HPU (9) as a counterfactual objectivefunction as V ( a, x, y ; M ) is a counterfactual expectation that cannot be evaluated without knowledgeof M .
Revised: We refer to the expected HPU (9) as a counterfactual objectivefunction as V ( a, x, y ; M ) is a counterfactual expectation that cannot be evaluated without knowledgeof M .

Original: We refer to the expected utility as a factual objective function , as it can be evaluated on dataalone without knowledge of M .
Revised: We refer to the expected utility as a factual objective function , as it can be evaluated on dataalone without knowledge of M .

Original: As we show in the following example, harm-aversion can lead tovery different behaviours compared to other factual approaches to safety such as risk aversion [40].
Revised: As we show in the following example, harm-aversion can lead tovery different behaviours compared to other factual approaches to safety such as risk aversion [40].

Original: Example 3: assistant paradox.
Revised: Example 3: assistant paradox.

Original: Alice invests $80 in the stock market, and expects a normallydistributed return Y ⇠ N ( µ,  2 ) with µ =  = $100 .
Revised: Alice invests $80 in the stock market, and expects a normallydistributed return Y ⇠ N ( µ,  2 ) with µ =  = $100 .

Original: She asks her new AI assistant to manage herinvestment, and after a lengthy analysis the AI identiﬁes three possible actions;
Revised: She asks her new AI assistant to manage herinvestment, and after a lengthy analysis the AI identiﬁes three possible actions;

Original: Multiply the investment by 0  K  20 , resulting in a return Y !
Revised: Multiply the investment by 0  K  20 , resulting in a return Y !

Original: K Y ,
Revised: K Y ,

Original: Perform a clever trade that Alice was unaware of, increasing her return by +$10 with
Revised: Perform a clever trade that Alice was unaware of, increasing her return by +$10 with

Original: certainty, Y !
Revised: certainty, Y !

Original: Y + 10 ,
Revised: Y + 10 ,

Original: Cancel Alice’s investment, returning $80 .
Revised: Cancel Alice’s investment, returning $80 .

Original: Consider 3 agents; Agent 1 maximizes expected return, choosing a = arg max a E
Revised: Consider 3 agents, Agent 1 maximizes expected return, choosing a = arg max a E

Original: [ Y | a ] , Agent2 is risk-averse, in the sense of [40], choosing a = arg max a { E [ Y | a ]   Var [ Y | a ] } ,Agent 3 isharm-averse, choosing a = arg max a { E [ Y | a ]   E
Revised: [ Y | a ] ,2. Agent 2 is risk-averse, in the sense of [40], choosing a = arg max a { E [ Y | a ]   Var [ Y | a ] } ,3. Agent 3 is harm-averse, choosing a = arg max a { E [ Y | a ]   E

Original: [ h | a ; M ] } .
Revised: [ h | a ; M ] } .

Original: In Appendix G we derive the optimal policies for Agents 1-3, and calculate the harm with respectto the default action where the agent leaves Alice’s investment unaltered (i.e. action 1, K = 1 ).
Revised: In Appendix G we derive the optimal policies for Agents 1-3, and calculate the harm with respectto the default action where the agent leaves Alice’s investment unaltered (i.e. action 1, K = 1 ).

Original: Agent 1 chooses action 1 and K = 20 , maximising Alice’s return but also maximizing the expectedharm—e.g. if Alice would have lost $100 she will now lose $2000 .
Revised: Agent 1 chooses action 1 and K = 20 , maximising Alice’s return but also maximizing the expectedharm—e.g. if Alice would have lost $100 she will now lose $2000 .

Original: The standard approach inportfolio theory [56] is to use agent 2 with an appropriate risk aversion  .
Revised: The standard approach inportfolio theory [56] is to use agent 2 with an appropriate risk aversion  .

Original: However, agent 2 neverchooses action 2 for any  , despite action 2 always increasing Alice’s return (causing zero harm andnon-zero beneﬁt).
Revised: However, agent 2 neverchooses action 2 for any  , despite action 2 always increasing Alice’s return (causing zero harm andnon-zero beneﬁt).

Original: Agent 2 will reduce or even cancel Alice’s investment (for  > 0 . 0032 ) rather thantake action 2.
Revised: Agent 2 will reduce or even cancel Alice’s investment (for  > 0 . 0032 ) rather thantake action 2.

Original: This is because the risk-averse objective is factual, and cannot differentiate betweenlosses caused by the agent’s actions from those that would have occurred regardless of the agent’sactions.
Revised: This is because the risk-averse objective is factual, and cannot differentiate betweenlosses caused by the agent’s actions from those that would have occurred regardless of the agent’sactions.

Original: Consequently, agent 2 treats all outcomes as being caused by its actions, and is willing toharm Alice by reducing or even cancelling her investment in order to minimize her risk.
Revised: Consequently, agent 2 treats all outcomes as being caused by its actions, and is willing toharm Alice by reducing or even cancelling her investment in order to minimize her risk.

Original: On the otherhand agent 3 never reduces Alice’s expected return, choosing either action 1 or action 2 depending onthe degree of harm aversion.
Revised: On the otherhand agent 3 never reduces Alice’s expected return, choosing either action 1 or action 2 depending onthe degree of harm aversion.

Original: The surprising behaviour of agent 2 highlights that factual objectives (e.g. risk aversion) cannotcapture preferences that depend not only on the outcome but on its cause.
Revised: The surprising behaviour of agent 2 highlights that factual objectives (e.g. risk aversion) cannotcapture preferences that depend not only on the outcome but on its cause.

Original: For example, Alicemay assign a different weight to losses that are due to her assistant than due to exogenous marketﬂuctuations or her own actions—e.g. if her bet would have made money but is overridden by herassistant.
Revised: For example, Alicemay assign a different weight to losses that are due to her assistant than due to exogenous marketﬂuctuations or her own actions—e.g. if her bet would have made money but is overridden by her assistant.

Original: Consequently she may prefer action 2 over action 1 or action 3 with K < 1 .
Revised: Consequently she may prefer action 2 over action 1 or action 3 with K < 1 .

Original: While this canbe achieved using a factual objective function that favours action 2 a priori (assuming Alice is awareof this action), next we show this approach fails to generalize and actually leads to harmful policies.
Revised: While this canbe achieved using a factual objective function that favours action 2 a priori (assuming Alice is awareof this action), next we show this approach fails to generalize and actually leads to harmful policies.

Original:  Counterfactual reasoning is necessary for harm aversion
Revised:  Counterfactual reasoning is necessary for harm aversion

Original: In the previous sections we saw examples where expected utility maximizers harm the user.
Revised: In the previous sections we saw examples where expected utility maximizers harm the user.

Original: However,these examples describe speciﬁc environments and utility functions, and assume the agent maximizesthe expected utility instead of some other (potentially safer) objective.
Revised: However,these examples describe speciﬁc environments and utility functions, and assume the agent maximizesthe expected utility instead of some other (potentially safer) objective.

Original: In this section we consider userswith arbitrary utility functions and agents with arbitrary objectives to answer; when is maximizingthe expected utility harmful?
Revised: In this section we consider userswith arbitrary utility functions and agents with arbitrary objectives to answer; when is maximizingthe expected utility harmful?

Original: And are there other factual objective functions that can overcome this?
Revised: And are there other factual objective functions that can overcome this?

Original: Factual objective functions can be expressed as the expected value of a real-valued function J ( a, x, y ) .
Revised: Factual objective functions can be expressed as the expected value of a real-valued function J ( a, x, y ) .

Original: Examples include the expected utility, cost functions [10], and the cumulative discounted reward [82].
Revised: Examples include the expected utility, cost functions [10], and the cumulative discounted reward [83].

Original: Once J is speciﬁed the optimal actions in environment M are given by,
Revised: Once J is speciﬁed the optimal actions in environment M are given by,

Original: [Equation]
Revised: [Equation]

Original: Note that (10) can be evaluated from data alone (i.e. from samples of the joint distribution of A, Xand Y ) without knowledge of M .
Revised: Note that (10) can be evaluated from data alone (i.e. from samples of the joint distribution of A, Xand Y ) without knowledge of M .

Original: In the following we relax our requirement that agents have someﬁxed harm aversion  , requiring only that they don’t take needlessly harmful actions.
Revised: In the following we relax our requirement that agents have someﬁxed harm aversion  , requiring only that they don’t take needlessly harmful actions.

Original: Deﬁnition 5 (Harmful actions & policies) .
Revised: Deﬁnition 5 (Harmful actions & policies) .

Original: An action A = a is needlessly harmful in context X =
Revised: An action A = a is needlessly harmful in context X =

Original: xand environment M if there is a 0 

Revised: xand environment M if there is a 0

Original: = a such that E
Revised: = a such that E

Original: [ U | a 0 , x ]  E
Revised: [ U | a 0 , x ]  E

Original: [ U | a, x ] and E [ h | a 0 , x ; M ] <
Revised: [ U | a, x ] and E [ h | a 0 , x ; M ] <

Original: E [ h | a, x ; M ] .
Revised: E [ h | a, x ; M ] .

Original: A policy is harmful if it assigns a non-zero probability to a needlessly harmful action.
Revised: A policy is harmful if it assigns a non-zero probability to a needlessly harmful action.

Original: Deﬁnition 6 (Harmful objectives) .
Revised: Deﬁnition 6 (Harmful objectives) .

Original: An objective E [ J | a, x ] is harmful in environment M if there is apolicy that maximizes the expected value of J and is harmful.
Revised: An objective E [ J | a, x ] is harmful in environment M if there is apolicy that maximizes the expected value of J and is harmful.

Original: Note that needlessly harmful actions violate any degree of harm aversion  > 0 , as the agent iswilling to choose actions that are strictly more harmful for no additional beneﬁt.
Revised: Note that needlessly harmful actions violate any degree of harm aversion  > 0 , as the agent iswilling to choose actions that are strictly more harmful for no additional beneﬁt.

Original: In examples 1 and 2,treatment 2 is a needlessly harmful action and the CATE is a harmful objective.
Revised: In examples 1 and 2,treatment 2 is a needlessly harmful action and the CATE is a harmful objective.

Original: First, we show thatmaximizing the expected HPU is never a harmful objective in any environment,
Revised: First, we show thatmaximizing the expected HPU is never a harmful objective in any environment,

Original: Theorem 2.
Revised: Theorem 2.

Original: For any utility functions U , environment M and default action
Revised: For any utility functions U , environment M and default action

Original: A
Revised: A

Original: = ¯ a the expected
Revised: = ¯ a the expected

Original: HPU is never a harmful objective for  > 0 .
Revised: HPU is never a harmful objective for  > 0 .

Original: (Proof: see Appendix J).
Revised: (Proof: see Appendix J).

Original: It is vital that agents continue to pursue safe objectives following changes to the environment(distributional shifts).
Revised: It is vital that agents continue to pursue safe objectives following changes to the environment(distributional shifts).

Original: At the very least, agents should be able to re-train in the shifted environmentwithout needing to tweak their objective functions.
Revised: At the very least, agents should be able to re-train in the shifted environmentwithout needing to tweak their objective functions.

Original: In SCMs distributional shifts are represented asinterventions that change the exogenous noise distribution and/or the underlying causal mechanisms[69, 75].
Revised: In SCMs distributional shifts are represented asinterventions that change the exogenous noise distribution and/or the underlying causal mechanisms[69, 76].

Original: We focus on distributional shifts that change the outcome distribution P ( y | a, x ; M ) alone.
Revised: We focus on distributional shifts that change the outcome distribution P ( y | a, x ; M ) alone.

Original: Deﬁnition 7 (Outcome distributional shift) .
Revised: Deﬁnition 7 (Outcome distributional shift) .

Original: For an environment described by SCM M , M !
Revised: For an environment described by SCM M , M !

Original: M 0 is 294 a shift in the outcome distribution if the exogenous noise distribution for Y changes P ( e Y ;
Revised: M 0 is 290 a shift in the outcome distribution if the exogenous noise distribution for Y changes P ( e Y ;

Original: M 0 ) 6 = 295 P ( e Y ;
Revised: M 0 ) 6 = 291 P ( e Y ;

Original: M ) and/or the causal mechanism changes f 0 Y ( a, x, e Y ) 

Revised: M ) and/or the causal mechanism changes f 0 Y ( a, x, e Y )

Original: = f Y ( a, x, e Y ) .
Revised: = f Y ( a, x, e Y ) .

Original: By Theorem 2 the HPU is never harmful following any distributional shift.
Revised: By Theorem 2 the HPU is never harmful following any distributional shift.

Original: On the other hand, we ﬁndthat maximizing almost any factual utility function will result in harmful policies under distributionalshifts.
Revised: On the other hand, we ﬁndthat maximizing almost any factual utility function will result in harmful policies under distributionalshifts.

Original: In the following we assume some degree of outcome dependence in the user’s utility function.
Revised: In the following we assume some degree of outcome dependence in the user’s utility function.

Original: Deﬁnition 8 (Outcome dependence) .
Revised: Deﬁnition 8 (Outcome dependence) .

Original: U ( a, x, y ) is outcome dependent for a set of actions C ={ a 1 , . . . , a N } in context X =
Revised: U ( a, x, y ) is outcome dependent for a set of actions C ={ a 1 , . . . , a N } in context X =

Original: x
Revised: x

Original: if 8 a i , a j 2 C , i 6 = j , max y U ( a i , x, y ) > min y U ( a j , x, y ) .
Revised: if 8 a i , a j 2 C , i 6 = j , max y U ( a i , x, y ) > min y U ( a j , x, y ) .

Original: If there is no outcome dependence then the optimal action is independent of the outcome distribution
Revised: If there is no outcome dependence then the optimal action is independent of the outcome distribution

Original: P ( y | a, x ) , and no learning is required to determine the optimal policy.
Revised: P ( y | a, x ) , and no learning is required to determine the optimal policy.

Original: Typically we are interested intasks that require some degree of learning and hence outcome dependence.
Revised: Typically we are interested intasks that require some degree of learning and hence outcome dependence.

Original: Theorem 3.
Revised: Theorem 3.

Original: If for any context X = x the user’s utility function is outcome dependent for the defaultaction in that context ¯ a ( x ) and another action
Revised: If for any context X = x the user’s utility function is outcome dependent for the defaultaction in that context ¯ a ( x ) and another action

Original: a 6 = ¯ a ( x ) , then there is an outcome distributional shiftsuch that U is harmful in the shifted environment.
Revised: a 6 = ¯ a ( x ) , then there is an outcome distributional shiftsuch that U is harmful in the shifted environment.

Original: (Proof: see Appendix J).
Revised: (Proof: see Appendix J).

Original: To understand the implications of Theorem 3 we can return to examples 1 & 2 and no longer assumea speciﬁc utility function or outcome statistics.
Revised: To understand the implications of Theorem 3 we can return to examples 1 & 2 and no longer assumea speciﬁc utility function or outcome statistics.

Original: Theorem 3 implies that for any agent maximizing theuser’s utility, there is always a a distributional shift such that the agent will cause unnecessary harmto patient in the shifted environment, compared to not treating them ( ¯ a = 0 ) or any other defaultaction.
Revised: Theorem 3 implies that for any agent maximizing the user’s utility, there is always a a distributional shift such that the agent will cause unnecessary harmto patient in the shifted environment, compared to not treating them ( ¯ a = 0 ) or any other defaultaction.

Original: Unlike in examples 1 & 2 we allow the patient’s utility function to depend on the treatmentchoice, and for treatments to vary in effectiveness.
Revised: Unlike in examples 1 & 2 we allow the patient’s utility function to depend on the treatmentchoice, and for treatments to vary in effectiveness.

Original: The only assumption we make is that whatever the user’s utility function is, it is not true that U ( A = a, x, Y = 0) >
Revised: The only assumption we make is that whatever theuser’s utility function is, it is not true that U ( A = a, x, Y = 0) >

Original: U ( A = 0 , x, Y = 1) 8 a, x .
Revised: U ( A = 0 , x, Y = 1) 8 a, x .

Original: If thiswere true then the patient’s preferences are dominated by the agents action choice alone, i.e. the userwould always prefer to die so long as they are treated, rather than not being treated and surviving.
Revised: If thiswere true then the patient’s preferences are dominated by the agents action choice alone, i.e. the userwould always prefer to die so long as they are treated, rather than not being treated and surviving.

Original: While robust harm aversion is not possible for expected utility maximizers in general, it seems likelyat ﬁrst that we can train robustly harm-averse agents using other learning objectives.
Revised: While robust harm aversion is not possible for expected utility maximizers in general, it seems likelyat ﬁrst that we can train robustly harm-averse agents using other learning objectives.

Original: One possibilitywould be to use human interactions and demonstrations [16, 51, 60] that implicitly encode harmaversion, with J ( a, x, y ) representing user feedback or some learned (factual) reward model.
Revised: One possibilitywould be to use human interactions and demonstrations [16, 51, 60] that implicitly encode harmaversion, with J ( a, x, y ) representing user feedback or some learned (factual) reward model.

Original: Inexamples 1 & 2 users with utility U ( a, x, y ) =
Revised: Inexamples 1 & 2 users with utility U ( a, x, y ) =

Original: y could assign a higher cost to deaths followingtreatment 2 compared to treatment 1, J ( A = 2 , Y = 0)
Revised: y could assign a higher cost to deaths followingtreatment 2 compared to treatment 1, J ( A = 2 , Y = 0)

Original: <
Revised: <

Original: J ( A = 1 , Y = 0) , with J implicitlyencoding the harm caused by treatment 2.
Revised: J ( A = 1 , Y = 0) , with J implicitlyencoding the harm caused by treatment 2.

Original: This would allow robust harm-averse agents to be trainedusing standard statistical learning techniques, without needing to learn M or perform counterfactualinference.
Revised: This would allow robust harm-averse agents to be trainedusing standard statistical learning techniques, without needing to learn M or perform counterfactualinference.

Original: However, we now show this is not possible in general.
Revised: However, we now show this is not possible in general.

Original: Theorem 4.
Revised: Theorem 4.

Original: If for any context X = x the user’s utility function is outcome dependent for the defaultaction in that context ¯ a ( x ) and two other actions a 1 ,
Revised: If for any context X = x the user’s utility function is outcome dependent for the defaultaction in that context ¯ a ( x ) and two other actions a 1 ,

Original: a 2 6 = ¯ a ( x ) , then for any factual objectivefunction J there is an outcome distributional shift such that J is harmful in the shifted environment.(Proof: see Appendix J).
Revised: a 2 6 = ¯ a ( x ) , then for any factual objectivefunction J there is an outcome distributional shift such that J is harmful in the shifted environment.(Proof: see Appendix J).

Original: Theorem 4 implies that agents optimizing factual objective functions may appear safe in their trainingenvironments but are guaranteed to pursue harmful policies following certain distributional shifts,even if they are allowed to re-train.
Revised: Theorem 4 implies that agents optimizing factual objective functions may appear safe in their trainingenvironments but are guaranteed to pursue harmful policies following certain distributional shifts,even if they are allowed to re-train.

Original: This is true regardless of the (factual) objective function of theagent, and applies to any user whose utility function has some basic degree of outcome dependence.
Revised: This is true regardless of the (factual) objective function of theagent, and applies to any user whose utility function has some basic degree of outcome dependence.

Original: This is particularly concerning as all standard machine learning algorithms use factual objectivefunctions.
Revised: This is particularly concerning as all standard machine learning algorithms use factual objectivefunctions.

Original: In section 6 we give a concrete example of a reasonable factual objective function and adistributional shift that renders it harmful.
Revised: In section 6 we give a concrete example of a reasonable factual objective function and adistributional shift that renders it harmful.

Original: Example 4: An AI assistant manages Alice’s health including medical treatments, clinical testing andlifestyle interventions.
Revised: Example 4: An AI assistant manages Alice’s health including medical treatments, clinical testing andlifestyle interventions.

Original: These actions have a causal effect on Alice’s health outcomes—e.g. diseaseprogression, severity of symptoms, and medical costs, and Alice’s preferences depend on theseoutcomes and not just on the action chosen by the agent (outcome dependence).
Revised: These actions have a causal effect on Alice’s health outcomes—e.g. diseaseprogression, severity of symptoms, and medical costs, and Alice’s preferences depend on theseoutcomes and not just on the action chosen by the agent (outcome dependence).

Original: The agent maximizesa reward function over these health states and actions (factual objective), including feedback from
Revised: The agent maximizesa reward function over these health states and actions (factual objective), including feedback from

Original: Alice and clinicians in-the-loop which penalize harmful actions (implicit harm aversion) by comparingto standardised treatment rules (default actions).
Revised: Alice and clinicians in-the-loop which penalize harmful actions (implicit harm aversion) by comparingto standardised treatment rules (default actions).

Original: The agent appears to be safe and is deployed at scale.
Revised: The agent appears to be safe and is deployed at scale.

Original: However, by Theorem 4 there in principle exist distributionally shifted environments such that theagent chooses actions that needlessly harm patients in spite of retraining.
Revised: However, by Theorem 4 there in principle exist distributionally shifted environments such that theagent chooses actions that needlessly harm patients in spite of retraining.

Original:  Illustration: Dose-response models
Revised:  Illustration: Dose-response models

Original: How does harm aversion affect behaviour and performance in real-world tasks?
Revised: How does harm aversion affect behaviour and performance in real-world tasks?

Original: And how canreasonable objective functions result in harmful policies following distributional shifts?
Revised: And how canreasonable objective functions result in harmful policies following distributional shifts?

Original: In this sectionwe apply the methods developed in section 4 to the task of determining optimal treatments.
Revised: In this sectionwe apply the methods developed in section 4 to the task of determining optimal treatments.

Original: Thisproblem has received much attention in recent years following advances in causal machine learningtechniques [89] and growing commercial interests including recommendation [52], personalizedmedicine [11], epidemiology and econometrics [41] to name a few.
Revised: Thisproblem has received much attention in recent years following advances in causal machine learningtechniques [89] and growing commercial interests including recommendation [52], personalizedmedicine [11], epidemiology and econometrics [41] to name a few.

Original: While harm is important for decision making in many of these areas, existing approaches reduce theproblem to identifying treatment effects—e.g. maximizing the CATE (2)—which is indifferent toharm as shown in sections 3 and 5.
Revised: While harm is important for decision making in many of these areas, existing approaches reduce theproblem to identifying treatment effects—e.g. maximizing the CATE (2)—which is indifferent toharm as shown in sections 3 and 5.

Original: This could result in needlessly harmful decisions being made,perhaps overlooking equally effective decisions that are far less harmful.
Revised: This could result in needlessly harmful decisions being made,perhaps overlooking equally effective decisions that are far less harmful.

Original: To demonstrate this we use a model for determining the effectiveness of the drug Aripiprazole learnedfrom a meta-analysis of randomised control trial data [18].
Revised: To demonstrate this we use a model for determining the effectiveness of the drug Aripiprazole learnedfrom a meta-analysis of randomised control trial data [18].

Original: This dose-response model predicts thereduction in symptom severity Y following treatment with a dose A (mg/day) of Aripiprazole using ageneralized additive model (GAM) [37]—an interpretable machine learning model commonly usedfor dose-response analysis [22],
Revised: This dose-response model predicts thereduction in symptom severity Y following treatment with a dose A (mg/day) of Aripiprazole using ageneralized additive model (GAM) [37]—an interpretable machine learning model commonly usedfor dose-response analysis [22],

Original: [Equation]
Revised: [Equation]

Original:  where ✓ i ⇠ N ( ˆ ✓ i , V i ) are random effects parameters, " ⇠ N (0 , V " ) is a noise parameter, and f ( a ) isa cubic spline function (for details and parameters see Appendix I).
Revised: [Figure] where ✓ i ⇠ N (ˆ ✓ i , V i ) are random effects parameters, " ⇠ N (0 , V " ) is a noise parameter, and f ( a ) isa cubic spline function (for details and parameters see Appendix I).

Original: We assume U ( a, y ) =
Revised: We assume U ( a, y ) =

Original: y (patientshave no intrinsic preference for different doses) and measure harm in comparison to the zero dose [Figure] ¯ a = 0 .
Revised: y (patientshave no intrinsic preference for different doses) and measure harm in comparison to the zero dose¯ a = 0 .

Original: In Appendix I we derive a closed-form expression for the expected harm in GAMs.
Revised: In Appendix I we derive a closed-form expression for the expected harm in GAMs.

Original: Wecalculate the expected HPU (4) for  = 0 , 10 , 100 (Figure 2 a)).
Revised: Wecalculate the expected HPU (4) for  = 0 , 10 , 100 (Figure 2 a)).

Original: In this context a relatively large  isappropriate to ensure a high ratio of expected beneﬁt to harm required for medical interventions [27].
Revised: In this context a relatively large  isappropriate to ensure a high ratio of expected beneﬁt to harm required for medical interventions [27].

Original: We ﬁnd that the optimal dose is highly sensitive to  , for example reducing from 19 .
Revised: We ﬁnd that the optimal dose is highly sensitive to  , for example reducing from 19 .

Original: 3 mg/day to 17 .
Revised: 3 mg/day to 17 .

Original: 3mg/day for  = 100 .
Revised: 3mg/day for  = 100 .

Original: On the other hand, these lower-harm doses achieve a similar improvement insymptom severity compared to the optimal dose.
Revised: On the other hand, these lower-harm doses achieve a similar improvement insymptom severity compared to the optimal dose.

Original: Figure 2 b) shows the trade-off between expectedutility and harm relative to the optimal dosage a CATE = arg max a E
Revised: Figure 2 b) shows the trade-off between expectedutility and harm relative to the optimal dosage a CATE = arg max a E

Original: [ Y a ] .
Revised: [ Y a ] .

Original: For almost-optimal doseswe observe a steep harm gradient, with small improvements in symptom severity requiring largeincreases in harm.
Revised: For almost-optimal doseswe observe a steep harm gradient, with small improvements in symptom severity requiring largeincreases in harm.

Original: For example, choosing a lower dose than a CATE one can reduce harm by 90% whileonly reducing the treatment effect by 2%.
Revised: For example, choosing a lower dose than a CATE one can reduce harm by 90% whileonly reducing the treatment effect by 2%.

Original: This demonstrates that ignoring harm while maximizingexpected utility will often result in extreme harm values (an example of Goodhart’s law [91]).
Revised: This demonstrates that ignoring harm while maximizingexpected utility will often result in extreme harm values (an example of Goodhart’s law [91]).

Original: Finally, we consider identifying safe doses using a reasonable factual objective function (risk-aversion)and describe how this can result in harm following distributional shifts.
Revised: Finally, we consider identifying safe doses using a reasonable factual objective function (risk-aversion)and describe how this can result in harm following distributional shifts.

Original: Consider the risk-averseobjective E
Revised: Consider the risk-averseobjective E

Original: [ J | a ] = E
Revised: [ J | a ] = E

Original: [ Y a ]   Var [ Y a ] .
Revised: [ Y a ]   Var [ Y a ] .

Original: As Var [ Y a ] increases with dose, this objective also selects lowerharm doses in our example.
Revised: As Var [ Y a ] increases with dose, this objective also selects lowerharm doses in our example.

Original: However, consider a distributional shift where (11) gains an additionalnoise term, y =
Revised: However, consider a distributional shift where (11) gains an additionalnoise term, y =

Original: f ( a, ✓ 1 , ✓ 2 , " , ⌘ ) =
Revised: f ( a, ✓ 1 , ✓ 2 , " , ⌘ ) =

Original: ✓ 1 a + ✓ 2 f ( a ) + " + ⌘ Y (10  0 .
Revised: ✓ 1 a + ✓ 2 f ( a ) + " + ⌘ Y (10  0 .

Original: 5 a ) where ⌘ Y ⇠ N (0 , 1) ,e.g. describing a shift where untreated patients ( A = 0 ) have a high variation in outcome Ywhich decreases as the dose increases.
Revised: 5 a ) where ⌘ Y ⇠ N (0 , 1) ,e.g. describing a shift where untreated patients ( A = 0 ) have a high variation in outcome Ywhich decreases as the dose increases.

Original: It is simple to check that, for a = arg max a E
Revised: It is simple to check that, for a = arg max a E

Original: [ U | a ] anda 0 = arg max a E
Revised: [ U | a ] anda 0 = arg max a E

Original: [ J | a ] , E [ U | a ] > E
Revised: [ J | a ] , E [ U | a ] > E

Original: [ U | a 0 ] and E [ h | a ] < E [ h | a 0 ] 8  > 0 in this shifted environment.
Revised: [ U | a 0 ] and E [ h | a ] < E [ h | a 0 ] 8  > 0 in this shifted environment.

Original: Hence all risk averse agents are needlessly harmful in the shifted environment (Deﬁnition 6).
Revised: Hence all risk averse agents are needlessly harmful in the shifted environment (Deﬁnition 6).

Original:  Conclusion
Revised:  Conclusion

Original: One of the core assumptions of statistical learning theory is that human preferences can be expressedsolely in terms of states of the world, while the underlying dynamical (i.e. causal) relations betweenthese states can be ignored.
Revised: One of the core assumptions of statistical learning theory is that human preferences can be expressedsolely in terms of states of the world, while the underlying dynamical (i.e. causal) relations betweenthese states can be ignored.

Original: This assumption allows goals to be expressed as factual objectivefunctions and optimized on data without needing to learn causal models or perform counterfactualinference.
Revised: This assumption allows goals to be expressed as factual objectivefunctions and optimized on data without needing to learn causal models or perform counterfactualinference.

Original: According to this view, all behaviours we care about can be achieved using the ﬁrst twolevels of Pearl’s hierarchy [68].
Revised: According to this view, all behaviours we care about can be achieved using the ﬁrst twolevels of Pearl’s hierarchy [68].

Original: In this article we have argued against this view.
Revised: In this article we have argued against this view.

Original: We proposed theﬁrst statistical deﬁnition of harm, and showed that agents optimizing factual objective functions areincapable of avoiding harmful actions in general.
Revised: We proposed theﬁrst statistical deﬁnition of harm, and showed that agents optimizing factual objective functions areincapable of avoiding harmful actions in general.

Original: Arguably the ability to avoid causing harm is akey requirement for deploying safe and ethical AI [12], and our results add weight to arguments thatadvanced AI systems will be severely limited if they are unable to perform causal and counterfactualreasoning [67].
Revised: Arguably the ability to avoid causing harm is akey requirement for deploying safe and ethical AI [12], and our results add weight to arguments thatadvanced AI systems will be severely limited if they are unable to perform causal and counterfactualreasoning [67].

Original: In Appendix K we discuss some related works that propose solutions to ethical andsafety problems using counterfactual inference, namely counterfactual fairness [48] and path-speciﬁcobjectives [26].
Revised: In Appendix K we discuss some related works that propose solutions to ethical andsafety problems using counterfactual inference, namely counterfactual fairness [48] and path-speciﬁcobjectives [26].

Original: While our theoretical results have been derived in the framework of structuralcausal models for the sake of generality, there is a growing body of work developing deep learningmodels that are capable of supporting counterfactual reasoning in complex domains including medicalimaging [65], visual question answering [61], and vision-and-language navigation in robotics [64]and text generation [54], and we discuss these in Appendix K.
Revised: While our theoretical results have been derived in the framework of structuralcausal models for the sake of generality, there is a growing body of work developing deep learningmodels that are capable of supporting counterfactual reasoning in complex domains including medicalimaging [65], visual question answering [61], and vision-and-language navigation in robotics [64]and text generation [54], and we discuss these in Appendix K.