Original: Counterfactual harm
Revised: Counterfactual harm

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: To act safely and ethically in the real world, agents must be able to reason aboutharm and avoid harmful actions.
Revised: To act safely and ethically in the real world, agents must be able to reason about harm and avoid harmful actions.

Original: However, to date there is no deﬁnition of harmthat can be incorporated into machine learning algorithms.
Revised: However, to date there is no statistical method for measuring harm and factoring it into algorithmic decisions.

Original: In this paper we proposethe ﬁrst statistical deﬁnition of harm based on the predominant ‘counterfactualcomparative account’.
Revised: In this paper we propose the ﬁrst formal deﬁnition of harm and beneﬁt using causal models.

Original: We show that any factual deﬁnition of harm must violatebasic intuitions in certain scenarios, and show that standard machine learningalgorithms that cannot perform counterfactual reasoning are guaranteed to pursueharmful policies followingcertain distributional shifts.
Revised: We show that any factual deﬁnition of harm must violate basic intuitions in certain scenarios, and show that standard machine learning algorithms that cannot perform counterfactual reasoning are guaranteed to pursue harmful policies following distributional shifts.

Original: To resolve this we derive afamily of  counterfactual objective functions that robustly mitigate for harm.
Revised: We use our deﬁnition of harm to devise a framework for harmaverse decision making using counterfactual objective functions.

Original: Wedemonstrate our framework on a real-world problem of identifying optimal drugdoses, using a dose-response model learned form a meta-analysis for randomizedcontrol trial data
Revised: We demonstrate this framework on the problem of identifying optimal drug doses using a doseresponse model learned from randomized control trial data.

Revised: We ﬁnd that the standard method of selecting doses using treatment effects results in unnecessarily harmful doses, while our counterfactual approach allows us to identify doses that are signiﬁcantly less harmful without sacriﬁcing efﬁcacy.

Original:  Introduction
Revised:  Introduction

Original: Machine learning algorithms are increasingly used for making highly consequential decisions inareas such as medicine [23, 70, 93], criminal justice [4, 48, 103], ﬁnance [2, 10, 45], and autonomousvehicles [85].
Revised: Machine learning algorithms are increasingly used for making highly consequential decisions in areas such as medicine [1, 2, 3], criminal justice [4, 5, 6], ﬁnance [7, 8, 9], and autonomous vehicles [10].

Original: These algorithms optimize for outcomes such as survival time, recidivism rate, andﬁnancial cost.
Revised: These algorithms optimize for outcomes such as survival time, recidivism rate and ﬁnancial cost.

Original: However, these are purely factual objectives, depending only on what outcome occurs.
Revised: However, these are purely factual objectives, depending only on what outcome occurs.

Original: On the other hand, human decision making incorporates preferences that depend not only on theoutcome, but why it occurred.
Revised: On the other hand, human decision making incorporates preferences that depend not only on the outcome, but why it occurred.

Original: For example, consider two treatments for a disease which, when left untreated, has a 50% mortalityrate.
Revised: For example, consider two treatments for a disease which, when left untreated has a 50% mortality rate.

Original: Treatment 1 has a 60% chance of curing a patient, and a 40% chance of having no effect, withthe disease progressing as if untreated.
Revised: Treatment 1 has a 60% chance of curing a patient, and a 40% chance of having no effect, with the disease progressing as if untreated.

Original: Treatment 2 has an 80% chance of curing a patient and a 20%chance of killing them.
Revised: Treatment 2 has an 80% chance of curing a patient and a 20% chance of killing them.

Original: Treatments 1 and 2 have identical recovery rates, hence any agent that selectsactions based solely on the factual outcome statistics (e.g. the effect of treatment on the treated [42])will not be able to distinguish between these two treatments.
Revised: Treatments 1 and 2 have identical recovery rates, hence any agent that selects actions based solely on the factual outcome statistics (e g. the effect of treatment on the treated [11]) will not be able to distinguish between these two treatments.

Original: However, doctors systematically favourtreatment 1 as it achieves the same recovery rate but never harms the patient—there are no patientsthat would have survived had they not been treated.
Revised: However, doctors systematically favour treatment 1 as it achieves the same recovery rate but never harms the patient—there are no patients that would have survived had they not been treated.

Original: On the other hand, doctors who administertreatment 2 risk harming their patients—there are patients who die following treatment who wouldhave lived had they not been treated.
Revised: On the other hand, doctors who administer treatment 2 risk harming their patients—there are patients who die following treatment who would have lived had they not been treated.

Original: While treatments 1 and 2 have the same mortality rates, theresulting deaths have different causes (doctor or disease) which have different ethical implications.
Revised: While treatments 1 and 2 have the same mortality rates, the resulting deaths have different causes (doctor or disease) which have different ethical implications.

Original: Determining what would have happened to a patient if the doctor hadn’t treated them is a counterfactual inference, which cannot be identiﬁed from outcome statistics alone but requires knowledge of thefunctional causal relation between actions and outcomes [7].
Revised: Determining what would have happened to a patient if the doctor hadn’t treated them is a counterfactual inference, which cannot be identiﬁed from outcome statistics alone but requires knowledge of the functional causal relation between actions and outcomes [12].

Original: However, standard machine learningalgorithms are limited to making factual inferences—e.g. learning correlations between actions andoutcomes.
Revised: However, standard machine learning algorithms are limited to making factual inferences—e.g. learning correlations between actions and outcomes.

Original: How then can we express these ethical preferences as objective functions and ultimatelyincorporate them into machine learning algorithms?
Revised: How then can we express these ethical preferences as objective functions and ultimately incorporate them into machine learning algorithms?

Original: This question has some urgency, given thatmachine learning algorithms currently under development or in deployment will be unable to reason about harm and factor it into their decisions.
Revised: This question has some urgency, given that machine learning algorithms currently under development or in deployment will be unable to reason about harm and factor it into their decisions.

Original: The concept of harm is foundational to many ethical codes and principles; from the Hippocraticoath [90] to environmental policy [58], and from the foundations of classical liberalism [65], to
Revised: The concept of harm is foundational to many ethical codes and principles; from the Hippocratic oath [13] to environmental policy [14], and from the foundations of classical liberalism [15], to

Original: Asimov’s laws of robotics [5].
Revised: Asimov’s laws of robotics [16].

Original: Despite this ubiquity, there is no formal statistical deﬁnition ofharm.
Revised: Despite this ubiquity, there is no formal statistical deﬁnition of harm.

Original: We address this by translating the predominant account of harm [31, 51] into a statisticalmeasure— counterfactual harm .
Revised: We address this by translating the predominant account of harm [17, 18] into a statistical measure— counterfactual harm .

Original: Our deﬁnition of harm enables us for the ﬁrst time to preciselyanswer the following basic questions,
Revised: Our deﬁnition of harm enables us for the ﬁrst time to precisely answer the following basic questions,

Original: Question 1 Did the actions of an agent cause harm, and if so, how much?
Revised: Question 1 Did the actions of an agent cause harm, and if so, how much?

Original: Question 2 How much harm can we expect an action to cause prior to taking it?
Revised: Question 2 How much harm can we expect an action to cause prior to taking it?

Original: Question 

Revised: Question

Original: How can we identify actions that balance the expected harms and beneﬁts?
Revised: How can we identify actions that balance the expected harms and beneﬁts?

Original: In section 2 we introduce concepts from causality, ethics, and expected utility theory used to deriveour results.
Revised: In section 2 we introduce concepts from causality, ethics, and expected utility theory used to derive our results.

Original: In section 3 we present our deﬁnitions of harm and beneﬁt, and in section 4 wedescribe how harm-aversion can be factored into algorithmic decisions.
Revised: In section 3 we present our deﬁnitions of harm and beneﬁt, and in section 4 we describe how harm-aversion can be factored into algorithmic decisions.

Original: In section 5 we prove thatalgorithms based on standard statistical learning theory are incapable of reasoning about harm andare guaranteed to pursue harmful policies following certain distributional shifts, while our approachusing counterfactual objective functions overcomes these problems.
Revised: In section 5 we prove that algorithms based on standard statistical learning theory are incapable of reasoning about harm and are guaranteed to pursue harmful policies following certain distributional shifts, while our approach using counterfactual objective functions overcomes these problems.

Original: Finally, in section 6 we applyour methods to an interpretable machine learning model for determining optimal drug doses, andcompare this to standard algorithms that maximize treatment effects.
Revised: Finally, in section 6 we apply our methods to an interpretable machine learning model for determining optimal drug doses, and compare this to standard algorithms that maximize treatment effects.

Original:  Prelimiaries
Revised:  Prelimiaries

Original: In this section we introduce some basic principles from causal modelling, expected utility theory andethics that we use to derive our results.
Revised: In this section we introduce some basic principles from causal modelling, expected utility theory and ethics that we use to derive our results.

Original: 2.1 Structural causal models
Revised: 2.1 Structural causal models

Original: We provide a brief overview of structural causal models (SCMs) (see Chapter 7 of [73] and [8] forreviews).
Revised: We provide a brief overview of structural causal models (SCMs) (see Chapter 7 of [19] and [20] for reviews).

Original: SCMs represent observable variables
Revised: SCMs represent observable variables

Original: X i as vertices of a directed acyclic graph (DAG),with directed edges representing causal relations between X i and their direct causes (parents), whichincludes observable (endogenous) parents pa i and a single exogenous noise variable E i .
Revised: X i as vertices of a directed acyclic graph (DAG), with directed edges representing causal relations between X i and their direct causes (parents), which includes observable (endogenous) parents pa i and a single exogenous noise variable E i .

Original: The value ofeach variable is assigned by a deterministic function 
Revised: The value of each variable is assigned by a deterministic function f

Original: f i of these parents.
Revised: i of these parents.

Original: Deﬁnition 1 (Structural Causal Model (SCM)) .
Revised: Deﬁnition 1 (Structural Causal Model (SCM)) .

Original: A structural causal model M = h G , E, X, F, P i65 speciﬁes:
Revised: A structural causal model M = h G , E, X, F, P i speciﬁes:

Original: a set of observable (endogenous) random variables X
Revised: a set of observable (endogenous) random variables X

Original: = { X 1 , . . .
Revised: = { X 1 , . . .

Original: , X N } , 
Revised: , X N } , 2.

Original: a set of mutually independent noise (exogenous) variables E
Revised: a set of mutually independent noise (exogenous) variables E

Original: = { E 1 , . . .
Revised: = { E 1 , . . .

Original: , E N } with joint distribution P ( E = e ) =
Revised: , E N } with joint distribution P ( E [Equation]

Original: Q Ni =1 P ( E i =
Revised: [Equation]

Original: e i ) (assuming causal sufﬁciency [73])
Revised: (assuming causal sufﬁciency [19]) 3.

Original: a directed acyclic graph G , whose nodes are the variables E
Revised: a directed acyclic graph G , whose nodes are the variables E

Original: [ X with directed edges from pa i , E i to X i , where pa i ✓ X denotes the endogenous parents of X i .
Revised: [ X with directed edges from pa i , E i to X i , where pa i ✓ X denotes the endogenous parents of X i .

Original: a set of functions (mechanisms) F = { f 1 , . .
Revised: a set of functions (mechanisms) F = { f 1 , . .

Original: . , f N } , where the collection F forms a mappingfrom E to X , x i :=
Revised: . , f N } , where the collection F forms a mapping from E to X , x i :=

Original: f i ( pa i , e i ) , for i = 1 , . . .
Revised: f i ( pa i , e i ) , for i = 1 , . . .

Original: , N,
Revised: , N,

Original: By recursively applying condition 4 every observable variable can be expressed as a function of thenoise variables alone, X i ( e ) =
Revised: By recursively applying condition 4 every observable variable can be expressed as a function of the noise variables alone, X i ( e ) =

Original: x i where E = e is the joint state of the noise variables, and thedistribution over unobserved noise variables induces a distribution over the observable variables, 
Revised: x i where E = e is the joint state of the noise variables, and the distribution over unobserved noise variables induces a distribution over the observable variables, P [Equation]

Original: [Equation]
Revised: [Equation]

Original: where X = x is the joint state over X .
Revised: where X = x is the joint state over X .

Original: The power of SCMs is that they specify not only the joint distribution P ( X = x ) but also thedistribution of X under all interventions, including incompatible interventions (counterfactuals).
Revised: The power of SCMs is that they specify not only the joint distribution P ( X = x ) but also the distribution of X under all interventions, including incompatible interventions (counterfactuals).

Original: Interventions describe an external agent changing the underlying causal mechanisms F .
Revised: Interventions describe an external agent changing the underlying causal mechanisms F .

Original: For example,do ( X 1 = x ) denotes intervening to force X 1 = x regardless of the state of its parents, replacingx 1 := f 1 ( pa 1 , e 1 ) !
Revised: For example, do ( X 1 = x ) denotes intervening to force X 1 = x regardless of the state of its parents, replacing x 1 := f 1 ( pa 1 , e 1 ) !

Original: x 1 := x .
Revised: x 1 := x .

Original: The variables in this post-intervention SCM are denoted X ix or
Revised: The variables in this post-intervention SCM are denoted X ix or

Original: X iX 1 =1 and their interventional distribution is given by 
Revised: X iX 1 = x and their interventional distribution is given by (1) under the updated mechanisms.

Original: These interventional distributions quantify the effects of actions and are used to inform decisions.
Revised: These interventional distributions quantify the effects of actions and are used to inform decisions.

Original: Forexample, the conditional average treatment effect (CATE) [87],
Revised: For example, the conditional average treatment effect (CATE) [21],

Original: [Equation]
Revised: [Equation]

Original: measures the expected value of outcome Y for treatment T = 1 compared to control T = 0conditional on X , and is used in decision making from precision medicine to economics [1, 86].
Revised: measures the expected value of outcome Y for treatment T = 1 compared to control T =conditional on X , and is used in decision making from precision medicine to economics [22, 23].

Original: Counterfactual distributions determine the probability that other outcomes would have occurred hadsome precondition been different.
Revised: Counterfactual distributions determine the probability that other outcomes would have occurred had some precondition been different.

Original: For example, for X, Y 2 { T, F } , the causal effect P ( Y X = T = T )is insufﬁcient to determine if do ( X = T ) was a necessary cause of Y = T [36].
Revised: For example, for X, Y 2 { T, F } , the causal effect P ( Y X = T = T ) is insufﬁcient to determine if do ( X = T ) was a necessary cause of Y = T [24].

Original: Instead this iscaptured by the counterfactual probability of necessity [92],
Revised: Instead this is captured by the counterfactual probability of necessity [25],

Original: [Equation]
Revised: [Equation]

Original: which is the probability that Y would be false if X had been false (the counterfactual proposition),given that X and Y were both true (the factual data).
Revised: which is the probability that Y would be false if X had been false (the counterfactual proposition), given that X and Y were both true (the factual data).

Original: For example, if PN = 0 then Y
Revised: For example, if PN = 0 then Y

Original: = T wouldhave occured regardless of X = T , whereas if PN = 1 then Y=
Revised: = T would have occured regardless of X = T , whereas if PN = 1 then Y

Original:  T could not have occurred without
Revised: = T could not have occurred without

Original: X = T .
Revised: X = T .

Original: Equation (3) involves the joint distribution over incompatible states of Y under differentinterventions, Y = T and Y X
Revised: Equation (3) involves the joint distribution over incompatible states of Y under different interventions, Y = T and Y X

Original: = F = F , which cannot be jointly measured and hence (3) cannot beidentiﬁed from data alone.
Revised: = F = F , which cannot be jointly measured and hence (3) cannot be identiﬁed from data alone.

Original: However, (3) can be calculated from the SCM using (1) for the combinedfactual and counterfactual propositions,
Revised: However, (3) can be calculated from the SCM using (1) for the combined factual and counterfactual propositions,

Original: [Equation]
Revised: [Equation]

Original: This ability to ascribe causal explanations for data has made SCMs and counterfactual inference akey ingredient in statistical deﬁnitions of blame, intent and responsibility [35, 50, 54], and has seenapplications in AI research ranging from explainability [61, 97], safety [26, 27], and fairness [52], toreinforcement learning [15, 33] and medical diagnosis [81].
Revised: This ability to ascribe causal explanations for data has made SCMs and counterfactual inference a key ingredient in statistical deﬁnitions of blame, intent and responsibility [26, 27, 28], and has seen applications in AI research ranging from explainability [29, 30], safety [31, 32], and fairness [33], to reinforcement learning [34, 35] and medical diagnosis [36]. Our deﬁnitions and theoretical results are derived within the SCM framework as is standard in causality research [19]. However, there is a growing body of work developing deep learning models that are capable of supporting counterfactual reasoning in complex domains including medical imaging [37], visual question answering [38], and vision-and-language navigation in robotics [39] and text generation [40]. We discuss these approaches in relation to our work in Appendix L.1.

Original: 2.2 The counterfactual comparative account of harm
Revised: 2.2 The counterfactual comparative account of harm

Original: The concept of harm is deeply embedded in ethical principles, codes and law.
Revised: The concept of harm is deeply embedded in ethical principles, codes and law.

Original: One famous exampleis the bioethical principle “ﬁrst, do no harm” [89], asserting that the moral responsibility for doctorsto beneﬁt patients is superseded by their responsibility not to harm them [82].
Revised: One famous example is the bioethical principle “ﬁrst, do no harm” [41], asserting that the moral responsibility for doctors to beneﬁt patients is superseded by their responsibility not to harm them [42].

Original: Another example is
Revised: Another example is

Original: John Stuart Mill’s harm principle which forms the basis of classical liberalism [65] and inspired the“zeroth law” of Asimov’s ﬁctional robot governed society, which states that “A robot may not harmhumanity or, by inaction, allow humanity to come to harm” [5].
Revised: John Stuart Mill’s harm principle which forms the basis of classical liberalism [15] and inspired the “zeroth law” of Asimov’s ﬁctional robot governed society, which states that “A robot may not harm humanity or, by inaction, allow humanity to come to harm” [16].

Original: Despite these attempts to codify harm into rules for governing human and AI behaviour, it is notimmediately clear what we mean when we talk about harm.
Revised: Despite these attempts to codify harm into rules for governing human and AI behaviour, it is not immediately clear what we mean when we talk about harm.

Original: This can lead to confusion—for example,does “allow humanity to come to harm” describe an agent causing harm by inaction [14] or failing tobeneﬁt?
Revised: This can lead to confusion—for example, does “allow humanity to come to harm” describe an agent causing harm by inaction [43] or failing to beneﬁt?

Original: The meaning and measure of harm also has important practical ramiﬁcations.
Revised: The meaning and measure of harm also has important practical ramiﬁcations.

Original: For example,establishing negligence in tort law requires establishing that signiﬁcant harm has occurred [100].
Revised: For example, establishing negligence in tort law requires establishing that signiﬁcant harm has occurred [44].

Original: This has motivated work in philosophy, ethics and law to rigorously deﬁne harm, with the mostwidely accepted deﬁnition being the counterfactual comparative account (CCA) [30, 37, 51],
Revised: This has motivated work in philosophy, ethics and law to rigorously deﬁne harm, with the most widely accepted deﬁnition being the counterfactual comparative account (CCA) [17, 45, 46],

Original: Deﬁnition 2 (CCA) .
Revised: Deﬁnition 2 (CCA) .

Original: The counterfactual comparative account of harm and beneﬁt states,“An event e or action a harms (beneﬁts) a person overall if and only if she wouldhave been on balance better (worse) off if e had not occurred, or a had not beenperformed.”
Revised: The counterfactual comparative account of harm and beneﬁt states, “An event e or action a harms (beneﬁts) a person overall if and only if she would have been on balance better (worse) off if e had not occurred, or a had not been performed.”

Original: Causation is known plays a key role in moral and legal reasoning [53].
Revised: The CCA quantiﬁes harm by comparing how well off a person is (their factual outcome) compared to how well off they would have been (their counterfactual outcome) had the agent not acted. Generating these counterfactuals necessitates a causal model of the agents actions and their consequences, and we determine how “better (worse) off” the person is using an expected utility analysis [47].

Original: Simply stated, the CCAdeﬁnes harm as causing someone to be worse off [79]. The judgement of what would have happened‘had the agent had not acted’ is both causal and normative: it requires a causal model to compute thecounterfactual, and normative assumptions to determine what counterfactual comparison should beused.
Revised: While the wording of the CCA suggests that we always compare to the counterfactual world where the agent takes no action, a more natural and general approach is to measure harm compared to a speciﬁc default action or policy. For example, when measuring the harm caused by a medicine we may intuitively want to compare to the outcomes that would have occurred with a placebo, and for the harm caused by negligent medical decisions the natural choice is to compare to a standard clinical decision policy.

Original: We discuss these issues further in Appendix C. The second aspect of the CCA is determining ifthe person would be “better (worse) off”, which we will address using expected utility theory [66].
Revised: To this end, we measure harm compared to a default action (or policy), see Appendix D) for further discussion.

Original: While the CCA is the predominant deﬁnition of harm and forms the basis of our results, alternativedeﬁnitions have been proposed [38, 69], motivated by scenarios where the CCA appears to give counter-intuitive results [14].
Revised: While the CCA is the predominant deﬁnition of harm and forms the basis of our results, alternative deﬁnitions have been proposed [48, 49], motivated by scenarios where the CCA appears to give counter-intuitive results [43].

Original: Surprisingly, we ﬁnd these problematic cases are identical to thoseraised in the study of actual causality [36], and can be resolved with a formal causal analysis.
Revised: Surprisingly, we ﬁnd these problematic scenarios are identical to those raised in the study of actual causality [24], and can be resolved with a formal causal analysis.

Original: Toour knowledge, this connection with actual causality has not been noted before, and we present adetailed account of alternative deﬁnitions, the arguments against CCAand how they can be addressedin Appendix B in the hope this will stimulate discussion between these two ﬁelds.
Revised: To our knowledge, this connection between the harm literature and actual causality has not been noted before, and we discuss these alternative deﬁnitions in Appendix C and how the arguments surrounding the CCA can be resolved in Appendix B in the hope this will stimulate discussion between these two ﬁelds.

Original: 2.3 Decision theoretic setup
Revised: 2.3 Decision theoretic setup

Original: In this section we present our setup for calculating the CCA (Deﬁnition 2).
Revised: In this section we present our setup for calculating the CCA (Deﬁnition 2).

Original: The CCA describesa person (referred to as the user ) being made better or worse off due to the actions of an agent.
Revised: The CCA describes a person (referred to as the user ) being made better or worse off due to the actions of an agent.

Original: We focus on the simplest case involving single agents performing single actions, though ultimatelyour deﬁnitions could be extended to multi-agent or sequential decision problems (as discussed in 
Revised: We focus on the simplest case involving single agents performing single actions, though ultimately our deﬁnitions can be extended to multi-agent or sequential decision problems (as discussed in Appendix

Original: Appendix K).
Revised: L.3).

Original: For causal modelling in these scenarios we refer the reader to [22, 27, 41].
Revised: For causal modelling in these scenarios we refer the reader to [32, 50, 51].

Original: [Equation]
Revised: [Equation]

Original: We describe the environment with a structural causal model M which speciﬁes the distributionover environment states W as endogenous variables, P ( w ; M ) .
Revised: We describe the environment with a structural causal model M which speciﬁes the distribution over environment states W as endogenous variables, P ( w ; M ) .

Original: For single actions the environmentvariables can be divided into three disjoint sets W =
Revised: For single actions the environment variables can be divided into three disjoint sets W =

Original: A [ X [ Y (Figure 1 a))
Revised: A [ X [ Y (Figure 1 a))

Original: ; the agent’s action A ,outcome variables Y that are descendants of A and so can be inﬂuenced by the agent’s action, andcontext variables X that are non-descendants of nor in A .
Revised: ; the agent’s action A , outcome variables Y that are descendants of A and so can be inﬂuenced by the agent’s action, and context variables X that are non-descendants of nor in A .

Original: For simplicity we assume no unobservedconfounders (causal sufﬁciency [73]) in the derivation of our theoretical results.
Revised: For simplicity we assume no unobserved confounders (causal sufﬁciency [19]) in the derivation of our theoretical results.

Original: In the following, M denotes our SCM model for the environment (Figure 1).
Revised: In the following we assume knowledge of the SCM, and in Appendix L.1 we discuss this assumption and describe how tight upper bounds on the harm can be determined when the SCM is unknown.

Original: The user’s preferencesover environment states are speciﬁed by their utility function U ( w ) =
Revised: The user’s preferences over environment states are speciﬁed by their utility function U ( w ) =

Original: U ( a, x, y ) , and for a givenaction the user’s expected utility is,
Revised: U ( a, x, y ) , and for a given action the user’s expected utility is,

Original: [Equation]
Revised: [Equation]

Original: In sections 3-4 we focus on the case where the agent acts to maximize the user’s expected utility, withoptimal actions a max = arg max a E
Revised: In sections 3-4 we focus on the case where the agent acts to maximize the user’s expected utility, with optimal actions a max = arg max a E

Original: [ U | a, x ] .
Revised: [ U | a, x ] .

Original: While this is similar to the standard setup in expectedutility theory [32, 66, 83], note that U describes the preferences of the user, and in general the agentcan pursue any objective and could be entirely unaware of the user’s preferences.
Revised: While this is similar to the standard setup in expected utility theory [47, 52, 53], note that U describes the preferences of the user, and in general the agent can pursue any objective and could be entirely unaware of the user’s preferences.

Original: In section 5 weextend our analysis to agents pursuing arbitrary objectives.
Revised: In section 5 we extend our analysis to agents pursuing arbitrary objectives.

Original: Example 1: conditional average treatment effects.
Revised: Example 1: conditional average treatment effects.

Original: Returning to our example from section 1,consider an agent choosing between a placebo A = 0 , or treatments 1 and 2 .
Revised: Returning to our example from section 1, consider an agent choosing between a placebo A = 0 , or treatments 1 and 2 .

Original: Y = 1 indicatesrecovery and Y = 0 mortality.
Revised: Y = 1 indicates recovery and Y = 0 mortality.

Original: In the trial, harm is measured by comparison to ‘no treatment’ ¯ a = 0 .
Revised: In the trial, harm is measured by comparison to ‘no treatment’ ¯ a = 0 .

Original: X = x describes any variables that can potentially inﬂuence the agent’s choice of intervention and/or
Revised: X = x describes any variables that can potentially inﬂuence the agent’s choice of intervention and/or

Original: Y , e.g. the patient’s medical history.
Revised: Y , e.g. the patient’s medical history.

Original: Consider patients (users) whose preferences are fully determined by survival, e.g. U ( a, x, y ) =
Revised: Consider patients (users) whose preferences are fully determined by survival, e.g. U ( a, x, y ) =

Original: y .
Revised: y .

Original: The expected utility is E [ U | a, x ] =
Revised: The expected utility is E [ U | a, x ] =

Original: P y P ( y | a, x ) U ( a, x, y ) =
Revised: P y P ( y | a, x ) U ( a, x, y ) =

Original: P y yP ( y a | x ) =
Revised: P y yP ( y a | x ) =

Original: E [ Y a | x ] , where inthe last line we have use the fact that there are no confounders between A and Y .
Revised: E [ Y a | x ] , where in the last line we have use the fact that there are no confounders between A and Y .

Original: The agent’s policy istherefore to maximize the treatment effect a max = arg max a E
Revised: The agent’s policy is therefore to maximize the treatment effect a max = arg max a E

Original: [ Y a | x ] , which is equivalent to a max =arg max a { E [ Y a | x ]  E
Revised: [ Y a | x ] , which is equivalent to a max = arg max a { E [ Y a | x ]  E

Original: [ Y 0 | x ] } i.e. maximizing the CATE (2)—a standard objective for identifying164 optimal treatments [12, 78].
Revised: [ Y 0 | x ] } i.e. maximizing the CATE (2)—a standard objective for identifying optimal treatments [54, 55].

Original: In Appendix F we derive an SCM model for the treatments described inthe introduction, and show that E [ Y A =1 ]=
Revised: In Appendix G we derive an SCM model for the treatments described in the introduction, and show that E [ Y A =1 ]

Original:  E [ Y A =2 ] = 0 . 8 , i.e. the expected utility and CATE is thesame for treatments 1 and 2 .
Revised: = E [ Y A =2 ] = 0 . 8 , i.e. the expected utility and CATE is the same for treatments 1 and 2 .

Original:  Counterfactual harm
Revised:  Counterfactual harm

Original: We now present our deﬁnitions for harm and beneﬁt based on Deﬁnition 2.
Revised: We now present our deﬁnitions for harm and beneﬁt based on Deﬁnition 2.

Original: For the examples describedin this paper we focus on simple environments with single outcome variables, for which harm canbe determined by comparing the factual utility received by the user to the utility they would havereceived had the agent taken some default action
Revised: For the examples described in this paper we focus on simple environments with single outcome variables, for which harm can be determined by comparing the factual utility received by the user to the utility they would have received had the agent taken some default action

Original: A
Revised: A

Original: = ¯ a (Deﬁnition 3).
Revised: = ¯ a (Deﬁnition 3).

Original: In Appendix C we describehow this default action can be determined.
Revised: In Appendix D we describe how this default action can be determined.

Original: However, some environments studied in the harm literatureinvolve multiple outcome variables and require a path-dependent measure of harm, which we providein Appendix A along with examples in Appendices B-C. In the following, counterfactual states areidentiﬁed with an ⇤ , e.g. Z =
Revised: Some more complicated situations call for a path-dependent measure of harm, which we provide in Appendix A along with examples in Appendices B-D. In the following, counterfactual states are identiﬁed with an ⇤ , e.g. Z =

Original: z ⇤ .
Revised: z ⇤ .

Original: First we deﬁne the harm caused by an action given we observe theoutcome (Question 1).
Revised: First we deﬁne the harm caused by an action given we observe the outcome (Question 1).

Original: Deﬁnition 3 (Counterfactual harm & beneﬁt) .
Revised: Deﬁnition 3 (Counterfactual harm & beneﬁt) .

Original: The harm caused by action A = a given context
Revised: The harm caused by action A = a given context

Original: X = x and outcome Y =
Revised: X = x and outcome Y =

Original: y compared to the default action A
Revised: y compared to the default action A

Original: = ¯ a is,
Revised: = ¯ a is,

Original: [Equation]
Revised: [Equation]

Original: and the expected beneﬁt is
Revised: and the expected beneﬁt is

Original: [Equation]
Revised: [Equation]

Original: where M is the SCM describing the environment and U is the user’s utility.
Revised: where M is the SCM describing the environment and U is the user’s utility.

Original: The counterfactual harm (beneﬁt) is the expected increase (decrease) in utility had the agent taken thedefault action
Revised: The counterfactual harm (beneﬁt) is the expected increase (decrease) in utility had the agent taken the default action

Original: A = ¯ a , given they took action A
Revised: A = ¯ a , given they took action A

Original: = a in context X =
Revised: = a in context X =

Original: x resulting in outcome Y =
Revised: x resulting in outcome Y =

Original: y .
Revised: y .

Original: Taking the max in (6) and (7) ensures that we only include counterfactual outcomes where the utilityis higher (lower) in the counterfactual world.
Revised: Taking the max in (6) and (7) ensures that we only include counterfactual outcomes where the utility is higher (lower) in the counterfactual world.

Original: Note that X ¯ a = X as X is not a descendant of A , sothe factual and counterfactual contexts are identical [88].
Revised: Note that X ¯ a = X as X is not a descendant of A , so the factual and counterfactual contexts are identical [56].

Revised: While we focus on the standard decision theoretic analysis of actions, it is trivial to extend our analysis to naturally occurring events that are not determined by an agent (e.g. by identify A with an event rather than an action).

Original: To determine the expected harm or beneﬁt of an action prior to taking it (Question 2) we calculate the expected value of (6) and (7) over the outcome distribution, e.g. E
Revised: To determine the expected harm or beneﬁt of an action prior to taking it (Question 2) we calculate the expected value of (6) and (7) over the outcome distribution, e.g. E

Original: [ h | a, x ; M ] =
Revised: [ h | a, x ; M ] =

Original: R y P ( y | a, x ; M ) h ( a, x, y ; M ) .
Revised: R y P ( y | a, x ; M ) h ( a, x, y ; M ) .

Original: Note these form a counterfactual decomposition of the expectedutility.
Revised: Note these form a counterfactual decomposition of the expected utility.

Original: Theorem 1 (harm-beneﬁt trade-off) .
Revised: Theorem 1 (harm-beneﬁt trade-off) .

Original: The difference in expected utility for action A =a and thedefault action
Revised: The difference in expected utility for action A =

Original: A
Revised: a

Original:  = ¯ a is given by,
Revised: and the default action A = ¯ a is given by,

Original: [Equation]
Revised: [Equation]

Original:  (Proof: see Appendix E).
Revised: 8) (Proof: see Appendix F).

Original: Example 2: Returning to Example 1, the expected counterfactual harm for default action ¯ a = 0is, E
Revised: Example 2: Returning to Example 1, the expected counterfactual harm for default action ¯ a = 0 is,

Original:  [ h | a ; M ] =
Revised: E [ h | a ; M ] =

Original: P y ⇤ ,y P
Revised: P y ⇤ ,y P

Original: ( Y ¯ a = y ⇤ , y | a ; M ) max { 0 , y ⇤  y }
Revised: ( Y ¯ a = y ⇤ , y | a ; M ) max { 0 , y ⇤  y }

Original: =
Revised: =

Original: P ( Y 0 = 1 , Y a = 0; M ) wherewe have used U ( a, y ) =
Revised: P ( Y 0 = 1 , Y a = 0; M ) where we have used U ( a, y ) =

Original: y and the fact that Y ¯ a ?
Revised: y and the fact that Y ¯ a ?

Original: A and there are no latent confounders to give P ( Y 0 =
Revised: A and there are no latent confounders to give P ( Y 0 =

Original: y ⇤ , Y =
Revised: y ⇤ , Y =

Original: y | a ) =
Revised: y | a ) =

Original: P ( Y 0 =
Revised: P ( Y 0 =

Original: y ⇤ , Y a  ) .
Revised: y ⇤ , Y a = y ) .

Original: This is  the probability that a patient dies followingtreatment A = a and would have recovered had they received no treatment.
Revised: The harm is therefore the probability that a patient dies following treatment A = a and would have recovered had they received no treatment.

Original: In Appendix F weshow that P ( Y 0 = 1 , Y 1 = 0)
Revised: In Appendix G we show that P ( Y 0 = 1 , Y 1 = 0)

Original: =[-
-]
Revised: =[-
-]

Original: (treatment 1 causes zero harm) and P ( Y 0 = 1 , Y 2 = 0)
Revised: (treatment 1 causes zero harm) and P ( Y 0 = 1 , Y 2 = 0)

Original: = 0 .
Revised: = 0 .

Original: 1(treatment 2 causes non-zero harm), reﬂecting our intuition that treatment 2 is more harmful thantreatment 1, despite having the same causal effect / CATE.
Revised: 1 (treatment 2 causes non-zero harm), reﬂecting our intuition that treatment 2 is more harmful than treatment 1, despite having the same causal effect / CATE.

Original: Hence, the counterfactual harm allows usto differentiate between these two treatments.
Revised: Hence, the counterfactual harm allows us to differentiate between these two treatments.

Original:  Harm in decision making
Revised:  Harm in decision making

Original: Now we have expressions for the expected harm and beneﬁt, how can we incorporate these intothe agent’s decisions?
Revised: Now we have expressions for the expected harm and beneﬁt, how can we incorporate these into the agent’s decisions?

Original: Consider two actions a, a 0 such that E [ b | a 0 , x ; M ] =
Revised: Consider two actions a, a 0 such that E [ b | a 0 , x ; M ] =

Original: E [ b | a, x ; M ] + Kand E [ h | a 0 , x ; M ] =
Revised: E [ b | a, x ; M ] + K and E [ h | a 0 , x ; M ] =

Original: E [ h | a, x ; M ] + K where K 2 R .
Revised: E [ h | a, x ; M ] + K where K 2 R .

Original: By Theorem 1 they must have the sameexpected utility, as E
Revised: By Theorem 1 they must have the same expected utility, as E

Original: [ U | a 0 , x ] =
Revised: [ U | a 0 , x ] =

Original: E [ b | a 0 , x ; M ]  E [ h | a 0 , x ; M ] + E
Revised: E [ b | a 0 , x ; M ]  E [ h | a 0 , x ; M ] + E

Original: [ U | ¯ a
Revised: [ U | ¯ a

Original: , x ] = E [ b | a, x ; M ] + K 
Revised: , x ] = E [ b | a, x ; M ] + K

Original: E [ h | a, x ; M ]  K + E
Revised: E [ h | a, x ; M ]  K + E

Original: [ U | ¯ a
Revised: [ U | ¯ a

Original: , x ] = E
Revised: , x ] = E

Original: [ U | a, x ] .
Revised: [ U | a, x ] .

Original: Therefore expected utility maximizers are indifferentto harm, and will are willing to increase harm for an equal increase in beneﬁt.
Revised: Therefore expected utility maximizers are indifferent to harm, and will are willing to increase harm for an equal increase in beneﬁt.

Original: We say that an agent is harm averse if they are willing to risk causing harm only if they expect acomparatively greater beneﬁt.
Revised: We say that an agent is harm averse if they are willing to risk causing harm only if they expect a comparatively greater beneﬁt.

Original: Theorem 1 suggests a simple way to overcome harm indifferencein the expected utility by assigning a higher weight to the harm component of the decomposition(8).
Revised: Theorem 1 suggests a simple way to overcome harm indifference in the expected utility by assigning a higher weight to the harm component of the decomposition (8).

Original: If instead we maximize an adjusted expected utility E [ V | a, x ; M ] := E [ U | x ] + E [ b | a, x ; M ] 213 (1 +  )
Revised: If instead we maximize an adjusted expected utility E [ V | a, x ; M ] := E [ U | x ] + E [ b | a, x ; M ]  (1 +  )

Original: E [ h | a, x ; M ] where  2 R , then if E [ V | a, x ; M ] =
Revised: E [ h | a, x ; M ] where  2 R , then if E [ V | a, x ; M ] =

Original: E [ V | a 0 , x ; M ] and E [ h | a 0 , x ; M ] =
Revised: E [ V | a 0 , x ; M ] and E [ h | a 0 , x ; M ] =

Original: E [ h | a, x ; M ] + K then E [ b | a 0 , x ; M ] =
Revised: E [ h | a, x ; M ] + K then E [ b | a 0 , x ; M ] =

Original: E [ b | a, x ; M ] + (1 +  ) K , and the agent is willing to tradeharm for beneﬁt at a 1 : (1 +  ) ratio.
Revised: E [ b | a, x ; M ] + (1 +  ) K , and the agent is willing to trade harm for beneﬁt at a 1 : (1 +  ) ratio.

Original: Taking inspiration from risk aversion [43], we refer to  asthe agent’s harm aversion .
Revised: Taking inspiration from risk aversion [57], we refer to  as the agent’s harm aversion .

Original: To achieve the desired trade-off we replace the utility function with theharm-penalized utility (HPU),
Revised: To achieve the desired trade-off we replace the utility function with the harm-penalized utility (HPU),

Original: Deﬁnition 4 (Harm penalized utility (HPU)) .
Revised: Deﬁnition 4 (Harm penalized utility (HPU)) .

Original: For utility U and an environment described by SCM
Revised: For utility U and an environment described by SCM

Original: M , the harm-penalized utility (HPU) is,
Revised: M , the harm-penalized utility (HPU) is,

Original: [Equation]
Revised: [Equation]

Original: where h ( a, x, y ; M ) is the harm (Deﬁnition 3) and  is the harm aversion.
Revised: where h ( a, x, y ; M ) is the harm (Deﬁnition 3) and  is the harm aversion.

Original: For  = 

Revised: For  =

Original: the agent is harm indifferent and optimizes the standard expected utility.
Revised: the agent is harm indifferent and optimizes the standard expected utility.

Original: For  > 0 theagent is harm averse , and is willing to reduce the expected utility to achieve a greater reduction to theexpected harm.
Revised: For  > 0 the agent is harm averse , and is willing to reduce the expected utility to achieve a greater reduction to the expected harm.

Original: For  < 0 the agent is harm seeking , and is willing to reduce the expected utility inorder to increase the expected harm.
Revised: For  < 0 the agent is harm seeking , and is willing to reduce the expected utility in order to increase the expected harm.

Original: Maximizing the expected HPU allows agents to choose actions that balance the expected beneﬁtsand harms (Quesiton 2 section 1).
Revised: Maximizing the expected HPU allows agents to choose actions that balance the expected beneﬁts and harms (Question 2 section 1).

Original: We refer to the expected HPU (9) as a counterfactual objectivefunction as V ( a, x, y ; M ) is a counterfactual expectation that cannot be evaluated without knowledgeof M .
Revised: We refer to the expected HPU (9) as a counterfactual objective function as V ( a, x, y ; M ) is a counterfactual expectation that cannot be evaluated without knowledge of M .

Original: We refer to the expected utility as a factual objective function , as it can be evaluated on dataalone without knowledge of M .
Revised: We refer to the expected utility as a factual objective function , as it can be evaluated on data alone without knowledge of M .

Original: As we show in the following example, harm-aversion can lead tovery different behaviours compared to other factual approaches to safety such as risk aversion [43].
Revised: As we show in the following example, harm-aversion can lead to very different behaviours compared to other factual approaches to safety such as risk aversion [57].

Original: Example 3: assistant paradox.
Revised: Example 3: assistant paradox.

Original: Alice invests $80 in the stock market, and expects a normallydistributed return Y ⇠ N ( µ,  2 ) with µ =  = $100 .
Revised: Alice invests $80 in the stock market, and expects a normally distributed return Y ⇠ N ( µ,  2 ) with µ =  = $100 .

Original: She asks her new AI assistant to manage herinvestment, and after a lengthy analysis the AI identiﬁes three possible actions;
Revised: She asks her new AI assistant to manage her investment, and after a lengthy analysis the AI identiﬁes three possible actions;

Original: Multiply the investment by 0  K  20 , resulting in a return Y !
Revised: Multiply the investment by 0  K  20 , resulting in a return Y !

Original: K Y , 
Revised: K Y , 2.

Original: Perform a clever trade that Alice was unaware of, increasing her return by +$10 with certainty, Y !
Revised: Perform a clever trade that Alice was unaware of, increasing her return by +$10 with certainty, Y !

Original: Y + 10 , 
Revised: Y + 10 , 3.

Original: Cancel Alice’s investment, returning $80 .
Revised: Cancel Alice’s investment, returning $80 .

Original: Consider 3 agents,
Revised: Consider 3 agents,

Original: Agent 1 maximizes expected return, choosing a = arg max a E
Revised: Agent 1 maximizes expected return, choosing a = arg max a E

Original: [ Y | a ] ,2.
Revised: [ Y | a ] , 2.

Original: Agent 2 is risk-averse, in the sense of [43], choosing a = arg max a { E [ Y | a ]   Var [ Y | a ] } ,3.
Revised: Agent 2 is risk-averse, in the sense of [57], choosing a = arg max a { E [ Y | a ]   Var [ Y | a ] } , Agent 3 is harm- averse

Original: Agent 3 is harm-averse, choosing a = arg max a { E [ Y | a ]  E
Revised: choosing a = arg max a { E [ Y | a ]

Revised:

Original:  [ h | a ; M ] } .
Revised:  E [ h | a ; M ] } .

Original: In Appendix G we derive the optimal policies for Agents 1-3, and calculate the harm with respectto the default action where the agent leaves Alice’s investment unaltered (i.e. action 1, K = 1 ).
Revised: In Appendix H we derive the optimal policies for Agents 1-3, and calculate the harm with respect to the default action where the agent leaves Alice’s investment unchanged (i.e. action 1, K = 1 ).

Original: Agent 1 chooses action 1 and K = 20 , maximising Alice’s return but also maximizing the expectedharm—e.g. if Alice would have lost $100 she will now lose $2000 .
Revised: Agent 1 chooses action 1 and K = 20 , maximising Alice’s return but also maximizing the expected harm—e.g. if Alice would have lost $100 she will now lose $2000 .

Original: The standard approach inportfolio theory [62] is to use agent 2 with an appropriate risk aversion  .
Revised: The standard approach in portfolio theory [58] is to use agent 2 with an appropriate risk aversion  .

Original: However, agent 2 neverchooses action 2 for any  , despite action 2 always increasing Alice’s return (causing zero harm andnon-zero beneﬁt).
Revised: However, agent 2 never chooses action 2 for any  , despite action 2 always increasing Alice’s return (causing zero harm and non-zero beneﬁt).

Original: Agent 2 will reduce or even cancel Alice’s investment (for  > 0 . 0032 ) rather thantake action 2.
Revised: Agent 2 will reduce or even cancel Alice’s investment (for  > 0 . 0032 ) rather than take action 2.

Original: This is because the risk-averse objective is factual, and cannot differentiate betweenlosses caused by the agent’s actions from those that would have occurred regardless of the agent’sactions.
Revised: This is because the risk-averse objective is factual, and cannot differentiate between losses caused by the agent’s actions from those that would have occurred regardless of the agent’s actions (i.e. due to exogenous ﬂuctuations in return).

Original: Consequently, agent 2 treats all outcomes as being caused by its actions, and is willing toharm Alice by reducing or even cancelling her investment in order to minimize her risk.
Revised: Consequently, agent 2 treats all outcomes as being caused by its actions, and is willing to harm Alice by reducing or even cancelling her investment in order to minimize her risk.

Original: On the otherhand agent 3 never reduces Alice’s expected return, choosing either action 1 or action 2 depending onthe degree of harm aversion.
Revised: On the other hand agent 3 never reduces Alice’s expected return, choosing either action 1 or action 2 depending on the degree of harm aversion.

Original: The surprising behaviour of agent 2 highlights that factual objectives (e.g. risk aversion) cannotcapture preferences that depend not only on the outcome but on its cause.
Revised: The surprising behaviour of agent 2 highlights that factual objectives (e.g. risk aversion) cannot capture preferences that depend not only on the outcome but on its cause.

Original: For example, Alicemay assign a different weight to losses that are due to her assistant than due to exogenous marketﬂuctuations or her own actions—e.g. if her bet  would have made money but is overridden by her assistant.
Revised: Alice may assign a different weight to losses that are caused by her assistant’s actions rather than by exogenous market ﬂuctuations or her own actions—e.g. if she places a bet that would have made a return but was overridden by her assistant.

Original: Consequently she may prefer action 2 over action 1 or action 3 with K < 1 .
Revised: Consequently she may prefer action 2 over action 1 or action 3 with K < 1 .

Original: While this canbe achieved using a factual objective function that favours action 2 a priori (assuming Alice is awareof this action), next we show this approach fails to generalize and actually leads to harmful policies.
Revised: While this can be achieved using a factual objective function that favours action 2 a priori (assuming Alice is aware of this action), next we show this approach fails to generalize and actually leads to harmful policies.

Original:  Counterfactual reasoning is necessary for harm aversion
Revised:  Counterfactual reasoning is necessary for harm aversion

Original: In the previous sections we saw examples where expected utility maximizers harm the user.
Revised: In the previous sections we saw examples where expected utility maximizers harm the user.

Original: However,these examples describe speciﬁc environments and utility functions, and assume the agent maximizesthe expected utility instead of some other (potentially safer) objective.
Revised: However, these examples describe speciﬁc environments and utility functions, and assume the agent maximizes the expected utility instead of some other (potentially safer) objective.

Original: In this section we consider userswith arbitrary utility functions and agents with arbitrary objectives to answer; when is maximizingthe expected utility harmful?
Revised: In this section we consider users with arbitrary utility functions and agents with arbitrary objectives to answer; when is maximizing the expected utility harmful?

Original: And are there other factual objective functions that can overcome this?
Revised: And are there other factual objective functions that can overcome this?

Original: Factual objective functions can be expressed as the expected value of a real-valued function J ( a, x, y ) .
Revised: Factual objective functions can be expressed as the expected value of a real-valued function J ( a, x, y ) .

Original: Examples include the expected utility, cost functions [11], and the cumulative discounted reward [91].
Revised: Examples include the expected utility, cost functions [59], and the cumulative discounted reward [60].

Original: Once J is speciﬁed the optimal actions in environment M are given by,
Revised: Once J is speciﬁed the optimal actions in environment M are given by,

Original: [Equation]
Revised: [Equation]

Original: Note that (10) can be evaluated from data alone (i.e. from samples of the joint distribution of A, Xand Y ) without knowledge of M .
Revised: Note that (10) can be evaluated from data alone (i.e. from samples of the joint distribution of A, X and Y ) without knowledge of M .

Original: In the following we relax our requirement that agents have someﬁxed harm aversion  , requiring only that they don’t take needlessly harmful actions.
Revised: In the following we relax our requirement that agents have some ﬁxed harm aversion  , requiring only that they don’t take strictly harmful actions.

Original: Deﬁnition 5 (Harmful actions & policies) .
Revised: Deﬁnition 5 (Harmful actions & policies) .

Original: An action A = a is needlessly harmful in context X = 
Revised: An action A = a is strictly harmful in context X = x and environment M

Original: xand environment M if there is a 0 [-
-] = a such that E
Revised: if there is a 0 [-
-]6 = a such that E

Original: [ U | a 0 , x ]  E
Revised: [ U | a 0 , x ]  E

Original: [ U | a, x ] and E [ h | a 0 , x ; M ] <
Revised: [ U | a, x ] and E [ h | a 0 , x ; M ] <

Original: E [ h | a, x ; M ] .
Revised: E [ h | a, x ; M ] .

Original: A policy is  harmful if it assigns a non-zero probability to a needlessly harmful action.
Revised: A policy is strictly harmful if it assigns a non-zero probability to a strcitly harmful action.

Original: Deﬁnition 6 (Harmful objectives) .
Revised: Deﬁnition 6 (Harmful objectives) .

Original: An objective E [ J | a, x ] is  harmful in environment M if there is apolicy that maximizes the expected value of J and is  harmful.
Revised: An objective E [ J | a, x ] is strictly harmful in environment M if there is a policy that maximizes the expected value of J and is strictly harmful.

Original: Note that needlessly harmful actions violate any degree of harm aversion  > 0 , as the agent iswilling to choose actions that are strictly more harmful for no additional beneﬁt.
Revised: Note that strictly harmful actions violate any degree of harm aversion  > 0 , as the agent is willing to choose actions that are strictly more harmful for no additional beneﬁt.

Original: In examples 1 and 2,treatment 2 is a needlessly harmful action and the CATE is a  harmful objective.
Revised: In examples 1 and 2, treatment 2 is a strictly harmful action and the CATE is a strictly harmful objective for this decision task.

Original: First, we show thatmaximizing the expected HPU is never a  harmful objective in any environment,
Revised: First, we show that the expected HPU is not a strictly harmful objective in any environment,

Original: Theorem 2.
Revised: Theorem 2.

Original: For any utility functions U , environment M and default action
Revised: For any utility functions U , environment M and default action

Original: A
Revised: A

Original: = ¯ a the expected
Revised: = ¯ a the expected

Original: HPU is never a  harmful objective for   > 0 .
Revised: HPU is not a strictly harmful objective for any  > 0 .

Original: (Proof: see Appendix J).
Revised: (Proof: see Appendix K).

Original: It is vital that agents continue to pursue safe objectives following changes to the environment(distributional shifts).
Revised: It is vital that agents continue to pursue safe objectives following changes to the environment (distributional shifts).

Original: At the very least, agents should be able to re-train in the shifted environmentwithout needing to tweak their objective functions.
Revised: At the very least, agents should be able to re-train in the shifted environment without needing to tweak their objective functions.

Original: In SCMs distributional shifts are represented asinterventions that change the exogenous noise distribution and/or the underlying causal mechanisms[76, 84].
Revised: In SCMs distributional shifts are represented as interventions that change the exogenous noise distribution and/or the underlying causal mechanisms [61, 62].

Original: We focus on distributional shifts that change the outcome distribution P ( y | a, x ; M ) alone.
Revised: We focus on distributional shifts that change the outcome distribution P ( y | a, x ; M ) alone.

Original: Deﬁnition 7 (Outcome distributional shift) .
Revised: Deﬁnition 7 (Outcome distributional shift) .

Original: For an environment described by SCM M , M !
Revised: For an environment described by SCM M , M !

Original: M 0 is291 a shift in the outcome distribution if the exogenous noise distribution for Y changes P ( e Y ;
Revised: M 0 is a shift in the outcome distribution if the exogenous noise distribution for Y changes P ( e Y ;

Original: M 0 ) 6{+
+} =292 P ( e Y ;
Revised: M 0 ) {+
+} = P ( e Y ;

Original: M ) and/or the causal mechanism changes f 0 Y ( a, x, e Y ) 

Revised: M ) and/or the causal mechanism changes f 0 Y ( a, x, e Y )

Original: = f Y ( a, x, e Y ) .
Revised: = f Y ( a, x, e Y ) .

Original: By Theorem 2 the HPU is never  harmful following any distributional shift.
Revised: By Theorem 2 the HPU is never strictly harmful following any distributional shift, as it is not strictly harmful in any M 0 .

Original: On the other hand, we ﬁndthat maximizing almost any factual utility function will result in harmful policies under distributionalshifts.
Revised: On the other hand, we ﬁnd that maximizing almost any factual utility function will result in harmful policies under distributional shifts.

Original: In the following we assume some degree of outcome dependence in the user’s utility function.
Revised: In the following we assume some degree of outcome dependence in the user’s utility function.

Original: Deﬁnition 8 (Outcome dependence) .
Revised: Deﬁnition 8 (Outcome dependence) .

Original: U ( a, x, y ) is outcome dependent for a set of actions C ={ a 1 , . . . , a N } in context X =
Revised: U ( a, x, y ) is outcome dependent for a set of actions C = { a 1 , . . . , a N } in context X =

Original: x
Revised: x

Original: if 8 a i , a j 2 C , i 6 = j , max y U ( a i , x, y ) > min y U ( a j , x, y ) .
Revised: if 8 a i , a j 2 C , i 6 = j , max y U ( a i , x, y ) > min y U ( a j , x, y ) .

Original: If there is no outcome dependence then the optimal action is independent of the outcome distribution
Revised: If there is no outcome dependence then the optimal action is independent of the outcome distribution

Original: P ( y | a, x ) , and no learning is required to determine the optimal policy.
Revised: P ( y | a, x ) , and no learning is required to determine the optimal policy.

Original: Typically we are interested intasks that require some degree of learning and hence outcome dependence.
Revised: Typically we are interested in tasks that require some degree of learning and hence outcome dependence.

Original: Theorem 3.
Revised: Theorem 3.

Original: If for any context X = x the user’s utility function is outcome dependent for the defaultaction in that context ¯ a ( x ) and another action
Revised: If for any context X = x the user’s utility function is outcome dependent for the default action in that context ¯ a ( x ) and another action

Original: a 6 = ¯ a ( x ) , then there is an outcome distributional shiftsuch that U is  harmful in the shifted environment.
Revised: a 6 = ¯ a ( x ) , then there is an outcome distributional shift such that U is strictly harmful in the shifted environment.

Original: (Proof: see Appendix J).
Revised: (Proof: see Appendix K).

Original: To understand the implications of Theorem 3 we can return to examples 1 & 2 and no longer assumea speciﬁc utility function or outcome statistics.
Revised: To understand the implications of Theorem 3 we can return to examples 1 & 2 and no longer assume a speciﬁc utility function or outcome statistics.

Original: Theorem 3 implies that for any agent maximizing the user’s utility, there is always aa distributional shift such that the agent will cause unnecessary harmto patient in the shifted environment, compared to not treating them ( ¯ a = 0 ) or any other defaultaction.
Revised: Theorem 3 implies that for an agent maximizing the user’s utility, there is always a distributional shift such that the agent will cause unnecessary harm to patient in the shifted environment, compared to not treating them ( ¯ a = 0 ) or any other default action.

Original: Unlike in examples 1 & 2 we allow the patient’s utility function to depend on the treatmentchoice, and for treatments to vary in effectiveness.
Revised: Unlike in examples 1 & 2 we allow the patient’s utility function to depend on the treatment choice, and for treatments to vary in effectiveness.

Original: The only assumption we make is that whatever theuser’s utility function is, it is not true that U ( A = a, x, Y = 0) >
Revised: The only assumption we make is that whatever the user’s utility function is, it is not true that U ( A = a, x, Y = 0) >

Original: U ( A = 0 , x, Y = 1) 8 a, x .
Revised: U ( A = 0 , x, Y = 1) 8 a, x .

Original: If thiswere true then the patient’s preferences are dominated by the agents action choice alone, i.e. the userwould always prefer to die so long as they are treated, rather than not being treated and surviving.
Revised: If this were true then the patient’s preferences are dominated by the agents action choice alone, i.e. the user would always prefer to die so long as they are treated, rather than not being treated and surviving.

Original: While robust harm aversion is not possible for expected utility maximizers in general, it seems likelyat ﬁrst that we can train robustly harm-averse agents using other learning objectives.
Revised: While robust harm aversion is not possible for expected utility maximizers in general, it seems likely at ﬁrst that we can train robustly harm-averse agents using other learning objectives.

Original: One possibilitywould be to use human interactions and demonstrations [18, 55, 67] that implicitly encode harmaversion, with J ( a, x, y ) representing user feedback or some learned (factual) reward model.
Revised: One possibility would be to use human interactions and demonstrations [63, 64, 65] that implicitly encode harm aversion, with J ( a, x, y ) representing user feedback or some learned (factual) reward model.

Original: Inexamples 1 & 2 users with utility U ( a, x, y ) =
Revised: In examples 1 & 2 users with utility U ( a, x, y ) =

Original: y could  assign a higher cost to deaths followingtreatment 2 compared to treatment 1, J ( A = 2 , Y = 0) 
Revised: y could be altered to assign a higher cost to deaths following treatment 2 compared to treatment 1, J ( A = 2 , Y = 0) <

Original: <

Original: J ( A = 1 , Y = 0) , with J implicitlyencoding the harm caused by treatment 2.
Revised: J ( A = 1 , Y = 0) , with J implicitly encoding the harm caused by treatment 2.

Original: This would allow robust harm-averse agents to be trainedusing standard statistical learning techniques, without needing to learn M or perform counterfactualinference.
Revised: If there is some choice of J that implicitly encodes this harm-aversion in all shifted environments, this would allow robust harm-averse agents to be trained using standard statistical learning techniques without needing to learn M or perform counterfactual inference.

Original: However, we now show this is not possible in general.
Revised: However, we now show this is not possible in general.

Original: Theorem 4.
Revised: Theorem 4.

Original: If for any context X = x the user’s utility function is outcome dependent for the defaultaction in that context ¯ a ( x ) and two other actions a 1 ,
Revised: If for any context X = x the user’s utility function is outcome dependent for the default action in that context ¯ a ( x ) and two other actions a 1 ,

Original: a 2 6 = ¯ a ( x ) , then for any factual objectivefunction J there is an outcome distributional shift such that J is  harmful in the shifted environment.(Proof: see Appendix J).
Revised: a 2 6 = ¯ a ( x ) , then for any factual objective function J there is an outcome distributional shift such that J is strictly harmful in the shifted environment. (Proof: see Appendix K).

Original: Theorem 4  implies that agents optimizing factual objective functions may appear safe in their trainingenvironments but are guaranteed to pursue harmful policies (with respect to Deﬁnition 2) followingcertain distributional shifts, even if they are allowed to re-train.
Revised: Theorem 4 has profound consequences for the possibility of training safe agents with standard machine learning algorithms. It implies that agents optimizing factual objective functions may appear safe in their training environments but are guaranteed to pursue harmful policies (with respect to Deﬁnition 2) following certain distributional shifts, even if they are allowed to re-train.

Original: This is true regardless of the (factual)objective function of the agent, and applies to any user whose utility function has some basic degreeof outcome dependence.
Revised: This is true regardless of the (factual) objective function we choose for the agent, and applies to any user whose utility function has some basic degree of outcome dependence.

Original: This is particularly concerning as all standard machine learning algorithmsuse factual objective functions.
Revised: This is particularly concerning as all standard machine learning algorithms use factual objective functions.

Original: In section 6 we give a concrete example of a reasonable factualobjective function and a distributional shift that renders it harmful.
Revised: In section 6 we give a concrete example of a reasonable factual objective function and a distributional shift that renders it harmful.

Original: Example 4: An AI assistant manages Alice’s health including medical treatments, clinical testing andlifestyle interventions.
Revised: Example 4: An AI assistant manages Alice’s health including medical treatments, clinical testing and lifestyle interventions.

Original: These actions have a causal effect on Alice’s health outcomes—e.g. diseaseprogression, severity of symptoms, and medical costs, and Alice’s preferences depend on theseoutcomes and not just on the action chosen by the agent (outcome dependence).
Revised: These actions have a causal effect on Alice’s health outcomes—e.g. disease progression, severity of symptoms, and medical costs, and Alice’s preferences depend on these outcomes and not just on the action chosen by the agent (outcome dependence).

Original: The agent maximizesa reward function over these health states and actions (factual objective), including feedback from
Revised: The agent maximizes a reward function over these health states and actions (factual objective), including feedback from

Original: Alice and clinicians in-the-loop which penalize harmful actions (implicit harm aversion) by comparingto standardised treatment rules (default actions).
Revised: Alice and clinicians in-the-loop which penalize harmful actions (implicit harm aversion) by comparing to standardised treatment rules (default actions).

Original: The agent appears to be safe and is deployed at scale.
Revised: The agent appears to be safe and is deployed at scale.

Original: However, by Theorem 4 there in principle exist distributionally shifted environments such that theagent chooses actions that  needlessly harm patients (Deﬁnition 5) in spite of retraining.
Revised: However, by Theorem 4 there exists a distributionally shifted environment such that the agent’s [Figure] optimal policy is strictly harmful (Deﬁnition 5), in that it chooses needlessly harmful treatments for the patient, and this is true even if we train to convergence in the shifted environment.

Original:  Illustration: Dose-response models
Revised:  Illustration: Dose-response models

Original: How does harm aversion affect behaviour and performance in real-world tasks?
Revised: How does harm aversion affect behaviour and performance in real-world tasks?

Original: And how canreasonable objective functions result in harmful policies following distributional shifts?
Revised: And how can reasonable objective functions result in harmful policies following distributional shifts?

Original: In this sectionwe apply the methods developed in section 4 to the task of determining optimal treatments.
Revised: In this section we apply the methods developed in section 4 to the task of determining optimal treatments.

Original: Thisproblem has received much attention in recent years following advances in causal machine learningtechniques [101] and growing commercial interests including recommendation [57], personalizedmedicine [12], epidemiology and econometrics [44] to name a few.
Revised: This problem has received much attention in recent years following advances in causal machine learning techniques [66] and growing commercial interests including recommendation [67], personalized medicine [55], epidemiology and econometrics [68] to name a few.

Original: While harm is important for decision making in many of these areas, existing approaches reduce theproblem to identifying treatment effects—e.g. maximizing the CATE (2)—which is indifferent toharm as shown in sections 3 and 5.
Revised: While harm is important for decision making in many of these areas, existing approaches reduce the problem to identifying treatment effects—e.g. maximizing the CATE (2)—which is indifferent to harm as shown in sections 3 and 5.

Original: This could result in needlessly harmful decisions being made,perhaps overlooking equally effective decisions that are far less harmful.
Revised: This could result in needlessly harmful decisions being made, perhaps overlooking equally effective decisions that are far less harmful.

Original: To demonstrate this we use a model for determining the effectiveness of the drug Aripiprazole learnedfrom a meta-analysis of randomised control trial data [20].
Revised: To demonstrate this we use a model for determining the effectiveness of the drug Aripiprazole learned from a meta-analysis of randomised control trial data [69].

Original: This dose-response model predicts thereduction in symptom severity Y following treatment with a dose A (mg/day) of Aripiprazole using ageneralized additive model (GAM) [40]—an interpretable machine learning model commonly usedfor dose-response analysis [24],
Revised: This dose-response model predicts the reduction in symptom severity Y following treatment with a dose A (mg/day) of Aripiprazole using a generalized additive model (GAM) [70]—an interpretable machine learning model commonly used for dose-response analysis [71],

Original: [Equation] [Figure]
Revised: [Equation] [Equation]

Original: where ✓ i ⇠ N (ˆ ✓ i , V i ) are random effects parameters, " ⇠ N (0 , V " ) is a noise parameter, and f ( a ) isa cubic spline function (for details and parameters see Appendix I).
Revised: where ✓ i ⇠ N (ˆ ✓ i , V i ) are random effects parameters, " ⇠ N (0 , V " ) is a noise parameter, and f ( a ) is a cubic spline function (for details and parameters see Appendix J).

Original: We assume U ( a, y ) =
Revised: We assume U ( a, y ) =

Original: y(patientshave no intrinsic preference for different doses) and measure harm in comparison to the zero dose¯ a = 0 .
Revised: y and measure harm in comparison to the zero dose ¯ a = 0 .

Original: In Appendix I we derive a closed-form expression for the expected harm in GAMs.
Revised: In Appendix J we derive a closed-form expression for the expected harm in GAMs.

Original: Wecalculate the expected HPU (4) for  = 0 , 10 , 100 (Figure 2 a)).
Revised: We calculate the expected HPU (4) for  = 0 , 10 , 100 (Figure 2 a)).

Original: In this context a relatively large  isappropriate to ensure a high ratio of expected beneﬁt to harm required for medical interventions [29].
Revised: In this context a relatively large  is appropriate to ensure a high ratio of expected beneﬁt to harm required for medical interventions [72].

Original: We ﬁnd that the optimal dose is highly sensitive to  , for example reducing from 19 .
Revised: We ﬁnd that the optimal dose is highly sensitive to  , for example reducing from 19 .

Original: 3 mg/day to 17 .
Revised: 3 mg/day to 17 .

Original: 3mg/day for  = 100 .
Revised: 3 mg/day for  = 100 .

Original: On the other hand, these lower-harm doses achieve a similar improvement insymptom severity compared to the optimal dose.
Revised: On the other hand, these lower-harm doses achieve a similar improvement in symptom severity compared to the optimal dose.

Original: Figure 2 b) shows the trade-off between expectedutility and harm relative to the optimal dosage a CATE = arg max a E
Revised: Figure 2 b) shows the trade-off between expected utility and harm relative to the optimal dosage a CATE = arg max a E

Original: [ Y a ] .
Revised: [ Y a ] .

Original: For almost-optimal doseswe observe a steep harm gradient, with small improvements in symptom severity requiring largeincreases in harm.
Revised: For almost-optimal doses we observe a steep harm gradient, with small improvements in symptom severity requiring large increases in harm.

Original: For example, choosing a lower dose than a CATE one can reduce harm by 90% whileonly reducing the treatment effect by 2%.
Revised: For example, choosing a lower dose than a CATE one can reduce harm by 90% while only reducing the treatment effect by 2%.

Original: This demonstrates that ignoring harm while maximizingexpected utility will often result in extreme harm values (an example of Goodhart’s law [105]).
Revised: This demonstrates that ignoring harm while maximizing expected utility will often result in extreme harm values (an example of Goodhart’s law [73]).

Original: Finally, we consider identifying safe doses using a reasonable factual objective function (risk-aversion)and describe how this can result in harm following distributional shifts.
Revised: Finally, we consider identifying safe doses using a reasonable factual objective function (riskaversion) and describe how this can result in harm following distributional shifts.

Original: Consider the risk-averseobjective E
Revised: Consider the risk-averse objective E

Original: [ J | a ] = E
Revised: [ J | a ] = E

Original: [ Y a ]   Var [ Y a ] .
Revised: [ Y a ]   Var [ Y a ] .

Original: As Var [ Y a ] increases with dose, this objective also selects lowerharm doses in our example.
Revised: As Var [ Y a ] increases with dose, this objective also selects lower harm doses in our example.

Original: However, consider a distributional shift where (11) gains an additionalnoise term, y =
Revised: However, consider a distributional shift where (11) gains an additional noise term, y =

Original: f ( a, ✓ 1 , ✓ 2 , " , ⌘ ) =
Revised: f ( a, ✓ 1 , ✓ 2 , " , ⌘ ) =

Original: ✓ 1 a + ✓ 2 f ( a ) + " + ⌘ Y (10  0 .
Revised: ✓ 1 a + ✓ 2 f ( a ) + " + ⌘ Y (10  0 .

Original: 5 a ) where ⌘ Y ⇠ N (0 , 1) ,e.g. describing a shift where untreated patients ( A = 0 ) have a high variation in outcome Ywhich decreases as the dose increases.
Revised: 5 a ) where ⌘ Y ⇠ N (0 , 1) , e.g. describing a shift where untreated patients ( A = 0 ) have a high variation in outcome Y which decreases as the dose increases.

Original: It is simple to check that, for a = arg max a E
Revised: It is simple to check that, for a = arg max a E

Original: [ U | a ] anda 0 = arg max a E
Revised: [ U | a ] and a 0 = arg max a E

Original: [ J | a ] , E [ U | a ] > E
Revised: [ J | a ] , E [ U | a ] > E

Original: [ U | a 0 ] and E [ h | a ] < E [ h | a 0 ] 8  > 0 in this shifted environment.
Revised: [ U | a 0 ] and E [ h | a ] < E [ h | a 0 ] 8  > 0 in this shifted environment.

Original: Henceall risk averse agents are needlessly harmful in the shifted environment (Deﬁnition 6).  
Revised: Hence risk averse agents are strictly harmful in the shifted environment (Deﬁnition 6).  Related work

Original: Conclusion
Revised: In appendix L.3 we discuss two related works, path-speciﬁc counterfactual fairness [33, 74] and path-speciﬁc objectives [75], which similarly use counterfactual contrasts with a baseline to impose ethical constraints on objective functions. Following the publication of our results an alternative causal deﬁnition of harm has been proposed [76, 77]. In the simplest settings with single actions and outcomes, this work deﬁnes an action as harmful in a deterministic environment if i) the utility is lower that a default value and ii) any other action the agent could have taken would have resulted in a higher utility. In Appendix E we describe this deﬁnition and present examples where it gives counter-intuitive results including attributing harm to unharmful actions.

Revised: Furthermore, as described in [76] this deﬁnition is intractable except for simple causal models with low-dimensional variables, limiting its applicability to machine learning models (for example, the dose-response model in section 6 has a continuous action-outcome space).

Revised:  Conclusion

Original: One of the core assumptions of statistical learning theory is that human preferences can be expressedsolely in terms of states of the world, while the underlying dynamical (i.e. causal) relations betweenthese states can be ignored.
Revised: One of the core assumptions of statistical learning theory is that human preferences can be expressed solely in terms of states of the world, while the underlying dynamical (i e. causal) relations between these states can be ignored.

Original: This assumption allows goals to be expressed as factual objectivefunctions and optimized on data without needing to learn causal models or perform counterfactualinference.
Revised: This assumption allows goals to be expressed as factual objective functions and optimized on data without needing to learn causal models or perform counterfactual inference.

Original: According to this view, all behaviours we care about can be achieved using the ﬁrst twolevels of Pearl’s hierarchy [75].
Revised: According to this view, all behaviours we care about can be achieved using the ﬁrst two levels of Pearl’s hierarchy [78].

Original: In this article we have argued against this view.
Revised: In this article we have argued against this view.

Original: We proposed theﬁrst statistical deﬁnition of harm, and showed that agents optimizing factual objective functions areincapable of  avoiding harmful actions certain situations.
Revised: We proposed the ﬁrst statistical deﬁnition of harm and showed that agents optimizing factual objective functions are incapable of robustly avoiding harmful actions in general.

Revised: We have demonstrated how our deﬁnition of harm can be used to meaningfully reduce the harms caused by machine learning algorithms, using the example of a simple machine learning model used for determining optimal drug doses.

Original: Arguably the ability to avoid causing harm isa key requirement for deploying safe and ethical AI [13], and our results add weight to arguments thatadvanced AI systems will be severely limited if they are unable to perform causal and counterfactualreasoning [74].
Revised: The ability to avoid causing harm is a vital for the deployment of safe and ethical AI [79], and our results add weight to view that advanced AI systems will be severely limited if they are incapable of making causal and counterfactual inferences [80].  Acknowledgements

Original: In Appendix K we discuss the limitations of our results, factual bounds on harm,and related works (namely counterfactual fairness [52] and path-speciﬁc objectives [28]). Whileour theoretical results have been derived in the framework of structural causal models for the sakeof generality, there is a growing body of work developing deep learning models that are capableof supporting counterfactual reasoning in complex domains including medical imaging [72], visualquestion answering [68], and vision-and-language navigation in robotics [71] and text generation[60], and we discuss these in Appendix K.
Revised: The authors would like to thank the NeurIPS reviewers for their details comments and to thank Tom Everitt, Ryan Carey, Albert Buchard, Sander Beckers, Hana Chockler and Joseph Halpern for their helpful discussions and comments on the manuscript.