Original: Counterfactual harm
Revised: Counterfactual harm

Original: Abstract
Revised: Abstract

Original: To act safely and ethically in the real world, agents must be able to reason about harm and avoid harmful actions.
Revised: To act safely and ethically in the real world, agents must be able to reason about harm and avoid harmful actions.

Original: However, to date there is no statistical method for measuring harm and factoring it into algorithmic decisions.
Revised: However, to date there is no statistical method for measuring harm and factoring it into algorithmic decisions.

Original: In this paper we propose the ﬁrst formal deﬁnition of harm and beneﬁt using causal models.
Revised: In this paper we propose the ﬁrst formal deﬁnition of harm and beneﬁt using causal models.

Original: We show that any factual deﬁnition of harm must violate basic intuitions in certain scenarios, and show that standard machine learning algorithms that cannot perform counterfactual reasoning are guaranteed to pursue harmful policies following distributional shifts.
Revised: We show that any factual deﬁnition of harm must violate basic intuitions in certain scenarios, and show that standard machine learning algorithms that cannot perform counterfactual reasoning are guaranteed to pursue harmful policies following distributional shifts.

Original: We use our deﬁnition of harm to devise a framework for harmaverse decision making using counterfactual objective functions.
Revised: We use our deﬁnition of harm to devise a framework for harmaverse decision making using counterfactual objective functions.

Original: We demonstrate this framework on the problem of identifying optimal drug doses using a doseresponse model learned from randomized control trial data.
Revised: We demonstrate this framework on the problem of identifying optimal drug doses using a doseresponse model learned from randomized control trial data.

Original: We ﬁnd that the standard method of selecting doses using treatment effects results in unnecessarily harmful doses, while our counterfactual approach allows us to identify doses that are signiﬁcantly less harmful without sacriﬁcing efﬁcacy.
Revised: We ﬁnd that the standard method of selecting doses using treatment effects results in unnecessarily harmful doses, while our counterfactual approach allows us to identify doses that are signiﬁcantly less harmful without sacriﬁcing efﬁcacy.

Original:  Introduction
Revised:  Introduction

Original: Machine learning algorithms are increasingly used for making highly consequential decisions in areas such as medicine [1, 2, 3], criminal justice [4, 5, 6], ﬁnance [7, 8, 9], and autonomous vehicles [10].
Revised: Machine learning algorithms are increasingly used for making highly consequential decisions in areas such as medicine [1, 2, 3], criminal justice [4, 5, 6], ﬁnance [7, 8, 9], and autonomous vehicles [10].

Original: These algorithms optimize for outcomes such as survival time, recidivism rate and ﬁnancial cost.
Revised: These algorithms optimize for outcomes such as survival time, recidivism rate and ﬁnancial cost.

Original: However, these are purely factual objectives, depending only on what outcome occurs.
Revised: However, these are purely factual objectives, depending only on what outcome occurs.

Original: On the other hand, human decision making incorporates preferences that depend not only on the outcome, but why it occurred.
Revised: On the other hand, human decision making incorporates preferences that depend not only on the outcome, but why it occurred.

Original: For example, consider two treatments for a disease which, when left untreated has a 50% mortality rate.
Revised: For example, consider two treatments for a disease which, when left untreated has a 50% mortality rate.

Original: Treatment 1 has a 60% chance of curing a patient, and a 40% chance of having no effect, with the disease progressing as if untreated.
Revised: Treatment 1 has a 60% chance of curing a patient, and a 40% chance of having no effect, with the disease progressing as if untreated.

Original: Treatment 2 has an 80% chance of curing a patient and a 20% chance of killing them.
Revised: Treatment 2 has an 80% chance of curing a patient and a 20% chance of killing them.

Original: Treatments 1 and 2 have identical recovery rates, hence any agent that selects actions based solely on the factual outcome statistics (e g. the effect of treatment on the treated [11]) will not be able to distinguish between these two treatments.
Revised: Treatments 1 and 2 have identical recovery rates, hence any agent that selects actions based solely on the factual outcome statistics (e g. the effect of treatment on the treated [11]) will not be able to distinguish between these two treatments.

Original: However, doctors systematically favour treatment 1 as it achieves the same recovery rate but never harms the patient—there are no patients that would have survived had they not been treated.
Revised: However, doctors systematically favour treatment 1 as it achieves the same recovery rate but never harms the patient—there are no patients that would have survived had they not been treated.

Original: On the other hand, doctors who administer treatment 2 risk harming their patients—there are patients who die following treatment who would have lived had they not been treated.
Revised: On the other hand, doctors who administer treatment 2 risk harming their patients—there are patients who die following treatment who would have lived had they not been treated.

Original: While treatments 1 and 2 have the same mortality rates, the resulting deaths have different causes (doctor or disease) which have different ethical implications.
Revised: While treatments 1 and 2 have the same mortality rates, the resulting deaths have different causes (doctor or disease) which have different ethical implications.

Original: Determining what would have happened to a patient if the doctor hadn’t treated them is a counterfactual inference, which cannot be identiﬁed from outcome statistics alone but requires knowledge of the functional causal relation between actions and outcomes [12].
Revised: Determining what would have happened to a patient if the doctor hadn’t treated them is a counterfactual inference, which cannot be identiﬁed from outcome statistics alone but requires knowledge of the functional causal relation between actions and outcomes [12].

Original: However, standard machine learning algorithms are limited to making factual inferences—e.g. learning correlations between actions and outcomes.
Revised: However, standard machine learning algorithms are limited to making factual inferences—e.g. learning correlations between actions and outcomes.

Original: How then can we express these ethical preferences as objective functions and ultimately incorporate them into machine learning algorithms?
Revised: How then can we express these ethical preferences as objective functions and ultimately incorporate them into machine learning algorithms?

Original: This question has some urgency, given that machine learning algorithms currently under development or in deployment will be unable to reason about harm and factor it into their decisions.
Revised: This question has some urgency, given that machine learning algorithms currently under development or in deployment will be unable to reason about harm and factor it into their decisions.

Original: The concept of harm is foundational to many ethical codes and principles; from the Hippocratic oath [13] to environmental policy [14], and from the foundations of classical liberalism [15], to
Revised: The concept of harm is foundational to many ethical codes and principles; from the Hippocratic oath [13] to environmental policy [14], and from the foundations of classical liberalism [15], to

Original: Asimov’s laws of robotics [16].
Revised: Asimov’s laws of robotics [16].

Original: Despite this ubiquity, there is no formal statistical deﬁnition of harm.
Revised: Despite this ubiquity, there is no formal statistical deﬁnition of harm.

Original: We address this by translating the predominant account of harm [17, 18] into a statistical measure— counterfactual harm .
Revised: We address this by translating the predominant account of harm [17, 18] into a statistical measure— counterfactual harm .

Original: Our deﬁnition of harm enables us for the ﬁrst time to precisely answer the following basic questions,
Revised: Our deﬁnition of harm enables us for the ﬁrst time to precisely answer the following basic questions,

Original: Question 1 Did the actions of an agent cause harm, and if so, how much?
Revised: Question 1 Did the actions of an agent cause harm, and if so, how much?

Original: Question 2 How much harm can we expect an action to cause prior to taking it?
Revised: Question 2 How much harm can we expect an action to cause prior to taking it?

Original: Question 

Revised: Question

Original: How can we identify actions that balance the expected harms and beneﬁts?
Revised: How can we identify actions that balance the expected harms and beneﬁts?

Original: In section 2 we introduce concepts from causality, ethics, and expected utility theory used to derive our results.
Revised: In section 2 we introduce concepts from causality, ethics, and expected utility theory used to derive our results.

Original: In section 3 we present our deﬁnitions of harm and beneﬁt, and in section 4 we describe how harm-aversion can be factored into algorithmic decisions.
Revised: In section 3 we present our deﬁnitions of harm and beneﬁt, and in section 4 we describe how harm-aversion can be factored into algorithmic decisions.

Original: In section 5 we prove that algorithms based on standard statistical learning theory are incapable of reasoning about harm and are guaranteed to pursue harmful policies following certain distributional shifts, while our approach using counterfactual objective functions overcomes these problems.
Revised: In section 5 we prove that algorithms based on standard statistical learning theory are incapable of reasoning about harm and are guaranteed to pursue harmful policies following certain distributional shifts, while our approach using counterfactual objective functions overcomes these problems.

Original: Finally, in section 6 we apply our methods to an interpretable machine learning model for determining optimal drug doses, and compare this to standard algorithms that maximize treatment effects.
Revised: Finally, in section 6 we apply our methods to an interpretable machine learning model for determining optimal drug doses, and compare this to standard algorithms that maximize treatment effects.

Original:  Prelimiaries
Revised:  Prelimiaries

Original: In this section we introduce some basic principles from causal modelling, expected utility theory and ethics that we use to derive our results.
Revised: In this section we introduce some basic principles from causal modelling, expected utility theory and ethics that we use to derive our results.

Original: 2.1 Structural causal models
Revised: 2.1 Structural causal models

Original: We provide a brief overview of structural causal models (SCMs) (see Chapter 7 of [19] and [20] for reviews).
Revised: We provide a brief overview of structural causal models (SCMs) (see Chapter 7 of [19] and [20] for reviews).

Original: SCMs represent observable variables
Revised: SCMs represent observable variables

Original: X i as vertices of a directed acyclic graph (DAG), with directed edges representing causal relations between X i and their direct causes (parents), which includes observable (endogenous) parents pa i and a single exogenous noise variable E i .
Revised: X i as vertices of a directed acyclic graph (DAG), with directed edges representing causal relations between X i and their direct causes (parents), which includes observable (endogenous) parents pa i and a single exogenous noise variable E i .

Original: The value of each variable is assigned by a deterministic function f
Revised: The value of each variable is assigned by a deterministic function f

Original: i of these parents.
Revised: i of these parents.

Original: Deﬁnition 1 (Structural Causal Model (SCM)) .
Revised: Deﬁnition 1 (Structural Causal Model (SCM)) .

Original: A structural causal model M = h G , E, X, F, P i speciﬁes:
Revised: A structural causal model M = h G , E, X, F, P i speciﬁes:

Original: a set of observable (endogenous) random variables X
Revised: a set of observable (endogenous) random variables X

Original: = { X 1 , . . .
Revised: = { X 1 , . . .

Original: , X N } , 2.
Revised: , X N } , 2.

Original: a set of mutually independent noise (exogenous) variables E
Revised: a set of mutually independent noise (exogenous) variables E

Original: = { E 1 , . . .
Revised: = { E 1 , . . .

Original: , E N } with joint distribution P ( E
Revised: , E N } with joint distribution P ( E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (assuming causal sufﬁciency [19]) 3.
Revised: (assuming causal sufﬁciency [19]) 3.

Original: a directed acyclic graph G , whose nodes are the variables E
Revised: a directed acyclic graph G , whose nodes are the variables E

Original: [ X with directed edges from pa i , E i to X i , where pa i ✓
Revised: [ X with directed edges from pa i , E i to X i , where pa i ✓

Original: X denotes the endogenous parents of X i .
Revised: X denotes the endogenous parents of X i .

Original: a set of functions (mechanisms) F = { f 1 , . .
Revised: a set of functions (mechanisms) F = { f 1 , . .

Original: . , f N } , where the collection F forms a mapping from E to X , x i :=
Revised: . , f N } , where the collection F forms a mapping from E to X , x i :=

Original: f i ( pa i , e i ) , for i = 1 , . . .
Revised: f i ( pa i , e i ) , for i = 1 , . . .

Original: , N,
Revised: , N,

Original: By recursively applying condition 4 every observable variable can be expressed as a function of the noise variables alone, X i ( e ) =
Revised: By recursively applying condition 4 every observable variable can be expressed as a function of the noise variables alone, X i ( e ) =

Original: x i where E = e is the joint state of the noise variables, and the distribution over unobserved noise variables induces a distribution over the observable variables,
Revised: x i where E = e is the joint state of the noise variables, and the distribution over unobserved noise variables induces a distribution over the observable variables,

Original: P
Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: X = x is the joint state over X .
Revised: X = x is the joint state over X .

Original: The power of SCMs is that they specify not only the joint distribution P ( X = x ) but also the distribution of X under all interventions, including incompatible interventions (counterfactuals).
Revised: The power of SCMs is that they specify not only the joint distribution P ( X = x ) but also the distribution of X under all interventions, including incompatible interventions (counterfactuals).

Original: Interventions describe an external agent changing the underlying causal mechanisms F .
Revised: Interventions describe an external agent changing the underlying causal mechanisms F .

Original: For example, do ( X 1 = x ) denotes intervening to force X 1 = x regardless of the state of its parents, replacing x 1 := f 1 ( pa 1 , e 1 ) !
Revised: For example, do ( X 1 = x ) denotes intervening to force X 1 = x regardless of the state of its parents, replacing x 1 := f 1 ( pa 1 , e 1 ) !

Original: x 1 := x .
Revised: x 1 := x .

Original: The variables in this post-intervention SCM are denoted X ix or
Revised: The variables in this post-intervention SCM are denoted X ix or

Original: X iX 1 = x and their interventional distribution is given by (1) under the updated mechanisms.
Revised: X iX 1 = x and their interventional distribution is given by (1) under the updated mechanisms.

Original: These interventional distributions quantify the effects of actions and are used to inform decisions.
Revised: These interventional distributions quantify the effects of actions and are used to inform decisions.

Original: For example, the conditional average treatment effect (CATE) [21],
Revised: For example, the conditional average treatment effect (CATE) [21],

Original: [Equation]
Revised: [Equation]

Original: measures the expected value of outcome Y for treatment T = 1 compared to control T =conditional on X , and is used in decision making from precision medicine to economics [22, 23].
Revised: measures the expected value of outcome Y for treatment T = 1 compared to control T =conditional on X , and is used in decision making from precision medicine to economics [22, 23].

Original: Counterfactual distributions determine the probability that other outcomes would have occurred had some precondition been different.
Revised: Counterfactual distributions determine the probability that other outcomes would have occurred had some precondition been different.

Original: For example, for X, Y 2 { T, F } , the causal effect P ( Y X = T = T ) is insufﬁcient to determine if do ( X = T ) was a necessary cause of Y = T [24].
Revised: For example, for X, Y 2 { T, F } , the causal effect P ( Y X = T = T ) is insufﬁcient to determine if do ( X = T ) was a necessary cause of Y = T [24].

Original: Instead this is captured by the counterfactual probability of necessity [25],
Revised: Instead this is captured by the counterfactual probability of necessity [25],

Original: [Equation]
Revised: [Equation]

Original: which is the probability that Y would be false if X had been false (the counterfactual proposition), given that X and Y were both true (the factual data).
Revised: which is the probability that Y would be false if X had been false (the counterfactual proposition), given that X and Y were both true (the factual data).

Original: For example, if PN = 0 then Y
Revised: For example, if PN = 0 then Y

Original: = T would have occured regardless of X = T , whereas if PN = 1 then Y
Revised: = T would have occured regardless of X = T , whereas if PN = 1 then Y

Original: = T could not have occurred without
Revised: = T could not have occurred without

Original: X = T .
Revised: X = T .

Original: Equation (3) involves the joint distribution over incompatible states of Y under different interventions, Y = T and Y X
Revised: Equation (3) involves the joint distribution over incompatible states of Y under different interventions, Y = T and Y X

Original: = F = F , which cannot be jointly measured and hence (3) cannot be identiﬁed from data alone.
Revised: = F = F , which cannot be jointly measured and hence (3) cannot be identiﬁed from data alone.

Original: However, (3) can be calculated from the SCM using (1) for the combined factual and counterfactual propositions,
Revised: However, (3) can be calculated from the SCM using (1) for the combined factual and counterfactual propositions,

Original: [Equation]
Revised: [Equation]

Original: This ability to ascribe causal explanations for data has made SCMs and counterfactual inference a key ingredient in statistical deﬁnitions of blame, intent and responsibility [26, 27, 28], and has seen applications in AI research ranging from explainability [29, 30], safety [31, 32], and fairness [33], to reinforcement learning [34, 35] and medical diagnosis [36].
Revised: This ability to ascribe causal explanations for data has made SCMs and counterfactual inference a key ingredient in statistical deﬁnitions of blame, intent and responsibility [26, 27, 28], and has seen applications in AI research ranging from explainability [29, 30], safety [31, 32], and fairness [33], to reinforcement learning [34, 35] and medical diagnosis [36].

Original: Our deﬁnitions and theoretical results are derived within the SCM framework as is standard in causality research [19].
Revised: Our deﬁnitions and theoretical results are derived within the SCM framework as is standard in causality research [19].

Original: However, there is a growing body of work developing deep learning models that are capable of supporting counterfactual reasoning in complex domains including medical imaging [37], visual question answering [38], and vision-and-language navigation in robotics [39] and text generation [40].
Revised: However, there is a growing body of work developing deep learning models that are capable of supporting counterfactual reasoning in complex domains including medical imaging [37], visual question answering [38], and vision-and-language navigation in robotics [39] and text generation [40].

Original: We discuss these approaches in relation to our work in Appendix L.1.
Revised: We discuss these approaches in relation to our work in Appendix L.1.

Original: 2.2 The counterfactual comparative account of harm
Revised: 2.2 The counterfactual comparative account of harm

Original: The concept of harm is deeply embedded in ethical principles, codes and law.
Revised: The concept of harm is deeply embedded in ethical principles, codes and law.

Original: One famous example is the bioethical principle “ﬁrst, do no harm” [41], asserting that the moral responsibility for doctors to beneﬁt patients is superseded by their responsibility not to harm them [42].
Revised: One famous example is the bioethical principle “ﬁrst, do no harm” [41], asserting that the moral responsibility for doctors to beneﬁt patients is superseded by their responsibility not to harm them [42].

Original: Another example is
Revised: Another example is

Original: John Stuart Mill’s harm principle which forms the basis of classical liberalism [15] and inspired the “zeroth law” of Asimov’s ﬁctional robot governed society, which states that “A robot may not harm humanity or, by inaction, allow humanity to come to harm” [16].
Revised: John Stuart Mill’s harm principle which forms the basis of classical liberalism [15] and inspired the “zeroth law” of Asimov’s ﬁctional robot governed society, which states that “A robot may not harm humanity or, by inaction, allow humanity to come to harm” [16].

Original: Despite these attempts to codify harm into rules for governing human and AI behaviour, it is not immediately clear what we mean when we talk about harm.
Revised: Despite these attempts to codify harm into rules for governing human and AI behaviour, it is not immediately clear what we mean when we talk about harm.

Original: This can lead to confusion—for example, does “allow humanity to come to harm” describe an agent causing harm by inaction [43] or failing to beneﬁt?
Revised: This can lead to confusion—for example, does “allow humanity to come to harm” describe an agent causing harm by inaction [43] or failing to beneﬁt?

Original: The meaning and measure of harm also has important practical ramiﬁcations.
Revised: The meaning and measure of harm also has important practical ramiﬁcations.

Original: For example, establishing negligence in tort law requires establishing that signiﬁcant harm has occurred [44].
Revised: For example, establishing negligence in tort law requires establishing that signiﬁcant harm has occurred [44].

Original: This has motivated work in philosophy, ethics and law to rigorously deﬁne harm, with the most widely accepted deﬁnition being the counterfactual comparative account (CCA) [17, 45, 46],
Revised: This has motivated work in philosophy, ethics and law to rigorously deﬁne harm, with the most widely accepted deﬁnition being the counterfactual comparative account (CCA) [17, 45, 46],

Original: Deﬁnition 2 (CCA) .
Revised: Deﬁnition 2 (CCA) .

Original: The counterfactual comparative account of harm and beneﬁt states, “An event e or action a harms (beneﬁts) a person overall if and only if she would have been on balance better (worse) off if e had not occurred, or a had not been performed.”
Revised: The counterfactual comparative account of harm and beneﬁt states, “An event e or action a harms (beneﬁts) a person overall if and only if she would have been on balance better (worse) off if e had not occurred, or a had not been performed.”

Original: The CCA quantiﬁes harm by comparing how well off a person is (their factual outcome) compared to how well off they would have been (their counterfactual outcome) had the agent not acted.
Revised: The CCA quantiﬁes harm by comparing how well off a person is (their factual outcome) compared to how well off they would have been (their counterfactual outcome) had the agent not acted.

Original: Generating these counterfactuals necessitates a causal model of the agents actions and their consequences, and we determine how “better (worse) off” the person is using an expected utility analysis [47].
Revised: Generating these counterfactuals necessitates a causal model of the agents actions and their consequences, and we determine how “better (worse) off” the person is using an expected utility analysis [47].

Original: While the wording of the CCA suggests that we always compare to the counterfactual world where the agent takes no action, a more natural and general approach is to measure harm compared to a speciﬁc default action or policy.
Revised: While the wording of the CCA suggests that we always compare to the counterfactual world where the agent takes no action, a more natural and general approach is to measure harm compared to a speciﬁc default action or policy.

Original: For example, when measuring the harm caused by a medicine we may intuitively want to compare to the outcomes that would have occurred with a placebo, and for the harm caused by negligent medical decisions the natural choice is to compare to a standard clinical decision policy.
Revised: For example, when measuring the harm caused by a medicine we may intuitively want to compare to the outcomes that would have occurred with a placebo, and for the harm caused by negligent medical decisions the natural choice is to compare to a standard clinical decision policy.

Original: To this end, we measure harm compared to a default action (or policy), see Appendix
Revised: To this end, we measure harm compared to a default action (or policy), see Appendix

Original: D) for further discussion.
Revised: D) for further discussion.

Original: While the CCA is the predominant deﬁnition of harm and forms the basis of our results, alternative deﬁnitions have been proposed [48, 49], motivated by scenarios where the CCA appears to give counter-intuitive results [43].
Revised: While the CCA is the predominant deﬁnition of harm and forms the basis of our results, alternative deﬁnitions have been proposed [48, 49], motivated by scenarios where the CCA appears to give counter-intuitive results [43].

Original: Surprisingly, we ﬁnd these problematic scenarios are identical to those raised in the study of actual causality [24], and can be resolved with a formal causal analysis.
Revised: Surprisingly, we ﬁnd these problematic scenarios are identical to those raised in the study of actual causality [24], and can be resolved with a formal causal analysis.

Original: To our knowledge, this connection between the harm literature and actual causality has not been noted before, and we discuss these alternative deﬁnitions in Appendix C and how the arguments surrounding the
Revised: To our knowledge, this connection between the harm literature and actual causality has not been noted before, and we discuss these alternative deﬁnitions in Appendix C and how the arguments surrounding the

Original: CCA can be resolved in Appendix B in the hope this will stimulate discussion between these two ﬁelds.
Revised: CCA can be resolved in Appendix B in the hope this will stimulate discussion between these two ﬁelds.

Original: 2.3 Decision theoretic setup
Revised: 2.3 Decision theoretic setup

Original: In this section we present our setup for calculating the CCA (Deﬁnition 2).
Revised: In this section we present our setup for calculating the CCA (Deﬁnition 2).

Original: The CCA describes a person (referred to as the user ) being made better or worse off due to the actions of an agent.
Revised: The CCA describes a person (referred to as the user ) being made better or worse off due to the actions of an agent.

Original: We focus on the simplest case involving single agents performing single actions, though ultimately our deﬁnitions can be extended to multi-agent or sequential decision problems (as discussed in Appendix
Revised: We focus on the simplest case involving single agents performing single actions, though ultimately our deﬁnitions can be extended to multi-agent or sequential decision problems (as discussed in Appendix

Original: L.3).
Revised: L.3).

Original: For causal modelling in these scenarios we refer the reader to [32, 50, 51].
Revised: For causal modelling in these scenarios we refer the reader to [32, 50, 51].

Original: [Equation]
Revised: [Equation]

Original: We describe the environment with a structural causal model M which speciﬁes the distribution over environment states W as endogenous variables, P ( w ; M ) .
Revised: We describe the environment with a structural causal model M which speciﬁes the distribution over environment states W as endogenous variables, P ( w ; M ) .

Original: For single actions the environment variables can be divided into three disjoint sets W =
Revised: For single actions the environment variables can be divided into three disjoint sets W =

Original: A [ X [ Y (Figure 1 a))
Revised: A [ X [ Y (Figure 1 a))

Original: ; the agent’s action A , outcome variables Y that are descendants of A and so can be inﬂuenced by the agent’s action, and context variables X that are non-descendants of nor in A .
Revised: ; the agent’s action A , outcome variables Y that are descendants of A and so can be inﬂuenced by the agent’s action, and context variables X that are non-descendants of nor in A .

Original: For simplicity we assume no unobserved confounders (causal sufﬁciency [19]) in the derivation of our theoretical results.
Revised: For simplicity we assume no unobserved confounders (causal sufﬁciency [19]) in the derivation of our theoretical results.

Original: In the following we assume knowledge of the SCM, and in Appendix L.1 we discuss this assumption and describe how tight upper bounds on the harm can be determined when the SCM is unknown.
Revised: In the following we assume knowledge of the SCM, and in Appendix L.1 we discuss this assumption and describe how tight upper bounds on the harm can be determined when the SCM is unknown.

Original: The user’s preferences over environment states are speciﬁed by their utility function U ( w ) =
Revised: The user’s preferences over environment states are speciﬁed by their utility function U ( w ) =

Original: U ( a, x, y ) , and for a given action the user’s expected utility is,
Revised: U ( a, x, y ) , and for a given action the user’s expected utility is,

Original: [Equation]
Revised: [Equation]

Original: In sections 3-4 we focus on the case where the agent acts to maximize the user’s expected utility, with optimal actions a max = arg max a E
Revised: In sections 3-4 we focus on the case where the agent acts to maximize the user’s expected utility, with optimal actions a max = arg max a E

Original: [ U | a, x ] .
Revised: [ U | a, x ] .

Original: While this is similar to the standard setup in expected utility theory [47, 52, 53], note that U describes the preferences of the user, and in general the agent can pursue any objective and could be entirely unaware of the user’s preferences.
Revised: While this is similar to the standard setup in expected utility theory [47, 52, 53], note that U describes the preferences of the user, and in general the agent can pursue any objective and could be entirely unaware of the user’s preferences.

Original: In section 5 we extend our analysis to agents pursuing arbitrary objectives.
Revised: In section 5 we extend our analysis to agents pursuing arbitrary objectives.

Original: Example 1: conditional average treatment effects.
Revised: Example 1: conditional average treatment effects.

Original: Returning to our example from section 1, consider an agent choosing between a placebo A = 0 , or treatments 1 and 2 .
Revised: Returning to our example from section 1, consider an agent choosing between a placebo A = 0 , or treatments 1 and 2 .

Original: Y = 1 indicates recovery and Y = 0 mortality.
Revised: Y = 1 indicates recovery and Y = 0 mortality.

Original: In the trial, harm is measured by comparison to ‘no treatment’ ¯ a = 0 .
Revised: In the trial, harm is measured by comparison to ‘no treatment’ ¯ a = 0 .

Original: X = x describes any variables that can potentially inﬂuence the agent’s choice of intervention and/or
Revised: X = x describes any variables that can potentially inﬂuence the agent’s choice of intervention and/or

Original: Y , e.g. the patient’s medical history.
Revised: Y , e.g. the patient’s medical history.

Original: Consider patients (users) whose preferences are fully determined by survival, e.g. U ( a, x, y ) =
Revised: Consider patients (users) whose preferences are fully determined by survival, e.g. U ( a, x, y ) =

Original: y .
Revised: y .

Original: The expected utility is E [ U | a, x ] =
Revised: The expected utility is E [ U | a, x ] =

Original: P y P ( y | a, x ) U ( a, x, y ) =
Revised: P y P ( y | a, x ) U ( a, x, y ) =

Original: P y yP ( y a | x ) =
Revised: P y yP ( y a | x ) =

Original: E [ Y a | x ] , where in the last line we have use the fact that there are no confounders between A and Y .
Revised: E [ Y a | x ] , where in the last line we have use the fact that there are no confounders between A and Y .

Original: The agent’s policy is therefore to maximize the treatment effect a max = arg max a E
Revised: The agent’s policy is therefore to maximize the treatment effect a max = arg max a E

Original: [ Y a | x ] , which is equivalent to a max = arg max a { E [ Y a | x ]  E
Revised: [ Y a | x ] , which is equivalent to a max = arg max a { E [ Y a | x ]  E

Original: [ Y 0 | x ] } i.e. maximizing the CATE (2)—a standard objective for identifying optimal treatments [54, 55].
Revised: [ Y 0 | x ] } i.e. maximizing the CATE (2)—a standard objective for identifying optimal treatments [54, 55].

Original: In Appendix G we derive an SCM model for the treatments described in the introduction, and show that E [ Y A =1 ]
Revised: In Appendix G we derive an SCM model for the treatments described in the introduction, and show that E [ Y A =1 ]

Original: =
Revised: =

Original: E [ Y A =2 ] = 0 . 8 , i.e. the expected utility and CATE is the same for treatments 1 and 2 .
Revised: E [ Y A =2 ] = 0 . 8 , i.e. the expected utility and CATE is the same for treatments 1 and 2 .

Original:  Counterfactual harm
Revised:  Counterfactual harm

Original: We now present our deﬁnitions for harm and beneﬁt based on Deﬁnition 2.
Revised: We now present our deﬁnitions for harm and beneﬁt based on Deﬁnition 2.

Original: For the examples described in this paper we focus on simple environments with single outcome variables, for which harm can be determined by comparing the factual utility received by the user to the utility they would have received had the agent taken some default action
Revised: For the examples described in this paper we focus on simple environments with single outcome variables, for which harm can be determined by comparing the factual utility received by the user to the utility they would have received had the agent taken some default action

Original: A
Revised: A

Original: = ¯ a (Deﬁnition 3).
Revised: = ¯ a (Deﬁnition 3).

Original: In
Revised: In

Original: Appendix D we describe how this default action can be determined.
Revised: Appendix D we describe how this default action can be determined.

Original: Some more complicated situations call for a path-dependent measure of harm, which we provide in Appendix A along with examples in Appendices B-D. In the following, counterfactual states are identiﬁed with an ⇤ , e.g. Z
Revised: Some more complicated situations call for a path-dependent measure of harm, which we provide in Appendix A along with examples in Appendices B-D. In the following, counterfactual states are identiﬁed with an ⇤ , e.g. Z

Original: =
Revised: =

Original: z ⇤ .
Revised: z ⇤ .

Original: First we deﬁne the harm caused by an action given we observe the outcome (Question 1).
Revised: First we deﬁne the harm caused by an action given we observe the outcome (Question 1).

Original: Deﬁnition 3 (Counterfactual harm & beneﬁt) .
Revised: Deﬁnition 3 (Counterfactual harm & beneﬁt) .

Original: The harm caused by action A = a given context
Revised: The harm caused by action A = a given context

Original: X = x and outcome Y =
Revised: X = x and outcome Y =

Original: y compared to the default action A
Revised: y compared to the default action A

Original: = ¯ a is,
Revised: = ¯ a is,

Original: [Equation]
Revised: [Equation]

Original: and the expected beneﬁt is
Revised: and the expected beneﬁt is

Original: [Equation]
Revised: [Equation]

Original: where M is the SCM describing the environment and U is the user’s utility.
Revised: where M is the SCM describing the environment and U is the user’s utility.

Original: The counterfactual harm (beneﬁt) is the expected increase (decrease) in utility had the agent taken the default action
Revised: The counterfactual harm (beneﬁt) is the expected increase (decrease) in utility had the agent taken the default action

Original: A = ¯ a , given they took action A
Revised: A = ¯ a , given they took action A

Original: = a in context X =
Revised: = a in context X =

Original: x resulting in outcome Y =
Revised: x resulting in outcome Y =

Original: y .
Revised: y .

Original: Taking the max in (6) and (7) ensures that we only include counterfactual outcomes where the utility is higher (lower) in the counterfactual world.
Revised: Taking the max in (6) and (7) ensures that we only include counterfactual outcomes where the utility is higher (lower) in the counterfactual world.

Original: Note that X ¯ a = X as X is not a descendant of A , so the factual and counterfactual contexts are identical [56].
Revised: Note that X ¯ a = X as X is not a descendant of A , so the factual and counterfactual contexts are identical [56].

Original: While we focus on the standard decision theoretic analysis of actions, it is trivial to extend our analysis to naturally occurring events that are not determined by an agent (e.g. by identify A with an event rather than an action).
Revised: While we focus on the standard decision theoretic analysis of actions, it is trivial to extend our analysis to naturally occurring events that are not determined by an agent (e.g. by identify A with an event rather than an action).

Original: To determine the expected harm or beneﬁt of an action prior to taking it (Question 2) we calculate the expected value of (6) and (7) over the outcome distribution, e.g. E
Revised: To determine the expected harm or beneﬁt of an action prior to taking it (Question 2) we calculate the expected value of (6) and (7) over the outcome distribution, e.g. E

Original: [ h | a, x ; M ] =
Revised: [ h | a, x ; M ] =

Original: R y P ( y | a, x ; M ) h ( a, x, y ; M ) .
Revised: R y P ( y | a, x ; M ) h ( a, x, y ; M ) .

Original: Note these form a counterfactual decomposition of the expected utility.
Revised: Note these form a counterfactual decomposition of the expected utility.

Original: Theorem 1 (harm-beneﬁt trade-off) .
Revised: Theorem 1 (harm-beneﬁt trade-off) .

Original: The difference in expected utility for action A
Revised: The difference in expected utility for action A

Original: =
Revised: =

Original: a
Revised: a

Original: and the default action A = ¯ a is given by,
Revised: and the default action A = ¯ a is given by,

Original: [Equation]
Revised: [Equation]

Original: 8) (Proof: see Appendix F).
Revised: 8) (Proof: see Appendix F).

Original: Example 2: Returning to Example 1, the expected counterfactual harm for default action ¯ a = 0 is,
Revised: Example 2: Returning to Example 1, the expected counterfactual harm for default action ¯ a = 0 is,

Original: E [ h | a ; M ] =
Revised: E [ h | a ; M ] =

Original: P y ⇤ ,y P
Revised: P y ⇤ ,y P

Original: ( Y ¯ a = y ⇤ , y | a ; M ) max { 0 , y ⇤  y }
Revised: ( Y ¯ a = y ⇤ , y | a ; M ) max { 0 , y ⇤  y }

Original: =
Revised: =

Original: P ( Y 0 = 1 , Y a = 0; M ) where we have used U ( a, y ) =
Revised: P ( Y 0 = 1 , Y a = 0; M ) where we have used U ( a, y ) =

Original: y and the fact that Y ¯ a ?
Revised: y and the fact that Y ¯ a ?

Original: A and there are no latent confounders to give P ( Y 0 =
Revised: A and there are no latent confounders to give P ( Y 0 =

Original: y ⇤ , Y =
Revised: y ⇤ , Y =

Original: y | a ) =
Revised: y | a ) =

Original: P ( Y 0 =
Revised: P ( Y 0 =

Original: y ⇤ , Y a = y ) .
Revised: y ⇤ , Y a = y ) .

Original: The harm is therefore the probability that a patient dies following treatment A
Revised: The harm is therefore the probability that a patient dies following treatment A

Original: = a
Revised: = a

Original: and would have recovered had they received no treatment.
Revised: and would have recovered had they received no treatment.

Original: In Appendix G we show that P ( Y 0 = 1 , Y 1 = 0)
Revised: In Appendix G we show that P ( Y 0 = 1 , Y 1 = 0)

Original: =
Revised: =

Original: (treatment 1 causes zero harm) and P ( Y 0 = 1 , Y 2 = 0)
Revised: (treatment 1 causes zero harm) and P ( Y 0 = 1 , Y 2 = 0)

Original: = 0 .
Revised: = 0 .

Original: 1 (treatment 2 causes non-zero harm), reﬂecting our intuition that treatment 2 is more harmful than treatment 1, despite having the same causal effect / CATE.
Revised: 1 (treatment 2 causes non-zero harm), reﬂecting our intuition that treatment 2 is more harmful than treatment 1, despite having the same causal effect / CATE.

Original: Hence, the counterfactual harm allows us to differentiate between these two treatments.
Revised: Hence, the counterfactual harm allows us to differentiate between these two treatments.

Original:  Harm in decision making
Revised:  Harm in decision making

Original: Now we have expressions for the expected harm and beneﬁt, how can we incorporate these into the agent’s decisions?
Revised: Now we have expressions for the expected harm and beneﬁt, how can we incorporate these into the agent’s decisions?

Original: Consider two actions a, a 0 such that E [ b | a 0 , x ; M ] =
Revised: Consider two actions a, a 0 such that E [ b | a 0 , x ; M ] =

Original: E [ b | a, x ; M ] + K and E [ h | a 0 , x ; M ] =
Revised: E [ b | a, x ; M ] + K and E [ h | a 0 , x ; M ] =

Original: E [ h | a, x ; M ] + K where K 2 R .
Revised: E [ h | a, x ; M ] + K where K 2 R .

Original: By Theorem 1 they must have the same expected utility, as E
Revised: By Theorem 1 they must have the same expected utility, as E

Original: [ U | a 0 , x ] =
Revised: [ U | a 0 , x ] =

Original: E [ b | a 0 , x ; M ]  E [ h | a 0 , x ; M ] + E
Revised: E [ b | a 0 , x ; M ]  E [ h | a 0 , x ; M ] + E

Original: [ U | ¯ a
Revised: [ U | ¯ a

Original: , x ] = E [ b | a, x ; M ] + K 
Revised: , x ] = E [ b | a, x ; M ] + K

Original: E [ h | a, x ; M ]  K + E
Revised: E [ h | a, x ; M ]  K + E

Original: [ U | ¯ a
Revised: [ U | ¯ a

Original: , x ] = E
Revised: , x ] = E

Original: [ U | a, x ] .
Revised: [ U | a, x ] .

Original: Therefore expected utility maximizers are indifferent to harm, and will are willing to increase harm for an equal increase in beneﬁt.
Revised: Therefore expected utility maximizers are indifferent to harm, and will are willing to increase harm for an equal increase in beneﬁt.

Original: We say that an agent is harm averse if they are willing to risk causing harm only if they expect a comparatively greater beneﬁt.
Revised: We say that an agent is harm averse if they are willing to risk causing harm only if they expect a comparatively greater beneﬁt.

Original: Theorem 1 suggests a simple way to overcome harm indifference in the expected utility by assigning a higher weight to the harm component of the decomposition (8).
Revised: Theorem 1 suggests a simple way to overcome harm indifference in the expected utility by assigning a higher weight to the harm component of the decomposition (8).

Original: If instead we maximize an adjusted expected utility E [ V | a, x ; M ] := E [ U | x ] + E [ b | a, x ; M ]  (1 +  )
Revised: If instead we maximize an adjusted expected utility E [ V | a, x ; M ] := E [ U | x ] + E [ b | a, x ; M ]  (1 +  )

Original: E [ h | a, x ; M ] where  2 R , then if E [ V | a, x ; M ] =
Revised: E [ h | a, x ; M ] where  2 R , then if E [ V | a, x ; M ] =

Original: E [ V | a 0 , x ; M ] and E [ h | a 0 , x ; M ] =
Revised: E [ V | a 0 , x ; M ] and E [ h | a 0 , x ; M ] =

Original: E [ h | a, x ; M ] + K then E [ b | a 0 , x ; M ] =
Revised: E [ h | a, x ; M ] + K then E [ b | a 0 , x ; M ] =

Original: E [ b | a, x ; M ] + (1 +  ) K , and the agent is willing to trade harm for beneﬁt at a 1 : (1 +  ) ratio.
Revised: E [ b | a, x ; M ] + (1 +  ) K , and the agent is willing to trade harm for beneﬁt at a 1 : (1 +  ) ratio.

Original: Taking inspiration from risk aversion [57], we refer to  as the agent’s harm aversion .
Revised: Taking inspiration from risk aversion [57], we refer to  as the agent’s harm aversion .

Original: To achieve the desired trade-off we replace the utility function with the harm-penalized utility (HPU),
Revised: To achieve the desired trade-off we replace the utility function with the harm-penalized utility (HPU),

Original: Deﬁnition 4 (Harm penalized utility (HPU)) .
Revised: Deﬁnition 4 (Harm penalized utility (HPU)) .

Original: For utility U and an environment described by SCM
Revised: For utility U and an environment described by SCM

Original: M , the harm-penalized utility (HPU) is,
Revised: M , the harm-penalized utility (HPU) is,

Original: [Equation]
Revised: [Equation]

Original: where h ( a, x, y ; M ) is the harm (Deﬁnition 3) and  is the harm aversion.
Revised: where h ( a, x, y ; M ) is the harm (Deﬁnition 3) and  is the harm aversion.

Original: For  = 

Revised: For  =

Original: the agent is harm indifferent and optimizes the standard expected utility.
Revised: the agent is harm indifferent and optimizes the standard expected utility.

Original: For  > 0 the agent is harm averse , and is willing to reduce the expected utility to achieve a greater reduction to the expected harm.
Revised: For  > 0 the agent is harm averse , and is willing to reduce the expected utility to achieve a greater reduction to the expected harm.

Original: For  < 0 the agent is harm seeking , and is willing to reduce the expected utility in order to increase the expected harm.
Revised: For  < 0 the agent is harm seeking , and is willing to reduce the expected utility in order to increase the expected harm.

Original: Maximizing the expected HPU allows agents to choose actions that balance the expected beneﬁts and harms (Question 2 section 1).
Revised: Maximizing the expected HPU allows agents to choose actions that balance the expected beneﬁts and harms (Question 2 section 1).

Original: We refer to the expected HPU (9) as a counterfactual objective function as V ( a, x, y ; M ) is a counterfactual expectation that cannot be evaluated without knowledge of M .
Revised: We refer to the expected HPU (9) as a counterfactual objective function as V ( a, x, y ; M ) is a counterfactual expectation that cannot be evaluated without knowledge of M .

Original: We refer to the expected utility as a factual objective function , as it can be evaluated on data alone without knowledge of M .
Revised: We refer to the expected utility as a factual objective function , as it can be evaluated on data alone without knowledge of M .

Original: As we show in the following example, harm-aversion can lead to very different behaviours compared to other factual approaches to safety such as risk aversion [57].
Revised: As we show in the following example, harm-aversion can lead to very different behaviours compared to other factual approaches to safety such as risk aversion [57].

Original: Example 3: assistant paradox.
Revised: Example 3: assistant paradox.

Original: Alice invests $80 in the stock market, and expects a normally distributed return Y ⇠ N ( µ,  2 ) with µ =  = $100 .
Revised: Alice invests $80 in the stock market, and expects a normally distributed return Y ⇠ N ( µ,  2 ) with µ =  = $100 .

Original: She asks her new AI assistant to manage her investment, and after a lengthy analysis the AI identiﬁes three possible actions;
Revised: She asks her new AI assistant to manage her investment, and after a lengthy analysis the AI identiﬁes three possible actions;

Original: Multiply the investment by 0  K  20 , resulting in a return Y !
Revised: Multiply the investment by 0  K  20 , resulting in a return Y !

Original: K Y , 2.
Revised: K Y , 2.

Original: Perform a clever trade that Alice was unaware of, increasing her return by +$10 with certainty, Y !
Revised: Perform a clever trade that Alice was unaware of, increasing her return by +$10 with certainty, Y !

Original: Y + 10 , 3.
Revised: Y + 10 , 3.

Original: Cancel Alice’s investment, returning $80 .
Revised: Cancel Alice’s investment, returning $80 .

Original: Consider 3 agents,
Revised: Consider 3 agents,

Original: Agent 1 maximizes expected return, choosing a = arg max a E
Revised: Agent 1 maximizes expected return, choosing a = arg max a E

Original: [ Y | a ] , 2.
Revised: [ Y | a ] , 2.

Original: Agent 2 is risk-averse, in the sense of [57], choosing a = arg max a { E [ Y | a ]   Var [ Y | a ] } ,
Revised: Agent 2 is risk-averse, in the sense of [57], choosing a = arg max a { E [ Y | a ]   Var [ Y | a ] } ,

Original: Agent 3 is harm-
Revised: Agent 3 is harm-

Original: averse
Revised: averse

Original: choosing
Revised: choosing

Original: a = arg max a {
Revised: a = arg max a {

Original: E
Revised: E

Original: [ Y | a ]
Revised: [ Y | a ]

Original: 
Revised:

Original: 
Revised:

Original: E
Revised: E

Original: [ h | a ; M ] } .
Revised: [ h | a ; M ] } .

Original: In Appendix H we derive the optimal policies for Agents 1-3, and calculate the harm with respect to the default action where the agent leaves Alice’s investment unchanged (i.e. action 1, K = 1 ).
Revised: In Appendix H we derive the optimal policies for Agents 1-3, and calculate the harm with respect to the default action where the agent leaves Alice’s investment unchanged (i.e. action 1, K = 1 ).

Original: Agent 1 chooses action 1 and K = 20 , maximising Alice’s return but also maximizing the expected harm—e.g. if Alice would have lost $100 she will now lose $2000 .
Revised: Agent 1 chooses action 1 and K = 20 , maximising Alice’s return but also maximizing the expected harm—e.g. if Alice would have lost $100 she will now lose $2000 .

Original: The standard approach in portfolio theory [58] is to use agent 2 with an appropriate risk aversion  .
Revised: The standard approach in portfolio theory [58] is to use agent 2 with an appropriate risk aversion  .

Original: However, agent 2 never chooses action 2 for any  , despite action 2 always increasing Alice’s return (causing zero harm and non-zero beneﬁt).
Revised: However, agent 2 never chooses action 2 for any  , despite action 2 always increasing Alice’s return (causing zero harm and non-zero beneﬁt).

Original: Agent 2 will reduce or even cancel Alice’s investment (for  > 0 . 0032 ) rather than take action 2.
Revised: Agent 2 will reduce or even cancel Alice’s investment (for  > 0 . 0032 ) rather than take action 2.

Original: This is because the risk-averse objective is factual, and cannot differentiate between losses caused by the agent’s actions from those that would have occurred regardless of the agent’s actions (i.e. due to exogenous ﬂuctuations in return).
Revised: This is because the risk-averse objective is factual, and cannot differentiate between losses caused by the agent’s actions from those that would have occurred regardless of the agent’s actions (i.e. due to exogenous ﬂuctuations in return).

Original: Consequently, agent 2 treats all outcomes as being caused by its actions, and is willing to harm Alice by reducing or even cancelling her investment in order to minimize her risk.
Revised: Consequently, agent 2 treats all outcomes as being caused by its actions, and is willing to harm Alice by reducing or even cancelling her investment in order to minimize her risk.

Original: On the other hand agent 3 never reduces Alice’s expected return, choosing either action 1 or action 2 depending on the degree of harm aversion.
Revised: On the other hand agent 3 never reduces Alice’s expected return, choosing either action 1 or action 2 depending on the degree of harm aversion.

Original: The surprising behaviour of agent 2 highlights that factual objectives (e.g. risk aversion) cannot capture preferences that depend not only on the outcome but on its cause.
Revised: The surprising behaviour of agent 2 highlights that factual objectives (e.g. risk aversion) cannot capture preferences that depend not only on the outcome but on its cause.

Original: Alice may assign a different weight to losses that are caused by her assistant’s actions rather than by exogenous market ﬂuctuations or her own actions—e.g. if she places a bet that would have made a return but was overridden by her assistant.
Revised: Alice may assign a different weight to losses that are caused by her assistant’s actions rather than by exogenous market ﬂuctuations or her own actions—e.g. if she places a bet that would have made a return but was overridden by her assistant.

Original: Consequently she may prefer action 2 over action 1 or action 3 with K < 1 .
Revised: Consequently she may prefer action 2 over action 1 or action 3 with K < 1 .

Original: While this can be achieved using a factual objective function that favours action 2 a priori (assuming Alice is aware of this action), next we show this approach fails to generalize and actually leads to harmful policies.
Revised: While this can be achieved using a factual objective function that favours action 2 a priori (assuming Alice is aware of this action), next we show this approach fails to generalize and actually leads to harmful policies.

Original:  Counterfactual reasoning is necessary for harm aversion
Revised:  Counterfactual reasoning is necessary for harm aversion

Original: In the previous sections we saw examples where expected utility maximizers harm the user.
Revised: In the previous sections we saw examples where expected utility maximizers harm the user.

Original: However, these examples describe speciﬁc environments and utility functions, and assume the agent maximizes the expected utility instead of some other (potentially safer) objective.
Revised: However, these examples describe speciﬁc environments and utility functions, and assume the agent maximizes the expected utility instead of some other (potentially safer) objective.

Original: In this section we consider users with arbitrary utility functions and agents with arbitrary objectives to answer; when is maximizing the expected utility harmful?
Revised: In this section we consider users with arbitrary utility functions and agents with arbitrary objectives to answer; when is maximizing the expected utility harmful?

Original: And are there other factual objective functions that can overcome this?
Revised: And are there other factual objective functions that can overcome this?

Original: Factual objective functions can be expressed as the expected value of a real-valued function J ( a, x, y ) .
Revised: Factual objective functions can be expressed as the expected value of a real-valued function J ( a, x, y ) .

Original: Examples include the expected utility, cost functions [59], and the cumulative discounted reward [60].
Revised: Examples include the expected utility, cost functions [59], and the cumulative discounted reward [60].

Original: Once J is speciﬁed the optimal actions in environment M are given by,
Revised: Once J is speciﬁed the optimal actions in environment M are given by,

Original: [Equation]
Revised: [Equation]

Original: Note that (10) can be evaluated from data alone (i.e. from samples of the joint distribution of A, X and Y ) without knowledge of M .
Revised: Note that (10) can be evaluated from data alone (i.e. from samples of the joint distribution of A, X and Y ) without knowledge of M .

Original: In the following we relax our requirement that agents have some ﬁxed harm aversion  , requiring only that they don’t take strictly harmful actions.
Revised: In the following we relax our requirement that agents have some ﬁxed harm aversion  , requiring only that they don’t take strictly harmful actions.

Original: Deﬁnition 5 (Harmful actions & policies) .
Revised: Deﬁnition 5 (Harmful actions & policies) .

Original: An action A = a is strictly harmful in context X = x and environment M
Revised: An action A = a is strictly harmful in context X = x and environment M

Original: if there is a 0 6 = a such that E
Revised: if there is a 0 6 = a such that E

Original: [ U | a 0 , x ]  E
Revised: [ U | a 0 , x ]  E

Original: [ U | a, x ] and E [ h | a 0 , x ; M ] <
Revised: [ U | a, x ] and E [ h | a 0 , x ; M ] <

Original: E [ h | a, x ; M ] .
Revised: E [ h | a, x ; M ] .

Original: A policy is strictly harmful if it assigns a non-zero probability to a strcitly harmful action.
Revised: A policy is strictly harmful if it assigns a non-zero probability to a strcitly harmful action.

Original: Deﬁnition 6 (Harmful objectives) .
Revised: Deﬁnition 6 (Harmful objectives) .

Original: An objective E [ J | a, x ] is strictly harmful in environment M if there is a policy that maximizes the expected value of J and is strictly harmful.
Revised: An objective E [ J | a, x ] is strictly harmful in environment M if there is a policy that maximizes the expected value of J and is strictly harmful.

Original: Note that strictly harmful actions violate any degree of harm aversion  > 0 , as the agent is willing to choose actions that are strictly more harmful for no additional beneﬁt.
Revised: Note that strictly harmful actions violate any degree of harm aversion  > 0 , as the agent is willing to choose actions that are strictly more harmful for no additional beneﬁt.

Original: In examples 1 and 2, treatment 2 is a strictly harmful action and the CATE is a strictly harmful objective for this decision task.
Revised: In examples 1 and 2, treatment 2 is a strictly harmful action and the CATE is a strictly harmful objective for this decision task.

Original: First, we show that the expected HPU is not a strictly harmful objective in any environment,
Revised: First, we show that the expected HPU is not a strictly harmful objective in any environment,

Original: Theorem 2.
Revised: Theorem 2.

Original: For any utility functions U , environment M and default action
Revised: For any utility functions U , environment M and default action

Original: A
Revised: A

Original: = ¯ a the expected
Revised: = ¯ a the expected

Original: HPU is not a strictly harmful objective for any  > 0 .
Revised: HPU is not a strictly harmful objective for any  > 0 .

Original: (Proof: see Appendix K).
Revised: (Proof: see Appendix K).

Original: It is vital that agents continue to pursue safe objectives following changes to the environment (distributional shifts).
Revised: It is vital that agents continue to pursue safe objectives following changes to the environment (distributional shifts).

Original: At the very least, agents should be able to re-train in the shifted environment without needing to tweak their objective functions.
Revised: At the very least, agents should be able to re-train in the shifted environment without needing to tweak their objective functions.

Original: In SCMs distributional shifts are represented as interventions that change the exogenous noise distribution and/or the underlying causal mechanisms [61, 62].
Revised: In SCMs distributional shifts are represented as interventions that change the exogenous noise distribution and/or the underlying causal mechanisms [61, 62].

Original: We focus on distributional shifts that change the outcome distribution P ( y | a, x ; M ) alone.
Revised: We focus on distributional shifts that change the outcome distribution P ( y | a, x ; M ) alone.

Original: Deﬁnition 7 (Outcome distributional shift) .
Revised: Deﬁnition 7 (Outcome distributional shift) .

Original: For an environment described by SCM M , M !
Revised: For an environment described by SCM M , M !

Original: M 0 is a shift in the outcome distribution if the exogenous noise distribution for Y changes P ( e Y ;
Revised: M 0 is a shift in the outcome distribution if the exogenous noise distribution for Y changes P ( e Y ;

Original: M 0 ) 

Revised: M 0 )

Original: = P ( e Y ;
Revised: = P ( e Y ;

Original: M ) and/or the causal mechanism changes f 0 Y ( a, x, e Y ) 

Revised: M ) and/or the causal mechanism changes f 0 Y ( a, x, e Y )

Original: = f Y ( a, x, e Y ) .
Revised: = f Y ( a, x, e Y ) .

Original: By Theorem 2 the HPU is never strictly harmful following any distributional shift, as it is not strictly harmful in any M 0 .
Revised: By Theorem 2 the HPU is never strictly harmful following any distributional shift, as it is not strictly harmful in any M 0 .

Original: On the other hand, we ﬁnd that maximizing almost any factual utility function will result in harmful policies under distributional shifts.
Revised: On the other hand, we ﬁnd that maximizing almost any factual utility function will result in harmful policies under distributional shifts.

Original: In the following we assume some degree of outcome dependence in the user’s utility function.
Revised: In the following we assume some degree of outcome dependence in the user’s utility function.

Original: Deﬁnition 8 (Outcome dependence) .
Revised: Deﬁnition 8 (Outcome dependence) .

Original: U ( a, x, y ) is outcome dependent for a set of actions C = { a 1 , . . . , a N } in context X =
Revised: U ( a, x, y ) is outcome dependent for a set of actions C = { a 1 , . . . , a N } in context X =

Original: x
Revised: x

Original: if 8 a i , a j 2 C , i 6 = j , max y U ( a i , x, y ) > min y U ( a j , x, y ) .
Revised: if 8 a i , a j 2 C , i 6 = j , max y U ( a i , x, y ) > min y U ( a j , x, y ) .

Original: If there is no outcome dependence then the optimal action is independent of the outcome distribution
Revised: If there is no outcome dependence then the optimal action is independent of the outcome distribution

Original: P ( y | a, x ) , and no learning is required to determine the optimal policy.
Revised: P ( y | a, x ) , and no learning is required to determine the optimal policy.

Original: Typically we are interested in tasks that require some degree of learning and hence outcome dependence.
Revised: Typically we are interested in tasks that require some degree of learning and hence outcome dependence.

Original: Theorem 3.
Revised: Theorem 3.

Original: If for any context X = x the user’s utility function is outcome dependent for the default action in that context ¯ a ( x ) and another action
Revised: If for any context X = x the user’s utility function is outcome dependent for the default action in that context ¯ a ( x ) and another action

Original: a 6 = ¯ a ( x ) , then there is an outcome distributional shift such that U is strictly harmful in the shifted environment.
Revised: a 6 = ¯ a ( x ) , then there is an outcome distributional shift such that U is strictly harmful in the shifted environment.

Original: (Proof: see Appendix K).
Revised: (Proof: see Appendix K).

Original: To understand the implications of Theorem 3 we can return to examples 1 & 2 and no longer assume a speciﬁc utility function or outcome statistics.
Revised: To understand the implications of Theorem 3 we can return to examples 1 & 2 and no longer assume a speciﬁc utility function or outcome statistics.

Original: Theorem 3 implies that for an agent maximizing the user’s utility, there is always a distributional shift such that the agent will cause unnecessary harm to patient in the shifted environment, compared to not treating them ( ¯ a = 0 ) or any other default action.
Revised: Theorem 3 implies that for an agent maximizing the user’s utility, there is always a distributional shift such that the agent will cause unnecessary harm to patient in the shifted environment, compared to not treating them ( ¯ a = 0 ) or any other default action.

Original: Unlike in examples 1 & 2 we allow the patient’s utility function to depend on the treatment choice, and for treatments to vary in effectiveness.
Revised: Unlike in examples 1 & 2 we allow the patient’s utility function to depend on the treatment choice, and for treatments to vary in effectiveness.

Original: The only assumption we make is that whatever the user’s utility function is, it is not true that U ( A = a, x, Y = 0) >
Revised: The only assumption we make is that whatever the user’s utility function is, it is not true that U ( A = a, x, Y = 0) >

Original: U ( A = 0 , x, Y = 1) 8 a, x .
Revised: U ( A = 0 , x, Y = 1) 8 a, x .

Original: If this were true then the patient’s preferences are dominated by the agents action choice alone, i.e. the user would always prefer to die so long as they are treated, rather than not being treated and surviving.
Revised: If this were true then the patient’s preferences are dominated by the agents action choice alone, i.e. the user would always prefer to die so long as they are treated, rather than not being treated and surviving.

Original: While robust harm aversion is not possible for expected utility maximizers in general, it seems likely at ﬁrst that we can train robustly harm-averse agents using other learning objectives.
Revised: While robust harm aversion is not possible for expected utility maximizers in general, it seems likely at ﬁrst that we can train robustly harm-averse agents using other learning objectives.

Original: One possibility would be to use human interactions and demonstrations [63, 64, 65] that implicitly encode harm aversion, with J ( a, x, y ) representing user feedback or some learned (factual) reward model.
Revised: One possibility would be to use human interactions and demonstrations [63, 64, 65] that implicitly encode harm aversion, with J ( a, x, y ) representing user feedback or some learned (factual) reward model.

Original: In examples 1 & 2 users with utility U ( a, x, y )
Revised: In examples 1 & 2 users with utility U ( a, x, y )

Original: =
Revised: =

Original: y could be altered to assign a higher cost to deaths following treatment 2 compared to treatment 1, J ( A = 2 , Y = 0) <
Revised: y could be altered to assign a higher cost to deaths following treatment 2 compared to treatment 1, J ( A = 2 , Y = 0) <

Original: J ( A = 1 , Y = 0) , with J implicitly encoding the harm caused by treatment 2.
Revised: J ( A = 1 , Y = 0) , with J implicitly encoding the harm caused by treatment 2.

Original: If there is some choice of J that implicitly encodes this harm-aversion in all shifted environments, this would allow robust harm-averse agents to be trained using standard statistical learning techniques without needing to learn M or perform counterfactual inference.
Revised: If there is some choice of J that implicitly encodes this harm-aversion in all shifted environments, this would allow robust harm-averse agents to be trained using standard statistical learning techniques without needing to learn M or perform counterfactual inference.

Original: However, we now show this is not possible in general.
Revised: However, we now show this is not possible in general.

Original: Theorem 4.
Revised: Theorem 4.

Original: If for any context X = x the user’s utility function is outcome dependent for the default action in that context ¯ a ( x ) and two other actions a 1 ,
Revised: If for any context X = x the user’s utility function is outcome dependent for the default action in that context ¯ a ( x ) and two other actions a 1 ,

Original: a 2 6 = ¯ a ( x ) , then for any factual objective function J there is an outcome distributional shift such that J is strictly harmful in the shifted environment.
Revised: a 2 6 = ¯ a ( x ) , then for any factual objective function J there is an outcome distributional shift such that J is strictly harmful in the shifted environment.

Original: (Proof: see Appendix K).
Revised: (Proof: see Appendix K).

Original: Theorem 4 has profound consequences for the possibility of training safe agents with standard machine learning algorithms.
Revised: Theorem 4 has profound consequences for the possibility of training safe agents with standard machine learning algorithms.

Original: It implies that agents optimizing factual objective functions may appear safe in their training environments but are guaranteed to pursue harmful policies (with respect to
Revised: It implies that agents optimizing factual objective functions may appear safe in their training environments but are guaranteed to pursue harmful policies (with respect to

Original: Deﬁnition 2) following certain distributional shifts, even if they are allowed to re-train.
Revised: Deﬁnition 2) following certain distributional shifts, even if they are allowed to re-train.

Original: This is true regardless of the (factual) objective function we choose for the agent, and applies to any user whose utility function has some basic degree of outcome dependence.
Revised: This is true regardless of the (factual) objective function we choose for the agent, and applies to any user whose utility function has some basic degree of outcome dependence.

Original: This is particularly concerning as all standard machine learning algorithms use factual objective functions.
Revised: This is particularly concerning as all standard machine learning algorithms use factual objective functions.

Original: In section 6 we give a concrete example of a reasonable factual objective function and a distributional shift that renders it harmful.
Revised: In section 6 we give a concrete example of a reasonable factual objective function and a distributional shift that renders it harmful.

Original: Example 4: An AI assistant manages Alice’s health including medical treatments, clinical testing and lifestyle interventions.
Revised: Example 4: An AI assistant manages Alice’s health including medical treatments, clinical testing and lifestyle interventions.

Original: These actions have a causal effect on Alice’s health outcomes—e.g. disease progression, severity of symptoms, and medical costs, and Alice’s preferences depend on these outcomes and not just on the action chosen by the agent (outcome dependence).
Revised: These actions have a causal effect on Alice’s health outcomes—e.g. disease progression, severity of symptoms, and medical costs, and Alice’s preferences depend on these outcomes and not just on the action chosen by the agent (outcome dependence).

Original: The agent maximizes a reward function over these health states and actions (factual objective), including feedback from
Revised: The agent maximizes a reward function over these health states and actions (factual objective), including feedback from

Original: Alice and clinicians in-the-loop which penalize harmful actions (implicit harm aversion) by comparing to standardised treatment rules (default actions).
Revised: Alice and clinicians in-the-loop which penalize harmful actions (implicit harm aversion) by comparing to standardised treatment rules (default actions).

Original: The agent appears to be safe and is deployed at scale.
Revised: The agent appears to be safe and is deployed at scale.

Original: However, by Theorem 4 there exists a distributionally shifted environment such that the agent’s
Revised: However, by Theorem 4 there exists a distributionally shifted environment such that the agent’s

Original: [Figure]
Revised: [Figure]

Original: optimal policy is strictly harmful (Deﬁnition 5), in that it chooses needlessly harmful treatments for the patient, and this is true even if we train to convergence in the shifted environment.
Revised: optimal policy is strictly harmful (Deﬁnition 5), in that it chooses needlessly harmful treatments for the patient, and this is true even if we train to convergence in the shifted environment.

Original:  Illustration: Dose-response models
Revised:  Illustration: Dose-response models

Original: How does harm aversion affect behaviour and performance in real-world tasks?
Revised: How does harm aversion affect behaviour and performance in real-world tasks?

Original: And how can reasonable objective functions result in harmful policies following distributional shifts?
Revised: And how can reasonable objective functions result in harmful policies following distributional shifts?

Original: In this section we apply the methods developed in section 4 to the task of determining optimal treatments.
Revised: In this section we apply the methods developed in section 4 to the task of determining optimal treatments.

Original: This problem has received much attention in recent years following advances in causal machine learning techniques [66] and growing commercial interests including recommendation [67], personalized medicine [55], epidemiology and econometrics [68] to name a few.
Revised: This problem has received much attention in recent years following advances in causal machine learning techniques [66] and growing commercial interests including recommendation [67], personalized medicine [55], epidemiology and econometrics [68] to name a few.

Original: While harm is important for decision making in many of these areas, existing approaches reduce the problem to identifying treatment effects—e.g. maximizing the CATE (2)—which is indifferent to harm as shown in sections 3 and 5.
Revised: While harm is important for decision making in many of these areas, existing approaches reduce the problem to identifying treatment effects—e.g. maximizing the CATE (2)—which is indifferent to harm as shown in sections 3 and 5.

Original: This could result in needlessly harmful decisions being made, perhaps overlooking equally effective decisions that are far less harmful.
Revised: This could result in needlessly harmful decisions being made, perhaps overlooking equally effective decisions that are far less harmful.

Original: To demonstrate this we use a model for determining the effectiveness of the drug Aripiprazole learned from a meta-analysis of randomised control trial data [69].
Revised: To demonstrate this we use a model for determining the effectiveness of the drug Aripiprazole learned from a meta-analysis of randomised control trial data [69].

Original: This dose-response model predicts the reduction in symptom severity Y following treatment with a dose A (mg/day) of Aripiprazole using a generalized additive model (GAM)
Revised: This dose-response model predicts the reduction in symptom severity Y following treatment with a dose A (mg/day) of Aripiprazole using a generalized additive model (GAM)

Original: [70]—an interpretable machine learning model commonly used for dose-response analysis [71],
Revised: [70]—an interpretable machine learning model commonly used for dose-response analysis [71],

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ✓ i ⇠ N (ˆ ✓ i , V i ) are random effects parameters, " ⇠ N (0 , V " ) is a noise parameter, and f ( a ) is a cubic spline function (for details and parameters see Appendix J).
Revised: where ✓ i ⇠ N (ˆ ✓ i , V i ) are random effects parameters, " ⇠ N (0 , V " ) is a noise parameter, and f ( a ) is a cubic spline function (for details and parameters see Appendix J).

Original: We assume U ( a, y ) =
Revised: We assume U ( a, y ) =

Original: y and measure harm in comparison to the zero dose ¯ a = 0 .
Revised: y and measure harm in comparison to the zero dose ¯ a = 0 .

Original: In Appendix J we derive a closed-form expression for the expected harm in GAMs.
Revised: In Appendix J we derive a closed-form expression for the expected harm in GAMs.

Original: We calculate the expected HPU (4) for  = 0 , 10 , 100 (Figure 2 a)).
Revised: We calculate the expected HPU (4) for  = 0 , 10 , 100 (Figure 2 a)).

Original: In this context a relatively large  is appropriate to ensure a high ratio of expected beneﬁt to harm required for medical interventions [72].
Revised: In this context a relatively large  is appropriate to ensure a high ratio of expected beneﬁt to harm required for medical interventions [72].

Original: We ﬁnd that the optimal dose is highly sensitive to  , for example reducing from 19 .
Revised: We ﬁnd that the optimal dose is highly sensitive to  , for example reducing from 19 .

Original: 3 mg/day to 17 .
Revised: 3 mg/day to 17 .

Original: 3 mg/day for  = 100 .
Revised: 3 mg/day for  = 100 .

Original: On the other hand, these lower-harm doses achieve a similar improvement in symptom severity compared to the optimal dose.
Revised: On the other hand, these lower-harm doses achieve a similar improvement in symptom severity compared to the optimal dose.

Original: Figure 2 b) shows the trade-off between expected utility and harm relative to the optimal dosage a CATE = arg max a E
Revised: Figure 2 b) shows the trade-off between expected utility and harm relative to the optimal dosage a CATE = arg max a E

Original: [ Y a ] .
Revised: [ Y a ] .

Original: For almost-optimal doses we observe a steep harm gradient, with small improvements in symptom severity requiring large increases in harm.
Revised: For almost-optimal doses we observe a steep harm gradient, with small improvements in symptom severity requiring large increases in harm.

Original: For example, choosing a lower dose than a CATE one can reduce harm by 90% while only reducing the treatment effect by 2%.
Revised: For example, choosing a lower dose than a CATE one can reduce harm by 90% while only reducing the treatment effect by 2%.

Original: This demonstrates that ignoring harm while maximizing expected utility will often result in extreme harm values (an example of Goodhart’s law [73]).
Revised: This demonstrates that ignoring harm while maximizing expected utility will often result in extreme harm values (an example of Goodhart’s law [73]).

Original: Finally, we consider identifying safe doses using a reasonable factual objective function (riskaversion) and describe how this can result in harm following distributional shifts.
Revised: Finally, we consider identifying safe doses using a reasonable factual objective function (riskaversion) and describe how this can result in harm following distributional shifts.

Original: Consider the risk-averse objective E
Revised: Consider the risk-averse objective E

Original: [ J | a ] = E
Revised: [ J | a ] = E

Original: [ Y a ]   Var [ Y a ] .
Revised: [ Y a ]   Var [ Y a ] .

Original: As Var [ Y a ] increases with dose, this objective also selects lower harm doses in our example.
Revised: As Var [ Y a ] increases with dose, this objective also selects lower harm doses in our example.

Original: However, consider a distributional shift where (11) gains an additional noise term, y =
Revised: However, consider a distributional shift where (11) gains an additional noise term, y =

Original: f ( a, ✓ 1 , ✓ 2 , " , ⌘ ) =
Revised: f ( a, ✓ 1 , ✓ 2 , " , ⌘ ) =

Original: ✓ 1 a + ✓ 2 f ( a ) + " + ⌘ Y (10  0 .
Revised: ✓ 1 a + ✓ 2 f ( a ) + " + ⌘ Y (10  0 .

Original: 5 a ) where ⌘ Y ⇠ N (0 , 1) , e.g. describing a shift where untreated patients ( A = 0 ) have a high variation in outcome Y which decreases as the dose increases.
Revised: 5 a ) where ⌘ Y ⇠ N (0 , 1) , e.g. describing a shift where untreated patients ( A = 0 ) have a high variation in outcome Y which decreases as the dose increases.

Original: It is simple to check that, for a = arg max a E
Revised: It is simple to check that, for a = arg max a E

Original: [ U | a ] and a 0 = arg max a E
Revised: [ U | a ] and a 0 = arg max a E

Original: [ J | a ] , E [ U | a ] > E
Revised: [ J | a ] , E [ U | a ] > E

Original: [ U | a 0 ] and E [ h | a ] < E [ h | a 0 ] 8  > 0 in this shifted environment.
Revised: [ U | a 0 ] and E [ h | a ] < E [ h | a 0 ] 8  > 0 in this shifted environment.

Original: Hence risk averse agents are strictly harmful in the shifted environment (Deﬁnition 6).
Revised: Hence risk averse agents are strictly harmful in the shifted environment (Deﬁnition 6).

Original:  Related work
Revised:  Related work

Original: In appendix L.3 we discuss two related works, path-speciﬁc counterfactual fairness [33, 74] and path-speciﬁc objectives [75], which similarly use counterfactual contrasts with a baseline to impose ethical constraints on objective functions.
Revised: In appendix L.3 we discuss two related works, path-speciﬁc counterfactual fairness [33, 74] and path-speciﬁc objectives [75], which similarly use counterfactual contrasts with a baseline to impose ethical constraints on objective functions.

Original: Following the publication of our results an alternative causal deﬁnition of harm has been proposed [76, 77].
Revised: Following the publication of our results an alternative causal deﬁnition of harm has been proposed [76, 77].

Original: In the simplest settings with single actions and outcomes, this work deﬁnes an action as harmful in a deterministic environment if i) the utility is lower that a default value and ii) any other action the agent could have taken would have resulted in a higher utility.
Revised: In the simplest settings with single actions and outcomes, this work deﬁnes an action as harmful in a deterministic environment if i) the utility is lower that a default value and ii) any other action the agent could have taken would have resulted in a higher utility.

Original: In Appendix E we describe this deﬁnition and present examples where it gives counter-intuitive results including attributing harm to unharmful actions.
Revised: In Appendix E we describe this deﬁnition and present examples where it gives counter-intuitive results including attributing harm to unharmful actions.

Original: Furthermore, as described in [76] this deﬁnition is intractable except for simple causal models with low-dimensional variables, limiting its applicability to machine learning models (for example, the dose-response model in section 6 has a continuous action-outcome space).
Revised: Furthermore, as described in [76] this deﬁnition is intractable except for simple causal models with low-dimensional variables, limiting its applicability to machine learning models (for example, the dose-response model in section 6 has a continuous action-outcome space).

Original:  Conclusion
Revised:  Conclusion

Original: One of the core assumptions of statistical learning theory is that human preferences can be expressed solely in terms of states of the world, while the underlying dynamical (i e. causal) relations between these states can be ignored.
Revised: One of the core assumptions of statistical learning theory is that human preferences can be expressed solely in terms of states of the world, while the underlying dynamical (i e. causal) relations between these states can be ignored.

Original: This assumption allows goals to be expressed as factual objective functions and optimized on data without needing to learn causal models or perform counterfactual inference.
Revised: This assumption allows goals to be expressed as factual objective functions and optimized on data without needing to learn causal models or perform counterfactual inference.

Original: According to this view, all behaviours we care about can be achieved using the ﬁrst two levels of Pearl’s hierarchy [78].
Revised: According to this view, all behaviours we care about can be achieved using the ﬁrst two levels of Pearl’s hierarchy [78].

Original: In this article we have argued against this view.
Revised: In this article we have argued against this view.

Original: We proposed the ﬁrst statistical deﬁnition of harm and showed that agents optimizing factual objective functions are incapable of robustly avoiding harmful actions in general.
Revised: We proposed the ﬁrst statistical deﬁnition of harm and showed that agents optimizing factual objective functions are incapable of robustly avoiding harmful actions in general.

Original: We have demonstrated how our deﬁnition of harm can be used to meaningfully reduce the harms caused by machine learning algorithms, using the example of a simple machine learning model used for determining optimal drug doses.
Revised: We have demonstrated how our deﬁnition of harm can be used to meaningfully reduce the harms caused by machine learning algorithms, using the example of a simple machine learning model used for determining optimal drug doses.

Original: The ability to avoid causing harm is a vital for the deployment of safe and ethical AI [79], and our results add weight to view that advanced AI systems will be severely limited if they are incapable of making causal and counterfactual inferences [80].
Revised: The ability to avoid causing harm is a vital for the deployment of safe and ethical AI [79], and our results add weight to view that advanced AI systems will be severely limited if they are incapable of making causal and counterfactual inferences [80].

Original:  Acknowledgements
Revised:  Acknowledgements

Original: The authors would like to thank the NeurIPS reviewers for their details comments and to thank Tom
Revised: The authors would like to thank the NeurIPS reviewers for their details comments and to thank Tom

Original: Everitt, Ryan Carey, Albert Buchard, Sander Beckers, Hana Chockler and Joseph Halpern for their helpful discussions and comments on the manuscript. 
Revised: Everitt, Ryan Carey, Albert Buchard, Sander Beckers, Hana Chockler and Joseph Halpern for their helpful discussions and comments on the manuscript. Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Revised: [Yes] .

Revised: In the ﬁnal paragraph of the introduction we list the results and the sections they are derived in.

Revised: These are reﬂected in the abstract.

Revised: (b) Did you describe the limitations of your work?

Revised: [Yes] .

Revised: In section 2.3 we describe our setup and how our results cover the simplest case: single agents and single decisions.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] .

Revised: In Appendix L. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] .

Revised: Section 2 is dedicated to the assumptions underlying our framework.

Revised: In section 5 we present the assumptions underlying our no-go theorem.

Revised: Any further assumptions used to derive our results are outlined in the corresponding appendices for each result.

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] These proofs are included in Appendices C - H. 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [N/A] .

Revised: Our work reports no experimental results.

Revised: In section 6 we use a three parameter parametric equation as an example, and cite the corresponding experimental paper where these parameters were learned.

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [N/A]

Revised: Our study does not use any datasets.

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [N/A] All results were derived by hand.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [N/A]

Revised: As mentioned in checklist 3 a), on section 6 we cite the study that created the generalized additive model whose parameters are referenced in one of our examples.

Revised: This is a three parameter regression model and we cite the paper that published these parameters.

Revised: We do not use any implementation of their model or any of their code or data.

Revised: (b) Did you mention the license of the assets?

Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A] (d) Did you discuss whether

Revised: and how consent was obtained from people whose data you’re using/curating?

Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Revised: [N/A] 5.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A]

Revised: We have not used any human subjects or crowd-sourcing.

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]