Original: Can Humans Be out of the Loop?
Revised: Can Humans Be out of the Loop?

Original: Author names withheld
Revised: Bernhard Sch¨olkopf

Original: Editor: Under Review for CLeaR
Revised: Caroline Uhler Kun Zhang

Original: Abstract
Revised: Abstract

Original: Recent advances in Reinforcement Learning have allowed automated agents (for short, agents) to achieve a high level of performance across a wide range of tasks, which when supplemented with human feedback has led to faster and more robust decision-making.
Revised: Recent advances in Reinforcement Learning have allowed automated agents (for short, agents) to achieve a high level of performance across a wide range of tasks, which when supplemented with human feedback has led to faster and more robust decision-making.

Original: The current literature, in large part, focuses on the human’s role during the learning phase: human trainers possess a priori knowledge that could help an agent to accelerate its learning when the environment is not fully known.
Revised: The current literature, in large part, focuses on the human’s role during the learning phase: human trainers possess a priori knowledge that could help an agent to accelerate its learning when the environment is not fully known.

Original: In this paper, we study an interactive reinforcement learning setting where the agent and the human have di ↵ erent sensory capabilities, disagreeing, therefore, on how they perceive the world (observed states) while sharing the same reward and transition functions.
Revised: In this paper, we study an interactive reinforcement learning setting where the agent and the human have di ↵ erent sensory capabilities, disagreeing, therefore, on how they perceive the world (observed states) while sharing the same reward and transition functions.

Original: We show that agents are bound to learn sub-optimal policies if they do not take into account human advice, perhaps surprisingly, even when human’s decisions are less accurate than their own.
Revised: We show that agents are bound to learn sub-optimal policies if they do not take into account human advice, perhaps surprisingly, even when human’s decisions are less accurate than their own.

Original: We propose the counterfactual agent who proactively considers the intended actions of the human operator, and proves that this strategy dominates standard approaches regarding performance.
Revised: We propose the counterfactual agent who proactively considers the intended actions of the human operator, and proves that this strategy dominates standard approaches regarding performance.

Original: Finally, we formulate a novel reinforcement learning task maximizing the performance of an autonomous system subject to a budget constraint over the available amount of human advice.
Revised: Finally, we formulate a novel reinforcement learning task maximizing the performance of an autonomous system subject to a budget constraint over the available amount of human advice.

Original: 1. Introduction
Revised: 1. Introduction

Original: Sequential decision-making plays a central role in the design of modern intelligent systems.
Revised: Sequential decision-making plays a central role in the design of modern intelligent systems.

Original: In a prototypical setting, for example, an agent is deployed in an unknown environment and needs to learn how to act such that a set of non-trivial goals are achieved.
Revised: In a prototypical setting, for example, an agent is deployed in an unknown environment and needs to learn how to act such that a set of non-trivial goals are achieved.

Original: A solution in these settings usually comes in the form of a policy, namely, a function mapping from the agent’s past observations (e.g., visited states, realized outcomes) to actions it should execute at each instant.
Revised: A solution in these settings usually comes in the form of a policy, namely, a function mapping from the agent’s past observations (e.g., visited states, realized outcomes) to actions it should execute at each instant.

Original: Human experts might be able to specify a policy in advance for simple tasks, but the policy usually needs to be learned from experimentation in most complex, real-world scenarios.
Revised: Human experts might be able to specify a policy in advance for simple tasks but the policy usually needs to be learned from experimentation in most complex, real-world scenarios.

Original: Reinforcement learning (RL) (Sutton and Barto, 1998) has emerged as the de facto framework to solve this problem, allowing agents to learn optimal policies by using through interactions with the environment, ideally, without explicit human instructions.
Revised: Reinforcement learning (RL) (Sutton and Barto, 1998) has emerged as the de facto framework to solve this problem, allowing agents to learn optimal policies by using through interactions with the environment, ideally, without explicit human instructions.

Original: One of the major challenges when applying RL methods in practice is the substantial amount of time and data required to ﬁnd a reasonable, hopefully optimal, policy.
Revised: One of the major challenges when applying RL methods in practice is the substantial amount of time and data required to ﬁnd a reasonable, hopefully optimal, policy.

Original: Many e ↵ orts have been made to accelerate the learning process by allowing agents to learn by interacting with human trainers.
Revised: Many e ↵ orts have been made to accelerate the learning process by allowing agents to learn by interacting with human trainers.

Original: One such approach is to learn from humans past behaviors, which includes learning from demonstration (Atkeson and Schaal, 1997; Chernova and Veloso, 2007; Argall et al., 2009; Hussein et al., 2017; Zhang and Bareinboim, 2017) and inverse reinforcement learning Ng and Russell (2000); Abbeel and Ng (2004).
Revised: One such approach is to learn from humans past behaviors, which includes learning from demonstration (Atkeson and Schaal, 1997; Chernova and Veloso, 2007; Argall et al., 2009; Hussein et al., 2017; Zhang and Bareinboim, 2017) and inverse reinforcement learning Ng and Russell (2000); Abbeel and Ng (2004).

Original: In general, these techniques learn a policy by taking as input observations of a human performing a task.
Revised: In general, these techniques learn a policy by taking as input observations of a human performing a task.

Original: Another approach that has shown promise is known as interactive reinforcement learning , where the agent is provided in real time with a series of critiques from (possibly non-expert) humans (Knox and Stone, 2009, 2012; Mandel et al., 2017).
Revised: Another approach that has shown promise is known as interactive reinforcement learning , where the agent is provided in real time with a series of critiques from (possibly non-expert) humans (Knox and Stone, 2009, 2012; Mandel et al., 2017).

Original: Techniques developed under this rubric include reward shaping (Skinner, 1990; Randløv and Alstrøm, 1998;
Revised: Techniques developed under this rubric include reward shaping (Skinner, 1990; Randløv and Alstrøm, 1998;

Original: Ng et al., 1999; Devlin and Kudenko, 2012), policy shaping (Gri  th et al., 2013; Loftin et al., 2014, 2016) and student-teacher training (Clouse, 1996; Torrey and Taylor, 2013).
Revised: Ng et al., 1999; Devlin and Kudenko, 2012), policy shaping (Gri  th et al., 2013; Loftin et al., 2014, 2016) and student-teacher training (Clouse, 1996; Torrey and Taylor, 2013).

Original: This body of the literature concerned with incorporating the “human in the loop” focuses on the role of human instructors in the learning phase of the system.
Revised: This body of the literature concerned with incorporating the “human in the loop” focuses on the role of human instructors in the learning phase of the system.

Original: By and large,(1) the challenge is that an agent has an incomplete model of the environment (e.g., unknown reward functions or system dynamics),(2) which can be improved by human trainers who possess a more detailed understanding about the environment and the task at hand.
Revised: By and large, the challenge is that an agent has an incomplete model of the environment (e.g., unknown reward functions or system dynamics), which can be improved by human trainers who possess a more detailed understanding about the environment and the task at hand.

Original: The translation of the trainer’s knowledge into insights for the agent would then allow a more aggressive pruning of the search space, naturally, leading to accelerated learning rates.
Revised: The translation of the trainer’s knowledge into insights for the agent would then allow a more aggressive pruning of the search space, naturally, leading to accelerated learning rates.

Original: This approach suggests that after serving as a data source for learning the model, human subjects have no additional value and, therefore, can be disposed of, i.e., moved “out of the loop.”
Revised: This approach suggests that after serving as a data source for learning the model, human subjects have no additional value and, therefore, can be disposed of, i.e., moved “out of the loop.”

Original: In this paper, we explore a novel aspect of the role of human instructors in RL, speciﬁcally after the automated agent learns a model of the environment.
Revised: In this paper, we explore a novel aspect of the role of human instructors in RL, speciﬁcally after the automated agent learns a model of the environment.

Original: 1 We show, by a simple example, that when the agent and the human have di ↵ erent perceptual capabilities (observed states), while sharing the same reward and transition functions, the agent could still be constrained tolearn sub-optimal policies if it does not consider the human’s instructions (regardless of the number of interactions allowed, number of demonstrations, etc).
Revised: 1 We show, by a simple example, that when the agent and the human have di ↵ erent perceptual capabilities (observed states), while sharing the same reward and transition functions, the agent could still be constrained to sub-optimal policies if it does not consider the human’s instructions (regardless of the number of interactions allowed, number of demonstrations, etc).

Original: Most closely related to our work, (Wray et al., 2016) considered a planning problem where the human and agent repeatedly transfer the control of the decisions, thus reaching a performance level not achievable by the standard autonomous approaches.
Revised: Most closely related to our work, (Wray et al., 2016) considered a planning problem where the human and agent repeatedly transfer the control of the decisions, thus reaching a performance level not achievable by the standard autonomous approaches.

Original: This paper explicitly accounts forthe capabilities of the human and agent using the language of causality (Pearl, 2000; Lee and Bareinboim, 2018, 2019; Zhang and Bareinboim, 2019, 2020; Zhang et al., 2020).
Revised: This paper explicitly accounts for capabilities of the human and agent using the language of causality (Pearl, 2000; Lee and Bareinboim, 2018, 2019; Zhang and Bareinboim, 2019, 2020b; Zhang et al., 2020; Kumor et al., 2021).

Original: This allows us to delineate a large number of instances where simple solutions are available, yet overlooked in previous work.
Revised: This allows us to delineate a large number of instances where simple solutions are available, yet overlooked in previous work.

Original: More speciﬁcally, our contributions are:
Revised: More speciﬁcally, our contributions are:

Original: We demonstrate the conditions under which human input can provide valuable information for agents to achieve optimal performance, even when the model of the environment is fully known.
Revised: We demonstrate the conditions under which human input can provide valuable information for agents to achieve optimal performance, even when the model of the environment is fully known.

Original: We deﬁne the counterfactual agent and identify a class of environments where it can be e  ciently optimized using standard MDP algorithms (albeit with changes).
Revised: We deﬁne the counterfactual agent and identify a class of environments where it can be e  ciently optimized using standard MDP algorithms (albeit with changes).

Original: We formulate a novel RL problem subject to a budget constraint of available human advice. We show that in Markovian environments, this problem can be reduced to a constrained MDP.
Revised: We formulate a novel RL problem subject to a budget constraint of available human advice, which is reducible to a constrained MDP.

Original: The optimal policy, in this case, is computable as a solution of a polynomial program.
Revised: The optimal policy, in this case, is computable as a solution of a polynomial program.

Revised: Given the space constraints, all proofs are provided in the complete technical report (Zhang and Bareinboim, 2020a)

Original: 1.1. Preliminaries
Revised: 1.1. Preliminaries

Original: We introduce in this section the basic notations and deﬁnitions used throughout the paper.
Revised: We introduce in this section the basic notations and deﬁnitions used throughout the paper.

Original: Variables are denoted by capital letters (e.g., X ), and their values by lowercase letters ( x ).
Revised: Variables are denoted by capital letters (e.g., X ), and their values by lowercase letters ( x ).

Original: Let
Revised: Let

Original: X denote the domain of a variable X .
Revised: X denote the domain of a variable X .

Original: Let v ([ i,j ]) stand for a sequence of variables ￿ v ( i ) , v ( i + ) , . .
Revised: Let v ([ i,j ]) stand for a sequence of variables ￿ v ( i ) , v ( i + ) , . .

Original: , v ( j ) ￿ (empty if j < i ), and V ([ i,j ]) =
Revised: , v ( j ) ￿ (empty if j < i ), and V ([ i,j ]) =

Original: v ([ i,j ]) for a sequence of assignments
Revised: v ([ i,j ]) for a sequence of assignments

Original: V ( i ) = v ( i ) ,
Revised: V ( i ) = v ( i ) ,

Original: V ( i + ) =
Revised: V ( i + ) =

Original: v ( i + ) , . . .
Revised: v ( i + ) , . . .

Original: , V ( j ) = v ( j ) .
Revised: , V ( j ) = v ( j ) .

Original: We will consistently use the abbreviation P ( x ) for the probabilities P (
Revised: We will consistently use the abbreviation P ( x ) for the probabilities P (

Original: X = x ) , ∀ x ∈
Revised: X = x ) , ∀ x ∈

Original: D (
Revised: D (

Original: X ) , so does P ( y ￿ x ) =
Revised: X ) , so does P ( y ￿ x ) =

Original: P (
Revised: P (

Original: Y =
Revised: Y =

Original: y ￿
Revised: y ￿

Original: X = x ) .
Revised: X = x ) .

Original: This paper deals extensively with ﬁnite (PO)MDPs models as the basis for designing RL algorithms.
Revised: This paper deals extensively with ﬁnite (PO)MDPs models as the basis for designing RL algorithms.

Original: Deﬁnition 1 (Partially Observed MDP)
Revised: Deﬁnition 1 (Partially Observed MDP)

Original: A ﬁnite POMDP (Cassandra et al., 1994) is a 6-tuple ￿ S ,
Revised: A ﬁnite POMDP (Cassandra et al., 1994) is a 6-tuple ￿ S ,

Original: X ,
Revised: X ,

Original: O , T, R, ⌦ ￿ in which
Revised: O , T, R, ⌦ ￿ in which

Original: S is a ﬁnite set of states;
Revised: S is a ﬁnite set of states;

Original: X a ﬁnite set of actions;
Revised: X a ﬁnite set of actions;

Original: O is a ﬁnite set of observations; T ∶ S × X × S →
Revised: O is a ﬁnite set of observations; T ∶ S × X × S →

Original: [ 0 ,] is a transition distribution; R ∶ S × X →
Revised: [ 0 ,] is a transition distribution; R ∶ S × X →

Original: R is a reward function; and ⌦ ∶ S × O → [ 0 ,] is an observation function.
Revised: R is a reward function; and ⌦ ∶ S × O → [ 0 ,] is an observation function.

Original: Formally, if states are fully observed (i.e.,
Revised: Formally, if states are fully observed (i.e.,

Original: O = S and ⌦ ( s, x, s )
Revised: O = S and ⌦ ( s, x, s )

Original: = 1), the 4-tuple ￿ S ,
Revised: = 1), the 4-tuple ￿ S ,

Original: X , T, R ￿ forms an MDP.
Revised: X , T, R ￿ forms an MDP.

Original: For simplicity, let T sx ( s ′ ) =
Revised: For simplicity, let T sx ( s ′ ) =

Original: T ( s, x, s ′ ) , R sx =
Revised: T ( s, x, s ′ ) , R sx =

Original: R ( s, x ) and ⌦ s ( o ) =
Revised: R ( s, x ) and ⌦ s ( o ) =

Original: ⌦ ( s, o ) .
Revised: ⌦ ( s, o ) .

Original: 2. The Value of Human Advice
Revised: 2. The Value of Human Advice

Original: We start the discussion with an example involving the sequential decision-making of patients’ treatment.
Revised: We start the discussion with an example involving the sequential decision-making of patients’ treatment.

Original: We show that an agent with the fully speciﬁed system dynamics could fail to ﬁnd an e ↵ ective policy if it ignores the advice of the human instructor.
Revised: We show that an agent with the fully speciﬁed system dynamics could fail to ﬁnd an e ↵ ective policy if it ignores the advice of the human instructor.

Original: Speciﬁcally, a physician treats each patient who visits the hospital regularly to maintain her long term health condition.
Revised: Speciﬁcally, a physician treats each patient who visits the hospital regularly to maintain her long term health condition.

Original: This sequential environment can be represented by an MDP M MDPUC described in
Revised: This sequential environment can be represented by an MDP M mdpuc described in

Original: Fig. 2(a); here, UC stands for unobserved confounders.
Revised: Fig. 2(a); here, UC stands for unobserved confounders.

Original: More speciﬁcally, the physician measures the patient’s corticosteroid level at the t -th visit, S ( t ) =
Revised: More speciﬁcally, the physician measures the patient’s corticosteroid level at the t -th visit, S ( t ) =

Original: s ( t ) ∈ { 0 ,} , where 0 stands for a low and 1 for a high level of corticosteroid.
Revised: s ( t ) ∈ { 0 ,} , where 0 stands for a low and 1 for a high level of corticosteroid.

Original: She then decides a treatment X ( t )
Revised: She then decides a treatment X ( t )

Original: = x ′ ( t ) ∈ { 0 ,} (1 for to give the drug, 0 for not to), and then measures an overall health score Y ( t ) =
Revised: = x ′ ( t ) ∈ { 0 ,} (1 for to give the drug, 0 for not to), and then measures an overall health score Y ( t ) =

Original: y ( t ) ∈ { 0 ,} (i.e., 1 for “healthy” and 0 for “not healthy”).
Revised: y ( t ) ∈ { 0 ,} (i.e.,for “healthy” and 0 for “not healthy”).

Original: In reality, the patient’s health score Y ( t ) is also a ↵ ected by a pair of variables
Revised: In reality, the patient’s health score Y ( t ) is also a ↵ ected by a pair of variables

Original: U ( t ) = {
Revised: U ( t ) = {

Original: M ( t ) , E ( t ) } , where M ( t ) = m ( t ) stands for patient’s psychological status (0 for a positive mood, 1 for negative) and E ( t ) =
Revised: M ( t ) , E ( t ) } , where M ( t ) = m ( t ) stands for patient’s psychological status (0 for a positive mood, 1 for negative) and E ( t ) =

Original: e ( t ) stands for her socioeconomic status (0 for wealthy, 1 for poor).
Revised: e ( t ) stands for her socioeconomic status (0 for wealthy, 1 for poor).

Original: The patient’s long-term health condition can be modeled by the long-term cumulative reward ∑ [Equation]
Revised: The patient’s long-term health condition can be modeled by the long-term cumulative reward ∑ ∞ t = 1  t Y ( t ) ,  = 0 .

Original: The physician decides a treatment X ( t ) =
Revised: The physician decides a treatment X ( t ) =

Original: x ′ ( t ) based on values of S ( t ) , M ( t ) , E ( t ) , which we summarize as a policy function x 
Revised: x ′ ( t ) based on values of S ( t ) , M ( t ) , E ( t ) , which we summarize as a policy function x ′ ( t )

Original: [Equation]
Revised: = ⇡ h ￿ s ( t ) , m ( t ) , e ( t ) ￿ .

Original: The detailed parametrizations of the transition function T , the reward function R , and the policy ⇡ h are described in Appendix A.
Revised: The detailed parametrizations of the transition function T , the reward function R , and the policy ⇡ h are described in Appendix A.

Original: To maximize the patient’s long-term health, the hospital’s administration aims to automate the decision procedure and deploy an autonomous agent A exp .
Revised: To maximize the patient’s long-term health, the hospital’s administration aims to automate the decision procedure and deploy an autonomous agent A exp .

Original: The agent learns the fully speciﬁed model (functions T and R ) from the physician, but it doesn’t have accessto the psychological mood M ( t ) and the socioeconomic status E ( t ) of the patient due to privacy concerns. That is, M ( t ) , E ( t ) are unobserved variables for the learning agent. We solve for this system using both the standard MDP and POMDP planning algorithms and
Revised: The agent learns the fully speciﬁed model (functions T and R ) from the physician, but it doesn’t have access

Original: [Figure]
Revised: [Figure]

Original: ( t )
Revised: ( t )

Original: X ( t ) , Y ( t ) , S ( t + ) }
Revised: X ( t ) , Y ( t ) , S ( t + ) }

Revised: (Thm

Original: + where U ( t ) = {
Revised: + where U ( t ) = {

Original: U t 1 , U t} a ↵ ects {
Revised: U t 1 , U t} a ↵ ects {

Original: X ( t ) , Y ( t ) , S ( t + ) } , and also S ( t ) .
Revised: X ( t ) , Y ( t ) , S ( t + ) } , and also S ( t ) .

Original:  label the resulting policies as exp and exp2 , respectively.
Revised: to the psychological mood M ( t ) and the socioeconomic status E ( t ) of the patient due to privacy concerns. That is, M ( t ) , E ( t ) are unobserved variables for the learning agent. We solve for this system using both the standard MDP and POMDP planning algorithms and label the resulting policies as exp and exp2 , respectively.

Original: The administration also considers a simple agent ( A ctf ), which observes the corticosteroid level S ( t ) and also take the physician’s decision as advice in real-time (labeled as ctf ).
Revised: The administration also considers a simple agent ( A ctf ), which observes the corticosteroid level S ( t ) and also take the physician’s decision as advice in real-time (labeled as ctf ).

Original: Two baseline policies are included for comparison: (1) the physician’s current policy (called human ), and (2) a policy picking the treatment at random ( random ).
Revised: Two baseline policies are included for comparison: (1) the physician’s current policy (called human ), and (2) a policy picking the treatment at random ( random ).

Original: The cumulative reward of all policies are shown in
Revised: The cumulative reward of all policies are shown in

Original: Fig.
Revised: Fig.

Original: Perhaps surprisingly, the performance of exp and exp2 coincides with the random policy, indicating that A exp is unable to learn a reasonable policy, which should be better than chance.
Revised: Perhaps surprisingly, the performance of exp and exp2 coincides with the random policy, indicating that A exp is unable to learn a reasonable policy, which should be better than chance.

Original: Moreover, the interactive approach ( ctf ) manages to outperform all the other policies, despite the fact that the physician ( human ) performs worse than random guessing.
Revised: Moreover, the interactive approach ( ctf ) manages to outperform all the other policies, despite the fact that the physician ( human ) performs worse than random guessing.

Original: Puzzling questions arise at this point – ﬁrst, how could the human advice be useful for the performance of the agent even when it has full knowledge of the reward function R and the world dynamics T ?
Revised: Puzzling questions arise at this point – ﬁrst, how could the human advice be useful for the performance of the agent even when it has full knowledge of the reward function R and the world dynamics T ?

Original: Second, how could we formalize the interactive agent in this novel setting and systematically ﬁnd its optimal policy?
Revised: Second, how could we formalize the interactive agent in this novel setting and systematically ﬁnd its optimal policy?

Original: Our goal for the remainder of the paper is to answer these fundamental questions in full generality (i.e., for any decision setting).
Revised: Our goal for the remainder of the paper is to answer these fundamental questions in full generality (i.e., for any decision setting).

Original: 3. Modeling Autonomous Systems
Revised: 3. Modeling Autonomous Systems

Original: We will use structural causal models (SCMs) (Pearl, 2000) as the basic semantical framework of our analysis.
Revised: Our analysis relies on the semantical framework of structural causal models (SCMs) (Pearl, 2000) .

Original: SCMs allow us to explicitly model the di ↵ erence in capabilities of the human operator and the autonomous agent, thus providing a novel perspective on the human’s role in these settings.

Original: We will describe sequential decision-making systems using dynamic SCMs, which describes causal dynamics of how the environment’s states evolve over time.
Revised: We will describe sequential decision-making systems using dynamic SCMs, which describes causal dynamics of how the environment’s states evolve over time.

Original: Deﬁnition 2 (Dynamic SCMs)
Revised: Deﬁnition 2 (Dynamic SCMs)

Original: A dynamic structural causal model (DSCM)
Revised: A dynamic structural causal model (DSCM)

Original: M is a pair of SCMs￿
Revised: M is a pair of SCMs

Original: M 1 , M → ￿ (Pearl, 2000, Ch. 7) with domains over actions
Revised: [Equation] ￿ (Pearl, 2000, Ch. 7) with domains over actions

Original: X , observed states
Revised: X , observed states

Original: S , unobserved states
Revised: S , unobserved states

Original: U , and rewards
Revised: U , and rewards

Original: Y .
Revised: Y .

Original: M 1 is a SCM over initial states X ( ) , U ( ) , actions
Revised: M 1 is a SCM over initial states X ( ) , U ( ) , actions

Original: X ( ) and rewards Y ( ) .
Revised: X ( ) and rewards Y ( ) .

Original: M → is a SCM describing the process: for t = 2 , 3 , . . .
Revised: M → is a SCM describing the process: for t = 2 , 3 , . . .

Original: U ( t ) is the unobserved state whose values u ( t ) ∈
Revised: U ( t ) is the unobserved state whose values u ( t ) ∈

Original: U is decided by a function u ( t 
Revised: U is decided by a function u ( t ) =

Original: [Equation]
Revised: f u ￿ x t − ) , s ( t − ) , u ( t − ) , ✏ ( t ) u ￿ , where E ( t ) u is an independent noise drawn from P ( ✏ u ) 2 ;

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: u

Original: [Equation]
Revised: [Equation]

Original: ✏

Original: [Equation]
Revised: [Equation]

Original: u ￿ , where

Original: E ( t ) u is an independent noise drawn from P ( ✏ u ) 2 ;

Original: [Equation]
Revised: [Equation]

Original: is the observed state whose values s
Revised: is the observed state whose values s

Original: [Equation]
Revised: [Equation]

Original: is decided by a transition function
Revised: is decided by a transition function

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: is the observed reward whose values y
Revised: is the observed reward whose values y

Original: [Equation]
Revised: [Equation]

Original: is decided by a reward function
Revised: is decided by a reward function

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: is the action of the human instructor where its values x
Revised: is the action of the human instructor where its values x

Original: [Equation]
Revised: [Equation]

Original: is decided by a policy function x
Revised: is decided by a policy function x

Original: [Equation]
Revised: [Equation]

Original: The deﬁnition of DSCM is a direct translation of POMDP formalism to causal language: the pair S ( t ) , U ( t ) represents the underlying states, and S (
Revised: The deﬁnition of DSCM is a direct translation of POMDP formalism to causal language: the pair S ( t ) , U ( t ) represents the underlying states, and S (

Original: [Equation]
Revised: [Equation]

Original: ) for the partial observation 3 .
Revised: ) for the partial observation 3 .

Original: Each
Revised: Each

Original: DSCM M is associated with a causal diagram G , which is a directed acyclic graph (DAG) where empty nodes correspond to unobserved variables, solid nodes correspond to observed variables, and edges represent functional (cause-and-e ↵ ect) relationships (see Fig. 2).
Revised: DSCM M is associated with a causal diagram G , which is a directed acyclic graph (DAG) where empty nodes correspond to unobserved variables, solid nodes correspond to observed variables, and edges represent functional (cause-and-e ↵ ect) relationships (see Fig. 2).

Original: We next model the standard autonomous agents as the experimental agents, which interact with the environment through a series of causal interventions, oblivious to the presence of the human instructor.
Revised: We next model the standard autonomous agents as the experimental agents, which interact with the environment through a series of causal interventions, oblivious to the presence of the human instructor.

Original: Deﬁnition 3 (Experimental Agent)
Revised: Deﬁnition 3 (Experimental Agent)

Original: An experimental agent A exp is deﬁned as a sequence of policies ⇧ exp = ⇡ ([ 1 ,t ]) .
Revised: An experimental agent A exp is deﬁned as a sequence of policies ⇧ exp = ⇡ ([ 1 ,t ]) .

Original: Each ⇡ ( t ) is a function deciding values of X ( t ) such that
Revised: Each ⇡ ( t ) is a function deciding values of X ( t ) such that

Original: [Equation]
Revised: [Equation]

Original: = ⇡ ( t ) ￿ h ( t )
Revised: = ⇡ ( t ) ￿ h ( t )

Original: exp ￿ , where the experimental history h
Revised: exp ￿ , where the experimental history h

Original: ( t ) exp =
Revised: ( t ) exp =

Original: ￿ s ([ 1 ,t ]) , x ([ 1 ,t − ])
Revised: ￿ s ([ 1 ,t ]) , x ([ 1 ,t − ])

Original: ￿ is the observed history of A exp up to time t .
Revised: ￿ is the observed history of A exp up to time t .

Original: Using Def. 3, we could model the interaction between an autonomous agent A exp and the environment M dscm as an intervention on actions nodes X ([ 1 ,t ]) , denoted by do ( ⇧ exp ) , which sets values of X ( t ) to ⇡ ( t ) ￿ h ( t ) exp ￿ regardless of how they were ordinarily determined by the human operator ⇡ h (Pearl, 2000, Ch. 7).
Revised: Using Def. 3, we could model the interaction between an autonomous agent A exp and the environment M dscm as an intervention on actions nodes X ([ 1 ,t ]) , denoted by do ( ⇧ exp ) , which sets values of X ( t ) to ⇡ ( t ) ￿ h ( t ) exp ￿ regardless of how they were ordinarily determined by the human operator ⇡ h (Pearl, 2000, Ch. 7).

Original: To understand the decision process encoded here, note that each step t , the agent starts from a state S ( t )
Revised: To understand the decision process encoded here, note that each step t , the agent starts from a state S ( t )

Original: =
Revised: =

Original: s ( t ) , U ( t ) =
Revised: s ( t ) , U ( t ) =

Original: u ( t ) , performs an intervention X ( t ) =
Revised: u ( t ) , performs an intervention X ( t ) =

Original: ⇡ ￿ s ( t ) ￿ , receives a reward Y ( t ) =
Revised: ⇡ ￿ s ( t ) ￿ , receives a reward Y ( t ) =

Original: y ( t ) , and observes S ( t + )
Revised: y ( t ) , and observes S ( t + )

Original: =
Revised: =

Original: s ( t + ) .
Revised: s ( t + ) .

Original: We use Y x ([ 1 ,t ])
Revised: We use Y x ([ 1 ,t ])

Original: ￿ u
Revised: ￿ u

Original: [ 1 ,t ]) ￿ to denote the response of a variable Y to a sequence of interventions
Revised: [ 1 ,t ]) ￿ to denote the response of a variable Y to a sequence of interventions

Original: X ( ) =
Revised: X ( ) =

Original: x ( ) , . . .
Revised: x ( ) , . . .

Original: , X ( t ) =
Revised: , X ( t ) =

Original: x ( t ) .
Revised: x ( t ) .

Original: This stochastic process is summarized using the counterfactual distribution P ￿
Revised: This stochastic process is summarized using the counterfactual distribution P ￿

Original: Y
Revised: Y

Original: [Equation]
Revised: [Equation]

Original: ) ￿ , abbreviated as P ￿ y ( t ) x ( t ) ￿ s ( t ) , u ( t ) ￿ ; so does P ￿ s ( t + ) x ( t ) ￿ s ( t ) , u ( t ) ￿ .
Revised: ) ￿ , abbreviated as P ￿ y ( t ) x ( t ) ￿ s ( t ) , u ( t ) ￿ ; so does P ￿ s ( t + ) x ( t ) ￿ s ( t ) , u ( t ) ￿ .

Original: We assess the performance of an agent ⇧ using the cumulative rewards V ⇧ =
Revised: We assess the performance of an agent ⇧ using the cumulative rewards V ⇧ =

Original: E ￿ ∑ ∞ t = 

Revised: E ￿ ∑ ∞ t =

Original:  t Y ( t ) x ( t ) =
Revised:  t Y ( t ) x ( t ) =

Original: ⇡ ( t ) ￿ where  ∈ [ 0 ,) is a discount factor.
Revised: ⇡ ( t ) ￿ where  ∈ [ 0 ,) is a discount factor.

Original: The Markov property delineates a set of decision-making systems that could be e  ciently solved.
Revised: The Markov property delineates a set of decision-making systems that could be e  ciently solved.

Original: We next deﬁne the Markov property associated with the experimental agents.
Revised: We next deﬁne the Markov property associated with the experimental agents.

Original: Deﬁnition 4 (Experimental Markov Property)
Revised: Deﬁnition 4 (Experimental Markov Property)

Original: An environment M is said to be Experimentally Markov (exp-Markov, for short) if and only if for t ≥ 1 ,
Revised: An environment M is said to be Experimentally Markov (exp-Markov, for short) if and only if for t ≥ 1 ,

Original: P ￿ s ( t + ) x ([ 1 ,t ])
Revised: P ￿ s ( t + ) x ([ 1 ,t ])

Original: ￿
Revised: ￿

Original: [Equation]
Revised: [Equation]

Original: ([ 1 ,t ]) x ([
Revised: ([ 1 ,t ]) x ([

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Y (
Revised: Y (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: x ([
Revised: x ([

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ￿ =
Revised: ￿ =

Original: E ￿
Revised: E ￿

Original: Y ( t ) x ( t ) ￿ s ( t ) ￿ , (1) where s ([ 1 ,t ]) x ([ 1 ,t − 1 ]) =
Revised: Y ( t ) x ( t ) ￿ s ( t ) ￿ , (1) where s ([ 1 ,t ]) x ([ 1 ,t − 1 ]) =

Original: ￿ s ( t ) x ([ 1 ,t − 1 ]) , s ( t − ) x
Revised: ￿ s ( t ) x ([ 1 ,t − 1 ]) , s ( t − ) x

Original: ([ 1 ,t − 2 ]) , . . .
Revised: ([ 1 ,t − 2 ]) , . . .

Original: , s ( ) ￿ .
Revised: , s ( ) ￿ .

Original: t = 1 , 2 ,... could be observable to
Revised: t = 1 , 2 ,... could be observable to

Original: In Def. 4, S ( t ) x ([ 1 ,t − 1 ]) is the potential response of the state S ( t ) to intervention do ￿ x ([ 1 ,t − ]) ￿ , which can be read as “The value that the state S ( t ) would be had X ([ 1 ,t − ]) been x ([ 1 ,t − ]) ”
Revised: In Def. 4, S ( t ) x ([ 1 ,t − 1 ]) is the potential response of the state S ( t ) to intervention do ￿ x ([ 1 ,t − ]) ￿ , which can be read as “The value that the state S ( t ) would be had X ([ 1 ,t − ]) been x ([ 1 ,t − ]) ”

Original: (Halpern, 1998; Pearl, 2000).
Revised: (Halpern, 1998; Pearl, 2000).

Original: The exp-Markov property says that for an experimental agent, the information in S ([ 1 ,t ]) and the causal inﬂuence of the intervention do ￿ x ([ 1 ,t ]) ￿ can be best summarized in the current state S ( t ) and intervention do ￿ x ( t ) ￿ .
Revised: The exp-Markov property says that for an experimental agent, the information in S ([ 1 ,t ]) and the causal inﬂuence of the intervention do ￿ x ([ 1 ,t ]) ￿ can be best summarized in the current state S ( t ) and intervention do ￿ x ( t ) ￿ .

Original: For example, the MDPUC + model of Fig.
Revised: For example, the MDPUC + model of Fig.

Original: 2( c ) is exp-Markov though the unobserved states U ( t )
Revised: 2( c ) is exp-Markov though the unobserved states U ( t )

Original: =
Revised: =

Original: ￿
Revised: ￿

Original: U ( t) , U ( t) ￿ could a ↵ ect the states S ( t − ) , S ( t ) (see Appendix B for a detailed proof).
Revised: U ( t) , U ( t) ￿ could a ↵ ect the states S ( t − ) , S ( t ) .

Original: Consequently, for an expMarkov environment M , the resulting autonomous system ￿
Revised: Consequently, for an exp-Markov environment M , the resulting autonomous system ￿

Original: M, A exp ￿ forms an MDP.
Revised: M, A exp ￿ forms an MDP.

Original: The optimal policy ⇧ for an MDP is stationary where every policy function ⇡ ( t ) only depends on the state s ( t ) , t = 1 , 2 , . . .
Revised: The optimal policy ⇧ for an MDP is stationary where every policy function ⇡ ( t ) only depends on the state s ( t ) , t = 1 , 2 , . . .

Original: , and is invariant to time t , which we denote by ⇡ .
Revised: , and is invariant to time t , which we denote by ⇡ .

Original: Armed with these deﬁnitions, we are ﬁnally ready to analyze the medical treatment example and see how the human advice provides critical information to the task.
Revised: Armed with these deﬁnitions, we are ﬁnally ready to analyze the medical treatment example and see how the human advice provides critical information to the task.

Original: First note that MDPUC + is a generalization of the MDPUC model of Fig.
Revised: First note that MDPUC + is a generalization of the MDPUC model of Fig.

Original: 2( a ), the environment
Revised: 2( a ), the environment

Original: M mdpuc is exp-Markov.
Revised: M mdpuc is exp-Markov.

Original: The optimal policy of the autonomous agent A exp is stationary and takes the form of x ( t ) =
Revised: The optimal policy of the autonomous agent A exp is stationary and takes the form of x ( t ) =

Original: ⇡ ￿ s ( t ) ￿ .
Revised: ⇡ ￿ s ( t ) ￿ .

Original: On the other hand, the physician follows a policy ⇡ h that takes as input the values s ( t ) , m ( t ) , e ( t ) .
Revised: On the other hand, the physician follows a policy ⇡ h that takes as input the values s ( t ) , m ( t ) , e ( t ) .

Original: The di ↵ erence in the domains of the policies ⇡ and ⇡ h represents di ↵ erent capabilities of the agent and the human.
Revised: The di ↵ erence in the domains of the policies ⇡ and ⇡ h represents di ↵ erent capabilities of the agent and the human.

Original: In this case, the physician possesses superior observation capacities as she could access the states M ( t ) , E ( t ) , which are unavailable for the agent.
Revised: In this case, the physician possesses superior observation capacities as she could access the states M ( t ) , E ( t ) , which are unavailable for the agent.

Original: The autonomous agent carries a series of interventions do ( ⇡ ) , which replaces the physician’s policy ⇡ h with its policy ⇡ .
Revised: The autonomous agent carries a series of interventions do ( ⇡ ) , which replaces the physician’s policy ⇡ h with its policy ⇡ .

Original: Since M ( t ) , E ( t ) a ↵ ect the values of reward Y ( t ) and the next state S ( t + ) , the physician’s decision x ′ ( t ) could contain critical information for the long-term health condition of the patients.
Revised: Since M ( t ) , E ( t ) a ↵ ect the values of reward Y ( t ) and the next state S ( t + ) , the physician’s decision x ′ ( t ) could contain critical information for the long-term health condition of the patients.

Original: Discarding the physician’s decision x ′ ( t ) thus leads to potentially sub-optimal policies.
Revised: Discarding the physician’s decision x ′ ( t ) thus leads to potentially sub-optimal policies.

Original: 4. Counterfactual Agents
Revised: 4. Counterfactual Agents

Original: We will account for the human interaction in the environment using the language of SCMs.
Revised: We will account for the human interaction in the environment using the language of SCMs.

Original: A counterfactual agent takes the human’s advice as its intended action, and adjusts this action using the counterfactual reasoning if a better decision is available (the intended action is called intuition in (Bareinboim et al., 2015; Forney et al., 2017).
Revised: A counterfactual agent takes the human’s advice as its intended action, and adjusts this action using the counterfactual reasoning if a better decision is available (the intended action is called intuition in (Bareinboim et al., 2015; Forney et al., 2017).

Original: Deﬁnition 5 (Counterfactual Agent)
Revised: Deﬁnition 5 (Counterfactual Agent)

Original: A counterfactual agent A ctf is deﬁned as a policy ⇧ ctf = ⇡ ([ 1 ,t ]) .
Revised: A counterfactual agent A ctf is deﬁned as a policy ⇧ ctf = ⇡ ([ 1 ,t ]) .

Original: ⇡ ( t ) is a function deciding values of X ( t ) such that x ( t ) =
Revised: ⇡ ( t ) is a function deciding values of X ( t ) such that x ( t ) =

Original: ⇡ ( t ) ￿ h ( t ) ctf ￿ , where the counterfactual history h ( t ) ctf =
Revised: ⇡ ( t ) ￿ h ( t ) ctf ￿ , where the counterfactual history h ( t ) ctf =

Original: ￿ s ([ 1 ,t ]) , x ([ 1 ,t − ]) , x ′ ([ 1 ,t ]) ￿ , and x ′ ([ 1 ,t ]) is a sequence of human’s decisions up to time t .
Revised: ￿ s ([ 1 ,t ]) , x ([ 1 ,t − ]) , x ′ ([ 1 ,t ]) ￿ , and x ′ ([ 1 ,t ]) is a sequence of human’s decisions up to time t .

Original: For an agent A ctf , the evaluation of the policy ⇧ ctf is indeed a counterfactual reasoning procedure: “Given that I am about to perform x ′ ( t ) (intended, factual) and the past history, what the world state and reward would be had I done the action x ( t ) (contrary to the fact, counter-factual)”.
Revised: For an agent A ctf , the evaluation of the policy ⇧ ctf is indeed a counterfactual reasoning procedure: “Given that I am about to perform x ′ ( t ) (intended, factual) and the past history, what the world state and reward would be had I done the action x ( t ) (contrary to the fact, counter-factual)”.

Original: For concreteness, consider a counterfactual agent A ctf in the MDPUC model with the goal of maximizing the reward at t = 1.
Revised: For concreteness, consider a counterfactual agent A ctf in the MDPUC model with the goal of maximizing the reward at t = 1.

Original: Given the observed state s ( ) and its intended action x ′ ( ) , the agent A ctf decides for a new action x ( ) to maximize the expected reward Y ( ) , i.e., the counterfactual statement E ￿
Revised: Given the observed state s ( ) and its intended action x ′ ( ) , the agent A ctf decides for a new action x ( ) to maximize the expected reward Y ( ) , i.e., the counterfactual statement E ￿

Original: Y ( ) x ( 1 ) ￿ s ( ) , x ′ ( ) ￿ , which can be read as “The value that the reward Y ( ) would attain had I done x ( ) , given that I am about to do x ′ ( ) and the current state is s ( ) .”
Revised: Y ( ) x ( 1 ) ￿ s ( ) , x ′ ( ) ￿ , which can be read as “The value that the reward Y ( ) would attain had I done x ( ) , given that I am about to do x ′ ( ) and the current state is s ( ) .”

Original: Note that the values of x ′ ( ) (factual) and x ( ) (counterfactual) could be di ↵ erent.
Revised: Note that the values of x ′ ( ) (factual) and x ( ) (counterfactual) could be di ↵ erent.

Original: It follows immediately after Def. 5 that a counterfactual agent consistently dominates an experimental agent in terms of the performance.
Revised: It follows immediately after Def. 5 that a counterfactual agent consistently dominates an experimental agent in terms of the performance.

Original: Corollary 6 Let the optimal policy for A exp denoted by ⇧ ∗ exp and for A ctf by ⇧ ∗ ctf .
Revised: Corollary 6 Let the optimal policy for A exp denoted by ⇧ ∗ exp and for A ctf by ⇧ ∗ ctf .

Original: For any DSCM model, V ⇧ ∗ exp ≤
Revised: For any DSCM model, V ⇧ ∗ exp ≤

Original: V ⇧ ∗ ctf .
Revised: V ⇧ ∗ ctf .

Original: Given the space constraints, all proofs are included in Appendix B. Corol.
Revised: Given the space constraints, all proofs are included in Appendix B. Corol.

Original: 6 reassures the intuition that, since the counterfactual history always covers the experimental one (i.e., h ( t ) exp ⊆
Revised: 6 reassures the intuition that, since the counterfactual history always covers the experimental one (i.e., h ( t ) exp ⊆

Original: h ( t ) ctf ), counterfactual agents are never worse than their experimental counterparts.
Revised: h ( t ) ctf ), counterfactual agents are never worse than their experimental counterparts.

Original: The following result provides a graphical condition for when the performance of A ctf and
Revised: The following result provides a graphical condition for when the performance of A ctf and

Original: A exp coincides, i.e., the human advice can be discarded without loss of information.
Revised: A exp coincides, i.e., the human advice can be discarded without loss of information.

Original: Theorem 7 For any DSCM model where the human decision is a ↵ ected only by observable variables (arrow U ( t ) →
Revised: Theorem 7 For any DSCM model where the human decision is a ↵ ected only by observable variables (arrow U ( t ) →

Original: X (
Revised: X (

Original: [Equation]
Revised: [Equation]

Original: ) is not present), denoted by DSCM
Revised: ) is not present), denoted by DSCM

Original: [Equation]
Revised: [Equation]

Original: In words, Corol. 6 and Thm.
Revised: In words, Corol. 6 and Thm.

Original: 7 challenge the somewhat popular belief that the role of human instructors in RL systems is to serve as data sources for agents with partially speciﬁed models, and it is dispensable once the learning procedure is complete.
Revised: 7 challenge the somewhat popular belief that the role of human instructors in RL systems is to serve as data sources for agents with partially speciﬁed models, and it is dispensable once the learning procedure is complete.

Original: These results imply that (1) it is still better to utilize an interactive approach ( A ctf ) even if the information a ↵ ecting the human decision has an adverse e ↵ ect on its performance; (2) the human advice is indispensable if it possesses some information about latent states (arrow
Revised: These results imply that (1) it is still better to utilize an interactive approach ( A ctf ) even if the information a ↵ ecting the human decision has an adverse e ↵ ect on its performance; (2) the human advice is indispensable if it possesses some information about latent states (arrow

Original: U ( t ) →
Revised: U ( t ) →

Original: X ( t ) is present), even when the agent has obtained the fully speciﬁed world model.
Revised: X ( t ) is present), even when the agent has obtained the fully speciﬁed world model.

Original: 4.1. Optimizing Counterfactual Agents
Revised: 4.1. Optimizing Counterfactual Agents

Original: We will next discuss methods to systematically compute the optimal policy of counterfactual agents.
Revised: We will next discuss methods to systematically compute the optimal policy of counterfactual agents.

Original: We ﬁrst characterize the Markov property associated with the counterfactual agent, as it outlines su  cient statistics for A ctf .
Revised: We ﬁrst characterize the Markov property associated with the counterfactual agent, as it outlines su  cient statistics for A ctf .

Original: Deﬁnition 8 (Counterfactual Markov Property)
Revised: Deﬁnition 8 (Counterfactual Markov Property)

Original: An environment M is said to be counterfactually Markov (ctf-Markov, for short) if and only if for t ≥ 1 ,
Revised: An environment M is said to be counterfactually Markov (ctf-Markov, for short) if and only if for t ≥ 1 ,

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: where x ′ ([ 1 ,t ]) x ([
Revised: where x ′ ([ 1 ,t ]) x ([

Original: [Equation]
Revised: [Equation]

Original: t − 1 ])
Revised: t − 1 ])

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: , x
Revised: , x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ￿ and s ([ 1 ,t ]) x ([
Revised: ￿ and s ([ 1 ,t ]) x ([

Original: [Equation]
Revised: [Equation]

Original: − 1 ]) follows Def. 4.
Revised: − 1 ]) follows Def. 4.

Original: Compared to the exp-Markov property (Def. 4), Eqs.
Revised: Compared to the exp-Markov property (Def. 4), Eqs.

Original: (2) and (3) condition not only on the observed states s ([ 1 ,t ]) x ([ 1 ,t − 1 ]) but all the past human’s decisions x ′ ([ 1 ,t ]) to interventions do ￿ x ([ 1 ,t − ])
Revised: (2) and (3) condition not only on the observed states s ([ 1 ,t ]) x ([ 1 ,t − 1 ]) but all the past human’s decisions x ′ ([ 1 ,t ]) to interventions do ￿ x ([ 1 ,t − ])

Original: ￿ as well.
Revised: ￿ as well.

Original: This additional term makes the independence conditions for ctfMarkov property somewhat more involved: models that are exp-Markov are not necessarily ctf-Markov, as shown in the next example.
Revised: This additional term makes the independence conditions for ctfMarkov property somewhat more involved: models that are exp-Markov are not necessarily ctf-Markov, as shown in the next example.

Original: Proposition 9 M mdpuc + (Fig. 2( c )) is not ctf-Markov.
Revised: Proposition 9 M mdpuc + (Fig. 2( c )) is not ctf-Markov.

Original: Prop. 9 conﬁrms that, in general, ctf-Markov property does not necessarily hold in a exp-Markov environment.
Revised: Prop. 9 conﬁrms that, in general, ctf-Markov property does not necessarily hold in a exp-Markov environment.

Original: However, the ctf-Markov property is satisﬁable in some settings, which is characterized by the general graphical condition given in the next result.
Revised: However, the ctf-Markov property is satisﬁable in some settings, which is characterized by the general graphical condition given in the next result.

Original: Theorem 10 M mdpuc (Fig.
Revised: Theorem 10 M mdpuc (Fig.

Original: 2( b )) is ctf-Markov.
Revised: 2( b )) is ctf-Markov.

Original: For a ctf-Markov environment M mdpuc , the interactive RL system ￿
Revised: For a ctf-Markov environment M mdpuc , the interactive RL system ￿

Original: M, A ctf ￿ forms an
Revised: M, A ctf ￿ forms an

Original: MDP where the human decision is included as an additional observed state.
Revised: MDP where the human decision is included as an additional observed state.

Original: More formally, the MDP should be constructed as follows:
Revised: More formally, the MDP should be constructed as follows:

Original: Proposition￿
Revised: Proposition￿

Original: M mdpuc , A ctf ￿ forms an MDP
Revised: M mdpuc , A ctf ￿ forms an MDP

Original: [Equation]
Revised: [Equation]

Original:  ctf 
Revised: S ctf ,

Original: [Equation]

Original: X ctf
Revised: X ctf

Original: [Equation]
Revised: [Equation]

Original: T [Equation] R
Revised: T, R

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: , where

Original: S ctf = S × X ,
Revised: S ctf = S × X ,

Original: X ctf = X , and for any x, s, x
Revised: X ctf = X , and for any x, s, x

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]

Original: x

Original: [Equation]
Revised: [Equation]

Original: Prop. 11 follows immediately after the ctf-Markov property (Thm. 10).
Revised: Prop. 11 follows immediately after the ctf-Markov property (Thm. 10).

Original: The transition T and the reward function R are both counterfactual quantities, which can be computed by the procedure provided in (Pearl, 2000, Thm. 7.1.7), whenever the SCM is known.
Revised: The transition T and the reward function R are both counterfactual quantities, which can be computed by the procedure provided in (Pearl, 2000, Thm. 7.1.7), whenever the SCM is known.

Original: Alternatively, they can be empirically estimated using the new counterfactual randomization procedure introduced in Bareinboim et al.
Revised: Alternatively, they can be empirically estimated using the new counterfactual randomization procedure introduced in Bareinboim et al.

Original: There are several standard ways of solving an MDP with the discounted expected cumulative rewards optimization criterion Puterman (2014).
Revised: There are several standard ways of solving an MDP with the discounted expected cumulative rewards optimization criterion Puterman (2014).

Original: Some use dynamic programming (value of policy iteration), others reduce MDPs to linear programs (LPs).
Revised: Some use dynamic programming (value of policy iteration), others reduce MDPs to linear programs (LPs).

Original: A discounted
Revised: A discounted

Original: MDP formed by an interactive RL system ￿
Revised: MDP formed by an interactive RL system ￿

Original: M mdpuc , A ctf ￿ can be formulated as the following LP d’Epenoux (1963); Kallenberg (1983) (this maximization LP is the dual to the more commonly used minimization LP in the value function coordinates):[Equation] subject to 
Revised: M mdpuc , A ctf ￿ can be formulated as the following LP d’Epenoux (1963); Kallenberg (1983) (this maximization LP is the dual to the more commonly used minimization LP in the value function coordinates):

Original: [Equation]
Revised: [Equation]

Original: where ↵ ￿ s,x
Revised: where ↵ ￿ s,x

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: specify the probility distribution over the initial state.
Revised: specify the probility distribution over the initial state.

Original: The optimization variables  x ( s, x ′ ) are called the occupation measure of a policy, where  x ( s, x ′ ) is the total discounted number of times action x is executed in state s, x ′ .
Revised: The optimization variables  x ( s, x ′ ) are called the occupation measure of a policy, where  x ( s, x ′ ) is the total discounted number of times action x is executed in state s, x ′ .

Original: An optimial policy for A ctf is stationary and can be computed from a solution to the above
Revised: An optimial policy for A ctf is stationary and can be computed from a solution to the above

Original: LP as ⇡ ∗ 
Revised: LP as ⇡ ∗ ctf

Original: [Equation]
Revised: [Equation]

Original: For a general environment M dscm , it immediately follows from Prop. 9 that M dscm is not ctf-Markov, since DSCM is a generalization of MDPUC + .
Revised: For a general environment M dscm , it immediately follows from Prop. 9 that M dscm is not ctf-Markov, since DSCM is a generalization of MDPUC + .

Original: To solve for the counterfactual agent in an environment where ctf-Markov doesn’t hold, a simple realization is to resort to
Revised: To solve for the counterfactual agent in an environment where ctf-Markov doesn’t hold, a simple realization is to resort to

Original: POMDPs by using the human’s decision X 
Revised: POMDPs by using the human’s decision X ( t ) = x ′

Original: [Equation]
Revised: [Equation]

Original:  ) as an additional observation.
Revised: t ) as an additional observation.

Original: Proposition 12 A RL system ￿
Revised: Proposition 12 A RL system ￿

Original: M dscm , A ctf ￿ forms a POMDP ￿ S ctf ,
Revised: M dscm , A ctf ￿ forms a POMDP ￿ S ctf ,

Original: X ctf ,
Revised: X ctf ,

Original: O ctf , T, R, ⌦ ￿ where
Revised: O ctf , T, R, ⌦ ￿ where

Original: S ctf = S × U ,
Revised: S ctf = S × U ,

Original: X ctf =
Revised: X ctf =

Original: X ,
Revised: X ,

Original: O ctf = S × X , and for any x, s, u, x ′ , s ′ , u ′ ,
Revised: O ctf = S × X , and for any x, s, u, x ′ , s ′ , u ′ ,

Original: T ￿ s,u ￿ x ￿ s ′ , u ′ ￿ =
Revised: T ￿ s,u ￿ x ￿ s ′ , u ′ ￿ =

Original: P ￿
Revised: P ￿

Original: S ( t + ) X ( t ) =
Revised: S ( t + ) X ( t ) =

Original: x
Revised: x

Original: = s ′ , U ( t + ) X ( t ) =
Revised: = s ′ , U ( t + ) X ( t ) =

Original: x = u ′ ￿
Revised: x = u ′ ￿

Original: S ( t ) =
Revised: S ( t ) =

Original: s, U ( t ) =
Revised: s, U ( t ) =

Original: u ￿ ,
Revised: u ￿ ,

Original: R ￿ s,u ￿ x =
Revised: R ￿ s,u ￿ x =

Original: E ￿
Revised: E ￿

Original: Y ( t ) X ( t ) =
Revised: Y ( t ) X ( t ) =

Original: x ￿
Revised: x ￿

Original: S ( t ) =
Revised: S ( t ) =

Original: s, U ( t ) =
Revised: s, U ( t ) =

Original: u ￿ , ⌦ s,u ￿ s ′ , x ′ ￿ =
Revised: u ￿ , ⌦ s,u ￿ s ′ , x ′ ￿ =

Original: P ￿
Revised: P ￿

Original: S ( t + )
Revised: S ( t + )

Original: = s ′ ,
Revised: = s ′ ,

Original: X ( t + ) = x ′ ￿
Revised: X ( t + ) = x ′ ￿

Original: S ( t + ) =
Revised: S ( t + ) =

Original: s, U ( t + ) =
Revised: s, U ( t + ) =

Original: u ￿ .
Revised: u ￿ .

Original: Since the underlying state is not perfrectly observable, the agent can maintain a belief
Revised: Since the underlying state is not perfrectly observable, the agent can maintain a belief

Original: B ( t ) about the current state ( s ( t ) , u
Revised: B ( t ) about the current state ( s ( t ) , u

Original: [Equation]
Revised: [Equation]

Original: t ) )
Revised: t ) )

Original: at each time step t , deﬁned as
Revised: at each time step t , deﬁned as

Original: [Equation]
Revised: [Equation]

Original: ￿ The belief can be updated as times progresses based on Bayes’ theorem.
Revised: ￿ The belief can be updated as times progresses based on Bayes’ theorem.

Original: Formally, the belief update step for t + 1 goes as follows:
Revised: Formally, the belief update step for t + 1 goes as follows:

Original: Theorem 13 Given the belief B (
Revised: Theorem 13 Given the belief B (

Original: [Equation]
Revised: [Equation]

Original: ) , action x ( t ) , next observed state s (
Revised: ) , action x ( t ) , next observed state s (

Original: [Equation]
Revised: [Equation]

Original: and next human’s decision x ′ ( t + ) , the next belief can be updated as:
Revised: and next human’s decision x ′ ( t + ) , the next belief can be updated as:

Original: [Equation]
Revised: [Equation]

Original: ) where ↵ is a normalizing constant.
Revised: ) where ↵ is a normalizing constant.

Original: Since the belief state is a su  cient statistic, the optimal policy of A ctf is then the solution of a “belief state MDP” over continous space.
Revised: Since the belief state is a su  cient statistic, the optimal policy of A ctf is then the solution of a “belief state MDP” over continous space.

Original: There are several standard methods for solving such MDPs, including witness algorithm Littman (1994), linear support algorithm Cheng (1988), and incremental pruning Cassandra et al. (1997), just to cite a few.
Revised: There are several standard methods for solving such MDPs, including witness algorithm (Littman, 1994), linear support algorithm (Cheng, 1988), and incremental pruning (Cassandra et al., 1997), just to cite a few.

Original: 5. The Trade-O ↵ between Autonomy and Optimality
Revised: 5. The Trade-O ↵ between Autonomy and Optimality

Original: [Table]
Revised: Environment

Original: Optimality
Revised: Optimality

Original: Autonomy A exp
Revised: Autonomy A exp

Original: A ctf
Revised: A ctf

Original: MDPUC −
Revised: MDPUC −

Original: MDPUC
Revised: MDPUC

Original: MDPUC +
Revised: MDPUC +

Original: DSCM −
Revised: DSCM −

Original: DSCM
Revised: DSCM

Original: [Table]
Revised: [Table]

Original: [Table]

Original: We start this section by summarizing in
Revised: We start this section by summarizing in

Original: Table 1 the results discussed so far.
Revised: Table 1 the results discussed so far.

Original: The “optimality” column indicates whether the corresponding agent could obtain the optimal policy (using the counterfactual agent
Revised: The “optimality” column indicates whether the corresponding agent could obtain the optimal policy (using the counterfactual agent

Original: A ctf as the baseline); a check (cross) mark in A exp under “optimality” represents that the experimental agent is (is not) able to achieve the optimal performance in the environment of the corresponding row, compared to its counterfactual counterpart.
Revised: A ctf as the baseline); a check (cross) mark in A exp under “optimality” represents that the experimental agent is (is not) able to achieve the optimal performance in the environment of the corresponding row, compared to its counterfactual counterpart.

Original: In an arbitrary environment, full autonomy (represented by the “autonomy” column) can only be achieved when it is not at the cost of the optimality, i.e., the performance of the experimental
Revised: In an arbitrary environment, full autonomy (represented by the “autonomy” column) can only be achieved when it is not at the cost of the optimality, i.e., the performance of the experimental

Original: A exp and counterfactual agent
Revised: A exp and counterfactual agent

Original: A ctf coincides.
Revised: A ctf coincides.

Original: These results conveys an inherent trade-o ↵ between optimality and autonomy when designing RL systems – while full autonomy is preferable, the agent could potentially achieve better performance by leveraging the human’s capabilities.
Revised: These results conveys an inherent trade-o ↵ between optimality and autonomy when designing RL systems – while full autonomy is preferable, the agent could potentially achieve better performance by leveraging the human’s capabilities.

Original: We model this trade-o ↵ as a constrained transfer of control (TOC) problem such that a system tries to maximize its rewards while repeatedly transferring between an experimental and a counterfactual agent, subject to the total time of the counterfactual agent (using human decision) is no more than  ratio of the total running time.
Revised: We model this trade-o ↵ as a constrained transfer of control (TOC) problem such that a system tries to maximize its rewards while repeatedly transferring between an experimental and a counterfactual agent, subject to the total time of the counterfactual agent (using human decision) is no more than  ratio of the total running time.

Original: To solve this problem, we ﬁrst deﬁne hybrid agents, which combines both the experimental A exp and counterfactual A ctf modes of reasoning.
Revised: To solve this problem, we ﬁrst deﬁne hybrid agents, which combines both the experimental A exp and counterfactual A ctf modes of reasoning.

Original: Deﬁnition 14 (Hybrid Agent)
Revised: Deﬁnition 14 (Hybrid Agent)

Original: A hybrid agent A hyb is an agent following a policy ⇧ hyb = ( ⇡ ○ f a ) ([ 1 ,t ]) and maintaining an internal state A ( t ) ∈ { a 0 , a} ( a 0 for A exp and a 1 for A ctf ).
Revised: A hybrid agent A hyb is an agent following a policy ⇧ hyb = ( ⇡ ○ f a ) ([ 1 ,t ]) and maintaining an internal state A ( t ) ∈ { a 0 , a} ( a 0 for A exp and a 1 for A ctf ).

Original: ( ⇡ ○ f a ) ( t ) is a composite function in the form of ⇡ ( t )
Revised: ( ⇡ ○ f a ) ( t ) is a composite function in the form of ⇡ ( t )

Original: ￿ h ( t ) hyb , x ′ ( t ) , f ( t [Equation] a ￿ h ( t ) hyb ￿￿ , where h ( t ) hyb is the history of the hybrid agent (includes s ([ 1 ,t ]) , x ([ 1 ,t ]) , and partially x ′ ([ 1 ,t ]) ), the function a ( t ) =
Revised: ￿ h ( t ) hyb , x ′ ( t ) , f ( t ) a ￿ h ( t ) hyb ￿￿ , where h ( t ) hyb is the history of the hybrid agent (includes s ([ 1 ,t ]) , x ([ 1 ,t ]) , and partially x ′ ([ 1 ,t ]) ), the function a ( t ) =

Original: f ( t ) a ￿ h ( t ) hyb ￿ decides a value of A ( t ) and ⇡ ( t ) decides the decision x ( t ) as follows:
Revised: f ( t ) a ￿ h ( t ) hyb ￿ decides a value of A ( t ) and ⇡ ( t ) decides the decision x ( t ) as follows:

Original: [Equation]
Revised: [Equation]

Original: A hybrid RL system is deﬁned as the pair ￿
Revised: A hybrid RL system is deﬁned as the pair ￿

Original: M dscm ,
Revised: [Equation]

Original: A hyb ￿ proactively deciding whether it considers the human decision.
Revised: ￿ proactively deciding whether it considers the human decision.

Original: The internal state A ( t ) is a switch variable indicating the current running mode the system is in ( a 0 for experimental or a 1 for counterfactual).
Revised: The internal state A ( t ) is a switch variable indicating the current running mode the system is in ( a 0 for experimental or a 1 for counterfactual).

Original: The human decision x ′ ( t ) is included as an evidence to decide for agent’s decision x ( t ) if A ( t ) = a 1 , and ignored otherwise.
Revised: The human decision x ′ ( t ) is included as an evidence to decide for agent’s decision x ( t ) if A ( t ) = a 1 , and ignored otherwise.

Original: We now consider algorithms to solve for such hybrid systems when the environment is ctf-Markov and full autonomy cannot be achieved (e.g., M mdpuc ).
Revised: We now consider algorithms to solve for such hybrid systems when the environment is ctf-Markov and full autonomy cannot be achieved (e.g., M mdpuc ).

Original: The following result establishes the relation between hybrid systems and MDPs.
Revised: The following result establishes the relation between hybrid systems and MDPs.

Original: Proposition 15 A hybrid system ￿
Revised: Proposition 15 A hybrid system ￿

Original: M mdpuc , A hyb ￿ forms an MDP ￿ S hyb ,
Revised: [Equation] ￿ forms an MDP ￿ S hyb ,

Original: X hyb , T, R ￿ where
Revised: X hyb , T, R ￿ where

Original: S hyb =
Revised: S hyb =

Original: S × X ,
Revised: S × X ,

Original: X hyb =
Revised: X hyb =

Original: X × A , and for any a, x, s, x ′ , s ′ , x ′′ , 
Revised: X × A , and for any a, x, s, x ′ , s ′ , x ′′ , T [Equation] x ′

Revised: [Equation]

Original: T ￿ s,x ′ ￿ ￿ x,a ￿ ￿ s ′ , x ′ ￿ =
Revised: ￿ s ′ [Equation] x ′ [Equation]

Revised: =

Original: P ￿
Revised: P ￿

Original: S ( t + ) X ( t ) =
Revised: S ( t + [Equation]

Original:  x = s ′ , X ( t + ) X ( t ) =
Revised: ( t ) = x = s ′ , X ( t + ) X ( t ) =

Original: x = x ′′ ￿
Revised: x = x [Equation] ￿

Original: S ( t ) =
Revised: S ( t ) =

Original: s, X ( t ) =
Revised: s, X ( t ) =

Original: x ′ ￿ ,
Revised: x ′ ￿ ,

Original: R￿ s,x ′ ￿ ￿ x,a ￿ = E ￿ Y ( t ) X ( t ) = x ￿
Revised: R

Original: S ( t ) = s, X ( t ) = x ′ ￿ .
Revised: [Equation]

Original: This result follows immediately after Prop. 11, as A ( t ) is an auxiliary action which does not directly a ↵ ect the environment.
Revised: This result follows immediately after Prop. 11, as A ( t ) is an auxiliary action which does not directly a ↵ ect the environment.

Original: Prop. 15 allows us to translate the constrained TOC problem to a constrained MDP problem.
Revised: Prop. 15 allows us to translate the constrained TOC problem to a constrained MDP problem.

Original: The optimal policy of A hyb can be computed as a solution to the LP speciﬁed in Eq.
Revised: The optimal policy of A hyb can be computed as a solution to the LP speciﬁed in Eq.

Original: 6 with additional constraints as follows:
Revised: (6) with additional constraints as follows:

Original: [Equation]
Revised: [Equation]

Original: ,a￿ ( i ) ≤  ￿ i,x,a  ￿ x,a ￿ ( i ) (9)  ￿ x,a￿ ( s, x ′ )
Revised: ,x  ￿ x,a￿ ( i ) ≤  ￿ i,x,a  ￿ x,a ￿ ( i ) (9)  ￿ x,a￿ ( s, x ′ )

Original: ￿ ￿ x  ￿ x,a￿ ( s, x ′ ) =
Revised: ￿ ￿ x  ￿ x,a￿ ( s, x ′ ) =

Original: ￿ x ′  ￿ x,a￿ ( s, x ′ ) ￿ ￿ x,x ′  ￿ x,a￿ ( s, x ′ )
Revised: ￿ x ′  ￿ x,a￿ ( s, x ′ ) ￿ ￿ x,x ′  ￿ x,a￿ ( s, x ′ )

Original: (10) ￿ x
Revised: (10) ￿ x

Original:  ￿ x,a ￿ ( s, x ′ )
Revised:  ￿ x,a ￿ ( s, x ′ )

Revised: [Equation]

Original: ￿￿ x,a  ￿ x,a ￿ ( s, x ′ ) =
Revised: ￿ x,a  ￿ x,a ￿ ( s, x ′ ) =

Original: ￿ x,x ′  ￿ x,a ￿ ( s, x ′ )
Revised: ￿ x,x ′  ￿ x,a ￿ ( s, x ′ )

Original: ￿ ￿ x,a,x ′  ￿ x,a ￿ ( s, x ′ ) (11)
Revised: ￿ ￿ x,a,x ′  ￿ x,a ￿ ( s, x ′ ) (11)

Original: Eq.
Revised: Eq.

Original: (9) ensures that for system ￿
Revised: (9) ensures that for system ￿

Revised: M

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ￿ , the total time running in counterfactual mode ( a 1 ) is no more than  ratio of the total running time (discounted so that future visits count less than present ones).
Revised: ￿ , the total time running in counterfactual mode ( a 1 ) is no more than  ratio of the total running time (discounted so that future visits count less than present ones).

Original: Eq.
Revised: Eq.

Original: (10) ensures that for A ( t ) =
Revised: (10) ensures that for A ( t ) =

Original: a 0 , the policy ⇡ does not take the human decision x ′ ( t ) as an input.
Revised: a 0 , the policy ⇡ does not take the human decision x ′ ( t ) as an input.

Original: Eq.
Revised: Eq.

Original: (11) reﬂects the functional constraint that f a is a Markovian policy only depends on the current state s ( t ) .
Revised: (11) reﬂects the functional constraint that f a is a Markovian policy only depends on the current state s ( t ) .

Original: This mathematical program forms a polynomial optimization problem Tuy et al. (1998), which is neither linear nor convex (due to Eqs.
Revised: This mathematical program forms a polynomial optimization problem (Tuy et al., 1998), which is neither linear nor convex (due to Eqs.

Revised: (10) to (11)).

Original: Despite its di  culty, there are several e  cient methods of polynomial optimization that can be used in this case, for example: the RLT method Sherali and Adams (2013), and a SDP relaxation method Lasserre (2001).
Revised: Despite its di  culty, there are several e  cient methods of polynomial optimization that can be used in this case, for example: the RLT method (Sherali and Adams, 2013), and a SDP relaxation method (Lasserre, 2001).

Original: [Figure]
Revised: [Figure]

Original: 6. Applications and Experiments
Revised: 6. Applications and Experiments

Original: In this section, we operationalize the learning algorithms for experimental ( A exp ), counterfactual ( A ctf ), and hybrid ( A hyb ) agents in the context of the medical treatment problem described in Sec. 2.
Revised: In this section, we operationalize the learning algorithms for experimental ( A exp ), counterfactual ( A ctf ), and hybrid ( A hyb ) agents in the context of the medical treatment problem described in Sec. 2.

Original: We assess agents’ performance using the cumulative reward (CR) and the average reward (AR).
Revised: We assess agents’ performance using the cumulative reward (CR) and the average reward (AR).

Original: Overall, we ﬁnd that simulation results conﬁrm that the counterfactual approach achieves a higher expected return, even when human decisions are poor; the polynomial program formulation e ↵ ectively balances between autonomy and optimality, subject to a budget constraint over available human input.
Revised: Overall, we ﬁnd that simulation results conﬁrm that the counterfactual approach achieves a higher expected return, even when human decisions are poor; the polynomial program formulation e ↵ ectively balances between autonomy and optimality, subject to a budget constraint over available human input.

Original: Experiment 1: O ✏ ine Planning.
Revised: Experiment 1: O ✏ ine Planning.

Original: Since M mdpuc is both exp-Markov and ctf-Markov (Thm. 10), the best possible policies of A exp and A ctf can be computed using standard
Revised: Since M mdpuc is both exp-Markov and ctf-Markov (Thm. 10), the best possible policies of A exp and A ctf can be computed using standard

Original: MDP algorithms following the construction given in Prop. 11, labeled as exp and ctf .
Revised: MDP algorithms following the construction given in Prop. 11, labeled as exp and ctf .

Original: The performance of agents A exp and A ctf are compared with the baseline policies described in Sec.
Revised: The performance of agents A exp and A ctf are compared with the baseline policies described in Sec.

Original: 2 ( human and random ).
Revised: 2 ( human and random ).

Original: An experimental policy computed through POMDP algorithms (called exp2 ) is also included.
Revised: An experimental policy computed through POMDP algorithms (called exp2 ) is also included.

Original: Simulations, shown in Fig. 1, support the e  ciency of the counterfactual approach, revealing that (1) both experimental policies obtained by standard MDP ( exp , CR = 1 .× 10 4 ) and POMDP ( exp2 , CR = 1 .× 10 4 ) coincide with random policy (CR = 1 .× 10 4 ); (2) the counterfactual agent A ctf ( ctf , CR = 1 .× 10 4 ) consistently outperforms the experimental agent A exp , even when the human performs worse than random guessing ( human , CR = 4 .× 10 3 ).
Revised: Simulations, shown in Fig. 1, support the e  ciency of the counterfactual approach, revealing that (1) both experimental policies obtained by standard MDP ( exp , CR = 1 .× 10 4 ) and POMDP ( exp2 , CR = 1 .× 10 4 ) coincide with random policy (CR = 1 .× 10 4 ); (2) the counterfactual agent A ctf ( ctf , CR = 1 .× 10 4 ) consistently outperforms the experimental agent A exp , even when the human performs worse than random guessing ( human , CR = 4 .× 10 3 ).

Original: Experiment 2: Online Learning.
Revised: Experiment 2: Online Learning.

Original: We next show that the challenge considered here is not only bounded to o ✏ ine-planning, but appears compounded in the online learning setting when the agent has only an incomplete model of the environment.
Revised: We next show that the challenge considered here is not only bounded to o ✏ ine-planning, but appears compounded in the online learning setting when the agent has only an incomplete model of the environment.

Original: We apply Mormax (Szita and Szepesv´ari, 2010) for experimental and counterfactual agents since M mdpuc is both expMarkov and ctf-Markov.
Revised: We apply Mormax (Szita and Szepesv´ari, 2010) for experimental and counterfactual agents since M mdpuc is both expMarkov and ctf-Markov.

Original: To empirically estimate system dynamics of A ctf (Prop. 11), we follow the counterfactual randomization procedure in (Bareinboim et al., 2015).
Revised: To empirically estimate system dynamics of A ctf (Prop. 11), we follow the counterfactual randomization procedure in (Bareinboim et al., 2015).

Original: We collect trajectories for both experimental and counterfactual agents, respectively, exp and ctf .
Revised: We collect trajectories for both experimental and counterfactual agents, respectively, exp and ctf .

Original: We also deploy a counterfactual agent A ctf where the transition function T and the reward function R are warm-started with 5000 observational samples (labeled as ctf + , following the composition axiom (Pearl, 2000, pp. 229).
Revised: We also deploy a counterfactual agent A ctf where the transition function T and the reward function R are warm-started with 5000 observational samples (labeled as ctf + , following the composition axiom (Pearl, 2000, pp. 229).

Original: That is, for any x, s, x ′ , s ′ , x ′′ ,
Revised: That is, for any x, s, x ′ , s ′ , x ′′ ,

Original: T ￿ s,x ￿ x ￿ s ′ , x ′′ ￿ =
Revised: T ￿ s,x ￿ x ￿ s ′ , x ′′ ￿ =

Original: P ￿
Revised: P ￿

Original: S ( t + )
Revised: S ( t + )

Original: = s ′ ,
Revised: = s ′ ,

Original: X ( t + ) = x ′′ ￿
Revised: X ( t + ) = x ′′ ￿

Original: S ( t ) =
Revised: S ( t ) =

Original: s, X ( t ) =
Revised: s, X ( t ) =

Original: x ￿ , (12)
Revised: x ￿ , (12)

Original: R ￿ s,x ￿ x =
Revised: R ￿ s,x ￿ x =

Original: E ￿
Revised: E ￿

Original: Y ( t ) ￿
Revised: Y ( t ) ￿

Original: S ( t ) =
Revised: S ( t ) =

Original: s, X ( t ) =
Revised: s, X ( t ) =

Original: x ￿ .
Revised: x ￿ .

Original: The two baseline policies previously described, obs and random , are also included.
Revised: The two baseline policies previously described, obs and random , are also included.

Original: The simulations (Fig. 3( a )) support the counterfactual approach.
Revised: The simulations (Fig. 3( a )) support the counterfactual approach.

Original: In fact, there is a signiﬁcant di ↵ erence in rewards experienced by counterfactual agents (AR = 79 .
Revised: In fact, there is a signiﬁcant di ↵ erence in rewards experienced by counterfactual agents (AR = 79 .

Original: 2438 for ctf + , AR = 79 .
Revised: 2438 for ctf + , AR = 79 .

Original: 2837 for ctf ) when compared to the experimental agent exp (AR = 50 . 1900), despite the fact that human (AR = 19 . 1913) performs poorly, even worse than random guessing ( random , AR = 49 . 2212).
Revised: 2837 for ctf ) when compared to the experimental agent exp (AR = 50 . 1900), despite the fact that human (AR = 19 . 1913) performs poorly, even worse than random guessing ( random , AR = 49 . 2212).

Original: In addition, ctf + demonstrates better convergence rate (afterepisodes) compared to ctf (does not converge until 25 episodes).
Revised: In addition, ctf + demonstrates better convergence rate (afterepisodes) compared to ctf (does not converge until 25 episodes).

Original: Experiment 3: Hybrid Planning.
Revised: Experiment 3: Hybrid Planning.

Original: We apply the methods discussed in Sec.
Revised: We apply the methods discussed in Sec.

Original: 5 to obtain the optimal policy for hybrid agents
Revised: 5 to obtain the optimal policy for hybrid agents

Original: A hyb in the constrained TOC setting.
Revised: A hyb in the constrained TOC setting.

Original: Recall that  is a constraint over the ratio between the total time of running the counterfactual approach (using human input) and the total running time of the system.
Revised: Recall that  is a constraint over the ratio between the total time of running the counterfactual approach (using human input) and the total running time of the system.

Original: We compute policies for three hybrid agents with the ratio constraint  set to 0 .
Revised: We compute policies for three hybrid agents with the ratio constraint  set to 0 .

Original: 9, labeled as hyb1 , hyb5 and hyb9 , respectively.
Revised: 9, labeled as hyb1 , hyb5 and hyb9 , respectively.

Original: The two baseline policies exp and ctf are also included for comparison.
Revised: The two baseline policies exp and ctf are also included for comparison.

Original: The simulations (Fig. 3( b )) reveal that the performance of hybrid agents converges to the counterfactual agent as  → 1.
Revised: The simulations (Fig. 3( b )) reveal that the performance of hybrid agents converges to the counterfactual agent as  → 1.

Original: In particular, hyb1 (  = 0 . 1) shows limited performance improvement (CR = 1 .× 10 4 ) over the autonomous approach ( exp , CR = 1 .× 10 4 ), while hyb9 (  = 0 . 9) experiences higher rewards (CR = 1 .× 10 4 ), which is comparable to the counterfactual agent ctf (CR = 1 .× 10 4 ).
Revised: In particular, hyb1 (  = 0 . 1) shows limited performance improvement (CR = 1 .× 10 4 ) over the autonomous approach ( exp , CR = 1 .× 10 4 ), while hyb9 (  = 0 . 9) experiences higher rewards (CR = 1 .× 10 4 ), which is comparable to the counterfactual agent ctf (CR = 1 .× 10 4 ).

Original: Predictably, the performance of hyb (  = 0 .
Revised: Predictably, the performance of hyb (  = 0 .

Original: 5, CR = 1 .× 10 4 ) lies in between hyb1 and hyb9 .
Revised: 5, CR = 1 .× 10 4 ) lies in between hyb1 and hyb9 .

Original: We show the composition graph of experimental and counterfactual modes in Fig.
Revised: We show the composition graph of experimental and counterfactual modes in Fig.

Original: 3( c ); we also includes two hybrid agents with  = 0 .
Revised: 3( c ); we also includes two hybrid agents with  = 0 .

Original: The simulations support that the polynomial optimization reduction worked as expected, where hyb1 ( P ( a) =
Revised: The simulations support that the polynomial optimization reduction worked as expected, where hyb1 ( P ( a) =

Original: P (
Revised: P (

Original: A ( t ) = a) = 0 . 88) and hyb3 ( P ( a)
Revised: A ( t ) = a) = 0 . 88) and hyb3 ( P ( a)

Original: = 0 . 6629) tend to stay in the autonomous mode, while hyb7 ( P ( a) = 0 . 3366) and hyb9 ( P ( a) = 0 .
Revised: = 0 . 6629) tend to stay in the autonomous mode, while hyb7 ( P ( a) = 0 . 3366) and hyb9 ( P ( a) = 0 .

Original: 1239) ask for human involvement frequently; unsurprisingly, hyb5 kept neutral.
Revised: 1239) ask for human involvement frequently; unsurprisingly, hyb5 kept neutral.

Original: 7. Conclusions
Revised: 7. Conclusions

Original: We investigated a novel interactive reinforcement learning setting where the human instructor and the RL agent do not perceive the world in the same fashion, but share the reward and transition functions.
Revised: We investigated a novel interactive reinforcement learning setting where the human instructor and the RL agent do not perceive the world in the same fashion, but share the reward and transition functions.

Original: We showed that the agent could be constrained to ﬁnd a sub-optimal policy if it does not take the human into the loop.
Revised: We showed that the agent could be constrained to ﬁnd a sub-optimal policy if it does not take the human into the loop.

Original: We proposed a new type of interactive agent, which could account for the human in real-time using counterfactual reasoning.
Revised: We proposed a new type of interactive agent, which could account for the human in real-time using counterfactual reasoning.

Original: Our analysis revealed that the counterfactual approach dominates standard methods even if the human instructor performs poorly, possibly worse than random guessing.
Revised: Our analysis revealed that the counterfactual approach dominates standard methods even if the human instructor performs poorly, possibly worse than random guessing.

Original: In fact, the human should be kept “in the loop” as long as it has access to information about the tasks at hand, even after the agent completes its learning and builds a model of the environment.
Revised: In fact, the human should be kept “in the loop” as long as it has access to information about the tasks at hand, even after the agent completes its learning and builds a model of the environment.

Original: To resolve the tension between the autonomy and optimality of the system, we proposed a novel RL task subject to a budget constraint.
Revised: To resolve the tension between the autonomy and optimality of the system, we proposed a novel RL task subject to a budget constraint.

Original: Automated decision-making systems are playing an increasingly prominent role in society, and we hope this work constitutes a step towards a better understanding of the principles underlying human-machine interactions.
Revised: Automated decision-making systems are playing an increasingly prominent role in society, and we hope this work constitutes a step towards a better understanding of the principles underlying human-machine interactions.