Original: Model-Free Model Reconciliation
Revised: Model-Free Model Reconciliation

Original: Anonymous

Original: Abstract
Revised: Abstract

Original: Designing agents capable of explaining complex sequential decisions remains a signiﬁcant open problem in automated decision-making.
Revised: Designing agents capable of explaining complex sequential decisions remains a signiﬁcant open problem in human-AI interaction.

Original: Recently, there has been a lot of interest in developing approaches for generating such explanations for various decision-making paradigms.
Revised: Recently, there has been a lot of interest in developing approaches for generating such explanations for various decision-making paradigms.

Original: One such approach has been the idea of explanation as model-reconciliation .
Revised: One such approach has been the idea of explanation as modelreconciliation .

Original: The framework hypothesizes that one of the common reasons for the user’s confusion could be the mismatch between the user’s model of the  task  and the one used by the system to generate the decisions.
Revised: The framework hypothesizes that one of the common reasons for a user’s confusion could be the mismatch between the user’s model of the agent’s task model and the model used by the agent to generate the decisions.

Original: While this is a general framework, most works that have been explicitly built on this explanatory philosophy have focused on  settings where the model of user’s knowledge is available in a declarative form.
Revised: While this is a general framework, most works that have been explicitly built on this explanatory philosophy have focused on classical planning settings where the model of user’s knowledge is available in a declarative form.

Original: Our goal in this pa- per is to adapt the model reconciliation approach to the cases where such  user models are no longer explicitly provided.
Revised: Our goal in this paper is to adapt the model reconciliation approach to a more general planning paradigm and discuss how such methods could be used when user models are no longer explicitly available.

Original: We present a simple and easy to learn labeling model that can help an explainer decide what information could help achieve model reconciliation between the user and the agent.
Revised: Speciﬁcally, we present a simple and easy to learn labeling model that can help an explainer decide what information could help achieve model reconciliation between the user and the agent with in the context of planning with MDPs.

Original:  Introduction
Revised:  Introduction

Original: A major barrier to integrating AI systems into our daily lives has been their inability to interact and work with us  in an intuitive and explicable manner.
Revised: A signiﬁcant barrier to integrating AI systems into our daily lives has been their inability to interact and work with us humans in an intuitive and explicable manner.

Original: Orchestrating such interactions would require the agents to have the ability to help users in the loop better understand the rationale behind their various actions.
Revised: Orchestrating such interactions would require the agents to have the ability to help users in the loop better understand the rationale behind their various actions.

Original: Thankfully there has been a lot of interest within the AI research community to develop systems capable of holding explanatory dialogue with users and thus help them understand the decisions under question (Miller 2017; Fox et al. 2017).
Revised: Thankfully there has been a lot of effort within the AI research community to develop systems capable of holding explanatory dialogues with users and thus help them understand the decisions under question [David W. Aha and Magazzeni, 2018; Daniele Magazzeni, 2018].

Original: When a user is stumped by an agent’s decision and asks for explanation, two common reasons as to why the user may have trouble understanding the agent’s actions could be due to either (1), the user’s lack of understanding (or even misunderstanding) of the task or (2), because of the user’s inability to understand the consequences of the actions due to theirlimited inferential capacity.
Revised: Such explanatory systems could help users resolve confusions regarding agent decisions that may stem from either a (1) lack of understanding (or even misunderstanding) of the task or (2) from their inferential limitations.

Original: While many earlier works in explanation have generally focused on the latter (cf.
Revised: While many earlier works in explanation have generally focused on the latter (cf.

Original: (Khan et al.  2009; Hayes and Shah 2017; Seegebarth et al. 
Revised: [ Khan et al. , 2009; Hayes and Shah, 2017; Seegebarth et al. , 2012;

Original: 2012; Topin and Veloso 2019)) , there is a growing consensus on the importance of explanatory mechanisms that can help bridge the knowledge asymmetry between the system and the user.
Revised: Topin and Veloso, 2019 ] ), there is a growing consensus on the importance of explanatory mechanisms that can help bridge the knowledge asymmetry between the system and the user.

Original: In particular, this problem has been studied within the context of classical planning and has been referred to as explanation as model-reconciliation (Chakraborti et al. 
Revised: In particular, in explanation as model-reconciliation [ Chakraborti et al. , 2017 ] we had studied the problem of reconciling knowledge asymmetries between the user and the agent within the context of planning.

Original: Most works in this direction have generally looked at cases where the user’s model of the task (i.e their belief about the initial state, the transition dynamics, and the goal) is known beforehand (in a representation scheme comparable to the one used by the agent) and do not match the agent’s model.
Revised: Works in this direction have generally looked at cases where the user’s model of the task (i.e their belief about the initial state, the transition dynamics, and the goal) is known beforehand (in a representation scheme comparable to the one used by the agent) and do not match the agent’s model.

Original: This mismatch means that the user would not be able to correctly evaluate the validity or the optimality of a given plan.
Revised: This mismatch means that the user would not be able to correctly evaluate the validity or the optimality of a given plan.

Original: Thus their explanations consist of information about the agent’s model that the user could incorporate into his/her own model to correctly evaluate the plan in question.
Revised: Thus in this paradigm the explanations consist of information about the agent’s model that the user could incorporate into his/her own model to correctly evaluate the plan in question.

Original: Unfortunately, it is not always possible to have access to such models.
Revised: Unfortunately, it is not always possible to have access to such models.

Original: In the most general case, we are dealing with the user’s model of the agent and hence the user may not be capable of presenting traces or decisions that could be generated from this model.
Revised: In the most general case, we are dealing with the user’s model of the agent and hence the user may not be capable of presenting traces or decisions that could be generated from this model.

Original: Even if the system tries to learn such a representation based on interactions with the user, there is no guarantee that the speciﬁc representational assumptions of the learned model and the vocabulary used would be satisﬁed by the user’s mental model.
Revised: Even if the system tries to learn such a representation based on interactions with the user, there is no guarantee that the speciﬁc representational assumptions of the learned model and the vocabulary used would be satisﬁed by the user’s mental model.

Original: The deﬁnition of explanation as model reconciliation may leave one with the idea that there is no way around it.
Revised: The deﬁnition of explanation as model reconciliation may leave one with the idea that there is no way around it.

Original: How could one ever truly perform effective reconciliation when there exists no user model guiding us to choose the parts of the model, which when revealed to the user will help them correctly evaluate the current decision?
Revised: How could one ever truly perform effective reconciliation when there exists no user model guiding us to choose the parts of the model, which when revealed to the user will help them correctly evaluate the current decision?

Original: Are we left with revealing the entire agent model to the user as the only option?
Revised: Are we left with revealing the entire agent model to the user as the only option?

Original: In this paper, we propose a simple and intuitive way we could still generate minimal explanations in the absence of declarative models.
Revised: In this paper, we propose a simple and intuitive way we could still generate minimal explanations in the absence of declarative models.

Original: We argue that we could drive such explanations by using simple and easy to learn models that can predict how human expectations could be affected by possible explanations (derived completely from information about the agent model) and in fact show how this method could be viewed as a variation of previous approaches that have been put forth to identify explicable behavior.
Revised: We argue that we could drive such explanations by using learned models that can predict how human expectations could be affected by possible explanations (derived completely from information about the agent model) and in fact show how this method could be viewed as a variation of previous approaches that have been put forth to identify explicable behavior.

Original: We willground this discussion within the context of MDPs and start by summarizing how model reconciliation could be achieved in such settings when the user model may be known (Section 4).
Revised: We will start by extending model reconciliation to the more general setting of planning with Markov Decision Processes (Section 4).

Original: The rest of the paper will investigate how these ideas could be used when the human mental model of the task is unavailable, and will formulate a learning problem that allows us to learn simple models that could be used to identify minimal explanations (Section 5).
Revised: The rest of the paper will investigate how these ideas could be used when the human mental model of the task is unavailable, and will formulate a learning problem that allows us to learn simple models that could be used to identify minimal explanations (Section 5).

Original: We will evaluate our method on a set of standard MDP benchmarks and perform user studies to validate its viability.
Revised: Finally, we will evaluate our method on a set of standard MDP benchmarks and perform user studies to validate its viability (Section 6).

Original:  Background
Revised:  Background

Original: Figure 1 presents a general schematic representation for explanation as model reconciliation.
Revised: Figure 1 presents a general schematic representation for explanation as model reconciliation.

Original: The automated agent (henceforth referred to as robot) starts with a model M R for coming up with a decision π (where depending on the context, π may be a single action, plan, policy or a label).
Revised: The automated agent (henceforth referred to as robot) starts with a model M R that can be used to generate a decision π (where depending on the context, π may be a single action, plan, policy or a program).

Original: The model is then projected into a space of interpretable models ˆ M R such that some desirable properties of the decision (say the optimality) may be conserved.

Original: In this setting,ˆ M H corresponds to the human’s preconceived notions about the robot model captured in the same interpretable space.
Revised: In this setting, M Rh corresponds to the human’s preconceived notions about the robot model.

Original: The explainer’s job then becomes providing information about the modelˆ M R , such that the updated human model can correctly evaluate the validity of the robot decisions.
Revised: The explainer’s job then becomes providing information about the model M R , such that the updated human model can correctly evaluate the validity of the robot decisions.

Original: Note that in this case, the robot could have chosen to provide the entire model, but for most realistic tasks, such models could be quite large, so dumping the entire model could be both unnecessary and impractical.
Revised: In this case, the robot could have chosen to provide the entire model, but for most realistic tasks, such models could be quite large, so dumping the entire model could be both unnecessary and impractical.

Original: It’s also well known that people generally prefer explanations that are in line with their beliefs (Miller 2017).
Revised: It’s also well known that people generally prefer explanations that are selective [ Miller, 2018; Lombrozo, 2006 ] .

Original: Thus the users would be happier with explanations that asks them to update a subset of beliefs as opposed to a complete update.
Revised: Thus the users would be happier with explanations that asks them to update a subset of beliefs as opposed to a complete update.

Original:  In (Chakraborti et al. 2017), M R was a classical planning model hence inherently interpretable and thus the reconciliation is performed with theoriginal domain model. This idea could be applied beyond just planning models, for example, one could understand the explanation methodology used by
Revised: Note that M R need not be the original agent model, but rather some abstraction/approximation of the underlying robot model (that conserves some desired property of the decision like optimality or validity). In [Chakraborti et al. , 2017] where model reconciliation was ﬁrst introduced, M R was a classical planning model hence inherently interpretable and thus the reconciliation could easily be achieved, but the idea could be applied beyond just planning models.

Original:  LIME (Ribeiro et al. 2016) as being a special case of model reconciliation.
Revised: For example, one could understand the explanation methodology used by LIME [Ribeiro et al. , 2016 ] as being a special case of model reconciliation.

Original: In their case, they assume the human model is empty andˆ M R is  automatically generated for each decision using a set of predeﬁned features.
Revised: In their case, they assume the human model is empty and M R is an approximation of the underlying decision model is automatically generated for each decision using a set of predeﬁned features.

Original: In this work, we will be looking at the agents that use discounted inﬁnite horizon Markov Decision Processes (or MDPs) (Russell and Norvig 2003) as the decision making framework.
Revised: In this work, we will be looking at the agents that use discounted inﬁnite horizon Markov Decision Processes (or MDPs) [Russell and Norvig, 2003] as the decision making framework.

Original: Each MDP M is deﬁned by a tuple (cid:104) S, A, T, R, γ, µ (cid:105) , where the S provides the set of possible atomic states, A deﬁnes the set of actions, T is the transition function, R the reward, γ the discounting factor (where 0 ≤ γ < 1 ) and µ corresponds to the distribution of possible initial states.
Revised: Each MDP M is deﬁned by a tuple (cid:104) S, A, T, R, γ, µ (cid:105) , where the S provides the set of possible atomic states, A deﬁnes the set of actions, T is the transition function, R the reward, γ the discounting factor (where 0 ≤ γ < 1 ) and µ corresponds to the distribution of possible initial states.

Original: T : S ×
Revised: T : S ×

Original: A × S → [0 , 1] provides the probability that for given state s ∈ S , the execution of an action a would induce a transition to a new state s (cid:48) , and R : S ×
Revised: A × S → [0 , 1] provides the probability that for given state s ∈ S , the execution of an action a would induce a transition to a new state s (cid:48) , and R : S ×

Original: A × S → R deﬁnes the reward corresponding to this transition.
Revised: A × S → R deﬁnes the reward corresponding to this transition.

Original: The solution concept in MDP takes the form of a policy π that maps each state to a potential action.
Revised: The solution concept in MDP takes the form of a policy π that maps each state to a potential action.

Original: A policy is said to be optimal for M (denoted as π ∗M ) if there exists no other policy that dominates the given policy in terms of the expected value of states.
Revised: A policy is said to be optimal for M (denoted as π ∗M ) if there exists no other policy that dominates the given policy in terms of the expected value of states (where the value of a state s under a policy π for a model M is denoted as V π M ( s ) ).

Original: Executing the policy in a state results in a sequence of state action state tuples called execution trajectory or simply a trajectory , denoted as τ = (cid:104) ( s 1 , a 1 , s 2 ) , ..., ( s n − 1 , a n − 1 , s n ) (cid:105) and we will use P M ( τ | π ) to denote the probability of sampling the given trajectory τ for a policy π  .
Revised: Executing the policy in a state results in a sequence of state action state tuples called execution trajectory or simply a trajectory , denoted as τ = (cid:104) ( s 1 , a 1 , s 2 ) , ..., ( s n − 1 , a n − 1 , s n ) (cid:105) and we will use P M ( τ | π ) to denote the probability of sampling the given trajectory τ for a policy π in model M .

Original: In the explanatory setting we are interested in, the robot uses a model M r = (cid:104) S, A, T r , R r , γ r (cid:105) of the task to come up with the policy to act on.
Revised: In the explanatory setting we are interested in, the robot uses a model M R = (cid:104) S, A, T R , R R , γ R (cid:105) of the task to come up with the policy to act on.

Original: For now we will assume this
Revised: For now we will assume this

Original: MDP already deﬁnes an interpretable model and the human uses a model M h = (cid:104) S, A, T h , R h , γ h (cid:105) to evaluate it (we will relax this assumption in later sections).
Revised: MDP already deﬁnes an interpretable model and the human uses a model M Rh = (cid:104) S, A, T Rh , R Rh , γ Rh (cid:105) to evaluate it (we will relax this assumption in later sections).

Original: Now the task ahead of us will be to  reinterpret the ideas of inexplicability and the idea of model reconciliation that was deﬁned in (Chakraborti et al. 2017) into the context of MDP, but before we go into the detail, let us consider a simpleexplanatory scenario.
Revised: Now the task ahead of us will be to formulate how we could still identify minimal information that could resolve user confusion when M Rh , but before we can do that we need to reinterpret the ideas of inexplicability and the idea of model reconciliation that was deﬁned in [ Chakraborti et al. , 2017 ] in the context of MDPs and we will start by considering a simple scenario.

Original:  Illustrative Example
Revised:  Illustrative Example

Original: Consider a warehouse scenario, where a robot is tasked with moving packages from racks and dropping them off at the dispatch chute.
Revised: Consider a warehouse scenario, where a robot is tasked with moving packages from racks and dropping them off at the dispatch chute.

Original: The robot is powered by a battery pack that can be recharged by visiting its docking station.
Revised: The robot is powered by a battery pack that can be recharged by visiting its docking station.

Original: The docking station also doubles as a quality assurance station that the robot needs to visit whenever it picks up a box labeled # (which means the box is fragile).
Revised: The docking station also doubles as a quality assurance station that the robot needs to visit whenever it picks up a box labeled # (which means the box is fragile).

Original: The robot’s operations are mostly deterministic, apart from a small probability of slipping (0.25) in some cells, that could leave the robot in the same position.
Revised: The robot’s operations are mostly deterministic, apart from a small probability of slipping (0.25) in some cells, that could leave the robot in the same position.

Original: Now suppose the warehouse has just hired a new parttime employee to oversee the operations.
Revised: Now suppose the warehouse has just hired a new part-time employee to oversee the operations.

Original: The employee is just getting used to this new setting and is puzzled by the robot’s decision to once in a while take a detour from the drop-off activity and visit a speciﬁc position of the factory ﬂoor (which is, in fact, the docking location).
Revised: The employee is just getting used to this new setting and is puzzled by the robot’s decision to once in a while take a detour from the drop-off activity and visit a speciﬁc position of the factory ﬂoor (which is, in fact, the docking location).

Original: If we wished the robot to be explainable, then it would need to be capable of helping the employee better understand the characteristics of its operations (i.e achieve some form of model reconciliation).
Revised: If we wished the robot to be explainable, then it would need to be capable of helping the employee better understand the underlying model used by the robot (i.e achieve some form of model reconciliation).

Original: Given the fact that the robot may not have an exact model of the user, one way to achieve this could be by providing robot’s entire model to the user.
Revised: Given the fact that the robot may not have an exact model of the user, one way to achieve this could be by providing robot’s entire model to the user.

Original: Unfortunately, this could easily overwhelm the user.
Revised: Unfortunately, this could easily overwhelm the user.

Original: Another possibility could be to allow the user to specify which robot actions appear inexplicable, and focus on providing facts relevant to those actions.
Revised: Another possibility could be to allow the user to specify which robot actions appear inexplicable, and focus on providing facts relevant to those actions.

Original: This explanation may still prove to be quite verbose and may in fact not help resolve their confusion.
Revised: This explanation may still prove to be quite verbose and may in fact not help resolve their confusion.

Original: For example, imagine a case where the robot is visiting the station to recharge its batteries and the human says that the visit action is inexplicable.
Revised: For example, imagine a case where the robot is visiting the station to recharge its batteries and the human says that the visit action is inexplicable.

Original: Now even if the robot mentions that visiting the station recharges it, the employee may still be confused if they are under the incorrect assumption that the robot is operating on full battery.
Revised: Now even if the robot mentions that visiting the station recharges it, the employee may still be confused if they are under the incorrect assumption that the robot is operating on full battery.

Original: Similarly, if the human had expected the robot to go to the docking station due to some confusion regarding the box codes, the human may mark the robot decision to not go to the dropoff as being inexplicable and the explanations that could resolve the confusion may have little to do with that speciﬁc action marked as inexplicable.
Revised: Similarly, if the human had expected the robot to go to the docking station due to some confusion regarding the box codes, the human may mark the robot decision to not go to the dropoff as being inexplicable and the explanations that could resolve the confusion may have little to do with that speciﬁc action marked as inexplicable.

Original:  Explanation as Model Reconciliation For MDPs
Revised:  Explanation as Model Reconciliation For MDPs

Original: In this paper, we will assume that the human and robot models (captured asMDPs) differ solely on the transition probabilities, the reward function and the discounting factor.
Revised: In this setting the human and robot models are captured as

Original: This means that we can characterize both models by the tuple θ( M ) = (cid:104) θ T , θ R , θ γ , θ µ (cid:105) , where the θ T provides the set of parameters that deﬁnes the transition probabilities P ( .
Revised: MDPs deﬁned over the same set of states and thus we can characterize both models by the tuple θ = (cid:104) θ T , θ R , θ γ , θ µ (cid:105) , where the θ T provides the set of parameters that deﬁnes the transition probabilities P ( .

Original: | s, a ) , while θ R the parameters corresponding to the reward function, θ γ the parameters corresponding to the discount factor and θ µ the parameters for the initial state distribution.
Revised: | s, a ) , while θ R the parameters corresponding to the reward function, θ γ the parameters corresponding to the discount factor and θ µ the parameters for the initial state distribution.

Original: For simple MDP models with atomic states, θ T contains parameters of a categorical distribution for each transition ( θ µ will contain similar parameters), θ R contains the reward associated with each transition (an (cid:104) s, a, s (cid:48) (cid:105) tuple) and θ γ just contains the value of the discount factor. 
Revised: For simple MDP models with atomic states, θ T contains parameters of the categorical distribution for each transition ( θ µ will contain similar parameters for the initial state distribution), θ R contains the reward associated with each transition (an (cid:104) s, a, s (cid:48) (cid:105) tuple) and θ γ just contains the value of the discount factor. The speciﬁc instantiations of the parameters for each model M is captured as θ ( M ) .

Original: For simplicity, we will denote each of the unique parameters in the larger set using indexes.
Revised: For simplicity, we will denote each of the unique parameters in the tuple θ using indexes.

Original: For example, θ s,aT  , will correspond to the parameters for the distribution P ( .
Revised: For example, θ s,aT ( M R ) , will correspond to the parameters for the distribution P ( .

Original: | s, a )  .
Revised: | s, a ) for the model M R .

Original: Mathematically, a model reconciliation operation will be captured as a function E (cid:104)M  h , M r (cid:105) : 2 Θ → M that takes in a set of model parameters and generates a new version of the model M h where the set of speciﬁed parameters will be set to values from M r .
Revised: If we use M to capture the set of all possible models and Θ = θ T × θ R × θ γ × θ µ , then model reconciliation operation can be captured as a function E (cid:104)M R h , M R (cid:105) : 2 Θ → M that takes in a set of model parameters and generates a new version of the model M Rh where the set of speciﬁed parameters will be set to values from M R .

Original: For example, ˆ M =E
Revised: For example, ˆ M =

Original: [Table]

Original: r [Table] [Table] ) will be a new model such that θ ( ˆ M ) will be identical to θ ( M h ) , except that θ s 1 ,a T ˆ M , will be equal to θ s 1 ,a T r .
Revised: E (cid:104)M Rh , M R (cid:105) ( θ s 1 ,a T ) will be a new model such that θ ( ˆ M ) will be identical to θ ( M Rh ) , except that θ s 1 ,a T ˆ M , will be equal to θ s 1 ,a T R .

Original: Practically, the model reconciliation operation corresponds to the robot informing the human about some part of its model.
Revised: Practically, the model reconciliation operation corresponds to the robot informing the human about some part of its model.

Original: We can assume that this communication incurs some cost and we can deﬁne a cost function C : 2 Θ → R that maps a given set of a threshold to a cost.
Revised: This communication could incur cost and we can capture this by using the cost function C : 2 Θ → R that maps a given set of a threshold to a cost.

Original: Now the question we need to ask is whether the agent is trying to explain its policy or if it is trying to explain some behavior (i.e an execution trace).
Revised: Now the question we need to ask is whether the agent is trying to explain its policy or if it is trying to explain some behavior (i.e an execution trace).

Original: Most of the earlier work that looks at model reconciliation explanation (cf.
Revised: Most of the earlier work that looks at model reconciliation explanation (cf.

Original: (Chakraborti et al.  2017; Sreedharan et al.  2018a; 2018b)) has looked at sequential plans and has generally ignored this differentiation and treated the problem of explaining plans to be same as that of explaining behavior.
Revised: [Chakraborti et al. , 2017; Sreedharan et al. , 2018a; 2018b]) has looked at sequential plans and has generally ignored this differentiation and treated the problem of explaining plans to be same as that of explaining behavior.

Original: In general, a given plan or policy compactly represents a set of possible behaviors and the choice of explaining behavior vs explaining the plans/policies could affect the content of the explanation being given.
Revised: In general, a given plan or policy compactly represents a set of possible behaviors and the choice of explaining behavior vs explaining the plans/policies could affect the content of the explanation being given.

Original: For example, when explaining policies there is the additional challenge of presenting the entire policy to the user and the explainer may need to justify action choices for extremely unlikely states or contingencies.
Revised: For example, when explaining policies there is the additional challenge of presenting the entire policy to the user and the explainer may need to justify action choices for extremely unlikely states or contingencies.

Original: On the other hand, when explaining a given set of behaviors the explainer needs to only justify their action choices for cases they actually witnessed.
Revised: On the other hand, when explaining a given set of behaviors the explainer needs to only justify their action choices for cases they actually witnessed.

Original: For example, when explaining traces from the warehouse scenario, given the small probability of slipping, the robot may never have to mention what to do when it slips, but on the other hand if we are dealing with full policies, the agent may need to talk about the states where the robot is in the slipped positions and they need to get up from that position and move on.
Revised: For example, when explaining traces from the warehouse scenario, given the small probability of slipping, the robot may never have to mention what to do when it slips, but on the other hand if we are dealing with full policies, the agent may need to talk about the states where the robot is in the slipped positions and they need to get up from that position and move on.

Original: Explaining policies or plans becomes more relevant when we consider explanatory dialogues where the agent and the user are trying to jointly come to agreement on what policy/plans to follow (eg: decision support systems), while the latter may be more useful when the user is observing some agent operating in an environment.
Revised: Explaining policies or plans becomes more relevant when we consider explanatory dialogues where the agent and the user are trying to jointly come to agreement on what policy/plans to follow (eg: decision support systems), while the latter may be more useful when the user is observing some agent operating in an environment.

Original: With respect to policies, we assume that the user is presented with the entire policy.
Revised: With respect to policies, we assume that the user is presented with the entire policy.

Original: A given policy is said to be explicable to the human, if the policy is optimal for the human model.
Revised: A given policy is said to be explicable to the human, if the policy is optimal for the human model.

Original: Therefore the goal of the explainer becomes that of ensuring the optimality of the given policy
Revised: Therefore the goal of the explainer becomes that of ensuring the optimality of the given policy

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A set of parameters θ E corresponds to a complete policy explanation for the given robot policy π ∗M r , if the policy is also optimal for E (cid:104)M h , M r (cid:105) ( θ E ) and is said to be the minimally complete policy explanation if there exists no other complete explanation θ E (cid:48) , such that, C ( θ E (cid:48) ) <
Revised: A set of parameters θ E corresponds to a complete policy explanation for the given robot policy π ∗M R , if the policy is also optimal for E (cid:104)M Rh , M R (cid:105) ( θ E ) and is said to be the minimally complete policy explanation if there exists no other complete explanation θ E (cid:48) , such that, C ( θ E (cid:48) ) <

Original: C ( θ E )
Revised: C ( θ E )

Original: Finding a complete policy explanation is relatively straightforward (the set of all parameters automatically meets this requirement).
Revised: Finding a complete policy explanation is relatively straightforward (the set of all parameters automatically meets this requirement).

Original: The more challenging case becomes that of ﬁnding the minimal or the cheaper explanations.
Revised: The more challenging case becomes that of ﬁnding the minimal or the cheapest explanations i.e. the minimally complete explanations.

Revised: Such minimally complete explanations can be calculated by adopting a search strategy similar to [ Chakraborti et al. , 2017 ] .

Revised: The search can start at the human model and try to ﬁnd the minimal number of parameters that needs to be updated in the human model for the current policy to become optimal.

Revised: Similar to generating minimally complete explanations, i.e, we can also generate monotonic explanations (i.e explanations where no further information about parameters in the robot model can affect the optimality of the current plan).

Revised: In the case of policies, we can also describe explicable planning and balancing cost of explanations with that of choosing policies that are inherently explicable, where inexplicablity score ( I E ) of a policy π is deﬁned as

Revised: [Equation]

Original:  For explaining behavior, we will look at the simplest case, namely the agent needs to explain a set of behaviors that the user has just observed.
Revised: Where π ∗ is the optimal policy in the human model. Explicable planning thus becomes the problem of choosing policies that minimize inexplicability score [ Kulkarni et al. , 2016 ] , while minimizing the potential loss in optimality due to the policy choice (since the most explicable plan may not be an optimal policy). Balanced planning, as studied in [ Chakraborti et al. , 2019; Sreedharan et al. , 2019 ] , proposes going one step further and also takes into account possible savings in inexplicability score that can be achieved by providing explanation (while incurring additional cost of communicating the required explanations). For explaining behavior, we will look at the simplest case, namely the agent needs to explain a set of behaviors that the user has just observed.

Original: We will assume that the observer has full observability of the state and is seeing the robot behavior for the ﬁrst time.
Revised: We will assume that the observer has full observability of the state and is seeing the robot behavior for the ﬁrst time.

Original: In such a setting, a given trace τ would appear explicable to the user if it could be sampled from the given MDP policy  or more generally, i.e P M H ( τ | π ) > δ , where δ is some small threshold
Revised: In such a setting, a given trace τ would appear explicable to the user if it could be sampled from their expected MDP policy (i.e a policy optimal in their model) or more generally, i.e P M Rh ( τ | π ) > δ , where δ is some small threshold.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: A set of parameter θ E corresponds to a complete behavior explanation for a set of traces T = { τ 1 , ...
Revised: A set of parameters θ E corresponds to a complete behavior explanation for a set of traces T = { τ 1 , ...

Original: τ n } , if ∀ τ ∈ T , ∃ π such that P E (cid:104)M h, M r (cid:105) ( θ E ) ( τ | π ) > δ and π is an optimal policy for the model P E (cid:104)M [Equation]
Revised: τ n } , if ∀ τ ∈ T , ∃ π such that P E (cid:104)M Rh, M R (cid:105) ( θ E ) ( τ | π ) > δ and π is an optimal policy for the model P E (cid:104)M Rh, M R (cid:105) ( θ E ) .

Original: The explanation is said to be the minimally complete behavior explanation if there exists no other complete explanation θ E (cid:48) , such that, C (θ E (cid:48) ) <
Revised: The explanation is said to be the minimally complete behavior explanation if there exists no other complete explanation θ E (cid:48) , such that, C (

Original: C ( θ E )
Revised: [Equation]

Original: Note that given the above deﬁnition,  it may notalways be possible to ﬁnd a complete explanation, as the trace may genuinely contain low probability transitions.
Revised: Note that given the above deﬁnition, if δ is set very high it may not be possible to ﬁnd a complete explanation, as the trace may genuinely contain low probability transitions.

Revised: In this work we will assume δ to be zero.

Original: While model reconciliation could be an important component of either policy or behavior explanation, the applicability of the model reconciliation explanations on its own for policies are limited by the fact that in all but states with the smallest state space, the user would have trouble just go- ing over the entire policy.
Revised: While model reconciliation could be an important component of either policy or behavior explanation, the applicabil- ity of the model reconciliation explanations on their own for policies is limited by the fact that in all but problems with the smallest state spaces, the user would have trouble going over the entire policy.

Original: Thus in these settings, explanatory systems would need to also utilize policy approximation  methods, then allow users the ability to drill down on policy details as required.
Revised: Thus in these settings, explanatory systems would need to also utilize policy approximation or summarization methods, then allow users the ability to drill down on policy details as required.

Original: Sinceone of our  goals is to focus on developing approaches that allow us to generate model reconciliation explanations without explicitly deﬁned user models, the rest of the paper will mostly focus on behavior explanation.
Revised: Since our main goals was to focus on developing approaches that allow us to generate model reconciliation explanations without explicitly deﬁned user models, the rest of the paper will mostly focus on behavior explanation.

Original: In section 8, we will have a brief discussion on how these methods could potentially be extended to policy explanation scenarios.
Revised: In Section 8, we will have a brief discussion on how these methods could potentially be extended to policy explanation scenarios.

Original:  Explaining Without Access to Human Mental Models
Revised:  Explaining Without Explicit Human Mental Models

Original: Now we will look at how we could identify cheaper complete behavior explanations when the human model is unknown.
Revised: Now we will look at how we can identify cheap complete behavior explanations when the human model is unknown.

Original: We will go one step further from identifying not only the parameters that need to be explained, but also capturing the right modality/abstractions to present the information about the parameters.
Revised: We will go one step further from identifying not only the parameters that need to be explained, but also capturing the right modality/abstractions to present the information about the parameters.

Original: That is, we will no longer assume that the human is using a full MDP model to come up with their decisions.
Revised: That is, we will no longer assume that the human is using a full MDP model to come up with their decisions.

Original: Instead,we will assume that the robot has access to a set of explanatory messages M = { m 1 , m 2 , ..., m n } that can be presented to the user.
Revised: Instead, the robot starts with a set of explanatory messages Ψ = { m 1 , m 2 , ..., m n } that can be presented to the user.

Original: Where the messages corresponds to a set of parameter values (the parameters corresponding to a set of messages { m 1 , ..m k } is denoted as E ( { m 1 , ..m k } ) ) of the model as captured in some abstraction of this model and has a corresponding cost ( C ) associated with it.
Revised: Where the messages correspond to a set of parameter values (the parameters corresponding to a set of messages { m 1 , ..m k } is denoted as E ( { m 1 , ..m k } ) ) of the model as captured in some abstraction of this model and has a corresponding cost ( C ) associated with it.

Original: The abstractions to consider may depend on the speciﬁc scenario and the previous information about the intended users (laypeople vs. experts).
Revised: The abstractions to consider may depend on the speciﬁc scenario and the previous information about the intended users (laypeople vs. experts).

Original: Some simple possibilities may be to consider qualitative models (say non-deterministic ones instead of stochastic) and considering state abstractions the given task.
Revised: Some simple possibilities may be to consider qualitative models (say non-deterministic ones instead of stochastic) and considering state abstractions the given task.

Original: Note that, technically E ( M ) , need not span the set of all possible model parameters, but could rather be limited to a subset of parameters identiﬁed to be relevant to the given problem.
Revised: Note that, technically E (Ψ) , need not span the set of all possible model parameters, but could rather be limited to a subset of parameters identiﬁed to be relevant to the given problem.

Original: One possible way may be to consider variations of explanation techniques like MSE (Khan et al. 2009) to identify set of possible factors that affect the optimality of each action.
Revised: One possible way may be to consider variations of explanation techniques like MSE [Khan et al. , 2009] to identify set of possible factors that affect the optimality of each action.

Original: In Figure 2, the subﬁgure (c) shows a set of possible explanatory messages for the warehouse domain, that consists of each parameter mapped to some english statement.
Revised: In Figure 2, the subﬁgure (c) shows a set of possible explanatory messages for the warehouse domain, that consists of each parameter mapped to some english statement.

Original: For models captured using factored representations that use relational or propositional variables, such statements could be easily generated using templates (cf.
Revised: For models captured using factored representations that use relational or propositional ﬂuents, such statements could be easily generated using templates (cf.

Original: (Hayes and Shah 2017)).
Revised: [Hayes and Shah, 2017]).

Original: Given this setting, we will now make two simplifying assumptions, namely, (1) The order in which the explanatory messages are presented does not matter (2) We have access to a set of observers with similar models and they share this model with the target user.
Revised: Given this setting, we will now make some simplifying assumptions, namely, (1) the order in which the explanatory messages are presented does not matter (2) we have access to a set of observers with similar models and they share this model with the target user (3) the robot is viewing the task at the same level or at a more detailed level of granularity than the user and (4) the user and robot have some shared vocabulary in regards to the task.

Revised: While assumption (1) is easily met since we are mostly dealing with model information and (4) is a prerequisite for most explanatory approaches, in section 8 we will discuss how we can possibly relax requirements (2) and (3).

Original: Now our goal is to learn a predictive model that is able to predict whether a given user would ﬁnd a given (cid:104) s, a, s (cid:105) tuple explicable and how the user’s perception changes with the given explanatory messages.
Revised: Now our goal is to learn a predictive model that is able to predict whether a given user would ﬁnd a given (cid:104) s, a, s (cid:105) tuple explicable and how the user’s perception changes with the given explanatory messages.

Original: For example, at the beginning of an episode the user may be presented with the following explanatory messages, ˆ M = { m 1 =
Revised: For example, at the beginning of an episode the user may be presented with the following explanatory messages, ˆΨ = { m 1 =

Original: “Robot slips with probability 0.25 at grey cells” } , which corresponds to the fact that P ( s i | a, s i )
Revised: “Robot slips with probability 0.25 at grey cells” } , which corresponds to the fact that P ( s i | a, s i )

Original: = 0 .
Revised: = 0 .

Original: 25 , for all states s i where the feature grey cell is true and for all actions a .
Revised: 25 , for all states s i where the feature grey cell is true and for all actions a .

Original: Now the user will be presented with a sequence of transitions, say(cid:104) (1 , 2) , right, (2 , 2) (cid:105) and asked whether the transition wasexplicable or not.
Revised: Now the user will be presented with a sequence of transitions, say , right and asked whether the transition was or not.

Original: Then the tuple(cid:104)(cid:104) (1 , 2) , right ,(2 , 2) (cid:105) , { m 1 } , l 1 (cid:105) , where l1 is the label assigned by the user to the transition, becomes input to our learning method [Equation]
Revised: Then the tuple right , where l label signed by the user to the transition, becomes input to our learning method.

Original: The exact function, we would want to learn would be
Revised: The exact function that we would want to learn would be

Revised: L

Original: [Equation]
Revised: [Equation]

Original: otherwise
Revised: s, a

Original:  Note that this is a modiﬁed version of the sequential model  introduced in (Zhang et al. 2017) for identifying whether a given plan is explicable or not.
Revised: [Equation] [Equation] Note that this is a modiﬁed version of the sequential model we introduced in [Zhang et al. , 2017] for identifying whether a given plan is explicable or not.

Original: Though our methods vary in some signiﬁcant aspects, namely, (1) We allow for the possibility that the explicability of the actions/traces could be affected by explanations provided by the system; (2) We no longer use labels of high level tasks as a proxy for the explicability of the trace.
Revised: Though our methods vary in some signiﬁcant aspects, namely, (1) we allow for the possibility that the explicability of the actions/traces could be affected by explanations provided by the system; (2) we no longer use labels of high level tasks as a proxy for the explicability of the trace.

Original: Instead, we just use a simple binary label on whether the transition is explicable or not; (3) We no longer consider sequence models but rather a much simpler labeling model that maps a single transition to the explicability label.
Revised: Instead, we just use a simple binary label on whether the transition is explicable or not; (3) we no longer consider sequence models but rather a much simpler labeling model that maps a single transition to the explicability label.

Original: We argue that in cases where the human is markovian on the same set of features as the agent, this rather simpler modelsufﬁces.
Revised: We argue that in cases where the human is markovian on the same set of features as the agent, this rather simpler model

Original: It is also important that our learning approach is more principled than the ones studied in (Zhang et al. 2017), since in their case to build a balanced dataset (of explicable and inexplicable plans), they would need to uniformly sample through the entire plan space (an extremely hard endeavour with no obvious known approaches), while we stick to traces generated from the optimal policy and only need to randomly generate possible sets of explanatory messages, which is clearly a smaller set.
Revised: Itisalso important that our learning approach is more tractable than the ones studied in [Zhang et al. , 2017], since in their case to build a balanced dataset (of explicable and inexplicable plans), they would need to uniformly sample through the entire plan space (an extremely hard endeavour with no obvious known approaches), while we stick to traces generated from the optimal policy and only need to randomly generate possible sets of explanatory messages, which is clearly a smaller set.

Original: Once we have learned an approximation of the above labeling function ˆ L , the problem of explanation generation for a trace τ  becomes that of ﬁnding the subset of M that balances the cost of communication with the reduction in the inexplicability of the given trace, i.e arg min[Equation] M
Revised: Once we have learned an approximation of the above labeling function ˆ L , the problem of explanation generation for a trace τ = (cid:104) s 0 , a 0 , s 1 , ..., s n , a n , s n +1 (cid:105) becomes that of ﬁnding the subset of Ψ that balances the cost of communication with the reduction in the inexplicability of the given trace, i.e arg min

Original: [Equation]
Revised: [Equation]

Original: Where ˆ M is a subset of M and α is some scaling factor that balances the cost of explanation with the number of inexplicable transitions for a given trace.
Revised: Where ˆΨ is a subset of Ψ and α is some scaling factor that balances the cost of explanation with the number of inexplicable transitions for a given trace.

Original:  Evaluation
Revised:  Evaluation

Revised: The success of the approach described above would be directly dependent on whether we can learn high accuracy labeling models.

Revised: Once we have access to such a model, we could be quite conﬁdent in our ability to generate useful explanation (provided the user’s model is the same as the one the labeler was trained on) and identifying the best explanation becomes a matter of just searching for the required subset of messages that minimizes the objective deﬁned in section 5.

Revised: So to evaluate the method our focus was on identifying if it was possible to learn high accuracy models.

Original: For evaluation, we validated our approach on both simulations and on data collected from users. 
Revised: We validated our approach on both simulations and on data collected from users. 6. Evaluation on Simulated Data

Original: For simulations, we used  slightly modiﬁed versions of the Taxi domain (Dietterich 1998) (of size 6*6), the Four rooms domains (Sutton et al. 1999) (of size 9*9) and the warehouse scenario (of size 9*9) described before.
Revised: For simulations, we used a slightly modiﬁed versions of the Taxi domain [ Dietterich, 1998 ] (of size 6*6), the Four rooms domain [ Sutton et al. , 1999 ] (of size 9*9) and the warehouse scenario (of size 9*9) described before (implemented using the SimpleRL framework [ Abel, 2019 ] ).

Original: For each domain, we start with an MDP instance (henceforth referred to as the robot model) and then create a space of possible user models by identifying a set of possible values for each MDP parameter.
Revised: For each domain, we start with an MDP instance (henceforth referred to as the robot model) and then create a space of possible user models by identifying a set of possible values for each MDP parameter.

Original: For example, in the taxi domain the parameters include position of the passengers, their destination, the step cost, discounting etc.., for the Four rooms this included the goal locations, locations with negative rewards, discounting, step cost, slip probability, etc.., and ﬁnally for the warehouse, the position of the box, the position of station #1, the step cost, slipping probabilities and the discounting factors were selected as potential parameters that can be updated.
Revised: For example, in the taxi domain the parameters include position of the passengers, their destination, the step cost, discounting etc., for the Four rooms this included the goal locations, locations with negative rewards, discounting, step cost, slip probability, etc., and ﬁnally for the warehouse, the position of the box, the position of station #1, the step cost, slipping probabilities and the discounting factors were selected as potential parameters that can be updated.

Original: In this setting, we assume that there exists a single explanatory message for each possible parameter.
Revised: In this setting, we assume that there exists a single explanatory message for each possible parameter.

Original: For each individual test, we select a random subset of three parameters and then randomly choose a value for each of these.
Revised: For each individual test, we select a random subset of three parameters and then randomly choose a value for each of these.

Original: We then treat this new MDP model as a stand-in for the user model and use it to label traces generated from the original MDP.
Revised: We then treat this new MDP model as a stand-in for the user model and use it to label traces generated from the original MDP.

Original: The traces were generated by choosing a random initial state and then following the optimal policy of the robot until either the terminal state is reached or the trace length reaches a predeﬁned limit.
Revised: The traces were generated by choosing a random initial state and then following the optimal policy of the robot until either the terminal state is reached or the trace length reaches a predeﬁned limit.

Original: For each trace, a random subset of the explanations was selected and presented to the human.
Revised: For each trace, a random subset of the explanations was selected and presented to the human.

Original: This means updating the MDP parameters to their corresponding values in the robot model only for the parameters speciﬁed by the current subset of explanation.
Revised: This means updating the MDP parameters to their corresponding values in the robot model only for the parameters speciﬁed by the current subset of explanation.

Original: Each individual transition was then labeled using this updated MDP.
Revised: Each individual transition was then labeled using this updated MDP.

Original: A transition was labeled as inexplicable if the action is not the optimal one in the human model (i.e. Q value is lower) or the next state had a probability of occurring of δ = 0 .
Revised: A transition was labeled as inexplicable if the action is not the optimal one in the human model (i.e. Q value is lower) or the next state had a probability of occurring of δ = 0 .

Original: We then used this set of labeled transitions to create a training set and test set for a decision tree.
Revised: We then used this set of labeled transitions to create a training set and test set for a decision tree learner.

Original: The input features to the decision tree consist of current state features, (just x and y for Four rooms and the position of the the taxi and passengers for the Taxi domain and for Warehouse it included the position of the robot and the fact whether the agent picked up the box or visited station #1), the index of the action and features capturing the current subset of explanations being considered.
Revised: The input features to the decision tree consist of current state features, (just x and y for Four rooms and the position of the the taxi and passengers for the Taxi domain and for Warehouse it included the position of the robot and the fact whether the agent picked up the box or visited station #1), the index of the action and features capturing the current subset of explanations being considered.

Original: In each Warehouse and Four rooms test instance, we collected 900 unique data points as training set and 100 data points as the test set.
Revised: In each Warehouse and Four rooms test instance, we collected 900 unique data points as training set anddata points as the test set.

Original: Due to the complexity of the taxi domain, we generated less data points (since for each different explanation subset we need to solve a new planning problem) and used close to 220 unique points as training data and on average 28 data points as the test set.
Revised: Due to the complexity of the taxi domain, we generated less data points (since for each different explanation subset we need to solve a new planning problem) and used close to 220 unique points as training data and on average 28 data points as the test set.

Original: We then tested on 20 such instances for each domain.
Revised: We then tested on 20 such instances for each domain.

Original: Figure 3 plots the average test accuracy for models trained with training sets of varying sizes.
Revised: Figure 3 plots the average test accuracy for models trained with training sets of varying sizes.

Original: As evident from the graph, a simple decision tree seems to be able to easily model the effect of explanations on labeling for  simulated scenarios.
Revised: As evident from the graph, a simple decision tree seems to be able to easily model the effect of explanations on labeling for these simulated scenarios.

Original: We chose a simple learning model to establish the viability of this method, but one could easily see that the use of more sophisticated learning methods and/or more informed features should lead to better results.
Revised: We chose a simple learning model to establish the viability of this method, but one could easily see that the use of more sophisticated learning methods and/or more informed features should lead to better results.

Original: User Studies: Next, we wanted to establish if we can still learn such simple models when the labels are collected from naive users.
Revised: Next, we wanted to establish if we can still learn such simple models when the labels are collected from naive users.

Original: Our goal here is not to consider scenarios with possible differences in the user’s knowledge, but rather cases where, even in the presence of a set of users with similar backgrounds, their responses to explanations would be too varied to learn useful models.
Revised: Our goal here is not to consider scenarios with possible differences in the user’s knowledge, but rather cases where, even in the presence of a set of users with similar backgrounds, their responses to explanations would be too varied to learn useful models.

Original: To test this, we used the Warehouse domain as a test bed and collected feedback on how users would view the explicability of traces generated from this domain when presented with explanatory messages detailed in Figure 2.
Revised: To test this, we used the Warehouse domain as a test bed and collected feedback on how users would view the explicability of traces generated from this domain when presented with explanatory messages detailed in Figure 2.

Original: For the study, we recruited 45 master turkers from the
Revised: For the study, we recruited 45 master turkers from the

Original: Amazon Mechanical Turk.
Revised: Amazon Mechanical Turk.

Original: Each participant was provided with the URL to a website ( https://goo.gl/Hun3ce ) where they could view and label various robot behaviors.
Revised: Each participant was provided with the URL to a website ( https://goo.gl/Hun3ce ) where they could view and label various robot behaviors.

Original: We considered a setting where the robot had a full battery, but was picking up a fragile box and thus still needs to visit the station #1.
Revised: We considered a setting where the robot had a full battery, but was picking up a fragile box and thus still needs to visit the station #1.

Original: The robot could slip on some cells marked in dark grey with probability 0.25 (slipping here meant the robot picture is tilted to give an impression that it slipped on the cell and didn’t prevent the robot from moving to the next cell).
Revised: The robot could slip on some cells marked in dark grey with probability 0.25 (slipping here meant the robot picture is tilted to give an impression that it slipped on the cell and didn’t prevent the robot from moving to the next cell).

Original: To make sure that all the users had similar mental models at the start, they were provided with the following facts, (a) That robot couldn’t pass through racks, (b) Whenever the robot runs low on battery it needs to get to Station 1, (c) Whenever the robot has a green battery sign next to the robot, that means their battery is full and (d) The robot needs to take the shortest route to the goal.
Revised: To make sure that all the users had similar mental models at the start, they were provided with the following facts, (a) that robot couldn’t pass through racks, (b) whenever the robot runs low on battery it needs to get to Station 1, (c) whenever the robot has a green battery sign next to the robot, that means their battery is full and (d) the robot needs to take the shortest route to the goal.

Original: Also, they were presented with an example trace in this instructions section and were made to take a small pre-test that allowed them to revise the above facts in various scenarios.
Revised: Also, they were presented with an example trace in this instructions section and were made to take a small pre-test that allowed them to revise the above facts in various scenarios.

Original: After the pre-test, they were shown eight traces from the robot policy sampled according to their probabilities.
Revised: After the pre-test, they were shown eight traces from the robot policy sampled according to their probabilities.

Original: After the ﬁrst one, the user was given an explanation message  taken from the seven possible messages (the order of the messages was always randomized).
Revised: After the ﬁrst trace, the user was given an explanation message before each trace, where the message was taken from the seven possible messages (the order of the messages was always randomized).

Original: From the data collected from 45 turkers, we removed data from seven users, based on the fact they didn’t ﬁnd any of the transition in the ﬁrst trace (i.e the case where no explanation was provided) inexplicable.
Revised: From the data collected from 45 turkers, we removed data from seven users, based on the fact they didn’t ﬁnd any of the transition in the ﬁrst trace (i.e the case where no explanation was provided) inexplicable.

Original: We imagine this number would go down when we move to expert users or users who are invested in the success of the robot.
Revised: We imagine this number would go down when we move to expert users or users who are invested in the success of the robot.

Original: The data generated for the remaining 38 users were then used to train a decision tree, where the  average 10-fold cross validation score for the model was at 0.935.
Revised: The data generated for the remaining 38 users were then used to train a decision tree. Since the placement of other objects in the environment were ﬁxed, we were able to use rather simple features for the model like the current position of the robot (x and y), previous position (again x and y), the action, whether they have slipped and ﬁnally the explanations given. We found the model to have an average 10-fold cross validation score of 0.935. For a randomly generated train and test split (where the test split was 10% and contained around 7% inexpicable labels) the precision score was 0.9637 and the recall score was 0.9568.

Original: Furthermore, we could see that the model was able to correctly predict the usefulness of intuitive minimal explanations for the given scenario.
Revised: Furthermore, we could see that the model was able to correctly predict the usefulness of intuitive minimal explanations for the given scenario.

Original: For example, it predicted that while the robots decision to visit station #1 would be considered inexeplicable by the user in the absence of any explanation, the user would mark it as explicable when they are explained about the box being fragile and that fragile boxes need to be inspected at station #1.
Revised: For example, it predicted that while the robots decision to visit station #1 would be considered inexeplicable by the user in the absence of any explanation, the user would mark it as explicable when they are explained about the box being fragile and that fragile boxes need to be inspected at station #1.

Original: In fact the model predicted that only the message that “fragile boxes need to be inspected at station #1” is enough to convince the user about the need for that action (i.e the user could deduce that the box must have been fragile).
Revised: In fact the model predicted that only the message that “fragile boxes need to be inspected at station #1” is enough to convince the user about the need for that action (i.e the user could deduce that the box must have been fragile).

Original: This shows that such learned models may help us generate cheaper explanations (the above set of explanations is smaller than the corresponding minimal complete behavior explanation for the domain), by taking into account the users ability to correctly predict missing information in simple cases.
Revised: This shows that such learned models may help us generate cheaper explanations (the above set of explanations is smaller than the corresponding minimal complete behavior explanation for the domain), by taking into account the users ability to correctly predict missing information in simple cases.

Original: Another pointpoint of interest was that the model predicted all slipping events as explainable even in the absence of all explanations.
Revised: Another point of interest was that the model predicted all slipping events as explainable even in the absence of any explanations.

Original: The cases where the user saw a slip before being told about the possibility of slipping was rare (since there are two explanatory messages related to slipping and the probability of slipping was 0.25).
Revised: The cases where the user saw a slip before being told about the possibility of slipping was rare (since there are two explanatory messages related to slipping and the probability of slipping was 0.25).

Original: Furthermore when we went over the data, we found that in most such cases, the users did mark it as explainable.
Revised: Furthermore when we went over the data, we found that in most such cases, the users did mark it as explainable.

Original: This may be because the effect of slipping may not have been that detrimental to the overall plan (it doesn’t take you off the current path).
Revised: This may be because the effect of slipping may not have been that detrimental to the overall plan (it doesn’t take you off the current path).

Original: It would be interesting to see if this result would be the same in cases where slipping was a more likely event and if it had a more apparent effect on the robot’s plan.
Revised: It would be interesting to see if this result would be the same in cases where slipping was a more likely event and if it had a more apparent effect on the robot’s plan.

Original:  Related Work
Revised:  Related Work

Original: To the best of our knowledge, this work represents the ﬁrst attempt at learning standins for user mental models that allows an agent to predict the potential impact of providing explanations as model reconciliation to observers.
Revised: To the best of our knowledge, this work represents the ﬁrst attempt at learning proxies for user mental models that allows an agent to predict the potential impact of providing explanations as model reconciliation to observers.

Original: With that said, there have been works that have looked at the problem of generating explanations in the presence of model uncertainty for human models.
Revised: With that said, there have been works that have looked at the problem of generating explanations in the presence of model uncertainty for human models.

Original: In particular,  works like (Sreedharan et al.  2018a; 2018b) looked at cases where the agent has access to a set of potential human models.
Revised: In particular, our previous works like [ Sreedharan et al. , 2018a; 2018b ] have looked at cases where the agent has access to a set of potential human models.

Original: One drawback of considering a set of possible models is either they would need to have explicit sensing to identify the user model (which could mean asking a large number of questions to the user) or providing a large amount of information to cover the space of all possible models.
Revised: One drawback of considering a set of possible models is either they would need to have explicit sensing to identify the user model (which could mean asking a large number of questions to the user) or providing a large amount of information to cover the space of all possible models.

Original: In our work, the problem of identifying the speciﬁcs of the user model is resolved through an ofﬂine training process.
Revised: In our work, the problem of identifying the speciﬁcs of the user model is resolved through an ofﬂine training process.

Original: Another work quite related to the discussion covered in this paper is (Reddy et al. 2018), wherein the authors tried to identify cases where they can learn a potential model for the human’s expectation of the task transition dynamics when they do not align with the real world dynamics.
Revised: Another work quite related to the discussion covered in this paper is [Reddy et al. , 2018], wherein the authors tried to identify cases where they can learn a potential model for the human’s expectation of the task transition dynamics when they do not align with the real world dynamics.

Original: Unlike their work, we do not assume that the user can provide traces for the given task, rather they may be able to provide some highlevel feedback on the action (i.e. they may not be able to do or even know the right action but may be able to point out actions or transitions that surprise them).
Revised: Unlike their work, we do not assume that the user can provide traces for the given task, rather they may be able to provide some highlevel feedback on the action (i.e. they may not be able to do or even know the right action but may be able to point out actions or transitions that surprise them).

Original: Moreover, their work requires that the user and the robot  have the same reward function, which is again an assumption we do not make.
Revised: Moreover, their work requires that the user and the robot must have the same reward function, which is again an assumption we do not make.

Original: Even if we had followed their technique to learn a potential model for the human’s transition function of the task, there is no guarantee that the learned representation aligns with the ones maintained by the human and the effects of explanatory actions may not be predictable.
Revised: Even if we had followed their technique to learn a potential approximation of the human’s transition model for the task, there is no guarantee that the learned representation would be one that makes sense to the human.

Original:  Discussions and Conclusion
Revised:  Discussion and Conclusion

Original: This paper proposes a possible way in which model reconciliation explanation could be applied to cases where the user model is unknown.
Revised: This paper proposes a possible way in which model reconciliation explanation could be applied to cases where the user model is unknown.

Original: The method described here is a rather simple and general method to identify information that could potentially affect the user’s mental model and produce effects that align with the agent’s requirements.
Revised: The method described here is a rather simple and general method to identify information that could potentially affect the user’s mental model and produce effects that align with the agent’s requirements.

Original: There is no requirement here that the messages have to align with actual facts about the world.
Revised: There is no requirement here that the messages have to align with actual facts about the world.

Original: This again points to the rather troubling similarities between the mechanisms needed to generate useful explanations and lies (Chakraborti and Kambhampati 2018).
Revised: This again points to the rather troubling similarities between the mechanisms needed to generate useful explanations and lies [Chakraborti and Kambhampati, 2018].

Original: Two important assumptions we made throughout the works is that the user only considers the current state (as deﬁned by the robot) to make their decisions and we have access to a model that was learned from interactions to previous users who had similar knowledge level to the current user.
Revised: Two important assumptions we made throughout the work is that the user only considers the current state (as deﬁned by the robot) to make their decisions and we have access to a model that was learned from interactions to previous users who had similar knowledge level to the current user.

Original: Relaxing the ﬁrst assumption would require us to go beyond learning models that map  transitions to labels.
Revised: Relaxing the ﬁrst assumption would require us to go beyond learning models that map each transitions to labels.

Original: We have to consider sequential  models  of the type considered in (Zhang et al. 2017) to capture the human’s expectations. 
Revised: Instead we have to consider sequential labeling models (for example models based on LSTM or CRF) of the type considered in [ Zhang et al. , 2017 ] to capture the human’s expectations. For example, we considered a simple extension of the warehouse domain where the human believes the robot should visit two locations (i.e the human state contains variables that record whether the user has visited the locations). Even though here the user is considering a more detailed model, we were able to learn labeling models of 80% accuracy by using simple CRFs.

Original: As for the second, instead of assuming that all users are of the same type, a more reasonable assumption may be that the users could be clustered into N groups and we could learn a different labeling model for each user type.
Revised: As for the second, instead of assuming that all users are of the same type, a more reasonable assumption may be that the users could be clustered into N groups and we could learn a different labeling model for each user type.

Original: Now we still have a challenge of identifying the user type of a new user and one way to overcome this would be by adopting a decision-theoretic approach to this problem and modeling it as a POMDP (wherethe user labels become observations andthe previously learned user modelsturn into the observation models).
Revised: Now we still have a challenge of identifying the user type of a new user and one way to overcome this would be by adopting a decision-theoretic approach to this problem and modeling it as a POMDP (where user labels become observations and previously learned user models the observation models).

Original: The work discussed in this paper only covers explanations that allow the user and the system to reconcile any model difference.
Revised: The work discussed in this paper only covers explanations that allow the user and the system to reconcile any model difference.

Original: This only covers a part of the entire explanatory dialogue.
Revised: This only covers a part of the entire explanatory dialogue.

Original: Even if there is no difference in models, the user may still have questions about parts of the policy or may raise alternative policies they think should be followed.
Revised: Even if there is no difference in models, the user may still have questions about parts of the policy or may raise alternative policies they think should be followed.

Original: This may arise from a difference in inferential abilities andthis may require providing information that is already part of their deductive closure eg: help them understand the long term consequences of taking some actions.
Revised: This may arise from a difference in inferential abilities and may require providing information that is already part of their deductive closure eg: help them understand the long term consequences of taking some actions.

Original: Once you have access to a set of such messages one could use a method similar to the one described in the paper to ﬁnd the set of helpful ones.
Revised: Once you have access to a set of such messages one could use a method similar to the one described in the paper to ﬁnd the set of helpful ones.

Original: Unlike the model reconciliation setting where the messages stand for information about the model, it is not quite clear how one could automatically generate such messages. 
Revised: Unlike the model reconciliation setting where the messages stand for information about the model, it is not quite clear how one could automatically generate such messages. Acknowledgments

Revised: This research is supported in part by the ONR grants

Revised: N00014-16-1-2892, N00014-18-1-2442, N00014-18-1-2840, the AFOSR grant FA9550-18-1-0067, and the NASA grant

Revised: NNX17AD06G.