Not all users are the same: Providing personalized explanations for sequential decision making problems
Abstract
There is a growing interest in designing autonomous agents that can work alongside humans.
Such agents will undoubtedly be expected to explain their behavior and decisions.
While generating explanations is an actively researched topic, most works tend to focus on methods that generate explana- tions that are one size ﬁts all.
As in the speciﬁcs of the usermodel are completely ignored.
The handful of works that look at tailoring their explanation to the user’s background rely on having speciﬁc models of the users (either analytic models or learned labeling models).
The goal of this work is thus to propose an end-to-end adaptive explanation generation sys- tem that begins by learning the different types of users that the agent could interact with.
Then during the interaction with the target user, it is tasked with identifying the type on the ﬂy and adjust its explanations accordingly.
The former is achieved by a data-driven clustering approach while for the latter, we compile our explanation generation problem into a POMDP.
We demonstrate the usefulness of our system on two domains using state-of-the-art POMDP solvers.
We also report the re- sults of a user study that investigates the beneﬁts of providing personalized explanations in a human-robot interaction
set- ting.
Recent successes in AI have sparked great interest for autonomous agents to be deployed into our day to day life.
Unfortunately, even the most powerful of AI systems fall quite short when it comes to the ability to work successfully with humans.
A capability sorely missing in most of these systems is the ability to make sure that the agent acts in a manner that a human observer would ﬁnd explicable.
When failing to do so, it must have the ability to explain its decisions to their teammates.
While the problem of explaining AI decisions itself has been getting a lot of attention (Gunning 2017), they have generally focused on generating explanations that are agnostic of the actual users of the system.
While studies in social sciences have repeatedly shown the need to generate explanations that are social, i.e., explanations tailored to speciﬁc background of the explainee (Miller 2019), most recent works in XAI have generally focused on generating either one-size-ﬁts-all explanation or ones that are meant exclusively for the designers of the system.
A recent trend in explanations, that have tried to look at the explainee’s understanding of the task has been the works done under the umbrella of model reconciliation (Chakraborti et al. 2017).
They generally look at the problem of explanation as one of correcting the human’s mental model about the task so they can correctly evaluate the agent’s decisions.
Unfortunately these works either assume that the agent knows the exact model of the user (or some proxy of it)
(c.f. (Chakraborti et al.
2017; Sreedharan et al. 2019)) or consider cases which allow for possible uncertainty about the user model but require that the uncertain model be explicitly represented in a speciﬁc declarative forms (c.f (Sreedharan, Chakraborti, and
Kambhampati 2018; Sreedharan, Srivastava, and Kambhampati 2018)).
In the absence of explicitly given models, the agent would need to learn a new model for each new user.
Realistically, this would be infeasible as learning a new model might require a lot of input from the user (not to mention the very act of interaction to learn these models could end up shifting their models).
Moreover, the agent would not leverage the existing knowledge it has gained from learning the models of other users.
We believe that a more reasonable setting is the one in which a given task will have categories of users such that the users in a category share a similar model of the task.
We refer to these categories as user types.
For example, for the task of robot-assisted cooking, one can imagine user types like novice, intermediate and expert chefs.
Each type of user might require different explanations.
Now if the robot has access to the models of each user type and knows the type of user it is interacting with, then it can successfully explain its behavior.
A signiﬁcant challenge in such a setting comes from the fact that the user types might not be known in advance.
In addition, even if they are, a user cannot be expected to know which type they belong to.
This is particularly true in cases where each user may not be aware of all the classes of users or how these classes are typiﬁed.
Given this complication, the system we design in this work is tasked with:
Identifying the types of users that exists for the task and learning their mental model of the task.
Inferring the type of user that the agent is interacting with, and using that to give personalized explanations.
In the following sections, we ﬁrst introduce one of the plan- ning domains that we use to evaluate our approach and then go over how an approximation of a user’s model is learned.
This is followed by a discussion on learning the model for each type of user.
We then go over how our system handles explaining a plan trace to the user.
Since the user’s type is not known, we setup a POMDP based formulation that can reason about the hidden user’s type and generate explanations accordingly.
Finally, we test a variety of POMDP solvers on two planning domains and present a user-study that uses certain objective metrics to determine the beneﬁts of providing personalized explanations.
Illustrative example
We will use a disaster-rescue domain to illustrate the ideas of the paper.
It is also one of the domains that we use to evaluate our solution approaches.
The domain represents a scenario where a rescue robot is tasked with evacuating a victim to its shelter.
There are also certain actions that the robot can do to get additional reward.
Some example actions include picking up the ﬁrst aid, extinguishing the ﬁre, refueling at the fuel station, which give a positive reward.
There are also certain obstacles like rubble and a puddle but the robot is capable of navigating through them.
Thus, the robot doesn’t incur a negative reward for navigating through any location with an obstacle.
Figure 1 (A) shows one example instance of the domain and an optimal plan executed by the robot.
Against each action, the corresponding reward is shown as well.
Outside the domain, we have a human supervisor whose task is to evaluate the robot’s behavior.
The supervisor can have inaccurate knowledge of certain facts about the domain.
As an example, we assume there are ﬁve types of supervisors identiﬁed as type-A to type-E .
Each type of supervisor can have different knowledge about the domain.
For instance, the supervisor of type-E thinks that refueling is unnecessary and associates a negative reward with it.
They also believe that the robot can break when moving into a location with puddle, and hence the corresponding transition has a negative reward in their model of the task.
Figure 1 (B) enumerates the inaccurate facts each type holds for the domain.
When evaluating the robot’s plan trace, the supervisors of each type may ﬁnd different parts of the plan inexplicable.
For e.g., the supervisor of type-B will be confused when the robot picks up the ﬁrst-aid.
Now, the robot isn’t aware of all the supervisor types, let alone the type of the supervisor with whom it is interacting.
One trivial way to make sure that the entire plan is explicable to the supervisor is to give them the entire model of the robot.
This, however, can overwhelm them.
We want the robot to provide only the required explanations to the supervisor.
For this to be possible, the robot needs to ﬁgure out what are the types of supervisors that exists and what explanation messages would make the plan explicable for each type.
Then, while interacting with the target supervisor, it needs to ﬁgure out their type and accordingly give personalized explanations.
Background
In this work, our focus is on robots that use discounted inﬁnite horizon Markov Decision Processes (MDPs) for solving a sequential decision making problems (Russell and
Norvig 2002).
An MDP M can be formally deﬁned as tuple (cid:104) S, A, T, R, γ (cid:105) , where: S is the set of states in the domain;
A is the set of actions that the robot can take in any state;
T is the transition function where T ( s, a, s (cid:48) ) gives the probability that the robot will reach state s (cid:48) after taking action a in state s ; R is the reward function such that R ( s, a, s (cid:48) ) gives the reward for a transition (cid:104) s, a, s (cid:48) (cid:105) ; γ is the discounting factor for the rewards obtained (where 0 ≤ γ < 1 ).
A policy π for an MDP maps each state to a recommended action.
The value for any state given π is the expected cumulative discounted reward obtained if the robot follows π .
For a given MDP M , the optimal policy π ∗M is simply the policy that would give the highest value from any state.
Lastly, the
Q-value function, Q π ( s, a ) , for an MDP gives the expected value obtained if the robot takes action a in state s and then follows the policy π .
We use Q ∗ to denote the Q-value function corresponding to the optimal policy for the MDP.
In many human-AI interaction scenarios, the model of the task used by the robot, M r may differ from how the human understands the task (including the robot’s capabilities).
This discrepancy could imply that even when the robot follows optimal policies, the robot behavior may appear to be surprising and inexplicable to the human.
Works like (Chakraborti et al. 2017) have looked at how to use explanations to address such scenarios.
Speciﬁcally they consider revealing information about parts of the robot’s model to correct the human’s misunderstanding about the task.
In this work, in particular, we use learned labeling models to ﬁgure out the speciﬁc model information to reveal to the user.
The learned model functions as a proxy to the user’s model of the task.
Speciﬁcally, we assume a set of explanatory messages E
= { e 1 , e 2 , . .
e n } , that include information about the robot’s model.
An example message, for the disaster-rescue domain, would be the robot obtains a reward of 1.0 for picking up the ﬁrst aid .
Given a speciﬁc transition τ = (cid:104) s, a, s (cid:48) (cid:105) , we learn a labeling model of the form L ( τ, ˆ E ) , that maps the transition and a set of messages ˆ E to either labels 1 or 0 .
Here 1 refers to the fact that once the user is given a set of messages ˆ E then they ﬁnd the transition to be explicable, i.e., there exists an optimal policy in their updated model that can generate this transition.
Note that our work accounts for the fact that different users of the system may have different backgrounds and mental model of the task.
Hence, they may require different sets of explanatory messages.
Our goal through this paper would be to see how we can set out to create an explanatory system that is able to create explanations that are tailor-made for speciﬁc users.
Explanation in the Presence of Multiple User Types
Our speciﬁc focus in this paper would be to handle cases where the robot needs to provide explanations to users with different backgrounds.
To simplify the setting, we will assume that each user’s understanding of the task could be mapped to one among a set of possible models, and thus maps to one of the possible labeling models in the set L .
Thus, for any user in the system of type u t , we will denote the corresponding labeling function as L t ∈ L .
Learning the set of labeling models
Now the ﬁrst step to setting up such an explanatory system for a task M r and a set of messages E will be to learn the corresponding labeling function set L .
Thus, the system needs to identify all the user types, and learn their respective labeling functions.
Similar to (Sreedharan et al. 2019), we will rely on data collected from users (referred to as observers) that include robot transitions along with messages and their corresponding labels provided by the observers.
We will additionally assume that we know which observer id generated the label.
Thus each datapoint takes the form d
= (cid:104) τ, ˆ E , l, o (cid:105) where τ = (cid:104) s, a, s (cid:48) (cid:105) is the transition, ˆ E is the message set, l is the corresponding label and o is the unique observer
id. The learner’s primary goal is to separate the collected data into k groups where ideally, k is equal to the true number of user types N u (which is not known), and each group has data only from a single user type.
This amounts to clustering the observers into k such groups.
To begin, each observer is represented as a vector ˙ E o ∈ { 0 , 1 } | E | where each bit corresponds to some message in E .
For a given ˙ E o , the set of messages whose bit is set to 1 represents the smallest set of messages which when given to the observer o will make every possible transition explicable to them.
To obtain ˙ E o , the learner ﬁrst uses the datapoints D o collected from the observer to train their corresponding labeling function L o .
Then, using L o , the smallest set of messages is computed that would make all the transitions in D o explicable to the observer.
Finally, the obtained message set is used as an approximation of ˙ E o .
Note that in order to learn a L o that is capable of computing the smallest message set, our data collection process is different than the one used in (Sreedharan et al. 2019).
In our case, the data is collected from the observers via counterfactual queries of the form, if the message set ˆ E was given to the user, then would they ﬁnd a transition τ to be explicable.
Now in this space of explanations, the observers belonging to the same type would tend to be closer to each other as they require similar explanations.
Hence, the learner can now apply k -means clustering in this space given any input number of clusters k .
Now that we have represented each observer in this space, the next step is to determine N u .
For this problem, we deﬁne an intra-cluster dissimilarity measure called disagreements that evaluates a given clustering of observers.
It is calculated as the total number of instances where two observers that have been clustered together and been given the same set of messages, disagree on the label of some transition common between their respective D o .
Formally we deﬁne disagreements as follows:
Deﬁnition 1 For a given clustering of observers, disagreements is equal to the total number of pairs of datapoints d i = (cid:104)
τ i , ˆ E i , l i , o i (cid:105) and d j = (cid:104)
τ j , ˆ E j , l j , o j (cid:105) that satisfy two conditions: ﬁrst, o i and o j belong to the same cluster, and second, τ i =
τ j , ˆ E i =
ˆ E j , and l i (cid:54) =
l j .
We expect the value of disagreements to be low when each cluster contains observers from a single type since they share a similar model of the task.
Consequently, the learner applies clustering to the set of vectors ˙
E o parameterized by the number of cluster k where k varies from 1 to the total number of observers.
For each clustering instance, the value of disagreements is computed.
At k = 1 , we expect the disagreements to be highest.
But as k increase, observers of different types would ideally be placed in separate clusters thus decreasing disagreements .
When k reaches N u , each cluster should ideally have observers from a single type.
Thus, further increasing k would not reduce the disagreements by too much (similar to the pattern followed by intra-cluster distance measures used in k -means clustering when increasing the number of clusters).
We therefore speculate that the elbow point of the plot of disagreements against increasing values of k would be equal to the N u .
The learner programmatically ﬁnds this elbow point and uses it as the number of user types, N u .
Given N u and its corresponding clustering output, the learner can now learn the labeling set L .
First, for each cluster, a union of all the datapoints for all the observer within that cluster is taken.
Each of this set now contains datapoints corresponding to an identiﬁed user type u t .
The corresponding labeling model L t is trained on the points in the set.
This gives us the required set L .
Additionally, we learn a conﬁdence measure which provides the conﬁdence for the prediction given by any learned model.
Speciﬁcally, for each
L t we learn a probabilistic model, P L t ( l | τ, ˆ E ) which gives the probability that the user of type t will give τ the label l provided they have been given the explanations ˆ E .
For notational convenience, we deﬁne a set U to include all the identiﬁed user types u t .
Explaining Behavior Traces
With the speciﬁc labeling models in place, we are now ready to generate explanations for a speciﬁc behavior trace
Π = (cid:104) τ 1 , ..., τ m (cid:105) .
An explanation in this setting consists of a sequence of explanatory messages of the form E =
(cid:104)
ˆ E 1 , ..., ˆ E m (cid:105) where the explanations E k are given to the user at step k of the interaction.
At each step k , the user is presented with the preﬁx (cid:104) τ 1 , ..., τ k (cid:105) denoted as π k , where the fact whether a user of type u t will ﬁnd any transition τ j to be explicable is given by the function L t ( τ j , (cid:83) ki
=1 (ˆ E i )) , i.e., the user would try to make sense of each transition using the current message as well as previous messages.
The user is expected to provide labels for each transition in π k .
Now each explanatory message may be associated with some communication cost.
To allow for reasoning regarding the tradeoffs between communication and inexplicability, we will assume access to C e , the function that gives the cost of presenting a set of messages ˆ E ∈ E to the user;
C inexp , the cost of showing an inexplicable plan step to the user, and λ , a constant value that decides the relative weight between the total communication and the total inexplicability cost for the entire interaction with the user.
Thus the cost of any step in the explanation sequence becomes a linear combination of these two costs and we can deﬁne the total cost of a explanation sequence as follows:
Deﬁnition
The cost of an explanation sequence E = (cid:104) ˆ E 1 , ..., ˆ E m (cid:105) , for a trace Π and a user type u t ∈ U is given by
[Equation]
Thus, if the user type were to be known, our goal would be to ﬁnd a sequence that minimizes this total cost.
Unfortunately, this is not true in our case and in addition to reasoning about the explanatory sequence, we would also need to reason about the hidden user types.
We assume that all we have access to is a possible prior over the user types.
A useful reasoning framework that we could leverage would be that of partially observable Markov decision process (POMDP) (Kaelbling, Littman, and Cassandra 1998).
We can cast this problem of reasoning about explanation into a meta POMDP of the form M E = (cid:104) S E , A E , T E , O E , Ω E , γ E (cid:105) .
Speciﬁcally, we turn the problem of identiﬁcation of explanation with unknown user type into a POMDP where the user type is the hidden part of the state.
Each of the components of M E are of the form:
S E : Each state in the meta-POMDP would contain a preﬁx that is part of the trace to be explained, two separate sets of explanatory messages denoted as E given and E next , a sequence L of ﬂags denoting whether the user found each of the transition within the preﬁx to be explicable or not given the messages in E given and ﬁnally the hidden user type.
The messages in E next are the ones that will be presented to the user in the next interaction step.
Note that the user type is unobservable to the robot.
Formally, let L π k denote the set of all possible sequence of explicability labels for the transitions within π k .
Then the state space is deﬁned as:
[Equation]
ˆ E given , ˆ E next , t (cid:105) | k ∈ [1 , m ] , L ∈ l π k , u t ∈
U
{+ˆ E given ∈ 2 | E | , ˆ E next ∈ 
 | E | and ˆ E given ∩ ˆ E next = ∅}+} A E : Actions in the POMDP correspond to the set of explanatory messages, E , that the robot can present to the user in addition to a special action called explain .
Thus, A E = E ∪ { explain } .
T E : The transition function can be described as acting in two different modes.
Say the interaction is in the state s = (cid:104)
π k , L, ˆ E next , ˆ E given , u t (cid:105) .
When the action, a = ˆ e (where ˆ e ∈ E − ˆ E given ) then the next state is simply (cid:104) π k , L, ˆ E next ∪ ˆ e, ˆ E given , u t (cid:105) i.e. the explanatory message is added to the set of messages to be given in the next interaction step.
In contrast, when a = explain or a ∈ ˆ E given (i.e. the robot chose an explanatory message that has already been given), then the transition function takes the interaction to its next step where the user is presented with the preﬁx π k +1 and given the messages in ˆ E next .
In this case, there will be multiple possible next states where each state’s label sequence correspond to a possible sequence of labels given to the π k + by the user of type u .
In all those states, the preﬁx is set to π k +1 , ˆ E given is set to ˆ E given ∪ ˆ E next and ˆ E next is set to ∅ in that order.
The user type is unchanged.
Assuming each transition will be labelled independently of each other by any user, the transition probabilities for each state would be computed as Π k +1 i
=1 P L t ( l i | ˆ E given ) where l i is the label given to τ i .
R E : The reward that the robot obtains will incorporate the cost associated with presenting an inexplicable transition, and the cost of communicating an explanatory message.
For a transition (cid:104) s, a, s (cid:48)
(cid:105) with π k as the preﬁx in s (cid:48) , the reward function is deﬁned as:
[Equation]
) if a = explain Ω E : The set of all observations would be all possible explicability labels for all preﬁxes π k in Π .
In addition, Ω E will contain a special observation denoted as l φ .
O E : The observation model is deterministic.
If the interaction moves to its next step, then the observation is the label sequence L associated with the next state.
Otherwise, it is simply l φ .
γ E : We leverage discounting to allow for on-time explanations of the form studied in (Zakershahrak et al. 2019).
Apart from the fact that we are explaining a sequence of traces, our embedding of the explanation reasoning into a sequential reasoning framework gives us additional advantages.
For one, we are able to leverage the POMDP’s ability to naturally generate information gathering strategies when appropriate to help proactively identify the user type.
In general; for each observation L , it provides us information about the underlying user type.
Assuming each transition is labelled independently, the belief about the user type is up- dated as follows in each interaction step:
[Equation]
Where b k ( t ) denotes the probability that the target user is of type t at k th interaction step, l i the label provided by the user for the transition
τ i and ˆ E given the explanations that have been presented so far.
To leverage the belief update, the
POMDP policy could choose to employ explanation messages at a step that are designed to help reduce the possible uncertainty regarding the user’s type.
For example in a case with two user types, say a given transition is inherently explicable for one type of user while it requires an explanation for the other type.
By withholding any explanation, the system would be able to tell the exact user type from the next observation.
Another advantage of using the sequential reasoning framework is that it allows us to leverage the fact that explanations may be non-monotonic (Chakraborti et al.
2017), i.e. the POMDP would prevent revealing unnecessary information up front that could potentially lead to the user getting confused.
While there exist exact algorithms for solving POMDPs, they are intractable for problems with long horizon (which in the case of M E is | Π | + | E | ).
Instead we will mostly focus on using approximate POMDP solvers.
In particular, we will consider approximations that are myopic, i.e. their ability to account for information gathering strategies is limited and we will also be looking at approximations that rely on at least some limited amount of lookahead.
Although reasoning about information gathering is useful, it is also one of the main complexities in POMDPs (Fern et al. 2014).
Thus, using myopic solvers can be highly efﬁcient.
In general, we believe that myopic behavior will not adversely affect the cost of the interaction for our problem.
Intuitively, this happens because unexpected labels can inform the robot about the actual user type.
For instance, say the robot’s current belief is more probable to an incorrect user type and it gives an explanation speciﬁc to that user type for a transition.
The actual user might still ﬁnd that transition to be inexplicable, thus driving the belief to the correct type.
We will next go over the POMDP solvers that we use for generating explanations.
Myopic Approximation: QMDP
QMDP (Littman, Cassandra, and Kaelbling 1995) is a highly efﬁcient way to learn a POMDP’s policy that uses the Qvalues of states in the underlying MDPs to approximate the
Q-value of a belief state.
If the user type of target user is known, then M E , simply becomes an MDP.
We use a variant of this MDP such that its optimal policy can be learned efﬁciently.
The state in this MDP would consist of state components of M E not including the labels sequence and the user type.
We ignore the labels to prevent the state space from blowing up in size (which would otherwise be exponential in length of the trace).
To retain the robot’s ability to prevent inexplicable labels, we instead, modify the reward function R E for the case when a = explain .
In that case, for the MDP corresponding to a user type t , the reward value for transitioning into a state with preﬁx π k is the expected inexplicability cost of the form − 1 .
(cid:80) ki =1 C inexp .P
L t ( l i =
| τ i , ˆ E given ) .
Let Q ∗ t denote the Q-value function for the MDP corresponding to user type t for the optimal policy. Then
the Q-value for a belief state b and an action a can be approximated as Q ( b, a ) ≈ (cid:80) t b ( t ) .Q
∗ t ( s o , a ) where s o are the state components corresponding to the MDP for user type t .
Intuitively, the Q-value is being computed for a scenario in which all the ambiguity of the user type would be resolved in the next step.
Thus, the Q-value estimation doesn’t account for the value provided by information gathering actions.
Myopic Approximation: QHR
This is a variant of a method proposed in (Fern et al. 2014) to approximate the Q ∗ t values.
Although ignoring the labels helps in reducing the state space, it still increases exponentially in the number of messages.
Thus in the QHR approach, instead of computing the optimal Q ∗ t , we will use the value of a suboptimal policy.
The suboptimal policy assigns an action to any state assuming that the robot will not be allowed to provide explanations from the next interaction step.
In this case, we let each action a be a set of explanatory messages i.e. a ∈ 2 E and calculate the value of the suboptimal policy in closed form as follows
[Equation]
where k is the current step in the interaction.
The estimate can be computed much faster than computing the entire policy for the corresponding MDP.
We recommend the use of this technique when the MDP’s state space is large and ofﬂine computation of the policy is restricted.
Once computed, the value can be used to get the Q-value estimate Q ( b, a ) for any belief state as shown in the QMDP technique.
POMCP with d -step lookahead
The POMCP algorithm (Silver and Veness 2010) has been shown to achieve high performance for POMDPs with large state space.
We used POMCP on M E as the approximation that allows for look ahead.
In our work, we made two modiﬁcations to the original POMCP approach.
First, while the original algorithm uses particle ﬁlter algorithm to approximate belief update during the search, we performed exact belief update as described in an earlier section.
Secondly, instead of always running the simulation until the terminal state, we only run the simulation till depth d in the search tree.
If the node corresponding to the history encountered till d is not present in the tree, then the simulation proceeds with the rollout.
Otherwise, the simulation is terminated and value for that belief node is computed using the QMDP technique.
Evaluation Computational experiments
We validate our approach by applying it on the disasterrescue domain and the four rooms domain (Sutton, Precup, and Singh 1999).
In the latter domain, the robot is simply tasked with navigating to a goal location using the shortest path in an area with four rooms while avoiding walls and locations with a negative reward.
The disaster-rescue domains had the 5 types of users described in Figure 1.
For the four rooms domain, we vary the number of user types and their models were selected randomly.
For evaluation, we deﬁne a model of the domain for each type as an MDP.
The parameter values of the model will depend on the knowledge the user type has about the domain.
For example, in the disasterrescue domain the MDP for user type B will have a negative reward for the transition in which the robot moves into a location with a fence.
The MDP models are used to simulate users for any type for evaluation.
For the four rooms domains, we varied the total number of user types between 2 , 3 and 4 .
Thus, we had three different settings for the domain (as opposed to the single setting of 5 user types for the other domain).
For each setting in the four rooms domain, the model of each user type was randomly selected.
First, we identify a parameter set
M that deﬁnes the domain’s MDP.
It consists of the goal locations, discounting factor, step cost, special locations with some penalty, and magnitude of the penalty.
We associate two different values with each parameter i in M denoted as M ri and M ti .
We then instantiate the robot’s model by setting its MDP parameters to their correponding values in
M ri .
For each of the user type t , some model parameters
M t ⊂ M are selected such that (cid:83) t M t =
M .
Then each of the selected model parameters i is assigned the value M ti .
The remaining MDP parameters are assigned the value M ri .
For each setting within the domains, the data required to identify the user types that exist in the domain and learn their labeling models, is collected by having simulated users of each type label plan traces.
We created 3 observers for each type (collecting 100 and 300 points per observer for four rooms and disaster-rescue).
For each observer, traces were generated where the robot starts at some random initial state and follows the optimal policy till it reaches a terminal state or a trace length of 10 .
For each trace, a set of randomly selected explanatory messages from the set E are given to the user.
The user’s model is updated according to the messages and labels are obtained for each step of the trace.
We label a transition to be explicable if the optimal policy in the updated user’s model can generate that transition.
Once the data is collected from the observers, we have access to a set of labeled transitions for each user type.
This reﬂects the ideal clustering that can be achieved as data per type is perfectly separated.
We learn the labeling models, L pc , corresponding to each user type.
These models represent the best possible labeling models that can be learned for each user type.
For a given set of labeled transitions, the labeling model is learned as a decision tree classiﬁer.
The input features to the decision tree consists of the transitions (where the state is deﬁned by the grid locations of the robot for both the domains), and the set of explanatory messages.
The output is the label given by the user.
The test set accuracy after training the classiﬁer is used as the conﬁdence measure P L t .
The testing accuracies for the L pc models across the domains for all types of users in each setting was ≥ 0 . 98 .
To evaluate our setup for scenarios where the explanatory system has no prior knowledge about the user types, we apply the clustering strategy described earlier for the data collected for each setting.
We used the approach presented in (Satopaa et al. 2011) to ﬁnd the elbow point on the disagreements vs k plots which was taken as N u .
We then learn the labeling models for the output clusters which are assumed to be the labeling models for the identiﬁed user types in the domain.
Our clustering approach was able to achieve ideal clustering i.e. it was able to ﬁnd the correct number of user types, and was able to group observers of same type together in both domains for all the cases except for the setting with 4 user types for four rooms domain where observers from two different types were clustered together.
On analyzing the last case, we found that the two user types were assigned model parameters such that all the transitions given to their observers were explicable to them and hence they didn’t require any explanations. So it was acceptable for them to be put in the same cluster and be considered as the same type.
Finally, for the labeling models learned for the identiﬁed user types in each setting, we got testing accuracies similar to the L pc models (i.e ≥ 0 . 98 ).
For a baseline technique, for each setting, we learn a single labeling model for the entire set of labeled transitions collected from all the observers.
This labeling model is then attributed to all the users in that setting.
This amounts to assuming that there exists only a single user type.
Since the labeling model of the target user is now known, the explanation can be generated easily by solving the problem as an
MDP (as described before in QMDP algorithm).
To evaluate the baseline and the POMDP solvers, we deﬁne a regret measure, R = C ( E ) − C ( E ) pc where C ( E ) is the cost of explanation sequence generated by the solver and
C ( E ) pc is the cost of explanation sequence generated by an oracle agent which has been given the user’s type and their labeling model in L pc .
We compute the regret value under different communication and inexplicability trade-off conditions by varying λ .
The cost of communicating an explanatory message and the cost of presenting an inexplicable transition to the user were both set to 1 unit.
For each value of λ , we generate 3 traces where the robot start with a random initial state and follows the optimal policy till it reaches a terminal state.
We then use our POMDP and baseline solvers to generate explanations for the traces for a user of each type for each setting across the two domains.
The POMCP solver was used with a lookahead of 2 steps.
Table 1 shows the average regret measure for both the domains for each value of λ .
The average was taken across all the plan traces, and all the users within each setting in the domain.
It also shows the average cost ( C ( E ) pc ) for the oracle.
The low regret values suggest that the our system is indeed able to ﬁgure out the user type and provide required explanations to the user.
Its performance is closer to the an oracle agent which has been given the best possible labeling model for the target user.
Moreover, all of our techniques signiﬁcantly outperform the baseline technique.
The poor performance of the baseline technique can be attributed to the low accuracy of the labeling model that was
Domain λ
[Table]
User study
We conducted a user study to investigate the effects of providing personalized explanations on the human-robot interaction when the robot is trying to explain a trace to the human.
We performed a between-subject study where the participants were provided explanations by one of two techniques: personalized and conformant for the disaster-rescue domain with the user types described in Figure 1.
We used the domain that we designed so as to avoid people having some prior background knowledge about the task.
Explanation techniques The personalized explanations were generated using the QMDP framework (with λ set to 1 ) where the participant was treated as the user whose type is unknown to the framework.
As explained in our methodology, the actual labels provided by the participant were used to update the robot’s belief about their type.
For learning the labeling model set L , in order to ensure maximal personalization, we assumed perfect clustering and collected around 1000 data points each from 3 simulated observers per type.
The accuracies for learned models were > 0 . 99 .
In the conformant explanation technique, the robot generates the explanations such that the presented preﬁx would be explicable to the user of any type.
Conformant explanations were generated by using the MDP formalism described in the QMDP algorithm with the reward function now computing the expected inexplicabilty cost of the preﬁx computed across all the user types, t .
Speciﬁcally, when a = explain , the reward for the transition is given as
[Equation]
Tested hypothesis We compare the interactions the participant have with the robot for the two explanation techniques along three dimensions by testing the following hypothesis:
Hypothesis 1 Personalizing explanations lead to a shorter interaction time with the robot.
Hypothesis 2 Personalizing explanations lead to higher explicability of the robot’s behavior.
Hypothesis 3 When given personalized explanation, users are able to retain the updated model of the robot at the end
[Table] Procedure The user study was conducted completely over a web based interface.
However, each participant was also on call with one of the authors in case they had queries about their task.
The participant was only allowed to ask queries to the author before their interaction with the robot began.
The participants didn’t have to share their screen while performing the study, and were allowed to keep their video disabled.
We asked the participant to keep their mic on at all time during the study.
In the user study, each participant is assigned one of the user types of the disaster-rescue domain and an explanation technique.
In the beginning of the study, the participant is provided with speciﬁc details about the domain that are known to the type assigned to the participant.
They are then trained on their task in the study which is to evaluate the robot behavior.
At each step of this evaluation, they need to label each transition they have seen so far as expected or unexpected based on both the information provided about the domain as well as any explanations that have been provided by the robot (we retain the labels from the previous step to reduce participant’s efforts in case they don’t want to change any label).
The participants were told that explanations are meant to correct the description of the domain and that they must update their domain knowledge when given any explanation.
We make sure the participant understand the task by having them label an action with and without explanation that was added to the domain for illustration.
The training is followed by an eligibility test where we test whether the participant remembers the task description accurately.
The participants are given a list of actions, and asked to choose whether the action has a reward ≥ 0 or not.
Upon passing the test, the participant gets to move on to the evaluation part of the study.
After the evaluation, we conduct a quiz to check whether they remember the updated task description.
In the quiz, the participant has to choose the correct statements among a set of 4 statements describing the domain.
Two of those statements are inaccurate, while the other two reﬂect the domain parameters that would be updated by the explanations that the assigned user type would need in order to ﬁnd the entire trace explicable.
For each correct response, the participant got a score of 1 .
Results We performed a between subject design with explanation types: personalized and conformant as the factors.
We collected data from a total of 40 participants (graduate students with 10 females, age ranging from 19 to 31 , mean age = 23 . 9 ) where each explanation type was assignedparticipants each.
The 5 user types of the domain were assigned uniformly among the 20 participants for each explanation type.
To test each hypothesis, we collected certain objective measures from the interaction.
For the ﬁrst hypothesis, we measured the total time the participant took for completing the robot’s evaluation.
For the second hypothesis, we used the number of transitions that remained inexplicable at the end of the interaction.
For the ﬁnal hypothesis, we used the score the participant got on the quiz.
The table 2) provides the average value of each measure across all participants for each explanation type (interaction time is in seconds).
We performed two tailed t-test and found the difference between explanation types to be signiﬁcant for all the measures (p-values reported in the Table 2).
As seen from the results, personalized explanations generated from our method outperforms conformant explanations on all the measures.
Related work
In this work our focus has been on generating explanations that try to resolve the user’s confusion that arise from the user’s incorrect or incomplete understanding of the task.
Such explanations have generally been referred to as model reconciliation explanation in the literature (Chakraborti et al.
In fact, we build on a speciﬁc variant of model reconciliation explanation that relies on labeling models instead of exact models (Sreedharan et al. 2019).
Model reconciliation forms only a type of explanations in the larger landscape of explanations (Chakraborti, Sreedharan, and Kambhampati 2020).
Another major form of explanatory information are the ones designed to provide inferential assistance to the users, for example works like (Seegebarth et al. 2012; Eiﬂer et al. 2020; Khan, Poupart, and Black 2009) etc.
There is no reason to believe that such explanatory messages could not be incorporated into the framework.
Many of these messages are generated independent of speciﬁc user information and thus are completely compatible with our framework.
Our methods should be able to easily identify when certain user groups would beneﬁt from such messages.
With respect to user speciﬁc explanations, while there have been a number of works that have pointed out the need for explanations being tailored to the end user’s background knowledge (c.f.
(Miller 2019; Zhou and Danks 2020))
there has been less number of works that have done this.
Within the context of sequential decision-making the closest work we are aware of is (Sreedharan, Chakraborti, and Kambhampati 2018).
Though as mentioned earlier they still require the possible multiple models to be representable in the form of an incomplete planning model.
Moreover, their user adaptive explanations require the ability for the robot to ask speciﬁc questions about user’s knowledge which may not be available to the system.
Similarly (Sreedharan, Srivastava, and Kambhampati 2018) assumes the user to belong to one of many types, but expect the user’s model to be an exact state abstraction of the current task model.
This requirement may not necessarily be always met.
Even outside explaining sequential decision making problems, we know of very few works that allow incorporation of user information.
One possible contender is the work done under the umbrella of
TCAV (Kim et al. 2018), where the user can specify concepts that can be used to build the explanation.
Even in this case, other than being able to build explanations in user’s vocabulary it doesn’t take into account any other information about the user’s background.
Finally, the idea of leveraging user types in an HRI setting has been previously explored in (Nikolaidis et al. 2015) where the human and the robot act collaboratively to complete a task.
The robot chooses its policy based on the target user’s type.
This work is complementary to our approach which focuses speciﬁcally on reconciling the model differences to make the robot’s behavior more explicable.
Our own method to generate the explanations rely on a compilation POMDP.
POMDP, even with the extremely high computational overhead has been a popular framework for many human aware application.
This is in many ways due to the fact that incomplete knowledge is a part and parcel of many real-world human-AI interaction scenarios.
An example was the use of POMDPs to create hand-washing assistants for people suffering from dementia (Hoey et al. 2010).
Our own use of various approximation is very much in keeping with techniques used by these earlier works (particularly those investigated in (Fern et al. 2014)).
Conclusion and future work
This paper proposes a way in which a robot can generate personalized explanations for the human in the loop for the case where different users can have different mental models of the task.
We argue that while one size ﬁts all explanations are easy to generate they can easily overwhelm the user of the system with unnecessary communication (as shown by our user study).
We develop an explanatory system that ﬁrst identiﬁes the user types that exist for a task, and then uses a POMDP based formulation that is capable of inferring the user’s type (and hence their model) on the ﬂy.
We show its competence on a disaster-rescue domain and four rooms domain under different trade-off conditions between communication and inexplicability cost.
We saw that in the cases considered, all our approaches are able to come quite close, in terms of interaction cost, to a comparable explanatory system that is operating with known user type.
We also presented a user-study where we saw a clear advantage of personalizing explanations in terms of some objective metrics.
We see this work as being just the ﬁrst step, and going forward we would like to extend this framework to more general settings.
First, we would like to relax the assumption that all users belong to a ﬁxed set of types.
We aim to investigate if Bayesian non parametric techniques would be useful in detecting novel user types.
Second, we aim to improve the sample efﬁciency of the learning algorithm used to identify all the user types for the task and learn their labeling models.
We believe that active learning (Settles 2009) based data collection could be useful to address this objective.
Acknowledgment
This research is supported in part by ONR grants
N00014-16-1-2892, N00014-18-1- 2442, N00014-18-12840, N00014-9-1-2119, AFOSR grant FA9550-18-1-0067,
DARPA SAIL-ON grant W911NF-19- 2-0006, and a JP
Morgan AI Faculty Research grant.