Explicable Policy Search
Anonymous Author(s)
AffiliationAddress email
Abstract
Human teammates often form conscious and subconscious expectations of eachother to better interact.
Teaming success is contingent on whether such expectationscan be met.
For an intelligent agent to operate with humans beside them, similarly,it must consider the human’s expectation of its behavior.
Otherwise, it may resultin loss of trust and degraded team performance.
A key challenge here is thatthe human’s expectation may not align with the agent’s optimal behavior, due to,for example, the human’s partial or inaccurate understanding of the task domain.
Prior work on explicable planning describes the ability of agents to respect theirhuman teammate’s expectations by trading off task performance for more expectedor “ explicable ” behaviors.
In this paper, we introduce Explicable Policy Search(EPS) to significantly extend such an ability to a reinforcement learning (RL)setting and to handle stochastic domains with continuous state and action spaces.
However, in contrast to the traditional RL method, EPS must at the same time inferthe human’s hidden expectations.
Such inferences require information about thehuman’s belief about domain dynamics and her reward model but directly queryingthem is impractical.
We demonstrate that they can be sufficiently encoded by asurrogate reward function, which can be learned based on the human’s feedbackon the agent’s behavior.
The surrogate reward function is then used to reshapethe agent’s reward function, which is shown to be equivalent to searching for anexplicable policy.
We evaluate our method for EPS in a set of continuous navigationdomains with synthetic human models and in an autonomous driving domain witha user study.
The results suggest that our method can generate explicable behaviorsthat reconcile task performance with human expectation intelligently and hasreal-world relevance in human-agent teaming domains.
Introduction
Intelligent agents are quickly becoming a part of our daily lives in a variety of domains, includingsmart home, autonomous driving, education and so on.
In such domains, the agents are expectedto perform in human inhabited environments and even collaborate closely with them.
Members insuch a collaborative setting often form conscious and subconscious expectations of each other and the success of collaboration depends on whether such expectations can be met.
A key challenge here is that the human’s expectation may not align with the agent’s optimal behavior.
Hence, agentschoosing their optimal behaviors without considering the human observers or collaborators could be seen as unexpected, leading to degraded team performance and loss of trust [16, 3].
Consider a drone navigation scenario in Fig.
1a where a drone is tasked to navigate from the starting position (as shown) to a destination.
On a good day, the drone would be expected to mostly navigate straight to the goal with some air perturbation.
However, there is a heavy wind today that changes the domain dynamics (which a human observer is unaware of) and it becomes impossible for the drone to navigate as usual.
Two alternatives are illustrated where the drone navigates in a zigzag and a curved
[Figure]
pathway. The curved pathway is more optimal since it makes (due to the windy condition) but the former is likely to be perceived as more expected to the human observer, given itscloseness to the expected behavior.
We note here that the explicable behavior goes beyond simplychoosing from or combining the agent’s optimal behavior with the human’s expected behavior.
Hence,achieving such a capability requires a more fundamental treatment.
In the experiments, we will showthat our learning agent can generate novel behaviors that are rarely seen in training scenarios.
The problem setting of explicable policy search (EPS) is illustrated in Fig.
1b, which is fundamentallya model reconciliation business [4].
In such a setting, considering the learning problem only fromthe perspective of the agent would sabotage the teaming when the agent’s behavior differs fromhuman’s expectation.
Our method considers any feedback the human may provide to the agent basedon her expectation of its behavior (generated under her models) to help the agent improve.
Hence,our method has a variety of applications where machines must be tuned to individual users, suchas an autonomous vehicle that learns to improve its driving behavior based on its owner’s feedbackbut at the same time must abide by standardized operating requirements to ensure safety, comfort,etc.
In our problem, we assume that the human is nosily rational at generating her expectation ofthe agent to accommodate her computational limitations.
While noisy rationality may not perfectlydescribe our computational model [33] due to various cognitive biases, modeling humans as suchhas been a common practice and demonstrated as a reasonable assumption in prior methods [28, 2].
Furthermore, similar to prior work on explicable planning [36, 22], we assume that the human is asole observer of the agent.
Extending it to an collaborative setting requires additional machinery andwill be discussed in future work, with steps already taken in a classical planning setting [35].
In this paper, we formulate explicable policy search (EPS) in Markov Decision Processes (MDPs).
The goal is to learn a policy that reconciles between maximizing the long-term return (based on theengineered reward model r A ) and minimizing the deviation of the agent’s behavior from the human’sexpectation of it, which is quantified by an explicability score.
We formulate the problem as a linearcombination of two objectives and show that such a simple treatment already yields substantialbenefits in our evaluation.
The challenge in this problem formulation mainly lies in the fact that thehuman’s expectation is hidden and must be learned.
A straightforward solution involves learningboth the human’s belief about the domain dynamics ( T HA ) and her reward model ( r H ) from humanfeedback, which is possible but impractical.
Instead, we show that the information needed fromthem for EPS can be sufficiently encoded by a surrogate reward function, which is much easier tolearn.
Such a reward function can then be incorporated into a policy search process to recover anexplicable policy.
The final modified reward function for EPS bears some similarity to maximumentropy reinforcement learning [17, 18] but is derived under a completely different motivation.
For evaluation, we design a set of navigation domains where the human’s feedback on her expectationof the agent’s behavior is synthetically generated based on the true domain dynamics modified withvarious “misunderstandings”.
We compare our method with three baselines that are selected to represent the traditional RL methods to demonstrate the key advantages of EPS under model differences.
Furthermore, to show its real-world relevance, we conduct a human study in an autonomous drivingdomain [26] where we design driving scenarios to elicit existing human biases about the vehicle’sdomain dynamics that could be dangerous to ignore during learning.
The results show that our methodcan intelligently generate explicable behaviors that are safe and preferred over those of the baselines.
The contribution of this paper is three-fold.
First, we introduce and formulate explicable policysearch, which extends explicable planning to a reinforcement learning setting and to stochasticdomains with continuous state and action spaces.
Second, we propose a practical solution for thischallenging problem by introducing a surrogate reward function learned from human feedback data,which sufficiently encodes the necessary information for explicable policy search.
Third, we evaluateour learning method extensively with simulations and human subjects.
Related Work
The problem of generating communicative actions or behaviors (a.k.a.
explainable planning)
hasbeen well studied as a subarea in explainable AI.
Various terms have been introduced for explainableplanning that are different but share similarities, such as legibility, explicability, transparency, etc.
For a review of their differences, refer to here [4].
For example, legible motion planning [9] isabout generating motion trajectories to better reveal the underlying goal of the agent.
Explicableplanning that we study in this work [36, 22, 13, 35] differs from legible planning in that it focuseson plans that better align with the human’s expectation given the goal .
The key characterizationof explicable planning methods revolves around the idea of model reconciliation where an agentmakes decisions based on two different models instead of one (with a focus on domain dynamics)[5, 3, 4].
Prior methods on explicable planning addressed the problem in a classical planning setting(e.g.
, PDDL [12]) under deterministic domains.
We argue that explicable planning should also beconsidered in a learning setting where human users can provide feedback on the agent’s behavior forthe agent to improve over time (i.e., for personalization).
Furthermore, extending explicable planningto a learning setting makes the approach applicable in stochastic environments with continuous stateand action spaces.
Differing from the prior work, we consider that the user’s reward model may alsodiffer from the agent’s, but assume the differences do not introduce a discrepancy in the perception ofthe agent’s “goal”.
Considering multiple candidate goals is the setting of legible planning [9].
As seen from Fig.
1b, estimating the human’s expectation requires learning both the human’s beliefabout domain dynamics and her reward model based on human feedback.
Existing work on inversereinforcement learning (IRL) [1, 29, 37] and reward learning [8, 32, 10] learns the reward modelfrom human data with the assumption that the human has access to an accurate model of domaindynamics.
It was shown that the human’s inaccurate belief of domain dynamics may skew the humanfeedback and lead to learning the opposite preferences with respect to the human’s true rewardmodel [14].
A similar situation exists in policy learning for which various methods have been devisedthat use human feedback, which include reward shaping, policy shaping [15], and interactive RL[21, 27, 7, 24].
When the human has an inaccurate belief of domain dynamics, as we will show, it canlead to non-convergence during learning or high variances in the learned behaviors for these methods.
While it may be possible to learn the human’s belief of the domain dynamics separately [30], thebelief and human’s reward model are generally tightly coupled in the human feedback and should bejointly learned.
Joint learning is possible [20, 14] but very challenging due to its high dimensionality,which makes it impractical for real-world domains.
When considering only differences in domaindynamics, our problem may be viewed as a form of off-dynamics learning [11] that addresses transferlearning from a source to a target domain.
However, we do not have direct access to the target domainin EPS, which is hidden in the human’s mind.
Problem Formulation
In this paper, we formulate any task domain as a Markov Decision Process (MDP).
An MDP isrepresented by a tuple M =
( S , A , T , r, ρ, γ ) , where S is the set of states, A the set of actions,
T ( s | s, a ) the transition function, r the reward function, ρ ( · ) the initial state distribution, and γthe discount factor.
For the problem setting as shown in Fig.
1b, we need to consider two MDPs.
Assuming the two MDPs share the same S , A , ρ , and γ , one is the agent’s model M A with the truedomain dynamics T A and the engineered reward function r A , and the other is the human’s model of expectation M HA with her belief about the domain dynamics T HA and the human’s reward functionr H .
For explicable policy search, T A , T HA , and r H are unknown .
Definition 1.
Explicable Policy Search (EPS) is the problem of searching for a policy via learningto maximize two objectives: the expected cumulative reward, and a policy explicability score betweenthe agent’s policy under M A and the expected policy under the human’s model M HA .
In this paper, we consider a linearly weighted sum of the two objectives:
[Equation]
[Table]
[Equation]
where π and π H denote the agent’s policy and human’s expected policy, respectively.
We combinethe two objectives linearly via a reconciliation factor λ to be consistent with the literature [36, 22]and simplify the technical development.
We introduce the policy explicability score for stochastic environments, which differs fundamentallyfrom the explicability scores defined in the prior work.
In [36, 22], explicability scores have beenconsidered as a similarity metric between the agent’s plan and the expected plan in the human’smind.
Working in a stochastic setting, foremost, requires us to consider distributions of trajectories.
Hence, a natural choice for the policy explicability score is the negative KL-divergence of the twodistributions of trajectories under the agent and human’s models and policies, respectively.
In such acase, the second objective in Eq.
(1) becomes a measurement of consistency between the generatedagent’s trajectories and the human’s expectation.
Intuitively, this means that an agent that only learnsto maximize it would learn to replicate (as closely as possible) what the human expects the agent tobehave–exactly what we set out to pursue!
In such a case, Eq.
(1) can be rewritten as:
[Equation]
where p A ( τ ) and p HA ( τ ) denote the distributions of trajectories for the agent and human, respectively.
Remark :
Depending on whether the agent’s state and action are observable, the distribution oftrajectories must be computed differently.
When both the state and action are observable, the humanwould be able to contrast both the agent’s action and resulting state with her expectation; otherwise,only the observable part needs to be considered.
In the following discussion, we assume that both thestate and action are observable.
For explicable policy search, the implication here is that the agentoptimizing Eq.
(2) would actively avoid parts of the state space where either its policy differs fromthe human’s expectation or the dynamics differs, which is reflected in the final solution derived.
Explicable Policy Search (EPS)
To present our learning method, we start by expressing the probability distribution of the agent’strajectories and human’s expectation.
We parameterize the agent’s policy using θ .
The probability ofrealizing the agent’s trajectory τ with π θ (and similarly for the human’s expectation) is:
[Equation]
Given these two distributions, we can now derive our solution for EPS.
Since both objectives in
Eq.
(2) are expressed as expectations over the same distribution of trajectories (i.e., p A ( τ ) ), we cancombine them after rewriting the policy explicability score in Eq.
(2) as follows:
[Equation]
where C is a constant.
The main challenge in solving the optimization problem in Eq.
(2) now lies inthe fact that neither the human’s policy π H nor her belief of the domain dynamics T HA are known.
4.1 Surrogate Reward Function
Eq.
(4) can be merged into Eq.
(2) such that the log terms essentially reshape the engineeredreward function.
In such a case, a straightforward approach is to learn the human’s belief about thedomain dynamics and her expected policy separately based on human feedback, while maintainingestimates of the true domain dynamics and current agent’s policy.
While possible, it is inefficientand unnecessary.
Instead, we propose to use a surrogate reward function.
The goal is to learn such afunction u H that retains the necessary information about the human’s belief of domain dynamics andexpected policy for policy search, i.e., u H .
=
log T HA ( s t +1 | s t , a t ) + log π HA ( a t
| s t ) .
Intuitively, we learn a reward function that alone can explain the human’s expectation of thetrajectories–a reward function that introduces the same distribution of the expected trajectories.
At the same time, we must take care to minimize the human feedback to make it practical.
A popularapproach that imposes feasible input requirement on human is preference-based RL [34, 7].
For EPS,instead of soliciting human preferred trajectories, we can present pairs of trajectories and ask humansto comment on which one is more expected .
Then, we can fit a reward function that would “prefer”more expected trajectories.
More specifically, we can try to correlate the distribution of expectedtrajectories with the surrogate reward function to be learned as follows:
[Equation]
Proposition 1.
There exists a unique reward function u H such that trajectory distribution underthe softmax human preference model with u H described above matches with the human’s expectedtrajectory distribution given in Eq.
When we compare mathematically the two equations of p HA ( τ ) (i.e., Eqs.
(3) and (5)), we see that theonly way to match the distributions is by satisfying:
[Equation]
One straightforward way for this is to set u H ( s t , a t )
=
log T HA ( s t +1 | s t , a t ) + log π HA ( a t
| s t ) .
In theother direction, since the trajectories may be of various lengths in general (including the length of 1as special cases), we can conclude further that:
[Equation]
Plugging this result into any trajectory of length greater than 1 , we can conclude that C 1 = 0 .
This means that u H in such a case becomes equivalent to log T HA and log π H for representing thedistribution of expected trajectories, which implies that the information needed from T HA and π Hfor optimizing Eq.
(2) can be substituted with u H .
u H can be learned by applying preference-basedlearning with a softmax preference model based on human feedback as described earlier (more detailsto follow).
To learn the unique surrogate reward function and avoid the non-identifiabillity issue [31],however, requires the learning to be globally optimized.
In our implementation, we simply normalizethe learned rewards.
Now, we rewrite Eq.
(2) based on the above result:
[Equation]
Note that Eq.
(6) is an approximation of the original objective in Eq.
(2) by ignoring the influence ofthe discount factor on the surrogate reward function and entropy terms.
This allows us to use them toreshape the reward function.
We can view this objective function as two parts.
The first two terms,r A ( s t , a t ) + λu H ( s t , a t ) , requires the agent to maximize rewards from two sources: the engineeredreward and the surrogate reward learned from human feedback.
These two reward functions areweighted according to the reconciliation parameter as expected.
The second part of this objectivefunction is
H T A [ s t +1 | s t , a t ]
+ H
π
θ
[ a t | s t ] , which are two entropy terms.
The first term is for theagent’s domain dynamics while the second term is for the target policy.
The second entropy term for the target policy (referred to as “ policy entropy ”) is well studied in themaximum entropy (MaxEnt) RL framework.
[17, 18], However, we note that this term in our workis derived for a completely different reason.
In MaxEnt RL, maximizing H π θ encourages the agent to explore during learning and increase robustness.
Given a fixed surrogate reward function, thisterm in EPS likewise encourages stochasticity in the agent’s policy to increase robustness.
A moredistinguishing feature of explicable policy search lies in the first term.
Maximizing H T
A encouragesthe agent to prefer parts of the environment where there is more stochasticity, which we refer to as“ environment entropy ”.
The incentive here is to reduce the influence to the policy explicability scoredue to differences in the agent’s policy and expected policy.
Intuitively, at more stochastic parts ofthe environment where action choices for the agent matter less, the agent’s trajectory is more likely tomatch with the human’s expectation even when the agent’s policy and the expected policy differ.
Expectation-based Preferences:
For learning u H , we apply a preference-based learning framework.
We request humans to provide their feedback on which segment in a pair of segments{ ( σ 1 , σ 2 ) } extracted from the agent’s trajectories is more expected .
To consider noise in humanfeedback, the human’s expectation preference is formulated as follows [7]:
[Equation]
We learn u H to minimize the cross entropy between our prediction of the expectation preferencesand feedback data.
In order to efficiently solicit the human’s expectation preference for the agent’sbehavior, we leverage uncertainty-based sampling [7, 25] to select traces for the human in an activelearning fashion.
To estimate T A , we assume it follows a Gaussian distributions and model it using atwo-headed neural network that outputs its mean and logarithm of standard deviation given the stateand action.
The agent interacts with the environment and collects transition data for learning.
T A isestimated using a data-driven method by minimizing the L 2 one-step prediction loss [23].
We use
Soft Actor Critic (SAC) [18] for policy learning but other policy search methods are also applicable.
To alleviate the issue of non-stationary prediction of expectation preferences during learning, werelabel the data samples every time u H is updated [24].
The algorithm for EPS is in the appendix.
Evaluation
We evaluate our method on a set of continuous navigation domains with synthetic human models anda simulated autonomous driving domain with a human subject study.
The study is IRB approved andall protocols have been followed.
The synthetic experiments are used to validate the effectiveness ofour method for searching for explicable policies.
The user study is to 1) confirm that our beliefs aboutdomain dynamics can be inaccurate or biased, which can affect our judgements and lead to severeconsequences if ignored, and 2) show that our method can effectively address such hidden issues byachieving an intelligent reconciliation between the task performance and human’s expectation.
5.1 Synthetic Navigation Domains
We conduct experiments on four navigation domains with continuous state and action spaces, asillustrated in Fig.
More detailed descriptions of our domains are in the appendix.
For all thedomains, the agent starts from the upper left corner and aims to navigate to the goal area (depictedin green).
We introduce various human inaccurate beliefs about domain dynamics for these fourdomains 2 .
For Domain 1 (D1) and Domain 2 (D2) (i.e., the first two rows in Fig. 2), the humanbelieves that the agent is more likely to fall into the pit when it is close by; the truth is everywhereis the same.
For Domain 3 (D3) and Domain 4 (D4) (i.e., the third and fourth row in Fig. 2), thehuman’s belief is that the agent could easily slip on ice.
Moreover, the human believes that the agentcan readily handle sandy roads.
The truth is that the agent is proficient on ice but can easily get stuckon sand in D3.
For D4, the agent is in addition capable of handling sand albeit being more costly.
The human’s expectation preferences for the agent’s behaviors is generated synthetically with respectto ground truth human model.
5.1.1 Baselines
We compared our method to three baselines to illustrate its advantages compared to the traditional RLmethods that do not consider model differences: Soft Actor Critic (SAC) [18], Deep RL from Human D
[Figure] Preferences (DRLHP) [7], and Policy Shaping (PS) [15].
SAC optimizes policy with respect to theengineered reward function and true domain dynamics without considering the human’s expectation.
DRLHP uses human expectation-based preferences for the agent’s behavior to estimate a rewardfunction and applies it to guide policy search.
It ignores the engineered reward function.
For SACand DRLHP, we could not use reward signals from both sources due to their different formats.
Theclosest competitor to EPS is policy shaping.
Policy shaping learns two policies using reward signalsfrom the environment and human expectation preferences, respectively.
A combination policy is thenobtained by multiplying them together.
It also seeks to combine different information sources thatare however assumed to be consistent.
That is why it has difficulty handling situations where thehuman’s expectation and agent’s optimal behavior conflict (see D3 in Fig. 2).
5.1.2 Results and Discussion
[Figure]
Table 1 shows the average return and policy explicability score of EPS compared to the baselines over 100 rollouts.
The policy explicabilityscore is computed using Eq.
(4) based on thesynthetic human models.
For EPS, the reconciliation parameter λ is tuned manually to show thedifferent behaviors compared to the baselines in
Fig. 2, with λ ∈ [2 . 0 , 2 . 8] 3 .
For all domains,
EPS achieved the highest policy explicabilityscore and followed the best performer for average return.
PS performed well on most taskswhile being considerably worse than others in D3.
Due to the conflict between the human’s expectation and task priorities in D3, the methodfailed to achieve a meaningful combination.
DRLHP performed poorly on all the domains becausepreference-based learning methods have difficulty dealing with domains with sparse or delayedrewards, which makes credit assignment challenging and results in large variations.
Since EPS alsoapplies a preference-based learning method to learn the surrogate reward function, we demonstratethis using the learned function.
As shown in Fig.
3, this function has a significant amount ofuncertainty so is not ideal for guiding policy search.
Although it adds variability into policy learning, [Table] it serves very well as an auxiliary objective by informing the agent where the human expects it toperform.
We show sampled trajectories for EPS, the baselines, and the human’s expectation computed usingthe synthetic human models in Fig.
In D1 and D2, SAC agent always chooses the shortest pathwhile EPS agent takes a detour that bypasses the pit and hence is more explicable.
In D3 and D4, SACagent chooses the path passing through the top since the agent can get stuck on sandy roads in D3and the top path is more efficient in D4.
It is worth noting that EPS agent makes different decisionson these two domains.
It selects the top path in D3 when the sandy road is difficult to navigate eventhough
it is against the human’s expectation.
However, when it can better navigate on sand in D4,it chooses to respect the human’s expectation to be more explicable, at the cost of lowering taskperformance.
Policy shaping agent is also successful in D4, but gets stuck in D3, because simplymultiplying two different policies could result in a new policy that is uninformative, irrespective ofthe λ used (e.g., when we have p 1 = (0 . 1 , 0 . 9) , p 2 = (0 . 9 , 0 . 1) , the resulting policy from multiplyingthem would always be p
= (0 . 5 , 0 . 5) ), which can lead to poor behaviors.
In summary, we show that
EPS can successfully generate effective behaviors that achieve an intelligent reconciliation betweenthe human’s expectation and task priorities.
It can generate novel reconciled behaviors (D1 and D2),stick to the task priorities (D3), or follow the human’s expectation (D4), as appropriately.
5.2 Autonomous Driving Domain
We used a simulated autonomous driving domain [26] to evaluate our method with human subjectsand demonstrate its real-world relevance, as illustrated in Fig. 4.
The state space is featured by theposition and velocity of the ego-vehicle and nearby vehicles.
The action space consists of five discreteactions.
Initially, the autonomous driving vehicle (green) is running on the highway with a car in frontof it (blue) on the same lane and with the same speed.
The task of the autonomous driving agent is tohandle situations when the front car slows down quickly and abruptly.
One common type of cognitivebias is the availability bias , which reflects humans’ tendency to overestimate the likelihood of eventswith greater availability in memory.
Such biases can occur in driving since we regularly drive underfamiliar conditions.
To design an experiment where such biases are present, we considered scenarioswith a regular driving condition and a sleety condition where the vehicle’s domain dynamics becamemore stochastic due to slippery roads.
With the availability bias, however, humans are likely to makethe same decisions under these similar but in actuality different conditions, leading to safety risks.
The user study consists of two phases: training and testing.
At the beginning of the training phase,we requested the participates to provide the importance ratings for several factors governing theautonomous driving behavior [6] in a 5-point Likert scale, such as average speed (3.42), distanceto the front car (4.57), relative speed to neighboring cars (3.28), and lane following (3.78).
Theaverage participants’ responses are shown in the parentheses above.
Their responses served as theengineered reward model ( r A ) and are linearly combined.
For example, the distance to the front car isdirectly associated with collision and hence given a larger weight.
Before human data collection, wealso presented the participants with the vehicle’s behavior after braking under the regular conditionwhen the front car slows down suddenly.
Then, given the information that it is driving on a sleetyday, we selected segments that showcased different vehicle behaviors after braking (i.e., brakingbehaviors under different effectiveness).
The participants were asked to select which one matchedtheir expectations the most.
Their responses were used to validate the availability biases in thisexperiment.
Then, we collected human data by actively selecting pairs of trajectory segments forthe participants to compare with.
The collected data was used to train our method and the baselines.
Then, we showed rollouts of the learned policy for each method to new participants and asked themto rate those rollouts in the testing phase. [Figure]
5.2.1 Results and Discussion
We published the experiments on Amazon Mechanical Turk (MTurk).
To sift out invalid responses,a sanity check demonstration was added that showed a collision, which should not be preferredunder any situation.
We recruited 15 participants for training, and one failed the sanity check.
Forthe bias validation question, 12 out of 14 valid participants chose that the ego-vehicle would slowdown effectively on a sleety day when braking.
This reflects the availability bias that the participantshad, which can lead to collisions on a slippery road if ignored.
We also noticed that the participantspreferred to brake than steer in general during training, which accords with the participants’ responsesthat attached a high importance rating to distance to the front car and lane following.
The testing phase occurs on a sleety day.
The behavior of EPS, SAC, and DRLHP agents areillustrated in Fig.
SAC agent steers immediately when the front car slows down.
It is the mostefficient behavior with the most return on a slippery road since braking would not be effective.
DRLHP agent, on the other hand, chooses to brake while staying in the same lane.
Such a behavior,however, is more likely to lead to a collision (risk to human passengers).
As we can see from Fig.
4,the agent is getting dangerously close to the vehicle in the front.
EPS agent first chooses to brake(to be explicable), and then steers to the other lane (to stay safe and continue moving forward),which maintains both explicability and task efficiency.
We demonstrated these rollouts in the testingphase to new participants.
We informed the participants that the vehicle was running on a sleetyday.
Each participant was required to provide ratings to all the demonstrations ranging from 1 to 10 .
We obtained 15 valid responses.
Interestingly, the participants rated EPS agent the most preferable,followed by DRLHP and SAC agents.
The average rating for each agent and its standard deviationare shown in Fig.
We note that the results contradict with the importance ratings solicited for theengineered reward model, which should have led to preferring the steering behavior as chosen by the
SAC agent on slippery roads.
This further suggests that the participants had a biased belief aboutthe vehicle’s dynamics (i.e., braking is as efficient under the regular condition as under the sleetycondition).
Regardless, our agent chose the behavior that was both explicable and safe.
Discussions and Conclusions
In this paper, we introduced and formulated the problem of explicable policy search that considersmodel differences between the learning agent and its human observer.
We developed an efficientsolution by learning a surrogate reward function that was then used to recover an explicable policy.
Our method significantly extends explicable planning to an RL setting and to stochastic environmentswith continuous state and action spaces.
We evaluated in simulations and with human subjects.
Results showed that our approach could better handle situations under model differences than severalbaselines and thus contributed a critical tool to achieving explainable human-agent interaction.
Limitations and Future Work.
We assume that the human’s (inaccurate) belief about the domaindynamics does not easily update as observations of the agent are made.
For scenarios whereinaccuracies stem from intrinsic cognitive biases that are difficult to change, this is a reasonableassumption.
In other cases where discrepancies were due to, e.g., information asymmetries, the beliefcan change dynamically and needs to be actively monitored [19].
Other possible directions includemore complex combinations of the expected return and policy explicability score, as well as weightedpolicy explicability scores to incorporate trajectory similarity as considered in prior work [36, 22].
Societal Impacts.
Our research is fundamental in nature without notable negative societal impacts.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary
Did you include the license to the code and datasets?
N/A
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplemental material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See supplemental material.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See results in table 1.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplemental material.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See supplementalmaterial.
(b) Did you mention the license of the assets?
[Yes] See supplemental material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See supplemental material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] No such info or content was used or curated.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[Yes] See supplemental material.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes] See supplemental material.