Explicable Policy Search
Anonymous Author(s)
AffiliationAddress email
Abstract
Human teammates often form conscious and subconscious expectations of eachother to better interact.
Teaming success is contingent on whether such expectationscan be met.
For an intelligent agent to operate with humans beside them, similarly,it must consider the human’s expectation of its behavior.
Otherwise, the loss oftrust and degraded team performance may be resulted.
A key challenge here isthat the human’s expectation may not align with the agent’s optimal behaviordue to, for example, the human’s partial or inaccurate understanding of the taskdomain.
Prior work on explicable planning describes the ability of agents torespect their human teammate’s expectations by trading off task performance formore expected or “ explicable ” behaviors.
In this paper, we introduce Explicable
Policy Search (EPS) to significantly extend such an ability to a reinforcementlearning (RL) setting and to handle stochastic domains with continuous state andaction spaces.
However, in contrast to the traditional RL method, EPS mustat the same time infer the human’s hidden expectations.
Such inferences requireinformation about the human’s belief about domain dynamics and her reward modelbut directly querying them is impractical.
We demonstrate that this informationcan be sufficiently encoded by a surrogate reward function, which can be learnedbased on the human’s feedback on the agent’s behavior.
The surrogate rewardfunction is then used to reshape the agent’s reward function, which is shown to beequivalent to searching for an explicable policy.
We evaluate our method for EPSin a set of continuous navigation domains with synthetic human models and in anautonomous driving domain with a user study.
The results suggest that our methodcan generate explicable behaviors that reconcile task performance with humanexpectation intelligently and has real-world relevance in human-agent teamingdomains.
Introduction
Intelligent agents are quickly becoming a part of our daily lives in a variety of domains, includingsmart home, autonomous driving, education and so on.
In such domains, the agents are expectedto perform in human inhabited environments and even collaborate closely with them.
Members insuch a collaborative setting often form conscious and subconscious expectations of each other andthe success of collaboration depends on whether such expectations can be met.
A key challengehere is that the human’s expectation may not align with the agent’s optimal behavior.
Hence, agentschoosing their optimal behaviors without considering the human observers or collaborators could beseen as unexpected, leading to degraded team performance and loss of trust [19, 6].
Consider a drone navigation scenario in Fig.
1a where a drone is tasked to navigate from the startingposition (as shown) to a destination.
On a good day, the drone would be expected to mostly navigatestraight to the goal with some air perturbation (thus possibly introducing a slightly curved trajectoryas illustrated in red in Fig. 1a).
However, there is a heavy wind today that changes the domain
[Figure]
dynamics (which a human observer is unaware of) and it becomes impossible for the drone to navigateas usual.
Two alternatives where the drone navigates in a zigzag and a curved pathway.
The curved pathway is more optimal since it makes fewer sharp turns (due to the windy condition)but the former is likely to be perceived as more expected to the human observer, given its closeness tothe expected behavior.
We note here that the explicable behavior goes beyond simply choosing fromor combining the agent’s optimal behavior with the human’s expected behavior.
Hence, achievingsuch a capability requires a more fundamental treatment.
In the experiments, we will show that ourlearning agent can generate novel behaviors that are rarely seen in training scenarios.
The problem setting of explicable policy search (EPS) is illustrated in Fig.
1b, which is fundamentallya model reconciliation business [7].
In such a setting, considering the learning problem only fromthe perspective of the agent would sabotage the teaming when the agent’s behavior differs fromhuman’s expectation.
Our method considers any feedback the human may provide to the agent basedon her expectation of its behavior (generated under her models) to help the agent improve.
Hence,our method has a variety of applications where machines must be tuned to individual users, suchas an autonomous vehicle that learns to improve its driving behavior based on its owner’s feedbackbut at the same time must abide by standardized operating requirements to ensure safety, comfort,etc.
In our problem, we assume that the human is nosily rational at generating her expectation ofthe agent to accommodate her computational limitations.
In general, noisy rationality is one type ofbounded rationality [45] due to a limited computational capability, resulting in sub-optimal behaviors.
While noisy rationality may not perfectly describe our computational model [43] due to variouscognitive biases, modeling humans as such has been a common practice and demonstrated as areasonable assumption in prior methods [34, 2].
Furthermore, similar to prior work on explicableplanning [48, 25], we assume that the human is a sole observer of the agent.
Extending it to ancollaborative setting requires additional machinery and will be discussed in future work, with stepsalready taken in a classical planning setting [47].
In this paper, we formulate explicable policy search (EPS) in Markov Decision Processes (MDPs).
The goal is to learn a policy that reconciles between maximizing the long-term return (based on theengineered reward model r A ) and minimizing the deviation of the agent’s behavior from the human’sexpectation of it, which is quantified by an explicability score.
We formulate the problem as a linearcombination of two objectives and show that such a simple treatment already yields substantialbenefits in our evaluation.
The challenge in this problem formulation mainly lies in the fact that thehuman’s expectation is hidden (but partially observable via human feedback) and must be learned.
A straightforward solution involves learning both the human’s belief about the domain dynamics( T HA ) and her reward model ( r H ) from human feedback, which is possible but impractical.
Instead,we show that the information needed from them for EPS can be sufficiently encoded by a surrogatereward function, which is much easier to learn.
Such a reward function can then be incorporated intoa policy search process to recover an explicable policy.
The final modified reward function for EPS bears some similarity to maximum entropy reinforcement learning [20, 21] but is derived under acompletely different motivation.
For evaluation, we design a set of navigation domains where the human’s feedback on her expectationof the agent’s behavior is synthetically generated based on the true domain dynamics modified withvarious “misunderstandings”.
We compare our method with three baselines that are selected to represent the traditional RL methods to demonstrate the key advantages of EPS under model differences.
Furthermore, to show its real-world relevance, we conduct a human study in an autonomous drivingdomain [29] where we design driving scenarios to elicit existing human biases about the vehicle’sdomain dynamics that could be dangerous to ignore during learning.
The results show that our methodcan intelligently generate explicable behaviors that are safe and preferred over those of the baselines.
The contribution of this paper is three-fold.
First, we introduce and formulate explicable policysearch, which extends explicable planning to a reinforcement learning setting and to stochasticdomains with continuous state and action spaces.
Second, we propose a practical solution for thischallenging problem by introducing a surrogate reward function learned from human feedback data,which sufficiently encodes the necessary information for explicable policy search.
Third, we evaluateour learning method extensively with simulations and human subjects.
Related Work
The problem of generating communicative actions or behaviors (a.k.a.
explainable planning)
hasbeen well studied as a subarea in explainable AI.
Various terms have been introduced for explainableplanning that are different but share similarities, such as legibility, explicability, transparency, etc.
For a review of their differences, refer to here [7].
For example, legible motion planning [12] isabout generating motion trajectories to better reveal the underlying goal of the agent.
Explicableplanning that we study in this work [48, 25, 16, 47] differs from legible planning in that it focuseson plans that better align with the human’s expectation given the goal .
The key characterizationof explicable planning methods revolves around the idea of model reconciliation where an agentmakes decisions based on two different models instead of one (with a focus on domain dynamics)[8, 6, 7].
While there exists prior work [33, 5] on training value-aligned learning agents that mustlearn to respect two different reward models, they do not consider the differences in the domaindynamics.
Prior methods on explicable planning addressed the problem in a classical planning setting(e.g.
, PDDL [15]) under deterministic domains.
We argue that explicable planning should also beconsidered in a learning setting where human users can provide feedback on the agent’s behavior forthe agent to improve over time (i.e., for personalization).
Furthermore, extending explicable planningto a learning setting makes the approach applicable in stochastic environments with continuousstate and action spaces.
Differing from the prior work, we consider that the user’s reward modelmay also differ from the agent’s, but assume the differences do not introduce a discrepancy in theperception of the agent’s “goal”.
Considering multiple candidate goals is the setting of legibleplanning [12].
Note that generating explainable behavior has a subtle but important difference fromgenerating interpretable behavior [44, 35, 38, 39], where the focus is to make the human understandthe agent’s behavior, not necessarily to “expect” its behavior.
In particular, an interpretable behaviormay not be an expected behavior.
However, an expected behavior must be interpretable.
In high-stakedomains where “surprises” should be minimized if at all possible, explicable behaviors would bemore desirable.
As seen from Fig.
1b, estimating the human’s expectation requires learning both the human’s beliefabout domain dynamics and her reward model based on human feedback.
Existing work on inversereinforcement learning (IRL) [1, 36, 49] and reward learning [11, 41, 13] learns the reward modelfrom human data with the assumption that the human has access to an accurate model of domaindynamics.
It was shown that the human’s inaccurate belief of domain dynamics may skew thehuman feedback and lead to learning the opposite preferences with respect to the human’s truereward model [17].
A similar situation exists in policy learning for which various methods havebeen devised that use human feedback, which include reward shaping, policy shaping [18], andinteractive RL [24, 32, 10, 27].
When the human has an inaccurate belief of domain dynamics, as wewill show, it can lead to non-convergence during learning or high variances in the learned behaviorsfor these methods.
While it may be possible to learn the human’s belief of the domain dynamicsseparately [37], the belief and human’s reward model are generally tightly coupled in the humanfeedback and should be jointly learned.
Joint learning is possible [23, 17] but very challenging due to its high dimensionality, which makes it impractical for real-world domains.
Another possibleapproach [30] (InfoGAIL) that may be leveraged uses latent code to discover salient semantic featuresto accommodate demonstrations from different experts.
Even though the aim is quite different fromours, one may consider the biases in the domain dynamics as captured by the latent code.
However,their work is focused on “discovering” the differences while we focus on “reconciling” the differences(without explicit modeling of the differences).
For example, it likely would fail in situations whereall experts are biased.
When considering only differences in domain dynamics, our problem may beviewed as a form of off-dynamics learning [14] that addresses transfer learning from a source to atarget domain.
However, we do not have direct access to the target domain in EPS, which is hidden inthe human’s mind.
Problem Formulation
In this paper, we formulate any task domain as a Markov Decision Process (MDP).
An MDP isrepresented by a tuple M =
( S , A , T , r, ρ, γ ) , where S is the set of states, A the set of actions,
T ( s | s, a ) the transition function, r the reward function, ρ ( · ) the initial state distribution, and γthe discount factor.
For the problem setting as shown in Fig.
1b, we need to consider two MDPs.
Assuming the two MDPs share the same S , A , ρ , and γ , one is the agent’s model M A with the truedomain dynamics T A and the engineered reward function r A , and the other is the human’s model ofexpectation M HA with her belief about the domain dynamics T HA and the human’s reward functionr H .
For explicable policy search, T A , T HA , and r H are unknown .
Note that our approach can also beeasily applied to situations where T A is known in model-based learning methods.
Definition 1.
Explicable Policy Search (EPS) is the problem of searching for a policy via learningto maximize two objectives: the expected cumulative reward, and a policy explicability score betweenthe agent’s policy under M A and the expected policy under the human’s model M HA .
In this paper, we consider a linearly weighted sum of the two objectives:
[Equation]
[Table]
[Equation]
where π and π H denote the agent’s policy and human’s expected policy, respectively.
We combinethe two objectives linearly via a reconciliation factor λ to be consistent with the literature [48, 25]and simplify the technical development.
Next, we motivate the design of our explicability metric.
Intuitively, a model can better explain thedata when it assigns the data a high probability.
Similarly, under the assumption that the humangenerates her expectations based on T HA , and r H , a behavior (i.e., trajectory) τ would be more likelyto be generated by the human’s model (i.e., expected by the human) when it is associated with a highprobability in the model.
In contrast, when the trajectory has a low probability, we would say that it is"inexplicable" with respect to the model.
As a result, the probability of the trajectory in the human’smodel ( T HA and r H ), denoted by p HA ( τ ) , may be considered as an explicability metric as the firstthought.
However, using p HA ( τ ) only ignores an important aspect of explicability under stochasticsetting since it is relative at the same time: a trajectory that is more expected should also appear morefrequently in the real-world.
Likewise, a trajectory that is less expected should appear less.
Thismeans that the explicability metric should be specified with respect to the relative frequency of thetrajectory in the human’s model and the agent’s model, e.g., p HA ( τ ) p A ( τ ) .
Hence, evaluating the explicabilitymetric under stochastic domains becomes computing the expectation of such an explicability metricunder a distribution of the trajectories.
This essential equates the choice for our explicability scoreto the negative KL-divergence of the two distributions of trajectories under the agent and human’smodels, respectively.
Hence, we introduce the policy explicability score for stochastic environments, which differs fundamentally from the explicability scores defined in the prior work.
In [48, 25], explicability scores have been considered as a similarity metric between the agent’s plan and the expected plan in the human’smind.
Policy explicability extends such a metric to consider distributions of plans.
With the newlyestablished policy explicability metric, the second objective in Eq.
(1) becomes a measurement ofconsistency between the generated agent’s trajectories and the human’s expectation.
Intuitively, thismeans that an agent that only learns to maximize it would learn to replicate (as closely as possible)what
the human expects the agent to behave–exactly what we set out to pursue!
Also note that whenthe two distributions match exactly, the policy explicability score would be 0 , exerting no influenceon the final policy as desired!
In such a case, Eq.
(1) can be rewritten as:
[Equation]
where p A ( τ ) and p HA ( τ ) denote the distributions of trajectories for the agent and human, respectively.
Remark :
Depending on whether the agent’s state and action are observable, the distribution oftrajectories must be computed differently.
When both the state and action are observable, the humanwould be able to contrast both the agent’s action and resulting state with her expectation; otherwise,only the observable part needs to be considered.
In the following discussion, we assume that both thestate and action are observable.
For explicable policy search, the implication here is that the agentoptimizing Eq.
(2) would actively avoid parts of the state space where either its policy differs fromthe human’s expectation or the dynamics differs, which is reflected in the final solution derived.
Explicable Policy Search (EPS)
To present our learning method, we start by expressing the probability distribution of the agent’strajectories and human’s expectation.
We parameterize the agent’s policy using θ .
The probability ofrealizing the agent’s trajectory τ with π θ (and similarly for the human’s expectation) is:
[Equation]
Given these two distributions, we can now derive our solution for EPS.
Since both objectives in
Eq.
(2) are expressed as expectations over the same distribution of trajectories (i.e., p A ( τ ) ), we cancombine them after rewriting the policy explicability score in Eq.
(2) as follows:
[Equation]
where C is a constant.
The main challenge in solving the optimization problem in Eq.
(2) now lies inthe fact that neither the human’s policy π H nor her belief of the domain dynamics T HA are known.
4.1 Surrogate Reward Function
Eq.
(4) can be merged into Eq.
(2) such that the log terms essentially reshape the engineeredreward function.
In such a case, a straightforward approach is to learn the human’s belief about thedomain dynamics and her expected policy separately based on human feedback, while maintainingestimates of the true domain dynamics and current agent’s policy.
While possible, it is inefficientand unnecessary.
Instead, we propose to use a surrogate reward function.
The goal is to learn such afunction u H that retains the sufficient and necessary information about the human’s belief of domaindynamics and expected policy for policy search, i.e., u H .
=
log T HA ( s t +1 | s t , a t ) + log π HA ( a t
| s t ) .
Next, we reveal how such a goal can be achieved by manipulating a preference-based formulation.
Intuitively, we learn a reward function that alone can explain the human’s expectation of thetrajectories–a reward function that introduces the same distribution of the expected trajectories.
At the same time, we must take care to minimize the human feedback to make it practical.
A popularapproach that imposes feasible input requirement on human is preference-based RL [46, 10].
For
EPS, instead of soliciting human preferred trajectories, we can present pairs of trajectories and askhumans to comment on which one is more expected .
Then, we can fit a reward function that would
“prefer” more expected trajectories.
More specifically, we can try to correlate the distribution ofexpected trajectories with the surrogate reward function to be learned as follows:
[Equation]
Proposition 1.
There exists a unique reward function u H such that trajectory distribution underthe softmax human preference model with u H described above matches with the human’s expectedtrajectory distribution given in Eq.
When we compare mathematically the two equations of p HA ( τ ) (i.e., Eqs.
(3) and (5)), we see that theonly way to match the distributions is by satisfying:
[Equation]
One straightforward way for this is to set u H ( s t , a t )
=
log T HA ( s t +1 | s t , a t ) + log π HA ( a t
| s t ) .
In theother direction, since the trajectories may be of various lengths in general (including the length of 1as special cases), we can conclude further that:
[Equation]
Plugging this result into any trajectory of length greater than 1 , we can conclude that C 1 = 0 .
Thisresult is similar to that in MaxEnt IRL [49].
Hence, our learning approach based on such a model isalso one that does not exhibit any additional preferences beyond matching the human feedback.
This means that u H in such a case becomes equivalent to log T HA and log π H for representing thedistribution of expected trajectories, which implies that the information needed from T HA and π Hfor optimizing Eq.
(2) can be substituted with u H .
u H can be learned by applying preference-basedlearning with a softmax preference model based on human feedback as described earlier (more detailsto follow).
To learn the unique surrogate reward function and avoid the non-identifiabillity issue [40],however, requires the learning to be globally optimized.
In our implementation, we simply normalizethe learned rewards.
Now, we rewrite Eq.
(2) based on the above result:
[Equation]
Note that Eq.
(6) is an approximation of the original objective in Eq.
(2) by applying discountingto the surrogate reward function and entropy terms.
This allows us to use them to reshape thereward function.
We can view this objective function as two parts.
The first two terms, r A ( s t , a t ) +λu H ( s t , a t ) , requires the agent to maximize rewards from two sources: the engineered rewardand the surrogate reward learned from human feedback.
These two reward functions are weightedaccording to the reconciliation parameter as expected.
The second part of this objective function is
H T A [ s t +1 | s t , a t ]
+ H
π
θ
[ a t | s t ] , which are two entropy terms.
The first term is for the agent’s domaindynamics while the second term is for the target policy.
The second entropy term for the target policy (referred to as “ policy entropy ”) is well studied in themaximum entropy (MaxEnt) RL framework.
[20, 21], However, we note that this term in our workis derived for a completely different reason.
In MaxEnt RL, maximizing H π θ encourages the agentto explore during learning and increase robustness.
Given a fixed surrogate reward function, thisterm in EPS likewise encourages stochasticity in the agent’s policy to increase robustness.
A moredistinguishing feature of explicable policy search lies in the first term.
Maximizing H T
A encouragesthe agent to prefer parts of the environment where there is more stochasticity, which we refer to as“ environment entropy ”.
The incentive here is to reduce the influence to the policy explicability scoredue to differences in the agent’s policy and expected policy.
Intuitively, at more stochastic parts ofthe environment where action choices for the agent matter less, the agent’s trajectory is more likely tomatch with the human’s expectation even when the agent’s policy and the expected policy differ.
Expectation-based Preferences:
For learning u H , we apply a preference-based learning framework.
We request humans to provide their feedback on which segment in a pair of segments{ ( σ 1 , σ 2 ) } extracted from the agent’s trajectories is more expected .
To consider noise in human feedback, the human’s expectation preference is formulated as follows, similar to that in [10, 31, 3, 4]:
[Equation]
We learn u H to minimize the cross entropy between our prediction of the expectation preferencesand feedback data.
In order to efficiently solicit the human’s expectation preference for the agent’sbehavior, we leverage uncertainty-based sampling [10, 28] to select traces for the human in an activelearning fashion.
To estimate T A , we assume it follows a Gaussian distributions and model it using atwo-headed neural network that outputs its mean and logarithm of standard deviation given the stateand action.
The agent interacts with the environment and collects transition data for learning.
T A isestimated using a data-driven method by minimizing the L 2 one-step prediction loss [26].
We use
Soft Actor Critic (SAC) [21] for policy learning but other policy search methods are also applicable.
To alleviate the issue of non-stationary prediction of expectation preferences during learning, werelabel the data samples every time u H is updated [27].
The algorithm for EPS is in the appendix.
Evaluation
We evaluate our method on a set of continuous navigation domains with synthetic human models anda simulated autonomous driving domain with a human subject study.
The study is IRB approved andall protocols have been followed.
The synthetic experiments are used to validate the effectiveness ofour method for searching for explicable policies.
The user study is to 1) confirm that our beliefs aboutdomain dynamics can be inaccurate or biased, which can affect our judgements and lead to severeconsequences if ignored, and 2) show that our method can effectively address such hidden issues byachieving an intelligent reconciliation between the task performance and human’s expectation.
5.1 Synthetic Navigation Domains
We conduct experiments on four navigation domains with continuous state and action spaces, asillustrated in Fig.
More detailed descriptions of our domains are in the appendix.
For all thedomains, the agent starts from the upper left corner and aims to navigate to the goal area (depictedin green).
We introduce various human inaccurate beliefs about domain dynamics for these fourdomains 2 .
For Domain 1 (D1) and Domain 2 (D2) (i.e., the first two rows in Fig. 2), the humanbelieves that the agent is more likely to fall into the pit when it is close by; the truth is everywhereis the same.
For Domain 3 (D3) and Domain 4 (D4) (i.e., the third and fourth row in Fig. 2), thehuman’s belief is that the agent could easily slip on ice.
Moreover, the human believes that the agentcan readily handle sandy roads.
The truth is that the agent is proficient on ice but can easily get stuckon sand in D3.
For D4, the agent is in addition capable of handling sand albeit being more costly.
The human’s expectation preferences for the agent’s behaviors is generated synthetically with respectto ground truth human model.
5.1.1 Baselines
We compared our method to three baselines to illustrate its advantages compared to the traditional RLmethods that do not consider model differences: Soft Actor Critic (SAC) [21], Deep RL from Human
Preferences (DRLHP) [10], and Policy Shaping (PS) [18].
SAC optimizes policy with respect to theengineered reward function and true domain dynamics without considering the human’s expectation.
DRLHP uses human expectation-based preferences for the agent’s behavior to estimate a rewardfunction and applies it to guide policy search.
It ignores the engineered reward function.
For SACand DRLHP, we could not use reward signals from both sources due to their different formats.
Theclosest competitor to EPS is policy shaping.
Policy shaping learns two policies using reward signalsfrom the environment and human expectation preferences, respectively.
A combination policy is thenobtained by multiplying them together.
It also seeks to combine different information sources thatare however assumed to be consistent.
That is why it has difficulty handling situations where thehuman’s expectation and agent’s optimal behavior conflict (see D3 in Fig. 2).
[Figure]
5.1.2 Results and Discussion
[Figure]
Table 1 shows the average return and policy explicability score of EPS compared to the baselines over 100 rollouts, with the sample standard deviations.
The policy explicability scoreis computed using Eq.
(4) based on the synthetic human models.
For EPS, the reconciliation parameter λ is tuned manually to show thedifferent behaviors compared to the baselines in
Fig. 2, with λ ∈ [2 . 0 , 2 . 8] 3 .
For all domains,
EPS achieved the highest policy explicabilityscore and followed the best performer for average return.
Note that SAC also achieved goodexplicability scores in the domains since the synthetic human models did not penalize the optimalbehavior as much as they could (i.e., the human beliefs were not modeled to be much more stochasticthan the ground truth).
PS performed well on most tasks while being considerably worse thanothers in D3.
Due to the conflict between the human’s expectation and task priorities in D3, themethod failed to achieve a meaningful combination.
DRLHP performed poorly on all the domainsbecause preference-based learning methods have difficulty dealing with domains with sparse ordelayed rewards, which makes credit assignment challenging and results in large variations.
Since
EPS also applies a preference-based learning method to learn the surrogate reward function, wedemonstrate this using the learned function.
As shown in Fig.
3, this function has a significantamount of uncertainty so is not ideal for guiding policy search.
Although it adds variability intopolicy learning, it serves very well as an auxiliary objective by informing the agent where the humanexpects it to perform.
We show sampled trajectories for EPS, the baselines, and the human’s expectation computed using thesynthetic human models in Fig.
In D1 and D2, SAC agent always chooses the shortest path while
EPS agent takes a detour that bypasses the pit and hence is more explicable.
In D3 and D4, SAC agentchooses the path passing through the top since the agent can get stuck on sandy roads in D3 and thetop path is more efficient in D4.
It is worth noting that EPS agent makes different decisions on these
[Table]
two domains.
It selects the top path in D3 when the sandy road is difficult to navigate even though itis against the human’s expectation.
This is because if the agent conforms to the human’s expectation,it would result in a huge loss to the engineered reward.
However, when it can better navigate on sandin D4, it chooses to respect the human’s expectation to be more explicable, at the cost of loweringtask performance.
Policy shaping agent is also successful in D4, but gets stuck in D3, because simplymultiplying two different policies could result in a new policy that is uninformative, irrespective ofthe λ used (e.g., when we have p 1 = (0 . 1 , 0 . 9) , p 2 = (0 . 9 , 0 . 1) , the resulting policy from multiplyingthem would always be p
= (0 . 5 , 0 . 5) ), which can lead to poor behaviors.
In summary, we show that
EPS can successfully generate effective behaviors that achieve an intelligent reconciliation betweenthe human’s expectation and task priorities.
It can generate novel reconciled behaviors (D1 and D2),stick to the task priorities (D3), or follow the human’s expectation (D4), as appropriately.
5.2 Autonomous Driving Domain
We used a simulated autonomous driving domain [29] to evaluate our method with human subjectsand demonstrate its real-world relevance, as illustrated in Fig. 4.
The state space is featured by theposition and velocity of the ego-vehicle and nearby vehicles.
The action space consists of five discreteactions.
Initially, the autonomous driving vehicle (green) is running on the highway with a car in frontof it (blue) on the same lane and with the same speed.
The task of the autonomous driving agent is tohandle situations when the front car slows down quickly and abruptly.
One common type of cognitivebias is the availability bias , which reflects humans’ tendency to overestimate the likelihood of eventswith greater availability in memory.
Such biases can occur in driving since we regularly drive underfamiliar conditions.
To design an experiment where such biases are present, we considered scenarioswith a regular driving condition and a sleety condition where the vehicle’s domain dynamics becamemore stochastic due to slippery roads.
With the availability bias, however, humans are likely to makethe same decisions under these similar but in actuality different conditions, leading to safety risks.
The user study consists of two phases: training and testing.
At the beginning of the training phase,we requested the participates to provide the importance ratings for several factors governing theautonomous driving behavior [9] in a 5-point Likert scale, such as average speed (3.42), distanceto the front car (4.57), relative speed to neighboring cars (3.28), and lane following (3.78).
Theaverage participants’ responses are shown in the parentheses above.
Their responses served as theengineered reward model ( r A ) and are linearly combined.
For example, the distance to the front car isdirectly associated with collision and hence given a larger weight.
Before human data collection, wealso presented the participants with the vehicle’s behavior after braking under the regular conditionwhen the front car slows down suddenly.
Then, given the information that it is driving on a sleetyday, we selected segments that showcased different vehicle behaviors after braking (i.e., brakingbehaviors under different effectiveness).
The participants were asked to select which one matchedtheir expectations the most.
Their responses were used to validate the availability biases in thisexperiment.
Then, we collected human data by actively selecting pairs of trajectory segments forthe participants to compare with.
The collected data was used to train our method and the baselines.
Then, we showed rollouts of the learned policy for each method to new participants and asked themto rate those rollouts in the testing phase.
5.2.1 Results and Discussion
We published the experiments on Amazon Mechanical Turk (MTurk).
To sift out invalid responses,a sanity check demonstration was added that showed a collision, which should not be preferredunder any situation.
We recruited 15 participants for training, and one failed the sanity check.
Forthe bias validation question, 12 out of 14 valid participants chose that the ego-vehicle would slowdown effectively on a sleety day when braking.
This reflects the availability bias that the participants
[Figure]
had, which can lead to collisions on a slippery road if ignored.
We also noticed that the participantspreferred to brake than steer in general during training, which accords with the participants’ responsesthat attached a high importance rating to distance to the front car and lane following.
The testing phase occurs on a sleety day.
The behavior of EPS, SAC, and DRLHP agents areillustrated in Fig.
SAC agent steers immediately when the front car slows down.
It is the mostefficient behavior with the most return on a slippery road since braking would not be effective.
DRLHP agent, on the other hand, chooses to brake while staying in the same lane.
Such a behavior,however, is more likely to lead to a collision (risk to human passengers).
As we can see from Fig.
4,the agent is getting dangerously close to the vehicle in the front.
EPS agent first chooses to brake(to be explicable), and then steers to the other lane (to stay safe and continue moving forward),which maintains both explicability and task efficiency.
We demonstrated these rollouts in the testingphase to new participants.
We informed the participants that the vehicle was running on a sleetyday.
Each participant was required to provide ratings to all the demonstrations ranging from 1 to 10 .
We obtained 11 valid responses.
Interestingly, the participants rated EPS agent the most preferable,followed by DRLHP and SAC agents.
The average rating for each agent and its standard deviationare shown in Fig.
We note that the results contradict with the importance ratings solicited for theengineered reward model, which should have led to preferring the steering behavior as chosen by the
SAC agent on slippery roads.
This further suggests that the participants had a biased belief aboutthe vehicle’s dynamics (i.e., braking is as efficient under the regular condition as under the sleetycondition).
Regardless, our agent chose the behavior that was both explicable and safe.
Discussions and Conclusions
In this paper, we introduced and formulated the problem of explicable policy search that considersmodel differences between the learning agent and its human observer.
We developed an efficientsolution by learning a surrogate reward function that was then used to recover an explicable policy.
Our method significantly extends explicable planning to an RL setting and to stochastic environmentswith continuous state and action spaces.
We evaluated in simulations and with human subjects.
Results showed that our approach could better handle situations under model differences than severalbaselines and thus contributed a critical tool to achieving explainable human-agent interaction.
Limitations and Future Work.
The assumptions underlying our work are that the human generatesher expectation based on her objective and belief about the domain dynamics in a noisily rationalway, and the human has a softmax preference model.
While such assumptions were commonlymade in prior work, it is worthwhile to consider situations where the human is cognitively bounded.
Furthermore, we assume that the human’s (inaccurate) belief about the domain dynamics does noteasily update as observations of the agent are made.
For scenarios where inaccuracies stem fromintrinsic cognitive biases that are difficult to change, this is a reasonable assumption.
In other caseswhere discrepancies were due to, e.g., information asymmetries, the belief can change dynamicallyand needs to be actively monitored [22].
Other possible directions include more complex (suchas state dependent) reconciliation of the expected return and policy explicability score, as well as weighted policy explicability scores to incorporate trajectory similarity as considered in priorwork [48, 25].
Our approach covers many real-world applications since the human understanding of the agent’smodel is inevitably different from the ground truth in real-world scenarios.
As a result, EPS maysometimes cause the user preferences to be ignored, which may be “unexplainable” but is a necessarycost to pay in many scenarios.
In particular, EPS combines the surrogate reward with an engineeredreward from the agent designer to ensure critical features (e.g., safety) while respecting user preferencewhenever feasible.
Note that the combination of such two reward functions should not be replacedby the user’s reward even if it can be perfectly estimated due to complexities such as human biasesand misunderstandings of the domain.
However, real-world deployment of EPS would require morework, such as an interactive design that explains to the user why her preferences are ignored.
Societal Impacts.
Our research is generally fundamental in nature without notable negative societalimpacts.
However, it is conceivable that our approach may be used to misguide RL agents to ignorehuman preferences in adversarial settings.
Further investigation must be conducted to understandsuch a possibility in more detail.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplemen-
tal material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See supplemental material.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See results in table 1.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplemental material.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See supplementalmaterial.
(b) Did you mention the license of the assets?
[Yes] See supplemental material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See supplemental material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] No such info or content was used or curated.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[Yes] See supplemental material.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes] See supplemental material.