Accounting for Human Learning when Inferring Human Preferences
Abstract
Inverse reinforcement learning (IRL) is a common technique for inferring human preferences from data.
Standard IRL techniques tend to assume that the human demonstrator is stationary, that is that their policy π doesn’t change over time.
In practice, humans interacting with a novel environment or performing well on a novel task will change their demonstrations as they learn more about the environment or task.
We investigate the consequences of relaxing this assumption of stationarity, in particular by modelling the human as learning.
Surprisingly, we ﬁnd in some small examples that this can lead to better inference than if the human was stationary.
That is, by observing a demonstrator who is themselves learning, a machine can infer more than by observing a demonstrator who is noisily rational.
In addition, we ﬁnd evidence that misspeciﬁcation can lead to poor inference, suggesting that modelling human learning is important, especially when the human is facing an unfamiliar environment.
Introduction
Directly specifying the desired task that a machine should perform can be challenging [14].
One approach to this problem is through Inverse Reinforcement Learning (IRL), where the task is to learn a reward function R which best explains the behaviour of a given (noisily-)optimal agent π [16], of which the standard model is the Boltzmann-rational model [21, 22, 10, 7, 11, 12, 5].
One strong assumption that we challenge in this work is that of stationarity .
Models of near-optimality tend to assume that the agent’s demonstrations are stationary; that is, that the policy does not change over time.
This may not be the case for several reasons, as the following examples show:
For a robotics task, the human demonstrator’s performance might improve during demonstrations, as they become quicker to complete a task as their understanding of the dynamics develops, or as they explore and then exploit better strategies 2.
With recommender systems, a human might still be learning their preferences; they might not know how much they enjoy a certain genre of music until they have tried it.
When labelling images [8] or ranking demonstrations [7, 5], a human labeller might not understand the instructions precisely at ﬁrst, but then they develop a better understanding by interacting with more examples.
In each of these examples, while a true human expert might not change their behavior through the course of generating demonstrations or comparisons, a human demonstrator interacting with a novel system might learn through the course of interacting.
Our key proposal is to account for non-stationary by modelling the demonstrator as learning.
We ﬁrst investigate how this modelling choice interacts with IRL.
Is there anything a machine inferrer can learn from human demonstrators that are themselves learning?
Our results show that the answer is yes.
Moreover, surprisingly we ﬁnd that non-stationary learning demonstrators can be more informative across several domains.
Further, we shed light on this result by showing that a learning demonstrator is able to disambiguate between rewards that stationary demonstrators cannot.
Next, we investigate the impacts of model misspeciﬁcation in relation to stationarity, and ﬁnd evidence that assuming stationarity is harmful when the human demonstrator is learning.
Encouragingly, we ﬁnd that merely modelling the demonstrator as learning is sufﬁcient for the inferrer to get good performance in most cases, even if the inferrer gets the parameters of the learning algorithm incorrect.
1.1 Related work
A large amount of recent work studies IRL with alternative assumptions.
To list a few, Bobu et al. [3] uses a modiﬁed version of the Boltzmann-rationality assumption; Evans et al.
[9] presents an algorithm for learning from various psychology-inspired algorithms; Reddy et al.
[19] studies the problem of IRL from an agent with false dynamics beliefs; Zhi-Xuan et al.
[20] presents an algorithm from learning from the trajectories of bounded planning agents; and Levine and Koltun [15] studies
IRL under the assumption that the demonstrations are merely locally (noisily-)optimal.
However, in the vast majority of work in this area, the agents are assumed to be stationary and don’t change their revealed preferences over time.
Chan et al.
[6] and Jacq et al.
[13] do study versions of the task of learning from a learner.
However,
Chan et al.
’s
Assistive Bandits studies the problem in the restricted domain of a multi-armed bandit, and thus makes the assumption that the human only learns about the reward and not the dynamics.
Jacq et al.
’s
Learning from a learner presents an algorithm that relies on the assumption that later actions have higher Q-value than earlier actions and then performs IRL from comparisons on top of that.
This work both considers sequential decision problems where the demonstrator does not know the dynamics and makes a different assumption about the learning algorithms.
Background
A ﬁnite Markov Decision Process is a tuple M =
( S , A , T, ρ, γ
, R θ , P 0 ) with • Finite sets of states S and actions A • Transition probabilities T ( s, a, s (cid:48) ) over s (cid:48) ∈ S , ∀ s ∈ S , a ∈ A • Initial state probabilities ρ over S • Discount rate γ ∈ [0 , 1) • Parameterised reward function R θ : S ×
A → R , for parameter θ ∈ Θ • Prior distribution over θ , P
A policy π ( a | s ) gives a probability vector over actions a ∈ A for a given state s ∈ S .
For policy π , the action-value function Q π :
S ×
A → R is the expected return of starting in state s , taking action a , and then following policy π :
[Equation]
The optimal action-value function Q ∗ is deﬁned to be Q ∗ ( s, a ) =
max π Q π ( s, a ) , and the optimal policy is π ∗ ( · | s ) ∼ Unif { argmax a Q ∗ ( s, a ) } .
A planner is a map D : Θ → Π from reward parameter θ to policy π .
The type of planner is assumed before doing IRL.
The optimal planner D ∗ has D ∗ ( θ ) =
π ∗ , the optimal policy.
The Boltzmann planner has D β ( θ ) =
π β , where π β is the Boltzmann policy parameterised by β , deﬁned to be the policy satisfying[-
-]
[Equation]
Input: Demonstrations ( τ 0 , . . . , τ n ) , prior P 0 ( θ ) , initial parameters w 0 , learning algorithm ( F, L )
Initialise parameters w θ = w 0 , P ( θ )
=
P 0 ( θ ) for τ in ( τ 0 , . . , τ n ) do π θ ← F ( w θ ) ; Compute log-likelihood (cid:96) ( θ ) ← (cid:80) ( s,a ) ∈
τ log π θ ( a | s ) ;
Update posterior log P ( θ ) ← log P ( θ ) + (cid:96) ( θ ) ;
Fill in the reward according to each θ , τ θ ;
Update agents w θ ← L ( w θ , τ θ ) end
The form of the IRL problem which we shall address is not that which has access to an entire policy π , but that which has access to only to a series of n demonstrations ( τ 0 , . . . , τ n ) which have been generated according to π .
As the primary form of non-stationarity we’d like to address is learning, we propose the following model of a learning agent.
Given some parameter space
W , a learner consists of a policy-generating-function
F : W → Π which takes parameters w to policy π ∈ Π , paired with an update-function L : W × { τ } → W for performing parameter updates, where { τ } is the set of all single trajectories.
The updates occur once at the end of each trajectory.
A diagram of the learning process being assumed is given in Figure 1.
Note that our formulation technically allows for arbitrary changes in policies, however, in this work we restrict our attention to “reasonable” learning algorithms.
Methods
To perform inference on θ , we use a variant on Bayesian Inverse Reinforcement Learning (BIRL) [18].
Given a sequence of trajectories ( τ 0 , τ 2 , . . . , τ n ) , we compute
[Equation]
A general algorithm is detailed in Algorithm 1.
3.1 Exploring inference performance in simulation
In an ideal world, we would be able to know a priori the reward parameters and learning algorithms that real human demonstrators are using, and then compare the performance of an inferrer that accounts for the correct learning algorithm, models learning incorrectly, or fails to account for learning at all.
However, because we don’t have access to the true reward parameters and learning algorithms of any particular real human, we start by studying simulated demonstrators with known learning algorithms and policy-generating functions, on environments where the ground truth reward is known.
3.2 Agents
For our stationary demonstrators we consider the Boltzmann planner D β from Section 2.
Our non-stationary demonstrators use the parameters w ∈ W to maintain a table of Q-value estimates, ˆ Q ( s, a ) .
These estimates are updated according to one of three different targets, corresponding with observed state s t , action a t , and reward r t :
L q — Q-learning with target y ( s t , a t ) =
r t + γ
[Equation]
V ( s t +1 ) 2.
L d — Direct Evaluation with reward-to-go target y ( s t , a t ) =
r t + γr t [-+
-]+1 + · · · + γ T − t r T , for horizon T 3.
L λ — TD ( λ ) with target y ( s t , a t ) =
r t + γ (cid:16) λy
( s t +1 , a t +1 )
+ (1 − λ )
ˆ V ( s t +1 ) (cid:17) .
This demonstrator interpolates between the Q-learning and Direct Evaluation agents.
Where for the current policy π , ˆ V ( s ) =
E [ ˆ Q ( s, a ) | a ∼ π ] .
Note that L q = L λ =
and L d = L λ =1 .
For each state action pair visited in trajectory τ i , the parameters are updated towards the target y
( s, a ) with a learning rate of 1 N ( s,a )+1 , where N ( s, a ) is the number of visits to ( s, a ) :
[Equation]
The policy function F = F β returns the Boltzmann policy corresponding with current Q-value estimates ˆ Q .
In this paper, we initialise our estimates to zero, ˆ Q 0 = 0 .
In simple environments we expect this algorithm to converge to the Boltzmann policy π β , but this is not guaranteed (for example because of insufﬁcient exploration).
3.3 Evaluation
To evaluate the quality of inference we look at the mutual information between reward parameters θ and the trajectories τ :
[Equation]
We also look at the mutual information of only one trajectory m i , as it will be instructive later to see how the mutual information varies as the agent converges:
[Equation]
[Table]
[Equation]
In some cases we look at the posterior probability of the true theta θ ∗ per-timestep, p i =
P ( θ ∗ |
τ i − 1 ) , or cumulatively P i
= P ( θ ∗ | τ 0 , . . . , τ i − 1 ) .
In both cases we consider the cumulative metric to be most important.
Illustrative example
Consider a simple two state MDP with two actions A, B where with probability 1 − (cid:15)
A leads to an apple and B leads to a banana, and with probability (cid:15) the outcome is switched.
Consider reward hypotheses | Θ | = 2 where θ = 1 speciﬁes a preference for apples and θ = 2 speciﬁes a preference for bananas.
See Figure 2 for a diagram of this environment.
Over a demonstration of two trajectories, we compare a stationary agent — the Boltzmann planner D β — with a Direct Evaluation learner ( F β , L d ) .
Note that the mutual information for a non-stationary agent in the ﬁrst trajectory is always zero.
The mutual information is shown in Figure 2 for varying temperature β and noise (cid:15) .
Interestingly, the mutual information for the non-stationary agent does not depend on (cid:15) : the agent will either update towards repeating the same action or update towards taking the opposite action, signalling their preference in each case.
On the other hand, the performance of the stationary agent decreases for larger (cid:15) ; this is because the action has less inﬂuence on the expected reward.
For small (cid:15) the stationary agent outperforms, and for large (cid:15) the non-stationary agent outperforms.
In the case of (cid:15) = 0 .
5 , the action has no inﬂuence on the outcome, and therefore the stationary agent exhibits no mutual information.
However, the non-stationary agent is still able to disambiguate in this case because they are signalling not only through their behaviour, but their change in behaviour.
Experiments
Motivated by our illustrative example, we compare the stationary and non-stationary models in three domains: an ambiguous environment, adversarially designed such that the stationary policy is invariant among θ ∈ Θ , a Gridworld with more interpretable policies, and random MDPs.
[Figure]
In each experiment, we sample θ ∼ Unif { Θ } , and record demonstrations for a stationary Boltzmann agent and a selection of the non-stationary agents described in Section 3.2.
Taking the demonstrations as input, we run BIRL, using the variant described in Algorithm 1 for the non-stationary agents.
We repeat each experiment n times and use a sample mean to estimate the metrics described in Section 3.3.
5.1 Ambiguous MDP
[Figure]
As discussed in Section 5.1, due to shaping, observing the stationary Boltzmann demonstrator gives no information about the reward parameter.
In our illustrative example we saw that that learning demonstrators can be more informative, including the particular ambiguous case where the stationary demonstrations don’t depend on θ .
We further verify this observation here through use of reward shaping [17].
We consider an MDP |S| = 4 , |A| = 3 with arbitrarily chosen transition dynamics.
We consider an arbitrary reward function which we shape three times, | Θ | = 4 , such that the Q-function in each state Q ( s, · ) is invariant up to a constant factor.
In particular this means that the stationary Boltzmann planner does not depend on the rewar parameter θ .
We run n = 10 experiments, each withtrajectories, and give the posterior at the true reward parameter θ ∗ in Figure 3.
We observe that the non-stationary agent exhibits enough information for good reward inference, converging to a posterior of 1 in the case of β = 5 , whereas in each case the stationary agent exhibits none.
In conclusion we see again that through changing behaviour the learner demonstrator is able to signal it’s preferences where the stationary demonstrator cannot.
[Figure]
As discussed in Section 4, we see that for low β the non-stationary demonstrator can be more informative than the
5.2 Gridworld experiment
Next we turn to study the implications from our illustrative example in a more natural setting.
We consider a 5 × 5 Gridworld environment based on OpenAI Gym’s ‘Frozen-Lake-v0’ [4].
The environment has one hole state and two reward states.
The reward parameter θ =
( θ 1 , θ 2 ) has θ j ∈ { 0 , 1 , 2 , 3 , 4 } , giving | Θ | = 25 .
A parameter (cid:15) controls the noise in the environment: with probability (cid:15) the direction taken is not the direction chosen by the agent, but one of the two perpendicular directions.
The environment is depicted in Figure 4.
We run n = 150 experiments and we include here the results for the Q-learning agent averaged over noise parameters (cid:15) = 0 .
The mutual information per trajectory m i and cumulative mutual information M i are given in
Figure 5.
Further experiments and experimental details can be found in the appendix, Section A.
We see that for low β = 2 the non-stationary demonstrator can be more informative than the
Boltzmann demonstrator, whereas for high β = 10 the stationary demonstrator outperforms.
A lower β leads to a weaker signal across the board as the demonstrator’s actions exhibit more noise.
In the appendix (Section A), we discuss how the non-stationary demonstrator performance is robust to higher amounts of action noise (cid:15) .
We also discuss an observed phenomenon where mutual information seems to spike for earlier demonstrations in the case of the Direct Evaluation agent.
5.2.1 Gridworld misspeciﬁcation
A natural question is whether modelling learning is necessary; perhaps a learning demonstrator is just as informative when the observer is modelling them as stationary.
To assess whether this is true, we look at the effect of having the wrong model for the Gridworld environment.
When a model is misspeciﬁed, mutual information is not a valid measure of performance, instead we look at the posterior metrics p
i , P i introduced in Section 3.3.
First, we study the effects of making an incorrect assumption as to whether a demonstrator is stationary or not.
We run n = 150 experiments for each form of misspeciﬁcation.
The cumulative posterior
[Figure] [Figure]
f e rr e r We see that misspeciﬁcation leads to poor inference, only very marginally improving on the posterior of 4% .
Furthermore, we see in Figure 6 that per-trajectory performance is worst for the earliest trajectories.
In the case where the non-stationary demonstrator is assumed to be stationary, inference improves over time as the agent converges to the policy expected by the inferrer.
This gives justiﬁcation for the common practice in IRL demonstrations whereby the demonstrator is given some time to understand the experiment setup and practice demonstrating before data collection begins (for example, see the instructions in Christiano et al.
In the other case, where a stationary agent is assumed to be non-stationary, inference improves over time because the inferrer, modelling the stationary demonstrator as converging, is increasingly expecting the (near-)optimal behaviour that it is observing.
Second we look at the effects of correctly allowing for non-stationarity but having the incorrect model.
We assess this by considering the TD ( λ ) model L λ where the demonstrator has parameter λ = λ D and the inferrer has parameter λ
= λ
I .
We run n = 150 experiments for a mesh of values λ D , λ I ∈ [0 , 1] .
Figure 8 gives a heatmap of the cumulative posterior after a demonstration of 181 trajectories, P 181 .
We observe that inference is mostly robust to misspeciﬁcation of λ .
For example, when the demonstrator has parameter λ D = 0 . 5 , the inferrer assigns a posterior ≥ 20% to the true reward for all λ I in the range [0 , 0 . 8] .
Only at the bottom left and bottom right (where the inferrer’s model is most misspeciﬁed) is performance is comparable to the performance of mismodelling the demonstrator as stationary.
In each case, the inferrer does better than the prior 4% .
5.3 Random MDPs
To explore a more general domain, next we consider randomly generated MDPs with |A|
= 2 and for |S| ∈ { 3 , 4 , 5 , 6 } .
Two of the states generate reward in {− 1 , 1 } , giving | Θ | = 16 .
We run n = 200 experiments, each with a demonstration of 200 trajectories.
The mutual information is given in Figure 9 for |S| = 3 , and additional results are available in Section A of the appendix.
1.0 λ D 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .
.λ
I
We observe that inference is robust to misspeciﬁcaiton of λ .
For example, when the demonstrator has parameter λ D = 0 . 5 , the inferrer assigns a posterior ≥ 20% to the true reward for λ I ∈ [0 , 0 . 8] .
[Figure]
Interestingly, we ﬁnd in all of the Random experiments
that although -trajectory information is lower for the non-stationary agent, the more important measure of cumulative mutual information is higher. This highlights a crucial difference between the two models. The stationary agent is giving answers to a single highly informative question, namely a request for a (near-)optimal demonstration.
Whereas the non-stationary demonstrator is giving answers to a multitude of different — in this case less informative — questions, namely the demonstrator’s preference for the trajectories experienced so far, as represented through changes in behaviour.
Limitations and Future work
Experiments with real humans.
As a ﬁrst step toward studying the interaction of demonstrator learning with the reward learning of an inferrer, we conducted experiments entirely in simulation.
Future work should perform experiments with real humans while ﬁnding ways to surmount the difﬁculties noted in Section 3.1.
Larger domains and approximate inference As we performed exact inference in this work, we were restricted to domains with a small set of discrete reward parameters.
Future work could study how well our results hold up when we instead perform approximate inference on domains with larger reward parameter spaces, or model the learning algorithm also as a random variable among a family of algorithms.
Broader Impact
Many real-world tasks we would like AIs to do are difﬁcult because they lack an easily-speciﬁed reward function.
For example, it’s very difﬁcult to specify a reward function that would induce courteous behavior in a self-driving car, and even harder to make one that correctly balances all the trade-offs that a car would face in the real world [2].
Reward inference is one approach to tackling this problem — instead of requiring a designer to specify the reward, it only requires a designer to specify a prior over reward functions and provide a dataset from which to learn the reward.
Thus, any work that contributes to better reward inference contributes to building better AIs in these real-world tasks, and our work is no exception.
As an example of the possible applications of our work, better understanding what the users of our AI desire could lead to recommender systems that can optimize for individual user preferences and autonomous cars that adapt to the risk and time preferences of their riders.
However, better reward inference could be used by bad actors in malicious ways.
For example, many intentionally malicious behaviors may be hard to specify manually.
Just like it is hard to specify the reward for a courteous car, it is hard to specify the reward of a purposely rude car.
Better reward inference would make it easier to specify either type of behavior.
A better understanding of what individual humans want could also be used to manipulate their behavior.
For example, a better model of person’s preferences for internet browsing could be used to develop more effective personalized online scams or spear phishing attacks.
A Gridworld experiments
Our Gridworld experiments were run for (cid:15) ∈ { 0 . 2 , 0 . 3 , 0 . 4 , 0 . 45 , 0 . 49 , 0 . 6 , 0 .
65 , 0 . 66 } , for stationary
Boltzmann demonstrator D β and learning algorithms L q , L d .
The initial state distribution is uniform over the non-terminal states.
We use γ = 0 .
Trajectories were cutoff at a maximum length oftimesteps.
Value iteration for stationary agents was run till convergence, with a tolerance of 10 − 9 for maximum Q-value divergence between iteration steps.
In the main body we include results for L q , and averaged over (cid:15)
= 0 .
We note that mutual information is generally higher for L q compared with L d .
To see a full array of results consult Figures 10, 11, 12, 13.
For misspeciﬁcation results consult Figures 14, 15, 16, 17.
We see from these results that as with the illustrative examples, more noise leads to lower mutual information in the stationary case.
The information from a Direct Evaluation demonstrator does not vary much with (cid:15) , however the Q-learning demonstrator appears to improve as more noise is introduced.
We suspect that this is in part because more noise leads to slower convergence and more changes in behaviour.
It is also evident, more-so from the misspeciﬁcation plots Figures 16, 17, that stationary inference is more sensitive to β compared with both non-stationary models.
In our experiments β is always correctly speciﬁed, but this suggests that non-stationary might suffer less from a misspeciﬁed β .
We emphasise another feature of the Direct Evaluation agents where there is a spike in mutual information towards the beginning of the demonstration, see Figures 10, 11.
Attributing behaviour to a speciﬁc past experience is easier in the early stages when the list is smaller.
Here this is overall more informative than later demonstrations for which individual trajectories have less impact on speciﬁc decisions.
B Random MDP experiments
Our random MDP experiments were run for |S| ∈ { 3 , 4 , 5 , 6 } , for a stationary Boltzmann demonstrator
D β and learning algorithms
L q , L d .
The initial state distribution is uniform over the non-terminal states.
We use γ = 0 .
Trajectories were cutoff at a maximum length of 100 timesteps.
Value iteration for stationary agents was run till convergence, with a tolerance of 10 − 9 for maximum
Q-value divergence between iteration steps.
To see a full array of results consult Figures 16, 17.
As with the Gridworld domain, we see a repeat of the phenomenon of the information spike for both the Direct Evaluation agent and the Q-learning agent, although it is less strong in the latter case.
The effect appears to be stronger for higher |S| , where the environment is more complex.
Mutual information in general appears to increase with |S| .
This is not surprising since with more starting states, and a larger set of possible trajectories, there are more ways for the demonstrator to signal their preferences.
As discussed in Section 5.3, the Q-learning demonstrator outperforms the stationary demonstrator in each case.
We see here that this cumulative outperformance of occurs very early on, in most cases within the ﬁrst 30 trajectories.
As with the Gridworld domain, per-trajectory mutual information appears generally to be higher for higher Boltzmann parameter β , however in this case we see that cumulative mutual information is generally lower.
Similar to the effect discussed in 5.3, this is because the demonstrator is very effectively signalling a near the optimal trajectory, but the scope of this information is narrow and does not reﬂect enough the reward along suboptimal trajectories.
In this domain, this results in less overall mutual information.
[Figure]
[Figure]
i β =β =e = 0 .
e = 0 .
e = 0 . e = 0 .
β =
[Figure] [Figure]
[Figure]
M i β =β =e = 0 .
e = 0 .
e = 0 . e = 0 .
β =
[Figure] [Figure]
[Figure]
learning
Stationary
[Figure]
= 0 .
e = 0 . e = 0 .
e = 0 .
Direct Eval
[Figure]
e
[Figure]
e
[Figure]
e
[Figure]
e
[Figure] [Figure]
[Figure] [Figure]
Q
[Figure]
learning
Stationary
[Figure] [Figure]
e
[Figure]
e
[Figure]
e
[Figure]
Direct Eval
[Figure]
m i β =β =| S | =| S | =| S |
=β =0.
m i | S | =| S | =| S | =0. m i | S | =| S | =| S | =   i 0.
m i
i   i | S | =| S |
=| S | =
Demonstrator
Stationary
Q-learning
Direct Eval
M i β =β =| S | =| S | =| S |
=β =
M i | S | =| S | =| S | = M i | S | =| S | =| S | =
i
M i   i   i | S | =| S |
=| S | =
Demonstrator
Stationary
Q-learning
Direct Eval