Learning Multimodal Rewards from Rankings
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Learning from human feedback has shown to be a useful approach inacquiring robot reward functions.
However, expert feedback is often assumed tobe drawn from an underlying unimodal reward function.
This assumption doesnot always hold including in settings where multiple experts provide data or whena single expert provides data for different tasks—we thus go beyond learning aunimodal reward and focus on learning a multimodal reward function.
We formulate the multimodal reward learning as a mixture learning problem and developa novel ranking -based learning approach, where the experts are only required torank a given set of trajectories.
Furthermore, as access to interaction data is oftenexpensive in robotics, we develop an active querying approach to accelerate thelearning process.
We conduct experiments and user studies using a multi-task variant of OpenAI’s LunarLander and a real Fetch robot, where we collect data frommultiple users with different preferences.
The results suggest that our approachcan efﬁciently learn multimodal reward functions, and improve data-efﬁciencyover benchmark methods that we adapt to our learning problem.
Introduction
Learning a reward function from different sources of humandata is a fundamental problem in robot learning.
In recentyears, there has been a large body of work that learns rewardfunctions using various forms of human input, such as expertdemonstrations [1], suboptimal demonstrations [2, 3, 4], pairwise comparisons [5, 6], physical corrections [7, 8], rankings[9] and trajectory assessments [10].
These works focus onlearning a unimodal reward function that models human preferences on a target task.
However, this unimodality assumption does not always hold: human preferences are usually morecomplex and need to be captured via a multimodal representation.
Further, even if the preferences of a human are truly unimodal, we often use a mixture of data from multiple humans,which can be difﬁcult to disentangle, leading to multimodality.
As an example, consider a robot placing a banana on one ofthe three shelves (see Fig. 1).
The middle shelf is often usedfor fruits, but it has no room left and if the robot tries to put thebanana there, it may cause other fruits to fall.
The top shelf hassome space
but it has been used for cooked meals.
The bottomshelf has a lot of free space, but is usually used only for toys.
In such a scenario, people may have very different preferences about what the robot should do.
Ifwe try to learn a unimodal reward using data collected from multiple people, the robot is likely tofail in the task, because the data will include inconsistent preferences.
One solution is of course to label the different modes in the data.
For example, one could separate thedata based on the preferred shelf, and learn different reward functions for each shelf.
However, thisseparation is not always straightforward.
For example in a driving dataset, it is unclear what should be labeled as aggressive or timid driving.
Clustering the data based on the human who providedthe data is also not viable as it will introduce data-inefﬁciency issues, and perhaps more importantly,humans are not always unimodal: a usually timid driver can drive more aggressively when in a hurry.
These examples motivate us to develop methods that can learn multimodal reward functions using datasets that are not speciﬁcally labeled with the modes.
To this end, previous work proposedlearning from demonstrations to learn multimodal policies [11, 12] or reward functions with multiple possible intentions [13, 14].
However, learning from expert demonstrations is often extremelychallenging in robotics as providing demonstrations on a robot with high degrees of freedom is nontrivial [5, 15], and humans have difﬁculty giving demonstrations that align with their preferencesdue to their cognitive biases [16, 17].
Thus, it is desirable to have methods that learn from othermore reliable sources of data.
For instance, humans can reliably compare two different trajectories,enabling a robot to learn from pairwise comparisons [5, 9, 18].
While learning from pairwise comparisons provides a rich source of data for learning reward functions, the theoretical results by Zhao et al.
[19] imply that extending the existing comparison-basedreward learning techniques to multimodal reward functions is not possible, i.e. failure cases can beconstructed, where pairwise comparisons are not sufﬁcient for identifying different modes of a multimodal function.
Our insight is that it is possible to learn a multimodal reward function by goingbeyond pairwise comparisons and instead using rankings .
To achieve this, we formulate multimodal reward learning as a mixture learning problem.
As datais a very expensive resource in robotics, we further develop an active querying method that aims toask the human users the most informative queries.
Our contributions are three-fold:
• We develop a method that uses rankings from humans to learn multimodal reward functions.•
We develop an active querying method to improve data-efﬁciency by asking the most informativeranking queries.• We conduct extensive experiments and user studies with OpenAI’s LunarLander and a real Fetchrobot to test our learning and querying methods in comparison to baselines.
Related Work
Reward Learning in Robotics.
Learning reward functions from human feedback is a fundamentalproblem in robot learning.
Ng and Russell [20] and Abbeel and Ng [21] introduce the problemof learning from demonstrations in the space of robotics.
Later works focus on improving inversereinforcement learning algorithms by reducing the ambiguity in learned rewards [1, 22].
Due to the difﬁculty in providing expert demonstrations in robotics, recent works attempt to learnreward functions using other forms of human feedback, such as physical corrections [7, 8] and pairwise comparisons where a human user compares the quality of two robot trajectories [23, 24, 25, 26].
Later works extend these algorithms for better time and data-efﬁciency [27, 28, 29].
Though therehave been works to extend this framework to reward functions modeled with Gaussian processes tocapture nonlinearities [6, 30, 31], the underlying reward function has always been unimodal.
Preference-based Learning.
Outside of robotics, preference-based learning, where data are in theform of comparisons, selections out of a set, or rankings, has attracted attention due to the ease ofcollecting data and its reliability.
Prior works have studied this in the classiﬁcation [32], bandits[33], and reinforcement learning settings [34].
In our problem, we have a continuous hypothesis space of reward functions.
In such cases, it iscommon to model human comparisons or rankings with a computational model.
Bradley-Terry isone such model for pairwise comparisons [35], which is easily extended to the queries where thehuman chooses the best of multiple items.
Known as multinomial logits (MNL) [36], this model hasbeen widely used for human preferences in many ﬁelds [37, 38, 39] including robotics [25, 27, 18].
To extend these models to rankings, Plackett-Luce [40, 41] and Mallows models [42, 43, 44, 45] arecommonly employed.
In this paper, we use the Plackett-Luce model as it is a natural extension of
MNL, which is widely used in robotics with great success.
We formalize this model in Section 3.
Even though there has been much research in this domain, all works we mentioned here focus onthe unimodal case, and do not work with the multimodal preferences.
Learning Mixture Models from Rankings.
One way to model multimodal reward functions isthrough mixture models, where the data is assumed to come from different individual models with some unknown probabilities.
To this end, previous works consider mixtures of MNLs [46, 47],
Plackett-Luce models [19], and Mallows models [48].
Other works adopt different methods tomodel multimodality, such as by assuming latent state dynamics that transition between differentmodes [49, 50] or by learning the different modes from labeled datasets [51, 52].
To avoid thesemodeling assumptions, we focus on directly learning the mixture model.
While Zhao et al.
[19] have theoretically studied the mixture of Plackett-Luce choice models, whichalso informs our algorithm in terms of the query sizes, they only focus on learning the rewards ofa discrete set of items.
To the best of our knowledge, our paper is the ﬁrst work that deals with acontinuous hypothesis space under a mixture of Plackett-Luce models.
Furthermore, we proposean active querying strategy for this mixture model to improve data-efﬁciency for human-in-the-looplearning, which is crucial in data-hungry applications such as robotics.
Problem Formulation
Setup.
We consider a fully-observable deterministic dynamical system.
A trajectory ξ in this systemis a series of states and actions, i.e., ξ = ( s 0 , a 0 , . . .
, s T , a T ) .
The set of feasible trajectories is Ξ .
We assume there is a set of M individual reward functions that are possibly different, each of whichencodes some preference between the trajectories in Ξ .
For the rest of the formalism, we refer toeach individual reward function as an expert for the clarity of the presentation.
Following the common linearity assumption in reward learning [1, 5, 28], we assume each preferencecan be modeled as a linear reward function over a known ﬁxed feature space Φ , so the rewardassociated with a trajectory ξ with respect to the m th expert is R m ( ξ )
= ω (cid:62) m Φ( ξ ) , where ω m is theunknown vector of weights.
Across the expert population, there exists some unknown distributionover the reward parameters, corresponding to the ratio of the data provided by the experts.
Werepresent this distribution with mixing coefﬁcients α m such that (cid:80)
Mm =1 α m = 1 .
We will then learnboth the unknown reward functions { ω m } Mm =1 and the mixing coefﬁcients { α m } Mm =1 , using rankingqueries made to the M experts.
This setup generalizes [5], which studied unimodal rewards.
Ranking Model.
We deﬁne a ranking query to be a set of the form Q = { ξ 1 , . . .
, ξ K } for a ﬁxedquery size K .
The response to a ranking query is a ranking over the items contained therein, of theform x =( ξ a 1 , . . .
, ξ a K ) , where a 1 is the index of the expert’s top choice, a 2 is the second top choice,and so on.
While it is not known which expert provided the response to the query, we know the priorthat a response comes from expert m with some unknown probability α m , i.e., Pr( R = R m )
=
α m .
Going back to our banana placing example, a ranking query of K robot trajectories is generated bythe algorithm, and a user—whose identity is unknown to the algorithm—responds to this query.
We then capture how human experts respond to these ranking queries by modeling a ranking distribution through an iterative process using Luce’s choice axiom [53].
In this process, the expertsrepeatedly select their top choice a m with a probability distribution generated with the softmax ruleto generate a ranking from the order items were selected:
[Equation]
In the following iterations, the experts select their top choice among the remaining trajectories:
[Equation]
This is known as the Plackett-Luce ranking model [40, 41].
Together with the prior over experts α m ,the resulting distribution over rankings x ∼ X is a mixture of Plackett-Luce models with mixingcoefﬁcients α m and weights proportional to e R m ( ξ ) .
Hence, the ranking distribution ﬁrst selects the reward function R m with probability α m , and thenselects trajectories from Q sequentially with probability proportional to the exponent of their reward,i.e., e R m , among the remaining trajectories until none is left, generating a ranking of the trajectories.
So given knowledge of the true reward function weights ω m and mixing coefﬁcients α m , we havethe following joint mass over observations x from a query Q :
[Equation]
Objective.
Our goal is to design a series of adaptive queries Q ( t ) to optimally learn the rewardweights ω m and corresponding mixing coefﬁcients α m upon observing the query responses x ( t ) .
We constrain all queries to consist of a ﬁxed number of elements K .
Active Learning of Multimodal Rewards from Rankings
In this section, we ﬁrst start with presenting our learning framework.
We then discuss how we canimprove data-efﬁciency, and propose an active querying approach.
4.1 Learning from Rankings
To learn the reward weights ω m and mixing coefﬁcients α m , we adopt a Bayesian learning approach.
For this, we maintain a posterior over the parameters ω m and α m .
Denoting the distribution over theparameters α i and ω i as Θ , this posterior can be written as
[Equation]
where we use the conditional independence of ranking queries x ( t ) given Θ and the conditionalindependence of the Q ( t ) on Θ given Q (1) , x (1) , .
, Q ( t − 1) , x ( t − 1) in the last equation.
To be ableto compute this posterior, we assume some prior distribution over the reward weights and the mixingcoefﬁcients, which is system-dependent and may come from domain knowledge, and use Eq.
(2) tocalculate the likelihood terms.
For example, in our simulations and user studies, we adopted a
Gaussian prior ω i ∼ N (0 , I ) and a uniform prior α ∼ Unif(∆ M − 1 ) where ∆ M − 1 is the unit M − 1simplex.
Learning this posterior distribution in Eq.
(3), one can compute a maximum likelihoodestimate (MLE) or expectation as the predicted reward weights and mixing coefﬁcients.
Equation (3) implies the queries made to the experts, Q ( t ) ’s, affect how well the posterior willbe learned.
Assuming a limited budget of queries, which is often the case in many real-worldapplications, including robotics, one would ideally ﬁnd an optimal adaptive sequence of queries suchthat the responses would give the highest amount of information about the reward weights and themixing coefﬁcients.
However, this is NP-hard, even in the unimodal case with pairwise comparisons[54].
We therefore resort to greedy optimization techniques to develop our active learning approach.
4.2 Active Querying via Information Gain
A query Q is desirable if observing its value x yields high information about the underlying modelparameters, α m and ω m .
Therefore, we propose using an information gain objective to adaptivelyselect the most informative query at each querying step, generalizing the approach of [5].
Assume at a ﬁxed timestep t we have made past query observations
D = (cid:8) Q ( t (cid:48) ) , x ( t (cid:48) )
(cid:9) t − 1 t (cid:48)
=1 .
The desired query is then Q ∗ = arg max Q I ( Q ; Θ | D ) where I ( · ; · ) denotes mutual information.
Equivalently, denoting conditional entropy with H [ · | · ] and the joint distribution over x andθ = { α m , ω m } Mm =1 conditioned on Q and D as P ( X, Θ | Q, D ) , we see [Equation] Pr[ X [Equation]
x | Q
[Equation]
The details of this derivation are presented in Appendix A.
4.3 Overall Algorithm
To efﬁciently solve the optimization in Eq.
(4), we ﬁrst note that we should use a Monte Carloapproximation since the expectations are taken over a continuous variable Θ and a discrete variable
X over an intractably large set of K !
alternatives.
To perform this Monte Carlo integration, werequire samples from the posterior Pr( X, Θ | Q, D ) .
Our key insight is that we can obtain joint samples from both posteriors by ﬁrst sampling from ¯Θ ∼
Pr(Θ | D ) and then sampling x ∼ Pr (cid:0) X | Q, Θ = ¯Θ (cid:1) since Θ ⊥ Q | D and X ⊥ D | Q, Θ .
Weperform the sampling x ∼ Pr (cid:0) X | Q, Θ = ¯Θ (cid:1) efﬁciently using Eq.
In general, exact samplingfrom the posterior Pr(Θ | D ) is intractable.
However, we note Eq.
(3) can be directly evaluated(using Eq.
and gives Pr(Θ | D ) up to a proportionality constant factor.
With this unnormalized posterior of Eq.
(3), we use the Metropolis-Hastings algorithm as describedin Appendix B to generate samples from the posterior Pr(Θ | D ) .
We see our optimization problem simpliﬁes to ﬁnding, for N ﬁxed samples ¯ θ i ∼ Pr(Θ | D ) andcorresponding samples: x i ∼ Pr (cid:0) X | Q, ¯ θ i (cid:1)
L ( Q ; x, ¯ θ )=
N (cid:88) i = (cid:20) log
[Equation]
Require: Observations D 1: (cid:8) ¯ θ i (cid:9) N i =1 ∼ Pr(Θ | D ) w/ Eq.
(3) via Metropolis-Hastings 2: procedure E VAL Q UERY ( Q ) 3: ∀ i, x i ∼ Pr (cid:0) x i | Q, ¯ θ i (cid:1) 4: return L ( Q ; x, ¯ θ ) (cid:46) Eq.
(5) 5: end procedure 6: Q ← M INIMIZE ( E VAL Q UERY )
(cid:46)
Simulated annealing 7: select query Q
We solve this optimization usingsimulated annealing [55] (see Appendix C).
Algorithm 1 goes overthe pseudocode of our approach, andwe discuss the hyperparameters inour experiments in Appendix D.
4.4 Analysis
We start the analysis by stating thebounds on the required number oftrajectories in each ranking query toachieve generic identiﬁability .
A Plackett-Luce model over Ξ is generically identiﬁable if for anysets of parameters Θ 1 and Θ 2 inducing the same distribution over the responses to all queries ofsize K on Ξ , the mixing coefﬁcients of Θ 1 and Θ 2 are the same and the induced rewards R m ( ξ ) areidentical across Ξ up to a constant additive scaling factor.
Theorem 1 (Zhao et al.
A mixture of M Plackett-Luce models with query size K and | Ξ | =
Kis generically identiﬁable if M ≤ (cid:4) K − 2 2 (cid:5) ! .
This statement follows directly from [19], which proves the above bound assuming that each queryto the Plackett-Luce mixture is a full ranking over the set of items (i.e. | Ξ | = K ).
However, theassumption | Ξ | = K is untenable in the active learning context, as it prevents any active queryselection.
To apply this result for our active learning algorithms, we relax the condition to | Ξ | ≥ K .
Corollary 1.1.
A mixture of M Plackett-Luce models with query size K is generically identiﬁableif M ≤ (cid:4) K − 2 2 (cid:5) ! .
We prove Corollary 1.1 in Appendix E.1.
In our context, generic identiﬁability implies if the humanresponse is modelled by a Plackett-Luce mixture, our Algorithm 1 will be able to recover its trueparameters (up to a constant additive factor for the rewards) in the limit of inﬁnite queries.
Remark 1.
Greedy selection of queries maximizing information gain in Eq.
(4) is not necessarilywithin a constant factor of optimality.
Appendix E.2 justiﬁes Remark 1.
In fact, greedy optimization of information gain for adaptive activelearning can be signiﬁcantly worse than a constant factor of optimality in pathological settings [56].
Despite its lack of theoretical guarantees, information gain is a commonly used effective approachin adaptive active learning [29, 57, 58].
Although other approaches like volume removal satisfyadaptive submodularity [25], they fail in settings with noisy observations by selecting high-noiselow-information queries, and in practice achieve far worse performance than information gain.
Experiments
Having presented our learning and active querying algorithms, we now evaluate their performancein comparison with other alternatives.
We start with describing the two tasks we experimented with:
LunarLander.
We used 1000 trajectories in
OpenAI Gym’s LunarLander environment [59]shown in Fig.
2 (see Appendix G.1 for detailson how they were generated).
Fetch Robot.
We generated 351 distinct trajectories of the Fetch robot [60] putting the banana on the shelves as shown in Fig.
1 (see Appendix G.2 for details).
5.1 Methods
We compare our active querying via information gain (IG) discussed in Algorithm 1 with two baselines:
A simple benchmark for active learning is random query selection without replacement.
Wealso benchmark against volume removal (VR), a common objective for active learning of robot reward functions [25].
See Appendix F for the details of these two baselines.
5.2 Metrics
We want to evaluate both the active querying and the learning performance.
The former requiresmetrics that assess the quality of the algorithm’s selected queries D = (cid:8) ( Q ( t ) , x ( t ) )
(cid:9) t in terms of theinformation they provide on the model parameters Θ .
We use two such metrics: mean squared error(MSE) and log-likelihood.
Since both active and non-active methods are expected to reach the sameperformance with a large number of queries, we look at the area under the curve (AUC) of these twometrics over number of queries.
To evaluate the learning performance, we quantify the success of arobot, which learned a multimodal reward, via the learned policy rewards on the actual task.
MSE.
Suppose we know the human is truly modeled by Θ ∗ adhering to the assumed model classof Section 3.
Given a set of observations D , we can compute an MLE estimate (cid:98) Θ of the modelparameters using Eq.
The MSE is then the squared error between (cid:98) Θ and Θ ∗ (see Appendix H.1).
While this metric cannot be evaluated with real humans, we can use this metric with synthetic humanmodels (model with known parameters Θ ∗ ) in simulation.
A lower MSE score means the selectedqueries D allow us to better learn a multimodal Plackett-Luce model close to the true model Θ ∗ .
Log-Likelihood.
The log-likelihood metric measures the log-likelihood of the response to a randomquery given the past observations D .
If the past observations D are informative, the true responseto a random query Q will in expectation be more likely, meaning the log-likelihood metric will begreater.
See Appendix H.2 for details on how we compute this metric.
Learned Policy Reward.
We take the MLE estimate of each reward weights vector and train a
DQN policy using them [61].
We then run these learned policies on the actual environment withthe corresponding true reward functions (see Appendix H.3) to obtain the learned policy rewards.
5.3 Results
Multimodal Learning is Necessary.
We ﬁrst compare unimodal and multimodal models to showthe insufﬁciency of unimodal rewards when the data come from a mixture.
To leave out any possiblebias due to active querying, we make this comparison using random querying.
We let the true reward function have M = 2 modes and set a query size of K = 6 items for identiﬁability as Section 4.4 suggests, and for acquiring high information from each query.
We simulate100 pairs of experts whose reward weights ω m
and the mixing coefﬁcients α m are sampled from theprior Pr(Θ) .
Having these simulated experts respond to 15 queries, we report the MSE in Fig.
The unimodal reward model causes an unstablyincreasing MSE.
This is mostly due to the outliers where the reward weights ω 1 and ω 2 are faraway from each other and the unimodal rewardfails to learn any of them.
We therefore alsoplot the median values and quartiles in Fig.
While the bimodal reward model learned usingour proposed approach decreases the MSE overtime, the unimodal model has a roughly constant MSE, which suggests it is unable to learnwhen the data come from a mixture.
We present an additional unimodal learning baseline evaluated on the user study data in Appendix K.
Active Querying with Information Gain is Data-Efﬁcient.
We next compare our informationgain-based active querying approach with the other baselines.
For this, we use the same experimentsetup as above with M = 2 reward function modes and ranking queries of size K = 6 , and simulate75 pairs of human experts.
We present the results in terms of MSE in Fig.
In LunarLander, theinformation gain objective signiﬁcantly outperforms both random querying and volume removal interms of the AUC MSE ( p < 0 .
005 , paired-sample t -test).
Notably, volume removal performs evenworse than the random querying method, which might be due to the known issues of volume removaloptimization as brieﬂy discussed in Appendix F.2.
On the other hand, the difference is not statistically signiﬁcant in the Fetch Robot experiment, which might be due to the small trajectory dataset, or because almost all trajectories in the dataset minimize or maximize some of the trajectory features, accelerating and simplifying learning under the linear reward assumption.
See Appendix G.2for details about the trajectory features and how we generated the trajectory dataset.
We further analyze the querying methods inthis multimodal setting under the log-likelihoodmetric in Fig. 4.
Information gain signiﬁcantlyoutperforms random querying and volume removal in both experiments with respect to the
AUC log-likelihood ( p < 0 .
With respectto the ﬁnal log-likelihood, information gain reduces the amount of required data in LunarLander by about 35% compared to random querying and about 60% to volume removal.
Similarly in the Fetch Robot, the improvement isapproximately 25% over both baselines.
Appendix J presents two additional experiments: one which clearly shows the effectiveness of our approach for learning a mixture ofmore than two reward functions (speciﬁcally,
M =5 ), and one which studies the robustness against misspeciﬁed M .
Information Gain Leads to Better Learning.
Having seen the superior predictive performance ofthe reward learned via information gain optimization, we next assess its performance in the actualenvironment.
As random querying outperforms volume removal in terms of log likelihood and MSEas in Fig.
4, we compare the information gain with random querying.
For this, we run the multimodal reward learning with 75 pairs of randomly generated reward weights( M = 2 and K = 6 ).
For each of the 150 individual reward functions, we compute the learnedpolicy rewards.
Fig.
5 shows the results.
While the standard errors in the plots seem high, thisis mostly because optimal trajectories for different reward weights differ substantially in terms ofrewards, which causes an irreducible variance.
However, since the underlying true rewards are thesame between the information gain and random querying methods, we ran the paired sample ttest between the results and observed statistical signiﬁcance ( p < 0 . 05 ).
This means although thelearned policy rewards between different runs differ substantially, the reward function learned viathe information gain method leads to better task performance compared to random querying.
User Studies
We now empirically analyze the performanceof our algorithm with two online user studies.
2 We again used the LunarLander and Fetch
Robot environments.
We provide a summaryand a video of the user studies and their results at https://sites.google.com/view/multimodal-reward-learning/ .
Experimental Setup.
For LunarLander, subjects were presented with either of the following instructions at every ranking query: “Land softly on the landing pad” or “Stay in the air as long aspossible”.
We randomized these instructions such that users get one of them with 0 .
6 and the otherwith 0 .
4 probability.
We kept the presented instructions hidden from the learning algorithms so thatthey need to learn a multimodal reward without knowing which mode each ranking belongs to.
For the Fetch robot environment, we recorded the 351 trajectories on the real robot as short videoclips so that the experiment can be conducted online under the pandemic regulations.
Human subjects participated in the experiment as groups of two to test learning from multiple users.
Eachparticipant was instructed that the robot needs to put the banana in one of the shelves and differentshelves have different conditions (the same as in our running example, see Fig.
1, Appendix I.1).
After emphasizing there is no one correct choice and it only depends on their preferences, we askedeach participant to indicate their preferences between the shelves on an online form.
Afterwards, each group of two subjects responded to 30 ranking queries in total where each query consisted of 6trajectories.
We selected who responds to each query randomly, with probabilities 0 .
6 and 0 . 4 .
Appendix I.2 presents details on the user interface used in our experiments.
Independent Variables.
We varied the querying algorithm: active with information gain and random querying.
We excluded the volume removal method to reduce the experiment completion timefor the subjects, as it already performed worse than random querying in our simulations.
Procedure.
We conducted the experiments as a within-subjects study.
We recruited 24 participants(ages 19 – 56; 9 female, 15 male) for LunarLander and 26 participants (ages 19 – 56; 11 female,15 male) for the Fetch robot.
Each subject in the LunarLander, and each group of two subjects inthe Fetch robot experiment responded to 40 ranking queries; 15 with each algorithm and 10 randomqueries for evaluation at the end.
The order of the ﬁrst 30 queries was randomized to prevent bias.
Dependent Measures.
Learning the multimodal reward functions via the 15 rankings collected byeach algorithm, we measured the log-likelihood of the ﬁnal 10 rankings collected for evaluation.
Hypotheses.
With LunarLander and Fetch robot, we test the following hypotheses respectively:
H
Querying the participants, who are trying to teach two different tasks, actively with informationgain will lead to faster learning than random querying.
H2.
While learning from two people with different preferences, active querying with informationgain will lead to faster learning than random querying.
Results.
Figure 6 visualizes how log-likelihoodof the evaluation queries changes over thecourse of learning by both algorithms.
Activequerying with information gain leads to signiﬁcantly faster learning compared to randomquerying in LunarLander.
Indeed, the difference in AUC log-likelihood is statistically signiﬁcant ( p < 0 . 05 ).
Furthermore, the activequerying method enabled reaching the ﬁnal performance of random querying after only 9 or 10queries, for around a 35% reduction in the amount of data needed, supporting H1 .
As the robot experiments have a an easier task with a small number of variables between the trajectories, both querying methods converge to similar performances by the end of 15 queries.
However,active querying accelerates learning in the early stages—the difference in AUC log-likelihood isagain statistically signiﬁcant ( p < 0 . 05 ).
Looking at the ﬁnal performance with random querying,improvement in data efﬁciency is about 10% , supporting H2 .
Conclusion
Summary.
This work presents a novel approach for learning multimodal reward functions.
We formulated the problem as a mixture learning problem solved using ranking queries that are answeredby experts.
We further developed an active querying method that maximizes information gain toimprove the quality of ranking queries made to the experts.
The results suggest our model learnsmultimodal reward functions, with data efﬁciency improved by our new active querying method.
Limitations and Future Work.
Our model for learning multimodal rewards requires knowingthe number of different modes (experts or tasks)
M in advance.
This might be difﬁcult in somesettings.
For example, when several experts belonging to different clusters, e.g., timid and aggressivedrivers, provide data, it might be difﬁcult to know the number of clusters in advance.
However, asimple approach that ﬁts the multimodal reward under various M could reveal the true number ofunderlying modes.
Another challenge is that learning a mixture reward model may contribute tothe reward ambiguity problem in inverse reinforcement learning: each individual reward may haveits own ambiguity.
Future work should investigate the practical implications of this.
In addition,theoretical results assert, to guarantee the reliable learning of a multimodal reward with M modes,ranking queries should consist of K queries such that M ≤ (cid:98) K − 2 2 (cid:99) ! .
While this is manageable bymultiple pairwise comparisons for each query or an iterative process where the expert selects thetop item, it might consume too much time for large M .
Thus, an interesting future direction is toinvestigate how to incorporate multiple forms of expert feedback, e.g., demonstrations in addition torankings, to pretrain and reduce the required interaction time with humans.