Learning Reward Functions from Scale Feedback
Abstract: Today’s robots are increasingly interacting with people and need to efficiently learn inexperienced user’s preferences.
A common framework is to iteratively query the user about which of two presented robot trajectories they prefer.
While this minimizes the users effort, a strict choice does not yield any information on how much one trajectory is preferred.
We propose scale feedback, where the user utilizes a slider to give more nuanced information.
We introduce a probabilistic model on how users would provide feedback and derive a learning framework for the robot.
We demonstrate the performance benefit of slider feedback in simulations, and validate our approach in two user studies suggesting that scale feedback enables more effective learning in practice.
Introduction
While autonomous robots are able to accomplish an increasing variety of tasks, a key challenge that still remains is how they should pursue and trade off between their goals.
In recent years, there has been significant work on interactively learning user preferences of robot behaviors [1–15].
Usually, the user is provided with one or more robot trajectories and is asked to provide feedback through pairwise comparisons [2–8], rankings of the trajectories [16, 17] or physical feedback [18, 12, 14].
The underlying reward function governing human preferences can then be learned through this implicit feedback.
Specifically, one framework with minimal complexity for the user is learning from choice feedback [2–8] where the robot demonstrates two alternative trajectories for some task.
The user then simply chooses their preferred behavior allowing the robot to infer an underlying reward function for the user preferences.
Choice feedback, although simple to collect, is limiting in a number of ways.
Consider the example shown in Fig.
1, where a robot is tasked to serve a drink to a customer.
The customer might have different preferences over the type of drink to have (milk, orange juice, or water), or the specifics of the trajectory the robot takes (e.g., if it goes over the stove or around it which can affect the temperature of the drink or the likelihood of the robot accidentally hitting the pan handle).
A strict choice feedback between two trajectories does not really capture these intricacies of human preferences.
We thus need to have a more expressive way of collecting data from humans.
Our key insight is that allowing users to provide a scaled approach on a slider (as shown in Fig. 1) can provide a more expressive medium for learning from humans and capture nuances in their preferences.
In this work, we propose scale feedback as a new mode of interaction: Instead of a strict question on which of the two proposed trajectories the user prefers, we allow for more nuanced feedback using a slider bar.
We design a Gaussian model for how users provide scale feedback, and learn a reward function capturing human preferences.
Similar to prior work in robotics, we assume this reward is a linear function of a set of features [19, 11, 13, 7], where the main task of learning from scale feedback is to recover the weights of this reward function.
To learn in a data-efficient manner, we actively generate our queries to the user, i.e., pairs of trajectories demonstrated to a user similar to
Fig. 1, by optimizing two well-known objectives of information gain [4] and max regret [5].
We demonstrate the performance benefit of scale feedback over choice in a driving simulation.
Further, we investigate its practicality in two user studies with the real robot experiment shown in
Fig.
Our results suggest scale feedback leads to significant improvements in learning performance.
Related Work
Learning from human feedback is an important problem in developing interactive robots that work alongside humans.
Researchers study learning from demonstrations [19–21], corrections [22, 23, 14], ordinal feedback [24, 8], rankings [16, 17, 25, 26], critiques [27, 28], and choices [2–5, 29].
While demonstrations usually are very informative, they are not always viable:
Demonstrating the desired behavior might require a high level of expertise [30], or can be difficult in high-order systems [31–33].
Choice questions minimize interface complexity and mental effort for the user.
However, when the user is indifferent towards both options, learning becomes difficult since users may become noisier in their responses.
Thus, [7, 4, 6] investigate modifications of learning from choice where users can also answer About Equal .
These two forms of choice feedback are usually referred to as strict and soft choice.
When the user chooses the neutral answer, the robot learns to assign about equal reward to the presented trajectories.
While choice feedback provides an easy medium for learning from humans, it provides at most one bit of information.
Thus, to effectively learn from choice feedback, we often need to actively generate the queries made to humans.
Previous work has investigated different auxiliary measures that are greedily optimized to enable efficient learning, including expected volume removal [2], information gain [4], and regret [5].
In the proposed scale feedback framework, we take the soft choice approach one step further: Instead of three discrete values for feedback (prefer A, prefer B, neutral) users give quasi-continuous feedback.
This allows the user to indicate by how much they prefer one option over the other.
Slider bars have been used in robotics for tuning parameters [34].
More related to our work, Cabi et al.
[35] proposed using them for reward sketching .
Instead of assigning a numerical preference between presented options, users continuously indicate the robot’s progress towards some goal.
However, this requires users to assign scores to different parts of trajectories.
Developing the scale feedback for preference-based learning, we retain the ease of comparing trajectories.
Problem Formulation
We now introduce the notation we use in this paper and formulate the learning problem.
Reward function.
We consider the scenario where a robot needs to customize its behavior to the preferences of a user Alice.
We assume Alice evaluates robot paths P ∈ P based on a vector of features ϕ P =
[ ϕ 1 ( P ) , . . .
, ϕ n ( P )] .
Similar to prior works in robotics [19, 11, 13, 7], we define a linear reward function r that assigns a numerical value to a path P by weighting a set of features:
[Equation]
These features are usually provided by a domain expert incorporating the core factors that the reward needs to capture, e.g., collision with other objects, or distance to the goal.
Further, the robot has access to a motion planner that finds an optimal path given a set of weights, i.e., the planner is a (deterministic) function ρ : R n → P where ρ ( w ) =
arg max P ∈P r ( P, w ) .
Regret.
Similar to [5], we define the regret between any two weights ( w , w ′ ) as the difference in the reward w ′ assigns to the paths ρ ( w ) and ρ ( w ′ ) :
[Equation]
which quantifies the suboptimality when the true weights are w ′ , but the path is optimized using w .
Learning.
Let w ∗
denote
Alice’s weights for the reward function.
These weights are not known to the robot; the only information initially available is a prior distribution P ( w = w ∗ ) .
The robot learns w ∗ by iteratively presenting her with two paths P and Q for K iterations.
We extend the learning from choice framework, where users simply indicate the path they prefer, to a setting where they instead provide a more finely detailed scale feedback .
Definition 1 (Scale feedback) .
Presented with two paths P and Q , Alice returns numerical feedback ψ ∈ [ − 1 , 1] .
If ψ = 0 , this means Alice has no preference between the paths, ψ = 1 equals a strong preference for path P and ψ = − 1 a strong preference for path Q .
From an interface design and expressiveness perspective, it is undesirable to have users give a numerical value for ψ .
Instead, they can express such a feedback with a slider bar with a more fine-grained set of options.
An example is illustrated in Fig.
We let D K
=
( P 1 , Q 1 , ψ 1 ) , . . .
, ( P K , Q K , ψ K ) } be the set of recorded user feedback.
Performance Measures.
Let w ˆ be the robot’s estimate of w ∗ , and ξ ( w ˆ , w ∗ ) be a performance measure for the learning process.
Previous works focused on the alignment of weights [2, 4],
Alignment = w ˆ · w ∗ / || w ˆ ||·|| w ∗
|| , measuring the cosine similarity of vectors w ˆ and w ∗ , i.e., how well the parameters of Alice’s reward function are learned.
Alternatively, Wilde et al.
[5] proposed the relative error in cost .
We adapt this as the Relative Reward = ϕ
( ρ ( w ˆ)) · w ∗ / ϕ ( ρ ( w ∗ )) · w ∗ , measuring how much Alice likes the trajectory optimized for w ˆ compared to the one optimized for w ∗ .
Problem Statement.
Let π be an adaptive policy for designing queries ( P, Q ) , and let D K ( π | w ∗ ) be the expected set of user feedback when a user w ∗ is queried by π for K iterations.
Given a robot motion planner ρ , a user with preferences w ∗ , and a budget of K rounds to query the user about their scale feedback on two presented paths, our goal is to find an adaptive policy π that solves max π
[Equation]
Approach
We now briefly review learning from choice, and then extend the framework to scale feedback.
4.1 Choice Feedback
When presented with two paths P and Q , a user returns an ordering P ⪰ Q ( P is preferred) or
P ⪯ Q ( Q is preferred).
In a noiseless setting, we have
[Equation]
That is, the path P has a reward that is at least as high as that of Q with respect to the hidden true user weights w ∗ .
Using r ( P, w ) =
ϕ P · w , we can tighten our notation and write ( ϕ P − ϕ Q ) · w ∗ instead of r ( P, w ∗ ) − r ( Q, w ∗ ) .
Equation (4) already contains an observation model:
If the user chooses path P , the robot can infer that P has a higher reward with respect to w ∗ .
This inequality defines a halfspace Λ( P, Q ) =
{ w | ( ϕ P − ϕ Q ) · w ∗ ≥ 0 } containing all weights that are feasible given the observed user choice.
Over k iterations, we can intersect the sets Λ( P 1 , Q 1 ) , . . .
, Λ( P k , Q k ) to obtain the feasible set F k shown in Fig. 2(a).
By definition, this feasible set is convex.
4.2 Scale Feedback
Scale feedback allows the robot to gain more information: the robot can also infer by how much the user prefers P , allowing for learning tighter feasible sets.
We extend the model in (4) and show how a noiseless user would provide scale feedback and then study how a robot can learn from it.
Definition 2 (Maximum Reward Gap) .
Given a user w ∗ , the maximum reward gap is
[Equation]
We notice that the maximum reward gap cannot be computed, since w ∗ is unknown to the robot.
Nevertheless, we can formulate the user choice model and then derive an observation model.
User model.
The maximum reward gap helps to define when a noiseless user would indicate a strong preference.
We assume this occurs if and only if the difference in reward of P and Q with respect to w ∗ is at least α ∗ δ ∗ for some 0 <α ∗ ≤ 1 .
Here α ∗ is a saturation parameter which governs at what reward difference (w.r.t.
to the maximum gap) the user’s feedback gets saturated to a strong preference.
For any other ( P, Q ) where | ( ϕ P − ϕ Q ) · w ∗ |∈ [0 , α ∗ δ ∗ ) we assume the user to linearly scale ψ between − 1 and 1 , which leads to the following model.
Definition 3 (Noiseless User Model) .
Presented with two paths P and Q , a noiseless user with parameter α ∗ ∈ (0 , 1] will always provide the following feedback:
[Equation]
We illustrate the noiseless user model in Fig.
3a under different saturation parameters α ∗ .
In Fig.
3b, we show a simulated example: for a fixed w ∗ we simulate how users with different values for α ∗ would provide scale feedback to the same 20 queries.
For larger α ∗ , they position the slider closer to the neutral position.
Finally, we derive an observation model for the noiseless user:
[Equation] [Equation]
Figures 2(b) and 2(c) illustrate the resulting feasible sets from (7).
Moreover, we notice the userspecific and unknown parameters α ∗ and δ ∗ always appear as a product.
Thus, we can introduce an auxiliary parameter β
= α ∗ δ ∗ to write (7) as [︁ − ψ, ϕ P − ϕ Q ]︁ · [ β, w ∗ ] ⋚ 0 .
As the model remains linear, the notion of halfspaces and the feasible set F can be extended to the augmented vector space.
4.3 Probabilistic User Feedback
In practice, users are often noisy; they might consider additional or slightly different features than the robot, not follow the linear reward function, or simply be uncertain in some answers.
Since we cannot expect users to always provide slider feedback following (6), we introduce a probabilistic model where we add uncertainty to the placement of the slider.
Another practical limitation is the fact that we cannot collect truly continuous feedback from the users.
Instead, the slider bar has a step size ϵ ∈ (0 , 1] such that the user provides feedback of the form nϵ for n ∈ Z and − ϵ − 1 ≤ n ≤ ϵ − 1 .
Note that ϵ → 0 retains the continuous scale feedback, whereas ϵ = 1 gives the soft choice model where the feedback is always in {− 1 , 0 , 1 } .
Definition 4 (Probabilistic User Model) .
Given a user w ∗ and a query ( P, Q ) , let ψ be the user feedback defined in the noiseless user model in (6).
A probabilistic user using a slider bar with a step size of ϵ then provides feedback
[Equation]
where ν is a zero-mean Gaussian noise, i.e., ν ∼ N (0 , σ 2 ) with standard deviation σ , and round ( x, ϵ ) outputs nϵ closest to x such that n ∈ Z ∩ [ − ϵ − 1 , ϵ − 1 ] .
Probabilistic Observation Model.
Given the probabilistic user model, we now show how a robot can infer about w ∗ from scale feedback.
In the noiseless case, user feedback defines a feasible set.
For the probabilistic case, we instead derive a distribution over w and α .
Let δ ( w )=max P,Q ∈P ( ϕ P − ϕ Q ) · w , similar to (5).
Then for 0 <α ≤ 1 , the belief is defined f ( w , α | ψ, P
[Equation]
where
[Equation]
Given noisy user feedback µ as in (8), we can define a probabilistic density function f ( ψ | µ ) .
Together with (9) we derive a compound probability distribution
[Equation]
where we can write f ( ψ | µ ) for ψ ∈ [ − 1 , 1] as
[Equation]
and f ( ψ | µ )=0 for ψ ̸∈ [ − 1 , 1] .
Here, Φ denotes the cdf of a standard normal distribution.
Finally, given a sequence D K = { ( P k , Q k , µ k ) } Kk =1 and some prior f ( w , α ) , the joint posterior is
[Equation]
[Equation]
Here, we can factor f ( w , α ) as P ( w ) P ( α ) by assuming w and α are independent and we also have a prior for α ∗ .
We then take the expectation of the posterior f ( w , α | D K ) as our learned user model.
Algorithm Design
We now outline the learning algorithm.
Over K iterations: (i) the robot actively generates a query ( P k , Q k ) given previous observations D k − 1 , (ii) the user provides feedback to the query in the form of the slider value µ k (in the noiseless case, µ k = ψ k ), and (iii) the robot updates its dataset D k using (13).
After iteration K , the algorithm returns the expected weight w ˆ = E
[ w | D K ] .
5.1 Worst Case Error Bound
To compare scale feedback to choice feedback, we establish a worst case bound on the performance measures for both frameworks.
We introduce the worst-case error as the maximum negative performance measure, 1 − ξ ( w , w ∗ ) .
The constant in front ensures a positive value, which we then discount with the posterior belief, given observations D :
[Equation]
This describes the worst w the robot could pick, discounted by the posterior distribution f learned from data D .
In the noiseless setting, this simplifies to max w ∈F 1 − ξ ( w , w ∗ ) .
Proposition 1 (Upper error bound) .
Let D S denote the observation made from scale feedback and D C be the observation from choice feedback for the same set of queries.
For any user weights w ∗ , it holds in the noiseless setting that Err max ( w ∗ , D S ) ≤ Err max ( w ∗ , D C ) .
The proof follows from the observation F Scale ⊆ F Choice , i.e., scale feedback removes more volume from the weight set.
Hence, the worst choice of an estimate w ˆ given observations is guaranteed to have a smaller worst case error when using scale feedback.
The full proof is in Appendix B.
5.2 Active Query Generation
To learn w ∗ efficiently, the robot chooses the query ( P, Q ) it presents to the user.
While randomly selected queries often lead to some learning progress, actively designing a query can drastically improve learning when the number of iterations is limited.
Two recent approaches for learning from choice are information gain [4] and max regret [5].
Information gain seeks to reduce the robot’s uncertainty over w while choosing queries that are easy to answer for the user.
Max regret, on the other hand, minimizes the maximum regret by showing mutual worst case paths, which also results in easy queries.
We leverage both of these methods for our active query generation in scale feedback.
We start with the information gain.
Let H denote Shannon’s information entropy [36].
As the outcome of the query is yet unknown, a greedy step takes the expectation over µ :
[Equation]
We approximate the computation of entropy by summing over a set Ω of M samples of ( w , α ) ∼ f .
Thus, following the derivation in Biyik et al.
[4], the new query ( P, Q ) solves
[Equation]
The max regret policy generates queries ( P, Q ) such that if the robot learned P but the user optimal solution would be Q is a worst case.
With a symmetric perspective over P and Q , we have
[Equation]
where R ( · , · ) is the reward difference defined in (2).
By observing feedback to such queries it greedily improves the probabilistic worst case error.
In contrast to the information gain approach, maximum regret requires P and Q to be optimal trajectories for some users ( w P , α P ) and ( w Q , α Q ) .
On the other hand, maximum regret does not require a one-step look-ahead and thus no summation over potential feedback values µ , making it computationally lighter.
Equations (16) and (17) now give us two different policies for solving the initial problem (3).
In the simulations, we compare how the performance of both benefits from scale feedback.
Simulation Results
We now present our main simulation results.
Additional results can be found in the Appendix.
Experiment Setup.
We simulate the presented framework using the Driver experiment used in [2, 4–6].
We modify the setup by adding 6 new features, obtaining a more challenging 10 -dimensional problem (details on the features, as well as results for the original driver can be found in the Appendix). 71 distinct user preferences w ∗ are drawn uniformly at random, and each user is simulated with α ∗ ∈ { . 25 , .
75 , 1 } , making it 284 runs for each method.
We set σ =0 .
1 for the noise level.
We generate a set of 200 distinct sample trajectories by drawing random weights w and then computing their optimal trajectories.
The active query generation methods then optimize over this set.
We evaluate learning using the alignment metric and the relative reward.
As a baseline we use soft choice (strict choice showed a slightly poorer performance).
To ensure a fair comparison, we emulate soft choice by setting the step size to ϵ = 1 and use the same noise model for both forms of feedback.
Results.
Fig.
4 shows the alignment and relative reward for the driver experiment for information gain, max regret and random query generation.
We observe that in all cases scale feedback significantly improves the performance over soft choice in both metrics ( p < .
001 in all cases with two-sample t -test).
When using the proposed scale feedback, the alignment after 20 iterations improves from .
77 to .
86 for information gain, from .
67 to .
76 for max regret, and from . 64 to .
75 for random queries.
The relative reward improves for information gain and max regret similarly from . 97 to 1 , i.e., the learned solution is optimal.
Both methods make most progress during the firstiterations.
Random queries improve the final relative reward from . 94 to . 97 .
Overall, the simulation showcases that scale feedback improves learning, independent of the query selection method.
For information gain and max regret, scale feedback allows for finding optimal solution, i.e., collecting 100% reward, within a small budget of iterations.
In Appendix D, we show additional simulation results for higher noise.
User Study
Finally, we analyze the scale feedback in comparison with choice feedback and under different active querying methods with two user studies.
In both studies, we used ϵ = 0 .
1 for scale queries.
Experiment Setup.
We designed a serving task with a Fetch robot [37] as shown in Fig. 1, and generated a dataset of 120 distinct trajectories.
Human subjects were told they should train the robot to bring the drink to the customer in the manner they prefer, paying attention to the following five factors: the drink (out of 3 options) to be served, the orientation of the pan in front of the robot, moving the drink behind or over the pan, the maximum height of the path, and the speed.
The subjects were also informed about the types of queries they will respond to.
Independent Variables.
In the first experiment, we wanted to compare scale and soft choice under random querying, and scale under random and information gain querying.
Hence, we varied the query type and the querying algorithm among: (i) soft choice with random querying, (ii) scale with random querying, and (iii) scale with information gain querying.
In the second experiment, we wanted to compare scale and soft choice under information gain querying.
Hence, we employed: (i) soft choice with information gain querying, and (ii) scale with information gain querying.
For all, we took σ = 0 .
35 based on pilot trials with different users (see Appendix E).
Procedure.
We recruited 18 participants ( 5 female, 13 male, ages 20 – 55) for the first, andparticipants ( 5 female, 9 male, ages 20 – 56) for the second experiment.
Due to the pandemic conditions, the subjects participated in the study remotely with an online interface as in Fig.
The study started with an instructions page with a two-question quiz to make sure the participants understood how to use the interface.
After reading the instructions, we had the subjects fill a form where they indicated their preferences for each of the five individual factors described above, to encourage them to be consistent in their responses during the data collection.
In the experiments, each participant responded to 10 queries generated with each of the algorithms.
After each of these 10 -query sets, they were shown the optimal trajectory from the dataset with respect to their learned reward function.
The participants responded to a 5 -point Likert scale survey (1-Strongly Disagree, 5-Strongly Agree) for this trajectory: “The displayed trajectory fits my preferences on the task.”
We also collected scale feedback for 10 more randomly-generated queries for validation in each experiment.
We randomized the order of these sets (of 10 queries) to prevent any bias.
The interface provided a “Sync Videos” button to restart both videos for easier comparison.
Dependent Measures.
As an objective measure of the learning performance, we calculated the loglikelihood of the validation set (of 10 scale queries 2 ) under the posterior f ( w , α | D ) learned using the 10 queries generated via each algorithm, i.e., we calculated:
[Equation]
We also used the responses to the 5 -point Likert scale survey questions to measure how well the learned rewards achieve the task.
Finally, the users took a post-experiment survey where they rated (from 1 to 5 ) the easiness and expressiveness of soft choice and scale questions.
Hypotheses.
We test the following hypotheses.
H1.
Scale feedback leads to faster learning than soft choice feedback.
H2.
Querying based on information gain accelerates learning compared to random querying.
H3.
Users will prefer information gain over random querying in terms of the optimized trajectories.
H4.
Users will prefer scale feedback over soft choice feedback in terms of the optimized trajectories.
H5.
Users will rate the scale feedback as easy as soft choice feedback.
H6.
Users will rate the scale feedback as expressive as soft choice feedback.
Results.
We present results of the first and the second experiments in Figs.
5 and 6, respectively.
It can be seen that the log-likelihood of the validation set after learning the reward function via scale feedback is higher than learning via soft choice feedback, under both random and information querying.
Besides, information gain based query generation accelerates the learning and leads to higher log-likelihood values compared to random querying.
All of these comparisons are statistically significant with p < .
001 (paired-sample t -test), so they strongly support H1 and H
In Fig. 5(b), it can be seen active querying led to learning reward functions that better optimize trajectories compared to random querying – this comparison was somewhat significant with p ≈ .
05 , supporting H3 .
In fact, when we fit a Gaussian distribution to the ratings, we observe that it is 1 .times as likely to get a better rating with information gain querying than random querying.
Surprisingly, learning via soft choice achieved slightly higher reward than learning via scale when queries were randomly selected, and slightly lower reward when queries were generated based on information gain.
However, these comparisons are not statistically significant.
This is indeed analogous to the relative reward comparisons in Fig.
4: more complex tasks might be needed to better analyze the difference between the two methods.
Thus, we neither reject nor accept H4 .
Finally, the subjective results in Fig. 5(c) and 6(c) suggest that users find the soft choice feedback slightly, but consistently, easier than the scale feedback ( p < . 01 ), rejecting H5 .
This is not surprising, as it is often easier to make a pairwise comparison and the “About Equal” option in the soft choice questions makes them even easier [4].
On the other hand, there was no statistically significant difference in terms of expressiveness of scale and soft choice feedback, partially supporting H6 .
In summary, it is interesting that our users perceived the soft choice as easier and even more expressive at times; even though quantitatively, the scale feedback significantly outperforms the soft choice.
Discussion
Summary.
We proposed scale feedback for reward learning where users provide more nuanced feedback than choice.
We introduced a user model and showed how a robot can infer reward from noisy scale feedback.
We adapted state-of-the-art query generation methods to accelerate learning.
In simulations and a user study, scale feedback significantly improved learning.
Users rank choice feedback as slightly easier, but both forms of feedback as equally expressive.
However, the minor decrease in ease of use is out-weighted by a strong improvement in learning performance.
Future Work.
We proposed scale queries as a way to give nuanced feedback between two trajectories.
It is possible to extend them to n + 1 trajectories, with specialized user interfaces that allow users to select a point from an n -simplex instead of a slider bar.
Future work should investigate this and if users can still give reliable feedback to these more complex queries.
In our experiments, we used a pre-computed trajectory set.
Alternatives, e.g., optimizing queries over action sets as in [2], or using planners as in [5], should be studied for real-time online learning systems.
The high estimate of σ in the user studies suggests the proposed probabilistic model may be inaccurate.
Future work should refine the user model, including interactively learning σ ; or fit a new user model that does not necessarily adopt a Gaussian noise.
Surprisingly, users did not perceive scale feedback as more expressive.
This could be addressed with improving interface design as well as designing a query generation method that actively exploits the slider’s expressiveness.
Acknowledgments
This research is partially supported by the Natural Sciences and Engineering Research Council of
Canada (NSERC).
The authors would also like to acknowledge funding by NSF grants # and #1941722, FLI grant RFP2-000, and DARPA.