Learning Reward Functions from Scale Feedback
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Today’s robots are increasingly interacting with people and need toefﬁciently learn inexperienced user’s preferences.
A common framework is toiteratively query the user about which of two presented robot trajectories theyprefer.
While this minimizes the users effort, a strict choice does not yield anyinformation on how much one trajectory is preferred.
We propose scale feedback,where the user utilizes a slider to give more nuanced information.
We introducea probabilistic model on how users would provide feedback and derive a learningframework for the robot.
We demonstrate the performance beneﬁt of slider feedback in simulations, and validate our approach in two user studies suggesting thatscale feedback enables more effective learning in practice.
Introduction
While autonomous robots are able to accomplish an increasing variety of tasks, a key challenge thatstill remains is how they should pursue and trade off between their goals.
In recent years, therehas been signiﬁcant work on interactively learning user preferences of robot behaviours [1–15].
Usually, the user is provided with one or more robot trajectories, and is asked to provide feedbackthrough pairwise comparisons [2–8] or rankings of the trajectories [16], or physical feedback [17,12, 14].
The underlying reward function governing human preferences can then be learned throughthis implicit feedback.
Speciﬁcally, one framework with minimal complexity for the user is learningfrom choice feedback [2–8], where the robot demonstrates two alternative trajectories for some task.
The user then simply chooses their preferred behaviour allowing the robot to infer an underlyingreward function for the user preferences.
Choice feedback, although simple to collect, is limiting in a number of ways.
Consider the exampleshown in Fig.
1, where a robot is tasked to serve a drink to a customer.
The customer might havedifferent preferences over the type of drink to have (milk, orange juice, or water), or the speciﬁcs ofthe trajectory the robot takes (e.g., if it goes over the stove or around it which can affect the temperature of the drink or the likelihood of the robot accidentally hitting the pan handle).
A strict choicefeedback between two trajectories does not really capture these intricacies of human preferences.
We thus need to have a more expressive way of collecting data from humans.
Our key insight isthat allowing users to provide a scaled approach on a slider (as shown in Fig. 1) can provide a moreexpressive medium for learning from humans and capture nuances in their preferences.
In this work, we propose scale feedback as a new mode of interaction: Instead of a strict question onwhich of the two proposed trajectories the user prefers, we allow for more nuanced feedback usinga slider bar.
We design a Gaussian model for how users provide scale feedback, and learn a rewardfunction capturing human preferences.
Similar to prior work in robotics, we assume this rewardis a linear function of a set of features [18, 11, 13, 7], where the main task of learning from scalefeedback is to recover the weights of this reward function.
To learn in a data-efﬁcient manner, weactively generate our queries to the user, i.e., pairs of trajectories demonstrated to a user similar to
Fig. 1, by optimizing two well-known objectives of information gain [4] and max regret [5].
We demonstrate the performance beneﬁt of scale feedback over choice in a driving simulation.
Further, we investigate its practicality in two user studies with the real robot experiment shown in
Fig.
Our results suggest scale feedback leads to signiﬁcant improvements in learning performance.
Related Work
Learning from human feedback is an important problem in developing interactive robots that workalongside humans.
Researchers study learning from demonstrations [18–20], corrections [21, 22,14], ordinal feedback [23, 8], rankings [16, 24, 25], critiques [26, 27], and choices [2–5, 28].
While demonstrations usually are very informative, they are not always viable:
Demonstrating thedesired behaviour might require a high level of expertise [29], or can be difﬁcult in high-ordersystems [30–32].
Choice questions minimize interface complexity and mental effort for the user.
However, when the user is indifferent towards both options, learning becomes difﬁcult since usersmay become noisier in their responses.
Thus, [7, 4, 6] investigate modiﬁcations of learning fromchoice where users can also answer About Equal .
These two forms of choice feedback are usuallyreferred to as strict and soft choice.
When the user chooses the neutral answer, the robot learns toassign about equal reward to the presented trajectories.
While choice feedback provides an easymedium for learning from humans, it provides at most one bit of information.
Thus to effectivelylearn from choice feedback, we often need to actively generate the queries made to humans.
Previouswork has investigated different auxiliary measures that are greedily optimized to enable efﬁcientlearning, including expected volume removal [2], information gain [4], and regret [5].
In the proposed scale feedback framework, we take the soft choice approach one step further: Instead of three discrete values for feedback (prefer A, prefer B, neutral) users give quasi-continuousfeedback.
This allows the user to indicate by how much they prefer one option over the other.
Slider bars have been used in robotics for tuning parameters [33].
More related to our work, Cabiet al.
[34] proposed using them for reward sketching .
Instead of assigning a numerical preferencebetween presented options, users continuously indicate the robot’s progress towards some goal.
However, this requires users to assign scores to different parts of trajectories.
Developing the scalefeedback for preference-based learning, we retain the ease of comparing trajectories.
Problem Formulation
We now introduce the notation we use in this paper and formulate the learning problem.
Reward function.
We consider the scenario where a robot needs to customize its behaviour to thepreferences of a user Alice.
We assume Alice evaluates robot paths P ∈ P based on a vector offeatures φ P
=
[ φ 1 ( P ) , . .
, φ n ( P )] .
Similar to prior works in robotics [18, 11, 13, 7], we deﬁne alinear reward function r that assigns a numerical value to a path P by weighting a set of features:
[Equation]
These features are usually provided by a domain expert incorporating the core factors that the reward needs to capture, e.g., collision with other objects, or distance to the goal.
Further, the robothas access to a motion planner that ﬁnds an optimal path given a weight, i.e., the planner is a (deterministic) function ρ : R n → P where ρ ( w ) =
arg max P ∈P r ( P, w ) .
Regret.
Similar to [5], we deﬁne the regret between any two weights ( w , w (cid:48) ) as the difference inthe reward w (cid:48) assigns to the paths P ( w ) and P ( w (cid:48) ) :
[Equation]
(2) which quantiﬁes the suboptimality when the true weights are w (cid:48) , but the path is optimized using w .
Learning.
Let w ∗
denote
Alice’s weights for the reward function.
These weights are not knownto the robot; the only information initially available is a prior distribution P ( w = w ∗ ) .
The robot learns w ∗ by iteratively presenting her with two paths P and Q for K iterations.
We extend thelearning from choice framework, where users simply indicate the path they prefer, to a setting wherethey instead provide a more ﬁnely detailed scale feedback .
Deﬁnition 1 (Scale feedback) .
Presented with two paths P and Q , Alice returns numerical feedbackψ ∈ [ − 1 , 1] .
If ψ = 0 , this means Alice has no preference between the paths, ψ = 1 equals a strongpreference for path P and ψ = − 1 a strong preference for path Q .
From an interface design and expressiveness perspective, it is undesirable to have users give a numerical value for ψ .
Instead, they can express such a feedback with a slider bar with a more ﬁne-grainedset of options.
An example is illustrated in Fig.
We let D K
=
( P 1 , Q 1 , ψ 1 ) , . . .
, ( P K , Q K , ψ K ) }be the set of recorded user feedback.
Performance Measures.
Let ˆ w be the robot’s estimate of w ∗ , and ξ ( ˆ w , w ∗ ) be a performancemeasure for the learning process.
Previous works focused on the alignment of weights [2, 4],
Alignment =
ˆ w · w ∗ / ||
ˆ w ||·|| w ∗
|| , measuring the cosine similarity of vectors ˆ w and w ∗ , i.e., how wellthe parameters of Alice’s reward function are learned.
Alternatively, Wilde et al.
[5] proposed therelative error in cost .
We adapt this as the Relative Reward =
φ ( P (ˆ w )) · w ∗ / φ ( P ( w ∗ )) · w ∗ , measuringhow much Alice likes the trajectory optimized for ˆ w compared to the one optimized for w ∗ .
Problem Statement.
Let π be an adaptive policy for designing queries ( P, Q ) , and let D K ( π | w ∗ )be the expected set of user feedback when a user w ∗ is queried by π for K iterations.
Given a robotmotion planner ρ , a user with preferences w ∗ , and a budget of K rounds to query the user abouttheir scale feedback on two presented paths, our goal is to ﬁnd an adaptive policy π that solves
[Figure]
[Equation]
Approach
[Figure]
We now brieﬂy review learning from choice, and then extend the framework to scale feedback.
4.1 Choice Feedback
When presented with two paths P and Q , a user returns an ordering P (cid:23) Q ( P is preferred) or
P (cid:22) Q ( Q is preferred).
In a noiseless setting, we have
[Equation]
That is, the path P has a higher reward than Q with respect to the hidden true user weights w ∗ .
Usingr ( P, w ) =
φ P · w , we can tighten ournotation and write ( φ P − φ Q ) · w ∗ instead of r ( P, w ∗ ) − r ( Q, w ∗ ) .
Equation (4) already contains an observation model:
If the user chooses path
P , the robot can infer that P hasa higher reward with respect to w ∗ .
This inequality deﬁnes a halfspaceΛ( P, Q )
=
{ w | ( φ P − φ Q ) · w ∗ ≥0 } containing all weights that are feasible given the observed user choice.
Over k iterations, we canintersect the sets Λ( P 1 , Q 1 ) , . . .
, Λ( P k , Q k ) to obtain the feasible set F k shown in Fig. 2(a).
Bydeﬁnition, the feasible set is convex.
4.2 Scale Feedback
[Figure]
Scale feedback allows the robot to gain more information: the robot can also infer by how much theuser prefers P , allowing for learning tighter feasible sets.
We extend the model in (4) and show howa noiseless user would provide scale feedback and then study how a robot can learn from it.
Deﬁnition 2 (Maximum Reward Gap) .
Given a user w ∗ , the maximum reward gap is
[Equation]
We notice that the maximum reward gap cannot be computed, since w ∗ is unknown to the robot.
Nevertheless, we can formulate the user choice model and then derive an observation model.
User model.
The maximum reward gap helps deﬁne when a noiseless user would indicate a strongpreference.
We assume this occurs if and only if the difference in reward of P and Q with respectto w ∗ is at least α ∗ δ ∗ for some 0 < α ∗ ≤ 1 .
Here α ∗ is a saturation parameter which governs at what reward difference (w.r.t.
to the maximum gap) the user’s feedback gets saturated to a strongpreference.
For any other ( P, Q ) where | ( φ P − φ Q ) · w ∗ |∈ [0 , α ∗ δ ∗ ) we assume the user to linearlyscale ψ between − 1 and 1 , which leads to the following model.
Deﬁnition 3 (Noiseless User Model) .
Presented with two paths P and Q a noiseless user withparameter α ∗ ∈ (0 , 1] will always provide the following feedback:
[Equation]
We illustrate the noiseless user model in Fig.
3a under different saturation parameters α ∗ .
In Fig.
3b,we show a simulated example: for a ﬁxed w ∗ we simulate how users with different values for α ∗would provide scale feedback to the same 20 queries.
For larger α ∗ , they position the slider closerto the neutral position.
Finally, we derive an observation model for the noiseless user:
[Equation] [Equation]
Figures 2(b) and 2(c) illustrate the resulting feasible sets from (7).
Moreover, we notice the userspeciﬁc and unknown parameters α ∗ and δ ∗ always appear as a product.
Thus, we can introduce anauxiliary parameter β
= α ∗ δ ∗ to write (7) as (cid:2) − ψ, φ P − φ Q (cid:3) · [ β, w ∗ ] (cid:81) 0 .
As the model remainslinear, the notion of halfspaces and the feasible set F can be extended to the augmented vector space.
4.3 Probabilistic User Feedback
In practise, users are often noisy; they might consider additional or slightly different features thanthe robot, not follow the linear reward function, or simply be uncertain in some answers.
Since wecannot expect users to always provide slider feedback following (6), we introduce a probabilisticmodel where we add uncertainty to the placement of the slider.
Another practical limitation is the fact that we cannot collect truly continuous feedback from theusers.
Instead, the slider bar has a step size (cid:15) ∈ (0 , 1] such that the user provides feedback of theform n(cid:15) for n ∈ Z and − (cid:15) − 1 ≤ n ≤ (cid:15) − 1 .
Note that (cid:15) → 0 retains the continuous scale feedback,whereas (cid:15) = 1 gives the soft choice model where the feedback is always in {− 1 , 0 , 1 } .
Deﬁnition 4 (Probabilistic User Model) .
Given a user w ∗ and a query ( P, Q ) , let ψ be the userfeedback deﬁned in the noiseless user model in (6).
A probabilistic user using a slider bar with astep size of (cid:15) then provides feedback
[Equation]
(8) where ν is a zero-mean Gaussian noise, i.e., ν ∼ N (0 , σ 2 ) with standard deviation σ , andround ( x, (cid:15) ) outputs n(cid:15) closest to x such that n ∈ Z ∩ [ − (cid:15) − 1 , (cid:15) − 1 ] .
Probabilistic Observation Model.
Given the probabilistic user model, we now show how arobot can infer about w ∗ from scale feedback.
In the noiseless case, user feedback deﬁnes afeasible set.
For the probabilistic case, we instead derive a distribution over w and α .
Letδ ( w )=max P,Q ∈P ( φ P − φ Q ) · w , similar to (5).
Then for 0 <α ≤ 1 , the belief is deﬁned
[Equation]
where
[Equation]
Given noisy user feedback µ as in (8), we can deﬁne a probabilistic density function f ( ψ | µ ) .
Together with (9) we derive a compound probability distribution
[Equation]
where we can write f ( ψ | µ ) for ψ ∈ [ − 1 , 1] asf
[Equation]
and f ( ψ | µ )=0 for ψ (cid:54)∈ [ − 1 , 1] .
Here, Φ denotes the cdf of a standard normal distribution.
Finally,given a sequence D K = { ( P k , Q k , µ k ) } Kk =1 and some prior f ( w , α ) , the joint posterior isf
[Equation]
w
[Equation] [Equation]
[Equation]
Here, we can factor f ( w , α ) as P ( w ) P ( α ) by assuming w and α are independent and we also have aprior for α ∗ .
We then take the expectation of the posterior f ( w , α | D K ) as our learned user model.
Algorithm Design
We now outline the learning algorithm.
Over K iterations: (i) the robot actively generates a query( P k , Q k ) given previous observations D k − 1 , (ii) the user provides feedback to the query in the formof the slider value µ k (in the noiseless case, µ k = ψ k ), and (iii) the robot updates its dataset D kusing (13).
After iteration K , the algorithm returns the expected weight ˆ w = E
[ w | D K ] .
5.1 Worst Case Error Bound
To compare scale feedback to choice feedback, we establish a worst case bound on the performancemeasures for both frameworks.
We introduce the worst-case error as the maximum negative performance measure, 1 − ξ ( w , w ∗ ) .
The constant in front ensures a positive value, which we thendiscount with the posterior belief, given observations D :
[Equation]
This describes the worst w the robot could pick, discounted by the posterior distribution f learnedfrom data D .
In the noiseless setting, this simpliﬁes to max w ∈F 1 − ξ ( w , w ∗ ) .
Proposition 1 (Upper error bound) .
Given any pair of paths ( P, Q ) , let D S denote the observationmade from scale feedback and D C be the observation from choice for the same set of queries.
Forany user weights w ∗ , it holds in the noiseless setting that Err max ( w ∗ , D S ) ≤ Err max ( w ∗ , D C ) .
The proof follows from the observation F Scale ⊆ F Choice , i.e., scale feedback removes more volumefrom the weight set.
Hence, the worst choice of an estimate ˆ w given observations is guaranteed tohave a smaller worst case error when using scale feedback.
The full proof is in the Appendix.
5.2 Active Query Generation
To learn w ∗ efﬁciently, the robot chooses the query ( P, Q ) it presents to the user.
While randomlyselected queries often lead to some learning progress, actively designing a query can drasticallyimprove learning when the number of iterations is limited.
Two recent approaches for learning fromchoice are information gain [4] and max regret [5].
Information gain seeks to reduce the robot’suncertainty over w while choosing queries that are easy to answer for the user.
Max regret, on theother hand, minimizes the maximum regret by showing mutual worst case paths, which also resultsin easy queries.
We leverage both of these methods for our active query generation in scale feedback.
We start with the information gain.
Let H denote Shannon’s information entropy [35].
As theoutcome of the query is yet unknown, a greedy step takes the expectation over µ :
[Equation]
We approximate the computation of entropy by summing over a set Ω of M samples of ( w , α ) ∼ f .
Thus, following the derivation in Biyik et al.
[4], the new query ( P, Q ) solves
[Equation]
The max regret policy generates queries ( P, Q ) such that if the robot learned P but the user optimalsolution would be Q is a worst case.
With a symmetric perspective over P and Q , we have
[Equation]
where R ( · , · ) is the reward difference deﬁned in (2).
By observing feedback to such queries it greedily improves the probabilistic worst case error.
In contrast to the information gain approach, maximum regret requires P and Q to be optimal trajectories for some users ( w P , α P ) and ( w Q , α Q ) .
On the other hand maximum regret does not require a one-step look-ahead and thus no integrationover potential feedback values ψ , making it computationally lighter.
Equations (16) and (17) now give us two different policies to solving the initial problem (3).
In thesimulations, we compare how the performance of both beneﬁts from scale feedback.
Simulation Results
[Figure]
We now present our main simulation results.
Additional results can be found in the Appendix.
Experiment Setup.
We simulate the presentedframework using the Driver experiment used in [2,4–6].
We modify the setup by adding 6 new features, obtaining a more challenging 10 -dimensionalproblem (details on the features, as well as resultsfor the original driver can be found in the Appendix).71 distinct user preferences w ∗ are drawn uniformlyat random, and each user is simulated with α ∗ ∈{ . 25 , .
75 , 1 } , making it 284 runs for each method.
We set σ =0 .
1 for the noise level.
We generate a setof 200 distinct sample trajectories by drawing random weights w and then computing their optimal trajectories.
The active query generation methodsthen optimize over this set.
We evaluate learning using the alignment metric and the relative reward.
As a baseline we use soft choice (strict choice showed a slightly poorer performance).
To ensurea fair comparison, we emulate soft choice by setting the step size to (cid:15)
= 1 and use the same noisemodel for both forms of feedback.
Results.
Fig.
4 shows the alignment and relative reward for the driver experiment for informationgain, max regret and random query generation.
We observe that in all cases scale feedback significantly improves the performance over soft choice in both alignment and relative reward metrics( p < .
001 in all cases with two-sample t -test).
When using the proposed scale feedback the alignment after 20 iterations improves from .
77 to .
86 for information gain, from .
67 to .
76 for maxregret, and from .
64 to .
75 for random queries.
The relative reward improves for information gainand max regret similarly from . 97 to 1 , i.e., the learned solution is optimal.
Both methods make mostprogress during the ﬁrst 10 iterations.
Random queries improve the ﬁnal relative reward from . 94to .
Overall, the simulation showcases that scale feedback improves learning, independent of thequery selection method.
For information gain and max regret, scale feedback allows for ﬁnding optimal solution, i.e., collecting 100% reward, within a small budget of iterations.
In the supplementarymaterial, we show additional simulation results for higher noise.
User Study
Finally, we analyze the scale feedback in comparison with choice feedback and under different activequerying methods with two user studies.
In all user studies, we used (cid:15) = 0 .
1 for scale queries.
Experiment Setup.
We designed a serving task with a Fetch robot [36] as shown in Fig.
1, andgenerated a dataset of 120 distinct trajectories.
Human subjects were told they should train the robotto bring the drink to the customer in the manner they prefer, paying attention to the following ﬁvefactors: the drink (out of 3 options) to be served, the orientation of the pan in front of the robot,moving the drink behind or over the pan, the maximum height of the path, and the speed.
Thesubjects were also informed about the types of queries they will respond to.
Independent Variables.
In the ﬁrst experiment, we wanted to compare scale and soft choice underrandom querying, and scale under random and information gain querying.
Hence, we varied thequery type and the querying algorithm among: (i) soft choice with random querying, (ii) scale withrandom querying, and (iii) scale with information gain querying.
In the second experiment, wewanted to compare scale and soft choice under information gain querying.
Hence, we employed: (i)soft choice with information gain querying, and (ii) scale with information gain querying.
For all,we took σ = 0 .
35 based on pilot trials with different users (see Appendix F).
Procedure.
We recruited 18 participants ( 5 female, 13 male, ages 20 – 55) for the ﬁrst, and 14participants ( 5 female, 9 male, ages 20 – 56) for the second experiment.
Due to the pandemicconditions, the subjects participated in the study remotely with an online interface as in Fig.
The study started with an instructions page with a two-question quiz to make sure the participantsunderstood how to use the interface.
After reading the instructions, we had the subjects ﬁll a formwhere they indicated their preferences for each of the ﬁve individual factors described above, toencourage them to be consistent in their responses during the data collection.
In the experiments, each participant responded to 10 queries generated with each of the algorithms.
After each of these 10 -query sets, they were shown the optimal trajectory from the dataset with respect to their learned reward function.
The participants responded to a 5 -point Likert scale survey(1-Strongly Disagree, 5-Strongly Agree) for this trajectory: “The displayed trajectory ﬁts my preferences on the task.”
We also collected scale feedback for 10 more randomly-generated queries forvalidation in each experiment.
We randomized the order of these sets (of 10 queries) to prevent anybias.
The interface provided a “Sync Videos” button to restart both videos for easier comparison.
Dependent Measures.
As an objective measure of the learning performance, we calculated the loglikelihood of the validation set (of 10 scale queries 2 ) under the posterior f ( w , α | D ) learned usingthe 10 queries generated via each algorithm, i.e., we calculated:
[Equation]
We also used the responses to the 5 -point Likert scale survey questions to measure how well thelearned rewards achieve the task.
Finally, the users took a post-experiment survey where they rated(from 1 to 5 ) the easiness and expressiveness of soft choice and scale questions.
Hypotheses.
We test the following hypotheses.
H1.
Scale feedback leads to faster learning than soft choice feedback.
H2.
Querying based on information gain accelerates learning compared to random querying.
H3.
Users will prefer information gain over random querying in terms of the optimized trajectories.
H4.
Users will prefer scale feedback over soft choice feedback in terms of the optimized trajectories.
H5.
Users will rate the scale feedback as easy as soft choice feedback.
H6.
Users will rate the scale feedback as expressive as soft choice feedback.
Results.
We present results of the ﬁrst and the second experiments in Figs.
5 and 6, respectively.
It can be seen that the log-likelihood of the validation set after learning the reward function viascale feedback is higher than learning via soft choice feedback, under both random and informationquerying.
Besides, information gain based query generation accelerates the learning and leads tohigher log-likelihood values compared to random querying.
All of these comparisons are statisticallysigniﬁcant with p < .
001 (paired-sample t -test), so they strongly support H1 and H2 .
In Fig. 5(b), it can be seen active querying led to learning reward functions that better optimize trajectories compared to random querying – this comparison was somewhat signiﬁcant with p ≈ .
05 ,supporting H3 .
In fact, when we ﬁt a Gaussian distribution to the ratings, we observe that it is 1 .
95times as likely to get a better rating with information gain querying than random querying.
Surprisingly, learning via soft choice achieved slightly higher reward than learning via scale when querieswere randomly selected, and slightly lower reward when queries were generated based on information gain.
However, these comparisons are not statistically signiﬁcant.
This is indeed analogous tothe relative reward comparisons in Fig.
4: more complex tasks might be needed to better analyze thedifference between the two methods.
Thus, we neither reject nor accept H4 .
Finally, the subjective results in Fig. 5(c) and 6(c) suggest that users ﬁnd the soft choice feedbackslightly, but consistently, easier than the scale feedback ( p < . 01 ), rejecting H5 .
This is not surprising, as it is often easier to make a pairwise comparison and the “About Equal” option in the softchoice questions makes them even easier [4].
On the other hand, there was no statistically signiﬁcantdifference in terms of expressiveness of scale and soft choice feedback, partially supporting H6 .
Insummary, it is interesting that our users perceived the soft choice as easier and even more expressiveat times; even though quantitatively, the scale feedback signiﬁcantly outperforms the soft choice.
Discussion
Summary.
We proposed scale feedback for reward learning where users provide more nuancedfeedback than choice.
We introduced a user model and showed how a robot can infer reward fromnoisy scale feedback.
We adapted state-of-the-art query generation methods to accelerate learning.
In simulations and a user study, scale feedback signiﬁcantly improved learning.
Users rank choicefeedback as slightly easier, but both forms of feedback as equally expressive.
However, the minordecrease in ease of use is out-weighted by a strong improvement in learning performance.
Future Work.
We proposed scale queries as a way to give nuanced feedback between two trajectories.
It is possible to extend them to n + 1 trajectories, with specialized user interfaces that allowusers to select a point from a n -simplex instead of a slider bar.
Future work should investigate thisand if users can still give reliable feedback to these more complex queries.
In our experiments, we used a pre-computed trajectory set.
Alternatives, e.g., optimizing queriesover action sets as in [2], or using planners as in [5], should be studied for real-time online learningsystems.
The high estimate of σ in the user studies suggests the proposed probabilistic model may beinaccurate.
Future work should reﬁne the user model, including interactively learning σ ; or ﬁt a newuser model that does not necessarily adopt a Gaussian noise.
Surprisingly, users did not perceivescale feedback as more expressive.
This could be addressed with improving interface design as wellas designing a query generation method that actively exploits the slider’s expressiveness.