Assisted Robust Reward Design
Anonymous Author(s)
AﬃliationAddress email
Abstract: Real-world robotic tasks require complex reward functions.
When wedeﬁne the problem the robot needs to solve, we pretend that a designer speciﬁesthis complex reward exactly, and it is set in stone from then on.
In practice,however, reward design is an iterative process: the designer chooses a reward,eventually encounters an “edge-case” environment where the reward incentivizesthe wrong behavior, revises the reward, and repeats.
What would it mean torethink robotics problems to formally account for this iterative nature of rewarddesign?
We propose that the robot not take the speciﬁed reward for granted, butrather have uncertainty about it, and account for the future design iterations asfuture evidence .
We contribute an Assisted Reward Design method that speedsup the design process by anticipating and inﬂuencing this future evidence: ratherthan letting the designer eventually encounter failure cases and revise the rewardthen, the method actively exposes the designer to such environments during thedevelopment phase.
We test this method in a simpliﬁed autonomous driving taskand ﬁnd that it more quickly improves the car’s behavior in held-out environmentsby proposing environments that are “edge cases” for the current reward.
Introduction
One popular way to design robotic agents is to program them to maximize the cumulative reward intheir environments.
While we have made great progress in solving this optimization problem, weoften delegate ﬁnding a good reward function to the human designer.
In reality, reward design is aniterative and challenging process.
Human designers spend tremendous eﬀort iterating on the rewardfunction as they test the robot in more and more environments.
What is more, once deployed, rewardfunctions can still be incorrect in edge-case environments, leading to failures.
Take, for instance, autonomous cars.
They have to correctly balance safety, eﬃciency, comfort andabiding by the law.
A human designer starts by looking at some representative traﬃc scenarios andspecifying a reward function that leads to desired behaviors in each of them.
But working well inthis set of environments is not enough — the reward function has to incentivize the right behaviorin any environment the car will encounter in its lifetime 1 .
The engineer will test-drive the car inboth simulation and in the real world.
Almost inevitably, optimizing for the speciﬁed reward willlead to some undesirable behavior in some further environments.
The engineer will then revise theinitial reward and repeat the process.
If they are lucky, they will have converged to a suitable rewardfunction by deployment time.
But for many systems, this is a never ending process: the car willencounter some edge-cases in the real world that they did not foresee — an unseen road layout ortraﬃc situation — and the engineer revises the reward again on the new environment.
In this work, we explore what it would mean for the robot to explicitly account for this iterative natureof reward design.
Rather than treating the current reward as set in stone, we envision a system inwhich the robot works together with the designer and helps them design the right reward function,as in Fig.
1: the robot takes the reward function speciﬁed by the designer, and proposes a newenvironment for the designer to investigate where the reward function might fail.
To capture thisformally, we ﬁrst enable the robot to explicitly account for the uncertainty in the reward function
[Figure]
itself: every reward revision does not deﬁne the reward function fully, but is instead an evidenceabout it.
Previous work on Inverse Reward Design [1] incorporates this uncertainty into the rewardfunction with the goal of making the robot plan more conservatively in new environments.
In thiswork, rather than passively updating the robot’s estimate of the reward, we argue that it is the robot’sjob to actively help the engineer design more robust reward functions, with the goal of performingwell at deployment time.
We refer to this as Assisted Reward Design .
The key insight is that:
By accounting for the iterative nature of reward design, the robot does not just passively estimatethe reward it receives but proactively inﬂuences the future evidence it can gather from the designer.
Our contributions are three-fold:1.
Formalizing reward design as an iterative process.
We formalize the problem of iterativereward design as a Meta MDP with hidden state.
Under this formulation, the robot can and shouldaccount for the future iterations of the reward as future evidence .
The robot can then make thesepieces of evidence more informative by exposing the designer to environments where revisions aremost likely needed.
Rather than letting the designer eventually encounter edge-case environments,our method proposes candidate environments and asks for reward designs, inserting itself in thedesign process loop, as in Fig.
This has two implications.
In cases where the designer wouldconverge to a good reward before deployment time, our method speeds up the iterations by focusingon informative environments.
Perhaps more importantly, in cases where the designer might stillotherwise have the wrong reward when deploying the system, our method has the potential to exposethe edge-case environments that cause failures, prompting ﬁxes in the reward ahead of time.
Providing an approximate algorithm for our method.
It is intractable to compute the rewarduncertainty given the high-dimensional nature of the reward space and the environment space.
Wepropose an approximate algorithm based on particle ﬁltering to enable eﬃcient computation.3.
Analyzing Assisted Reward Design in a simpliﬁed autonomous car domain.
We evaluate ourmethod’s practicality and eﬀectiveness in designing reward functions for autonomous cars, usingthe open-source environment proposed by Sadigh et al.
We conduct experiments with bothsimulated reward designers and robotics experts.
We ﬁnd that our method uncovers more diﬃculttest scenarios and speeds up the reward design process.
We also showcase that it proposes interestingenvironments that tend to be edge cases where the current reward estimate fails.
Overall, this paper takes a step towards making our robotic systems understand reward speciﬁcationas an iterative process.
In the end, we would like robotic systems not just to optimize what we designfor them but help us design better and more robust reward functions.
Even though admittedly morework is needed to put these ideas in real-world systems, especially in light of the need for a simulatoror learned model of environments, we are excited to see our method assist in diﬃcult reward designtasks and produce edge-case environments without any hand-coded heuristics.
Related Work
Designing robots to work reliably in an open world remains a challenging problem.
Our workbridges three lines of work towards this goal: teaching robots the correct behaviors, enabling robotsto actively learn from humans, and ensuring safety and robustness when deploying robots.
Learning Rewards from Diverse Human Input Given the diﬃculty of specifying reward functions,researchers have explored a wide variety of diﬀerent methods to enable robots to learn rewards instead from human input, especially demonstrations [3, 4, 5, 6].
Using this idea, recent works have exploredlearning from a diverse set of modalities.
These include language instructions [7, 8], numericalfeedback [9, 10], comparisons [11, 12], advice [13], facial expressions [14], collaborative behavior[15], or corrections [16, 17].
In contrast, our work builds on [1], which uses the speciﬁed rewardfunction itself as the human input to learn from.
We generalize this idea to an iterative, active process.
Active Reward Learning
Our work is an instance of active reward learning.
Prior work has focusedon actively querying for comparisons [18, 19], demonstrations [20, 21, 22, 23], or reward labels [24].
Such queries are typically for trajectories or states in a given environment (MDP).
Recent work hasproposed active learning to synthesizes the environment itself to gain more information about thereward (either the initial conﬁguration and behavior of other agents in the scene, such as [12], orthe entire environment [24]).
We also propose to actively query new environments, but the humanfeedback is a proxy reward designed to work only in that environment.
This is interesting for tworeasons.
First, we show that it leads to the robot proposing edge cases to the designer where theprevious reward is likely to fail, thus aiding in revising the reward.
Second, we argue that the sameactive reward learning ideas from the literature can be applied to the reward engineering process —when we design a reward function, we should not just have the robot go optimize it blindly.
Weshould instead think about it as an iterative process through the lens of active reward learning.
Reward Shaping Many prior works have also focused on recovering reward functions for suboptimalplanners, given a ground truth reward function that is diﬃcult to optimize, e.g. due to sparsity ornon-diﬀerentiability [25, 26].
Note that in many real-world problems, such as autonomous cars, theground truth reward is not obvious to write down.
Assisted Reward Design is diﬀerent from thereward shaping problem in that we do not assume access to a ground-truth reward function, but seektechniques that leverage proxy rewards that are the designer’s best guesses.
Safety in Real-World Robots Robots deployed in the open-world need to handle a large set ofpossible environment conﬁgurations, be it a busy road for an autonomous car or a messy kitchen fora home robot.
Given this challenge, there has been a rising number of work on safeguarding robotbehaviors by generating edge cases.
Koren et al.
[27] and O’Kelly et al.
[28] studied using black-boxoptimizers to perform adaptive testing and generate failure cases.
[29] trains a generative model ofenvironments from datasets.
Uesato et al.
[30] generates edge cases by learning a failure predictor.
These methods require deﬁning hard constraints that the robot should not violate.
Similarly, rewardshaping work has looked at how to incorporate such constraints into the reward [31].
In contrast, we focus on settings where such hard constraints are incomplete or diﬃcult to specify.
Our method does not assume access to such constraints.
We show that our method can learn todevelop edge-case scenarios that implicitly violate constraints without having to specify them.
Problem Statement
We ﬁrst describe the Unassisted Reward Design problem works, then we formulate the Assisted
Reward Design problem and introduce our method.
3.1 The Unassisted Reward Design Problem
Problem Setup Let an “environment” 𝑀 be a Markov Decision Process without the reward function.
At development time, we assume access to a large set of such environments, M devel .
We assume thisis a large set that the designer cannot exhaustively test.
This can be, for instance, all the environmentsin a several-million-mile autonomous driving dataset.
Or, it can be a parameteric set of possibleenvironment conﬁgurations, such as placements of objects, roads, and other agents.
Finally, it canalso be a generative model of the world.
The robot will run at deployment time in a diﬀerent set ofenvironments M deploy , which we do not have access to during development time.
We further assume access to a space of reward functions parameterized by 𝑤 ∈ 𝑊 — these can belinear weights on pre-deﬁned features or weights in a neural network mapping raw input to scalarreward.
We denote by 𝑤 ∗ weights of the ground truth reward function, which induces the desiredbehavior in M devel and M deploy .
We denote by 𝜉 𝑤,𝑀 the trajectory (or policy, but for this work, weconsider deterministic MDPs with set initial states) optimal to the cumulative reward induced by 𝑤 in 𝑀 , 𝜉 𝑤,𝑀 = argmax 𝜉 𝑅 𝑤 ( 𝜉 ; 𝑀 ) .
Our assumption is that there exists a 𝑤 ∗ such that 𝜉 𝑤 ∗ ,𝑀 is thedesired behavior for any 𝑀 ∈ M devel and any 𝑀 ∈ M deploy .
We do not have access to 𝑤 ∗ .
The Process of Unassisted Reward Design The designer takes a subset of environments M ⊆ M develand specify a reward function (via parameters ˜ 𝑤 ) that leads to good behavior in those environments.
In a one-shot design, they would select a set M , assume it is representative of M deploy , design a ˜ 𝑤 forthat M , and deploy the system with ˜ 𝑤 (never changing the reward again).
However, in reality, this isan iterative process.
They start with an M 0 , design a ˜ 𝑤 0 , and eventually encounter during their testinga new environment 𝑀 (cid:48) on which optimizing ˜ 𝑤 0 does not lead to desirable behavior.
Then, they wouldaugment their set to M 1 =
M 0 ∪ { 𝑀 (cid:48) } , and re-design the reward: M {+
+}
→ ˜ 𝑤 0 → M 1 → ˜ 𝑤 1 → ... .
This might continue into deployment if the reward at the time of deployment still fails on M deploy .
Implicitly, at every step along the way, the robot treats the current ˜ 𝑤 𝑖 as equivalent to 𝑤 ∗ .
3.2 The Assisted Reward Design Problem
We formulate the Assisted Reward Design Problem by introducing a meta-MDP problem, wherethe meta-agent receives proxy rewards as observations from the designer and acts by proposing newenvironments.
The goal is to minimize regret of the reward design over deployment M deploy .
Meta-States, Meta-Actions, Observations, Transitions.
In the Assisted Reward Design problem,the notion of a “state” is a set M of environments — this is what changes over time and is directlyobservable — along with the hidden state 𝑤 ∗ .
The meta-agent starts at state (M 0 ,𝑤 ∗ ) (environmentseither chosen by the designer to be representative, or simply the empty set).
It transitions to thenext state by “acting”, i.e. adding one more environment 𝑀 𝑖 to the set: M 𝑖 + 1 = M 𝑖 ∪ 𝑀 𝑖 .
Everytime it enters a state (M 𝑖 + 1 ,𝑤 ∗ ) , it receives an observation ˜ 𝑤 𝑖 + 1 about 𝑤 ∗ from the designer.
For themeta-agent to assist the designer, it needs to treat the observed reward ˜ 𝑤 𝑖 not the same as 𝑤 ∗ , but asa proxy that emulates 𝑤 ∗ only on the current M 𝑖 .
It can then interpret ˜ 𝑤 as evidence about 𝑤 ∗ anduse all past evidence to obtain a belief of 𝑤 ∗ .
To assist the designer, the key step is to account for(and thus inﬂuence) the future evidence in order to get a better reward estimate.
Observation Model.
Reward designers tend to select proxy rewards that work well in trainingenvironments.
With this intuition, previous work [1] introduced an observation model that computesthe probability of observing proxy reward ˜ 𝑤 under 𝑤 ∗ and designer’s training environment set M :
[Equation]
(1) with 𝛽 controlling how close to optimal we assume the person to be.
This distribution informs themeta-agent that it is receiving proxy rewards from an approximately optimal designer who looks atthe current M .
We can then leverage this model to infer a belief over 𝑤 ∗ : 𝑏 M ( 𝑤 ∗ ) .
Objective.
The objective of Assisted Reward Design is to achieve high reward at deployment time.
What the meta-agent controls are a sequence of actions, i.e. a sequence of environments M 𝑇 it canpropose.
These lead to observations about 𝑤 ∗ .
At deployment time, the meta-agent uses its beliefto generate optimal trajectories and accumulates reward according to 𝑤 ∗ .
We want to ﬁnd the setof environments M 𝑇 such that when given to the designer, these environments induce a belief thatleads to the best performance in M deploy :
[Equation]
where 𝑏 M 𝑇 ( 𝑤 ∗ ) is the meta-agent’s belief at deployment time, based on the evidence gathered fromthe states M 0 ,.., M 𝑇 ; 𝜉 𝑤,𝑀 = argmax 𝜉
𝑅 𝑤 ( 𝜉 ; 𝑀 ) is the optimal trajectory.
As currently stated, this objective is impossible to optimize directly because the meta-agent doesnot know M deploy .
In what follows, we introduce a solution to Assisted Reward Design based onthe heuristic objective of identifying 𝑤 ∗ — if the meta-agent uses its actions to disambiguate what𝑤 ∗ should be, then it can achieve optimum performance on the objective in Eq.
2 because it canplan optimally to 𝑤 ∗ .
We ﬁrst introduce the Maximal Information objective.
Then we show how themeta-agent can track a belief and perform future belief updates in order to propose new environments.
Assisted Reward Design via Active Info-Gathering.
Maximal Information.
We invert the probability model of Eq.
1 to obtain a belief over 𝑤 ∗ :𝑃 ( 𝑤 ∗ | ˜ 𝑤, M) .
Our key insight is that we can reduce the belief uncertainty by proposing futureenvironments to inﬂuence the designer.
In other words, we would like to ﬁnd environments that candisambiguate the reward as much as possible — if we can identify 𝑤 ∗ exactly, the meta-agent attainszero regret at deployment.
Thus, we introduce an approximate strategy for the meta-agent: proposingenvironments that provide the most information gain over its belief distribution, as commonly donein active learning [32, 33] and robot active exploration [34].
For computational tractability, we do aone-step look-ahead only, although the meta-agent could beneﬁt from reasoning over multiple futureiterations.
Given an action 𝑀 (MDP), we compute its mutual information with 𝑤 :
[Equation] [Equation]
, ˜ 𝑤 0: 𝑖 + 1 ]] Here H [ 𝑤 |M 0: 𝑖 , ˜ 𝑤 0: 𝑖 ] is the entropy of posterior distribution of 𝑃 𝑖 ( 𝑤 ) , and ˜ 𝑤 𝑖 + 1 is a new observationsampled from the updated designer model 𝑃 design ( ˜ 𝑤 𝑖
+ 1 | 𝑤 ∗ , M 𝑖 + 1 ) .
After observing ˜ 𝑤 𝑖 + 1 , we cancompute the conditional entropy H [ 𝑤 |M 0: 𝑖 + 1 , ˜ 𝑤 0: 𝑖 + 1 ] using methods from Sec. 4.1.
In our implementation, we set ˜ [-F+ 
-]𝑤 𝑖
+ 1 =
𝑤 ∗ to make Eq.
3 more tractable and ﬁnd that it empirically works well.
Next we introduce two update rules to compute conditional entropy — joint and independent mode.
4.1 Updating Beliefs over 𝑤 ∗
Joint reward design.
At iteration 𝑖 + 1 of Assisted Reward Design, the meta-agent takes an action𝑀 𝑖 + 1 (proposes an environment), and receives an observation ˜ 𝑤 𝑖 + 1 – a reward that works over allenvironments M 𝑖
+ 1 = M 𝑖 ∪{ 𝑀 𝑖 + 1 } .
Assuming that ˜ 𝑤 𝑖 + 1 is independent from previous observations,we can use ˜ 𝑤 𝑖 + 1 to update the belief over 𝑤 using Eq. 1:
[Equation]
(4) with [-% 
-]𝑃 0 the meta-agent’s prior (uniform in our experiments).
We take a sequential-importanceresampling particle ﬁltering approach to approximate 𝑃 𝑖 + 1 : we sample 𝑁 𝑝 particles from 𝑃 𝑖 , computeimportance weights based on Eq.
1 for each particle, and resample using the importance weights.
Inorder to compute the normalized 𝑃 design , we approximate the normalizer in Eq.
1 via samples.
Independent reward design.
We also allow designers to provide an observation ˜ 𝑤 𝑖 based solelyon the new environment 𝑀 𝑖 .
Prior work [35] ﬁnds that this model puts less burden on the designer.
Under this assumption, our belief update simpliﬁes to relying on a single new environment:
[Equation]
4.2 Adding New Features
Reward engineering often requires adding new features to the environment.
Our method is compatiblewith the designer adding features.
See appendix for more details.
Experiments with Simulated Designers
We evaluate Assisted Reward Design in a simpliﬁed autonomous driving task.
In this section, wehypothesize a ground truth reward function and simulate designer behaviors.
Note that our methoddoes not require such unique ground truth 𝑤 ∗ – this is merely used here for evaluation.
In Sec. 6, weconduct an expert study where we do not have the ground truth.
5.1 Experiment Setup
Driving task.
As illustrated in Fig.
2, the autonomous car is to merge to the leftmost lane.
Sincethere may be other human vehicles (simpliﬁed as constant speed) and obstacles, the autonomous car
9 Iteration 0.
R e g r e t
PosteriorRegret infogain random difﬁcult infogainproxy randomproxy difﬁcultproxy infogain random difﬁcult Method
R e l a t i v e
R e g r e t ( h
i g h e
r
i s b e tt e r )
Relativeregretonnexttask        9 Iteration 0.
R e g r e t
PosteriorRegret infogain random difﬁcult Method 0.
R e l a t i v e
R e g r e t ( h
i g h e
r
i s b e tt e r )
Relativeregretonnexttask needs to decide whether to overtake or to slow down. Our reward function based on Sadigh et al. [2]is a weighted sum of six features, including vehicle distance, progress, control eﬀort, etc.
Please referto supplementary material for their details.
A good reward function should correctly balance theseterms and reﬂect our desired behaviors in all diﬀerent situations.
To obtain M devel , we parametrizeenvironments over the initial placements of obstacles and other vehicles.
Environment Diﬃculty.
Successful merging tends to be more diﬃcult when human vehicles andobstacles are closer to the autonomous vehicle.
We can use a simple metric to describe eachenvironment: diﬃculty = (cid:205)
𝑖 1 / 𝑑 human vehicle 𝑖 + (cid:205)
𝑗 1 / 𝑑 traﬃc cone 𝑗 .
We visualize the distribution of
M devel and M deploy in Fig.
2 using this metric.
Note that both M devel and M deploy have a “long-tail”distribution of events — events that are rare to encounter daily, but could be highly useful for thereward design.
An ideal acquisition function in should be able to identify these events.
Baselines.
We compare our Maximal Information acquisition function with two baselines thatrepresent common design paradigms where the designer selects (1) the most diﬃcult tasks rankedby 𝑀 ∈ M devel (2) random environments from M devel .
5.2 Experiment Results
Evaluation.
A good Assisted Reward Design method should help the reward designer quicklyrecover high-quality reward functions.
To evaluate this, we specify one set of ground truth reward𝑤 ∗ .
At every iteration, we simulate the designer using 𝑤 ∗ under both independent mode and jointmode as described in Sec. 4.1.
Please refer to the appendix for experiment details.
We plot theperformance of posterior mean and the proxy reward ˜ 𝑤 under diﬀerent methods in Fig.
To mitigatethe eﬀect of the choice of initial environments, we experiment on six random initial environmentsand ﬁnd that our active method outperforms the proxy reward ˜ 𝑤 , as well as the random and diﬃcultbaseline in both joint and independent modes (p < 0.05).
Our active method (orange) outperforms both the random baseline (grey) and the diﬃculty heuristic(blue) as shown in Fig.
This means that certain environments, despite being rated “easy” byheuristics, turn out useful for the reward design.
These environments are often edge-cases that subtlytrigger undesirable behaviors — in Fig.
1 a rather empty road with two human vehicles triggered theautonomous vehicle to crash.
Next we further investigate this.
Edge-Case Nature of Proposed Environments To study why our method performs betterthan the baselines, we measure the quality of the proposed environments using: 𝑟 ( 𝑀 next ) =
245 E ˜ 𝑤 ∼ 𝑃 ( 𝑤 = 𝑤 ∗ ) (cid:2) Regret ( ˜ 𝑤 ; 𝑤 ∗ , 𝑀 next ) (cid:3)(cid:14)
E ˜ 𝑤 ∼ 𝑃 ( 𝑤 =
𝑤 ∗ )
E 𝑀 ∼M deploy (cid:2) Regret ( ˜ 𝑤 ; 𝑤 ∗ , 𝑀 ) (cid:3) , the ratio of regreton next environment versus average regret in deployment.
Higher 𝑟 ( 𝑀 next ) means the next environment better uncovers the overall regret of the current posterior.
As shown in Fig.
3, our activemethod proposes environments of highest relative regret.
This suggests that to ﬁnd the most usefulenvironment for Assisted Reward Design, simply relying on heuristic environment ranking is notenough.
It is more eﬀective to utilize all proxy rewards and their induced uncertainty over 𝑤 ∗ .
Experiments on Real Reward Design
[Figure]
the relative value of proposed environments. We conducted a study with 14 robotics experts 4on the simpliﬁed autonomous driving tasks,where we ask them to design reward functionsto generate their preferred driving behaviors.
6.1 Experiment Setup
We give the experts 10 minutes to familiarizewith the features.
We then let them design reward functions over four consecutive iterations,under three diﬀerent methods and three diﬀerentrandom seeds.
This gives a total of 36 individual reward designs.
We use counterbalancing torandomize the sequence of diﬀerent methods.
All our experiments are conducted over Zoom.
Violation.
Since we do not have the ground truth reward function, we cannot measure posteriorregret as in Sec. 5.
For evaluation purposes only, we introduce seven violation criteria to measurethe quality of our reward design, including collision, driving oﬀ track, stopping, etc.
Please refer tothe appendix for detailed deﬁnitions.
We use these criteria as unit tests on the behavior to captureunwanted behaviours.
We then count the total number of violations of each trajectory.
Better rewardfunctions should induce lower violation counts.
Although we could explicitly incorporate these testsas hard constraints in the “reward” function [31], we keep the reward features oblivious to them.
This is for two main reasons.
First, in the real world, constraint violations are acceptable in someedge-case scenarios: slamming on the brake, despite uncomfortable, is justiﬁable at the sight of traﬃcaccidents.
Such trade-oﬀs are diﬃcult to specify and it is our goal to exploit potential edge-cases.
Second, violation tests only capture a subset of unwanted behaviors.
Tailgating, for instance, doesnot break the tests.
Yet, it is undesirable in most cases for the autonomous vehicle.
In Sec.
6.2 weprovide qualitative examples from the experiment to illustrate such undesirable behaviors.
6.2 Experiment Results
Evaluation.
We perform Assisted Reward Design where experts answer the queries stemming fromour algorithm, as well as the baselines.
We visualize the result in Fig.
5 left.
Maximal Informationoutperforms the other acquisition functions in causing fewest average violations on M deploy .
Thisshows that Assisted Reward Design can help us tune a reward function that fails fewer tests as ameasurable proxy for the good driving behaviors in the real world.
Edge-Case Nature of Proposed Environments.
To measure the quality of the proposed environments at each iteration, we ask the experts to watch the trajectories that their proxy designs generateon the proposed tasks.
We then use the Likert scale questions in Fig.
4 and ask the users to rate howmuch they agree with the statement on a scale of 1-7.
We then compute the normalized score 𝑄 1 − 𝑄 2as a measure of how good the behavior is.
A lower score indicates better behavior.
We use the inverseof the score to indicate edge-caseness in Fig.
Our maximal information method outperforms theheuristic diﬃculty metric in ﬁnding edge-case environments that contain more violations.
Qualitative Analysis of Proposed Environments We randomly sample an initial environmentand query designer.
We then use maximal information, heuristic diﬃculty, and random acquisitionfunction to propose the next environments.
We visualize the trajectory of MAP estimate 𝑤 MAP on eachof these new environments in Sec. 6.
Notice that interestingly, our method tends to ﬁnd environmentswhere the current MAP estimate fails, even though we do not explicitly optimize for ﬁnding failurecases.
It turns out that these environments help reward designers eﬀectively narrow down on the truereward.
In comparison, heuristic diﬃculty ﬁnds environments with dense interactions.
Yet, they donot induce failure in the resulting trajectories likely because such diﬃculties have been addressed bypast reward designs.
Heuristic diﬃculty tends to generate repetitive environments.
Can Violation Count Replace the Maximal Information?
Sec.
6.2 uses violation counts toevaluate trajectories.
One may ask the question: can we simply use the violation count as anacquisition function to ﬁnd edge-case environments?
In other words, environments where the designleads to high collision, oﬀ-track counts, etc. should be proposed as edge cases.
On a surface level,this seems to be a reasonable approach.
In fact, this is commonly used in industry [36].
However, weargue that setting violation counts as the acquisition function limits the scope that we can identify edgecases.
We highlight this in Fig.
6, where we visualize selected environments proposed by Maximal
Information.
They exhibit subtle failures and lead to the highest Likert-scale ratings despite obeyingall violation criteria.
Interestingly in the second, fourth, and seventh examples, the vehicle narrowlyavoids the collision.
Such behaviors found by Maximal Information are hard to capture with handdesigned violation metrics.
By acting on the edge of violations, the algorithm pushes the rewarddesigner to better specify her “decision boundaries”.
As a result, Maximal Information can proposemore diverse edge cases and gather designer preference more quickly.
Discussion
We contribute Assisted Reward Design, an active reward learning method that generates environmentsand queries for reward functions as user inputs.
We ﬁnd experimentally that our method exposesedge cases — environments where the current proxy reward leads to high regret.
Experiments withrobotic experts conﬁrmed that these environments are edge cases that lead to undesirable behaviors.
Our method is applicable to model predictive control problems such as drone, indoor robot navigation,motion planning for manipulation [35] where we have known dynamics, as well as RL problemsgiven the environment simulator or learned model.
Given that naively applying RL in the inner loopof reward inference is intractable, it remains an open question how to eﬃciently combine with RL.
There are several limitations to our approach.
Firstly, our environment proposal algorithm couldbe more eﬃcient.
One may explore synthesis and continuous optimization over the environmentparameters.
Secondly, we need of formal analysis and guarantee of generating edge cases.
Thirdly,our method assumes access to a distribution of environments.
Obtaining such realistic distributions[37, 38, 29] for simulation remain an open problem for the community. B Driving Environment Implementation
This section provides the details of the driving environment used in the paper.
We introducedeﬁnitions of environment dynamics, feature and reward functions, the environment distributions,and how we implement the environment eﬃciently for trajectory optimization.
Features
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Dynamics We represent the forward dynamics of the MDP as 𝑥 𝑡 + 1 = 𝑓 ( 𝑥 𝑡 ,𝑢 𝑡 ) , where we have thefull knowledge of 𝑥 .
To model system dynamics, we use simple point-mass vehicle model withholonomic constraint.
Let 𝑥 car =
[ 𝑥, 𝑦,𝜃,𝑣 ] (cid:62) , where 𝑥, 𝑦 are the corrdinates of the vehicle, 𝜃 is theheading
[Table]
and 𝑣 is the speed.
We let 𝑢 =
[ 𝑢 steer ,𝑢 acc ] (cid:62) be the control input, where 𝑢 steer is the steeringinput and 𝑢 acc is the acceleration.
We also use 𝛼 as the friction coeﬃcient.
The model for a singlevehicle is:
[Equation]
We model human vehicles as moving forward at constant speed and traﬃc cones as static objects.
Feature and Reward Functions.
In the following Table 1 we introduce the environment features𝜙 in the driving environment.
Each feature composes of 𝜙 raw , a subset of the current state 𝑥 𝑡 andcontrol 𝑢 𝑡 , and a non-linear transformation on 𝜙 raw .
The nonlinear transformation is designed suchthat when maximized, it induces the desired behaviour in that feature, such as moving to the targetlane.
These environment features are are diﬀerentiable and that we can characterize the desireddriving behavior via a linear weighed sum 𝑤 (cid:62) 𝜙 .
Violations.
Here we provide in Table 2 the deﬁnition of environment constraint functions used in theexperiment section to evaluate driving quality.
Each constraint is a boolean function that returns trueor false for one timestep, and we compute the ﬁnal violation count for each trajectory by summingover constraints over the time horizon.
These constraint functions are not used for optimization, butas an evaluation criterion for the case study experiment.
Lower violation counts correspond to betterdriving behavior.
[Table]
The vehicle is driving over the maximal allowed speed.
[Table]
The vehicle is driving below the minimum speed on highway (i
e. backing up).
[Table]
|
[Equation]
∞ >
[Equation]
| 𝑢 max || ∞
The vehicle is applying too much force that it’s uncomfortable (i.e. accerlating too much or jerky).
[Table]
||
[Table]
𝑥 − (cid:174)
[Table]
car i || ≤
[Table]
The vehicle crashes into the other vehicles.
[Table]
||
[Table]
[Equation]
𝑥 obj
[Equation]
≤ 𝑑
[Table]
The vehicle crashes into the obstacles.
[Table]
The vehicle drives oﬀ the left or the right fence
[Table]
vehicle drives drives onto the wrong lane
[Table]
Planning Speed.
We implement the dynamics and reward function using JAX [40] and leverage
[Table]
the JIT compilation to speed up running time.
We use shooting method and perform gradient-based
[Table]
planning using the Adam optimizer for 200 steps.
We plan at a horizon of 10 timesteps and replan every 5 timesteps.
When planning for a single environment, we can generate 2.57 trajectories persecond.
Because we can vectorize the planning problem and plan for multiple trajectories at once, wecan achieve 157.72 trajectories per second, by planning for 500 trajectories in batch.
Computationsare benchmarked on the c2-standard-4 instance on Google Cloud.
We use batch planning forcomputing the belief distribution and environment proposals, as discussed in Sec.
A as the mainbottleneck of our algorithm.
Environment Distribution.
We use a simple method to deﬁne the distribution of environments
M design , M deploy .
There are two human vehicles and two traﬃc cones positioned on the three-lanehighway.
We assume that the autonomous vehicle starts from the center of the scene ( 𝑥 = 0 , 𝑦 = 0 ) , andsample the starting position of the other vehicles and obstacles.
The other vehicles can start anywherein 𝑥 min car
≤ 𝑥 car ≤ 𝑥 max car , 𝑦 min car ≤ 𝑦 ≤ 𝑦 max car and the obstacles can be initialized anywhere in canstart anywhere in 𝑥 min obs ≤ 𝑥 obs ≤ 𝑥 max obs , 𝑦 min obs ≤ 𝑦 ≤ 𝑦 max obs .
We ﬁlter out situations where theother vehicles or obstacles are initialized to be colliding with the main vehicle.
Note that this is a very coarsely designed distribution to showcase our method.
We have severallimitations.
For instance, we do not exclude the environments where the other vehicles runs intoobstacles.
These are relatively unlikely environments, and in principle we can deﬁne more realisticdistributions by more careful environment engineering or by loading real world driving datasets.
The diﬀerence between M devel and M deploy is that in M deploy , we deﬁne a tighter feasible range of 𝑥 carcentered around the autonomous vehicle.
This results in a shifted distribution of pseudo-diﬃcultymetric with long-tail events.
After discretization, M devel has 274k environments and M test has91k environments.
Given these large number of possible environments, it is impossible for rewarddesigners to enumerate them manually.
C Experiment DetailsOptimal Planning
For the autonomous driving task, we compute trajectories of 10 timesteps.
We useﬁnite-horizon optimal planning with regard to given rewards.
We plan at a horizon of 10 timesteps,and replan every 5 timesteps.
Reward Design We use rationality factor 𝛽 = 0 .
1 ||M|| to simulate noisy designer, and 𝛽 = 1 in theinverse model.
This is because we ﬁnd empirically that the proxy reward quickly approaches theground truth reward when we have multiple proxy environments.
We thus divide the rationalityfactor 𝛽 it by the number of proxy environments to maintain its noisiness.
To compute the posteriorprobability in Eq.
4 or Eq.
5, we need to approximate the normalizing constant.
We follow theapproach in [1] by sampling 𝑤 .
In the section 4.2 of [1], they ﬁnd empirically that it helped to includethe candidate sample 𝑤 in the normalizing sum.
This requires planning with 𝑤 and computing itsfeature sum in the MCMC inner loop, which largely slows down the inference.
Instead, we includethe candidate 𝑤 , but multiplies it with proxy ˜ 𝑤 ’s feature sum in the normalizing constant.
We then compare three methods (random, diﬃculty, maximal information) in iterative fashions.
Weuse the same initial environment for the three methods and perform 9 iterations.
We aggregate resultsover ﬁve random seeds.
Environment Proposal.
During environment proposal, we use 𝑁 𝑝 = 500 particles on 𝑁 cand =
64environments.
We implement a vectorized car dynamic model using JAX [40] and Ray [41] forbatch planning.
Our environment proposal step takes 4 minutes on c2-standard-4 instance on Google
Cloud.
Evaluation.
To examine the quality of our inferred posterior of the reward function, we need toevaluate the posterior in new driving environments.
We thus sample 500 environments uniformlyfrom M deploy as the deployment set for evaluation.
We compute the regret, note that because ofthe diﬀerent placement of vehicles and objects, diﬀerent environment have diﬀerent maximum andminimum reward they can produce.
Thus it is unfair to compute the absolute regret 𝑟 max − 𝑟 𝑤 .
Wethus compute the relative regret, by ﬁrst taking the worst cases reward 𝑟 min in each environment.
Then the relative regret is regret 𝑤 = ( 𝑟 max − 𝑟 𝑤 )/
( 𝑟 max − 𝑟 min ) =
( 𝑟 𝑤 ∗ − 𝑟 𝑤 )/( 𝑟 𝑤 ∗ − 𝑟 min ) .