Training Robots to Evaluate Robots: Example-Based Interactive Reward Functions for Policy Learning
Abstract: Physical interactions can often help reveal information that is not readily apparent.
For example, we may tug at a table leg to evaluate whether it is built well, or turn a water bottle upside down to check that it is watertight.
We propose to train robots to acquire such interactive behaviors automatically, for the purpose of evaluating the result of an attempted robotic skill execution.
These evaluations in turn serve as “interactive reward functions” (IRFs) for training reinforcement learning policies to perform the target skill, such as screwing the table leg tightly.
In addition, even after task policies are fully trained, IRFs can serve as veriﬁcation mechanisms that improve online task execution.
For any given task, our IRFs can be conveniently trained using only examples of successful outcomes, and no further speciﬁcation is needed to train the task policy thereafter.
In our evaluations on door locking and weighted block stacking in simulation, and screw tightening on a real robot, IRFs enable large performance improvements, even outperforming baselines with access to demonstrations or carefully engineered rewards.
Project website: https://sites.google.com/view/lirf-corl-2022/
Introduction
Consider a kitchen robot that must perform a large number of tasks, such as opening a refrigerator, cutting vegetables, tightening a water bottle lid or ﬂipping a pancake.
How might this robot acquire these skills?
Common skill acquisition approaches involve heavy engineering and expertise for each skill, directed either towards developing model-based control policies or towards specifying dense reward functions for reinforcement learning (RL).
These approaches do not scale well to the goal of acquiring large numbers of skills.
Consequently, there is growing interest in scalable approaches for
RL-based skill acquisition that permit easy task speciﬁcation by non-experts
A particularly promising task speciﬁcation framework that we call exemplar rewards, involves specifying tasks merely by showing the robot learner what the environment should look like after a well-executed skill [1, 2, 3, 4, 5, 6, 7].
For example, to specify the task of opening a refrigerator door, it would sufﬁce to show the learner some images of open refrigerators.
Then, once the skill is speciﬁed, the robot performs RL with the aim of altering the environment to generate observations similar to those examples.
When this framework is successful, a human user teaching the robot does not need technical expertise and does not need to even demonstrate full behaviors for task execution.
Instead, they need only capture images of the outcome.
Although this framework is appealing, many tasks cannot be speciﬁed by image observations of task success, or any other ﬁxed sensor setup.
For example, consider the task of closing a water bottle, as in Figure 1.
What image might specify a successful execution of this skill?
An image of a water bottle with a slightly loose cap would look identical to one with a tight cap, but a bottle closing policy that does not fully tighten the cap would be useless.
Thus, in this setting, an observation of the task success state from a camera constitutes only a partial observation and does not sufﬁce to specify the task.
Speciﬁcally, when non-success states might look visually similar to success states, image-based task speciﬁcation and assessment fails.
Figure 1 shows a few examples of this state aliasing problem: Is the bottle cap tight?
Is the door locked properly?
Is the tower of objects with unknown masses stable?
There are many more: Is a screw tight?
Is the leg of the table secured well?
Does the smoothie have the right consistency?
None of these questions can be answered correctly from passive image observations.
The failures of passive image-based assessment in these settings preclude robotic skill learning from image examples of task success.
To ﬁx this while retaining the versatility and user-friendliness of the exemplar rewards framework, we propose an interactive approach for evaluating task success.
As motivation, consider how a human might turn a water bottle upside down to conﬁrm that it does not leak, and is therefore closed well.
Here, the interaction reveals the key determinant of task success, namely, whether or not the cap is watertight.
We design an approach to automatically learn such interaction behaviors to evaluate task success.
Our examples are no longer mere images, but instead real actionable physical instances . In the water bottle setting, rather than image snapshots of closed water bottles, our approach receives actual physical bottles with tightly closed caps.
The robot can interact with these objects, such as by turning them upside down, to discover what about them constitutes successful execution of the desired “bottle closing” skill (i.e., the bottle does not leak) and how they behave differently from the outcomes of unsuccessful execution (i.e., it leaks).
In the other examples in Figure 1, a robot might try to pull the door open or nudge the tower to evaluate whether the task is complete.
We make two key contributions.
First, we propose an approach to train interactive reward functions , which are policies that specify what it means to correctly perform a skill and thus drive the improvement of the task policy learners in a reinforcement learning framework.
Next, we show how those same reward policies can be reused to enable in-the-loop introspective veriﬁcation behaviors during online skill execution, making robots more scrupulous and reliable.
We evaluate these contributions on three simulated and real robotic control settings, and show substantial improvements in task policy learning and execution.
Background and Problem Setup 2.1 Classiﬁer-Based Exemplar Rewards in fully observed MDPs
Reinforcement learning (RL) methods offer the promise of scalable data-driven synthesis of robot controllers for arbitrary new tasks.
Consider robotic task settings formalized as Markov decision processes (MDPs), deﬁned by the tuple ( S , A , T , R, µ ) .
At each time t , an agent selects an action a t 2 A in state s t 2 S , transitions to the next state s t +1 with probability T ( s t +1 | s t , a t ) , and receives reward r t =
R ( s t , a t , s t +1 ) .
Thus, the agent emits actions into the environment and receives two things in return: new state observations and rewards.
Ignoring discounting, a good RL task policy ⇡ T ( a t | s t ) selects actions to maximize the sum of rewards over time
P t r t , starting at a state drawn from an initial state distribution µ .
Thus, it is the reward function R that speciﬁes what task to perform in a given environment, and RL approaches in practice require expertise and effort for reward engineering [8] to specify each new task, avoiding misspeciﬁcation and guiding efﬁcient learning.
This often takes the form of tuned weighted combinations of many carefully constructed heuristic terms in the RL reward objective [9, 10, 11, 12], or even privileged task-speciﬁc sensors added to the environment during training [13, 14].
To circumvent laborious manual reward design, many methods aim to learn rewards from data.
Inverse reinforcement learning methods [15, 16, 17, 18] learn task reward functions from optimal demonstrations, but such demonstrations are typically expensive and may even be impossible to obtain.
Other methods train RL agents by learning rewards explicitly or implicitly from interactive human feedback [19, 20, 21, 22, 23], but these have the drawback of requiring in-the-loop queryable human teachers.
We build upon the popular “exemplar rewards” framework, which explores task speciﬁcations purely through examples [1, 2, 3, 4, 5, 6, 7].
In classiﬁer-based exemplar rewards approaches [1, 3], the human teacher trains a classiﬁer ˆ R ( s ) to label task outcomes as successes or failures, by gathering some success examples manually, and then generating failure examples from the task policy as it trains.
Armed with this learned reward function ˆ R ( s ) , the agent trains its task policy ⇡ T ( a | s ) to generate success outcomes and thereby maximize the estimated rewards.
An appealing property of this framework is that training a policy for each new task only requires a speciﬁcation of the ﬁnal goal state that should be achieved, without any need to specify how to achieve it.
Our work extends this exemplar rewards framework: as motivated in Section 1, rather than relying on passive observations, we will train interactive evaluation policies to evaluate task success, and these evaluations will in turn serve as “interactive reward functions” for training task policies.
This permits applying exemplar rewards to “partially observed” settings that do not permit accurate task success evaluation from images alone.
We expand on the partial observability problem below.
2.2 Partial Observability, State Aliasing, and History-Based POMDP Policies
Why are classiﬁer-based exemplar rewards approaches not suited to settings such as the door locking task in Figure 1 (middle)?
The root cause is “partial observability.”
To solve this task, the robot must push the door into the closed position and then rotate the latch into the locking position.
It only sees images from a ﬁxed camera in front of the door.
The latch, however, is occluded, and the robot can only observe and act upon the four handles on the front of the door, which rotate the latch through an axle mechanism.
The handles are visually identical, so any visual conﬁguration of the handles corresponds to any one of four conﬁgurations of the latch behind the door.
While this setting is constructed as a pedagogical example to expose and study the state aliasing problem, it is representative of several real-world settings, as discussed in Section 1 and Figure 1.
Such settings are modeled as partially observable MDPs, or POMDPs [24].
At each time, the agent only partially observes the underlying latent task state s t (i.e. the latch conﬁguration), through an observation o t ⇠ O ( o | s t ) , where the observation function O generates samples from the observation space ⌦ (i.e, images of the handle conﬁguration).
The original task state s t is not directly observable.
Thus, the task POMDP is deﬁned by ( S , A , T , R, ⌦ , O, µ ) .
Several prior works deal with the question of how the task policy in a POMDP might map to good actions despite only seeing partial observations at each time.
The key to this is the assumption, very often reasonable, that while individual observations are indeed incomplete, observation histories contain much more information about the state.
Then, a POMDP policy can perform well by relying on a history of observations and actions, either directly [25, 26, 27, 28] as ⇡ T ( a t | o t  H : t , a t  H  1: t  1 ) , or by ﬁrst estimating a belief distribution b t ( s t | o t  H : t , a t  H  1: t  1 ) over the state and then mapping to actions [24, 29, 30, 31, 32], as ⇡ T ( a t | b t ) .
In the door locking setting, the robot might observe the latch position before closing the door.
Therefore, even if the latch is occluded after the door is closed, a history-based policy could still track its position.
In other words, if trained well with the right reward function, such history-based POMDP policies can still learn good task behaviors.
Exemplar Interactive Reward Functions
This brings us to the key question: how to specify reward functions R for training POMDP policies?
In particular, could we extend the appealing exemplar rewards framework to POMDPs?
First, note that partial observability has to do with what policies can see, not where rewards come from.
Therefore, it is feasible in theory to provide state-based exemplar rewards R ( s ) that are not limited by partial observability. This implies relying on additional sensors for reward estimation during training that are not available to the policy ⇡ T that is being trained.
In other words, while the states s t are no longer observable to ⇡ T , it may still be possible to learn exemplar rewards ˆ R ( s t ) through privileged instrumentation of the environment during training to allow the reward function to observe the full state s t .
For example, privileged thermal cameras could measure ﬂuid levels to train an RGB image-based ﬂuid pouring policy [13].
However, such additional instrumentation is labor-intensive and task-speciﬁc, and sacriﬁces the appealing scalability and ease of use of the exemplar rewards framework.
How might we do this?
Learning exemplar rewards ˆ R ( o t ) based on the agent’s instantaneous observations would suffer from state aliasing: in the door locking setting, such a reward function would not be able to distinguish a correctly latched door from one in which the latch is off by a factor of 90°!
Nor described POMDP policies this exemplar trained samples of successful task outcomes full demonstrations so
3.1 Actionable Examples and Interactive Reward Functions (IRFs)
Our primary contribution is a solution to this conundrum: rather than presenting successful outcome examples as singular pre-recorded observations, we propose to present them as “actionable examples” , with which the robot can interact and generate new observations.
In other words, to learn a reward function for locking a door, the robot gets access to several physical instances of successfully locked doors.
Then, instead of training passive reward functions from image observations, we propose learning “interactive reward functions” (IRF), which consist of robot action policies ⇡ R that reveal the task rewards.
See the schematic in Figure 2.
We expand upon this idea below.
First, we gather “actionable success examples” P = {  +1 , ...,  + K } and “actionable failure examples”
N = {   1 , ...,   K } .
P and N contain object or environment conﬁgurations: in Figure 1, each  + might be a tight bottle cap, a well-locked door, a stable tower, and so on.
Now, the IRF policy may be seen as the solution to a new “IRF POMDP”.
This is identical to the task POMDP, except for two key differences. First, it is initialized to an actionable example state s
=
⇠ P [ N .
Next, its reward function R is based on discovering the label of s 0 : was it drawn from P or from N ?[-
-]
All other elements ( S , A , T , ⌦ , O ) of the task POMDP are retained.
Algorithm 1 shows pseudocode for our “Learning from IRFs” (LIRF) approach.
We point out three key features here.
(1) First, we bootstrap the task policy ⇡ T by training against a single-observationbased reward classiﬁer D ( o t ) .
This corresponds to running prior exemplar reward approaches; we use VICE [1].
(2) Next, in keeping with the exemplar rewards framework, we provide only the positive examples P .
Negative examples N are instead generated by the above-initialized task policy.
Thus, the IRF policy ⇡ R serves as a GAN-like adversarial critic [33, 18] for training the task policy ⇡ T .
Finally, how could we discover the true task completion reward, i.e., the label P (corresponding to positive reward) or the label N (negative reward) at the end of IRF execution?
For this, we could, in theory, train a classiﬁer from the full IRF trajectory history.
Instead, we ﬁnd that it sufﬁces to reuse the single-observation classiﬁer D ( o t ) from (1) above, and apply it only to the ﬁnal state after executing ⇡ R .
Intuitively, ⇡ R learns to modify the environment state  ⇡ R !
s ⇤
such that o
⇤ ⇠ O ( s ⇤ ) reveals whether  was drawn from P or N .
For example, in door locking, ⇡ R might learn to tug at the door: if the door is correctly locked ( P ), it stays closed, and if it is not locked ( N ), it opens.
Thus, a single post-IRF observation sufﬁces to classify  .
Figure 2 shows a schematic.
[Equation] [Equation] 3.2 IRFs as Veriﬁcation Mechanisms for Introspective, Fault-Tolerant Behaviors
Next, we observe that IRFs ⇡ R rely on no privileged information beyond the task policy ⇡ T ’s own observations.
Thus, while their primary purpose is to provide reward functions for training task policies, IRFs can also be deployed together with the task policies after training.
We propose a simple approach to use IRFs as veriﬁcation mechanisms in-the-loop during task execution.
Once the task policy ⇡ T is executed for a ﬁxed episode length, we will run the IRF ⇡ R to determine whether the task is complete.
If it is detected as incomplete, we will resume executing the task policy.
This perform-verify cycle continues until the IRF is satisﬁed that the task is complete (or timeout).
Appendix A.1 contains pseudocode for this procedure.
Other Related Work
LIRF trains task policies through rewards computed by an interactive reward function, which is itself a robot policy trained from examples.
While Section 2 extensively discussed connections to reward design, exemplar rewards, and POMDP policy learning to set up our approach, we now position our contributions against two other key related areas of prior work.
Adversarial Robot RL.
Like LIRF, some prior works train a target task policy against another “adversary” policy [34, 35, 36, 37].
Pinto et al.
[37] train a grasping robot to select more stable grasps by competing against an adversary robot that tugs at the grasped objects to dislodge them.
Rather than adversary policies that destabilize the task policy to increase robustness, we train adversary policies that provide outcome-based rewards to train the task policy.
Interactive Perception and Veriﬁcation.
Our method draws on ideas from interactive perception [38], where agents in partially observed settings act to obtain information about the latent state.
Interactive task veriﬁcation mechanisms (see [39] for a survey) employ human-speciﬁed interactive perception behaviors to assess the state of a task.
For example, a robot may use engineered motion primitives or perturbations to detect and correct task failures in manipulation [40, 41].
In contrast, we learn interactive veriﬁcation behaviors from examples, and use them not only during task execution, but also to deﬁne the reward function for training task skill policies.
One way to frame our contribution here is as follows: in reinforcement learning, agents emit actions into the environment and receive in return not just new states, but also task rewards .
To our knowledge, LIRF is the ﬁrst approach that employs interactive perception to directly estimate task rewards, rather than the state.
Experiments
We design various simulated and real-world experiments to evaluate LIRF for example-based task speciﬁcation in partially observable tasks that manifest the state aliasing issue.
We aim to answer the following questions: (1) Does our interactive reward learning framework enable us to learn policies to solve partially observable tasks that alternative approaches cannot, with comparable or even greater task supervision?
(2) Does the learned IRF policy sensibly evaluate task completion and provide good rewards?
(3) Can using the IRF policy in-the-loop for task veriﬁcation improve task execution performance?
Task Setups. We evaluate LIRF on 3 tasks, illustrated in Figure 3, and described below.
See
Appendix A.2 for more details and the project website for experiment videos.
• Door Locking (Sim): We instantiate the door locking task from sections 2.2 and 3 with an Adroit Hand using MuJoCo [42].
The robot has a ﬁxed viewpoint in front of the door, and must close the door and rotate the symmetric, four-handled doorknob to fully lock it (see Figure 3).
The observation space mimics perfect vision: it consists of the door orientation, the door frame position, the aliased cross-handle orientation [ 0 , 90 °) 2 and the Adroit hand pose.
The door is initialized as open, and the position of the door frame and the orientation of the latch are randomized.
• Weighted Block Stacking (Sim): We further test our algorithm on a weighted block stacking task based on the MuJoCo Fetch environment [43] as shown in Figure 3.
The goal of this task is to stack 3 visually identical blocks into a “stable” tower.
One block signiﬁcantly heavier than the other two blocks, so the optimal strategy is to place it at the bottom.
Again, the observation space mimics perfect vision: only the unordered poses of blocks are observable.
The weight of a block is only revealed after it is picked up, to mimic physically plausible “hefting” behavior.
• Screw Tightening (Real): To examine the robustness and generality of our algorithm, we test it on a real-robot screwing experiment using a D’Claw [44] that has 9 joints (Figure 3).
The objective of this task is to turn the 4-prong valve clock-wise for around 180 ° into the “tightened” state (white line on the valve base).
We engage a motor underneath the valve to mimic screw locking.
Again, the observation consists of the historical aliased valve angles, i.e. [0 , 90 ° ) and robot joint angles.
LIRF Implementation Details.
We use the soft actor-critic (SAC) algorithm [45] to train both the task policies and the IRF policies for each experiment and report results with mean and variance over 5 seeds for each simulated task.
The weight  for the sparse IRF reward in Eq 1 needs to be “large enough” (Appendix A.6) to have a substantial effect on LIRF training, but beyond this, the algorithm is not very sensitive to  ; we set  = 1000 .
All policies are trained until convergence, which takes around 2 M simulation steps for door locking and block stacking LIRF policies, 50 k steps for screwing LIRF policies ( ⇠ 10 hours on the real robot), and around 20 k steps for all IRF policies.
More details in Appendix A.5.
We will release all code and environments.
Baselines.
For evaluating our LIRF algorithm, we compare against RL policies learned with VICE exemplar rewards [1] and with ground truth state-based rewards (“GT State Reward”).
In addition, we compare against GAIfO [46], an imitation-from-observation algorithm that learns from demonstrations.
We provide as many demonstrations to GAIfO as the number of episodes with positive examples provided to LIRF (namely, 250, 10k, and 100 for door locking, weighted block stacking, and screwing respectively).
Note that LIRF actually uses fewer positive examples than the number of positive episodes, (110, 7K, and 1 actionable positive examples respectively) since the same example can be reused across episodes if it is not destroyed.
Finally, when feasible, we engineer an interactive reward function policy (Manual IRF) that performs hand-coded actions to evaluate the task: hand-coded unscrewing behavior for screwing, and a random poking action for the block [Table] stacking.
3 To isolate the effect of reward functions, all methods share the same input observation space for their policies.
See Appendix A.4 for more details on the baselines.
Note that GAIfO, GT State Reward [Table] and Manual IRF all involve more painstaking and comprehensive task speciﬁcations than LIRF, but offer useful comparison points nevertheless.
5.1 Results
Does LIRF succeed in partially observable settings?
Table 1 shows task success rates for our approach and baselines.
LIRF outperforms VICE across all tasks, showing that LIRF successfully extends the exemplar reward framework into partially observable settings where prior exemplar rewards methods like VICE fail.
LIRF also performs on par or better than baselines that use additional / more expensive forms of task supervision, such as demonstrations, human engineering, and privileged sensing of ground truth state.
Qualitatively, LIRF policies show good task-solving behaviors for each task - they learn to lock the door, construct towers with the heaviest block at the bottom, and fully tighten the screw (see Figure 4).
LIRF even demonstrates sophisticated self-correction behavior in the block stacking task by rebuilding the tower if the heavier block is incorrectly stacked on top of a lighter block.
Among baselines, VICE and GAIfO perform poorly due to state aliasing, and LIRF is competitive or superior to Manual IRF, showing that learning an IRF policy with RL may yield better rewards for training partially observable task policies.
[Table] Do the IRF policies ⇡ R sensibly evaluate task completion?
First, we evaluate the success rate of the IRF policy ⇡ R at distinguishing between positive and negative examples for each task in
Table 2.
IRF-based task success classiﬁcation is highly accurate for all tasks.
Figure 6 shows examples of learned reward function policies.
The IRF policies learn sensible strategies to separate visually indistinguishable positives and negatives.
In the door locking task, IRF pulls the ﬁxed handle; in stacking, it pokes
Finally, IRF policy learning is quite efﬁcient in terms of the number of actionable positive examples required.
Door locking, screwing, and weighted block stacking need 110, 1, and 7k actionable positive examples respectively (note that the same positive example can be reused for multiple IRF training episodes, unless its “positiveness” is destroyed at the end of that episode).
Block stacking is the hardest, since distinguishing stable from unstable towers requires a ﬁnely tuned poking force and motion, and even “stable” block tower examples are prone to toppling quite easily.
See Appendix A. for more details.
Can the IRF policy be used for in-the-loop veriﬁcation?
As seen in Table 1, LIRF+Verify, which reuses the learned
IRF policy during task execution, outperforms all baselines, even beating the ground truth reward baseline in door locking and screwing.
Figure 5 shows how LIRF+Verify improves performance with more perform-verify iterations.
LIRF+Verify makes the biggest difference in door locking: here, plain LIRF fails mainly because the policies operating on partial observations do not learn precisely how much to turn the latch without the aid of ⇡ R to declare task completion. This is also why GT State Reward performs quite poorly on this task despite training on the true state-based reward function (see also Appendix A.8). LIRF+Verify’s gains over LIRF are smaller in block stacking and screwing, because LIRF failures here occur from low-level imprecision in pick-and-place or valve-turning.
Here, LIRF+Verify helps by allowing the task policy ⇡ T to try again when it fails.
We show video examples on the project website.
Conclusions
We have presented LIRF, a framework for conveniently training example-based interactive robot policies to evaluate robot task policies, in order both to provide rewards to train them, and to verify their execution in partially observed tasks.
While our results show substantial improvements over baselines, LIRF currently has two drawbacks.
LIRF is best applied to settings where the initial task policies trained from single-image-based rewards fail most of the time. When this is not the case, such as in fully observable settings, IRF policies yield marginal or no beneﬁts over the initial policies (Appendix A.9), and therefore may not be the most frugal algorithmic choice.
Next, LIRF requires the physical storage of “actionable outcomes” as positive examples.
While the number of such examples in some tasks may be small enough to not be a major concern (e.g., among our experiments, 1 for screwing, 110 for door locking), in other cases, storing and presenting a large number of positive example objects to train the IRF might be cumbersome (e.g., 7k for block stacking).
We discuss scalability in more detail in Appendix A.10.
Future work overcoming these limitations could further expand the domain of applicability of LIRF.  Acknowledgement
This work was supported by the U.S. Ofﬁce of Naval Research under grant ONR N00014-22-1-2677, and a gift from NEC Laboratories America.