Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Interventions are an intuitive way for a human to convey informationabout desired behaviors to a robotic agent and improve the performance of thedeployed robot policy.
However, such interventions cost signiﬁcant human time,giving rise to the central challenge in interactive imitation learning: how can wecontrol the timing and length of interventions to facilitate learning and limit burdenon the human supervisor?
To address this challenge, we present ThriftyDAgger, anovel algorithm for actively querying a human supervisor for interventions givena desired human intervention budget.
ThriftyDAgger uses a learned switchingpolicy to solicit interventions only at states that are sufﬁciently (1) novel , wherethe robot policy has no reference behavior to imitate, or (2) risky , where therobot has low conﬁdence in task completion.
Experiments in simulation and ona physical cable routing experiment suggest that ThriftyDAgger balances taskperformance and supervisor burden more effectively than prior algorithms.
Aftertraining, ThriftyDAgger applied at execution time achieves a 100% success rateon both the simulation and physical tasks.
A user study ( N = 10 ) in which userscontrol a three-robot ﬂeet while also performing a concentration task suggeststhat ThriftyDAgger increases human and robot performance by 58% and 80%respectively compared to the next best algorithm while reducing supervisor burden.
See https://tinyurl.com/thrifty-dagger for supplementary material.
Introduction
Imitation learning (IL) [1, 2, 3] has seen success in a variety of robotic tasks ranging from autonomousdriving [4, 5, 6] to robotic manipulation [7, 8, 9, 10, 11].
In its simplest form, the human providesan ofﬂine set of task demonstrations to the robot, which the robot uses to match human behavior.
However, this ofﬂine approach can lead to low task performance due to a mismatch between thestate distribution encountered in the demonstrations and that visited by the robot [12, 13], resultingin brittle policies that cannot be effectively deployed in real-world applications [14].
Interactiveimitation learning , in which the robot periodically cedes control to a human supervisor for correctiveinterventions, has emerged as a promising technique to address these challenges [15, 16, 17, 18].
However, while interventions make it possible to learn robust policies, these interventions requiresigniﬁcant human time.
Thus, the central challenge in interactive IL algorithms is to control thetiming and length of interventions to balance task performance and the burden imposed on the humansupervisor [19, 18].
Achieving this balance is even more critical if the human supervisor must overseemultiple robots at once [20, 21, 22], for instance supervising a robot ﬂeet in a warehouse.
Since evenrelatively reliable robot policies inevitably encounter new situations that must fall back on humanexpertise, this problem is immediately relevant to modern day industrial applications.
One way to determine when to solicit interventions is to allow the human supervisor to decide when toprovide the corrective interventions.
However, these approaches—termed “human-gated” interactive
IL algorithms [15, 16, 23]—require the human supervisor to continuously monitor the robot todetermine when to intervene.
This imposes signiﬁcant burden on the supervisor and cannot effectivelyscale to settings in which a small number humans supervise a large number of robots.
To address thischallenge, there has been recent interest in approaches that enable the robot to actively query humans for
-gated” algorithms [19, 24, 25, 18].
Robot-gated methods allow therobot to reduce burden on the human supervisor by only requesting interventions when necessary,switching between robot control and human control based on some intervention criterion.
Hoqueet al.
[18] formalize the idea of supervisor burden as the expected total cost incurred by the humanin providing interventions, which consists of the expected cost due to context switching betweenautonomous and human control and the time spent actually providing interventions.
However, itis difﬁcult to design intervention criteria that limit this burden while ensuring that the robot gainssufﬁcient information to imitate the supervisor’s policy.
This paper makes several contributions.
First, we develop intervention criteria based on a synthesis oftwo estimated properties of a given state: novelty , which measures whether the state is signiﬁcantlyout of the distribution of previously encountered states, indicating that the robot policy should notbe trusted; and risk , which indicates whether the robot is unlikely to make task progress.
Whilestate novelty has been considered in prior work [25], the key insight in our intervention criteria liesin combining novelty with a new risk metric to estimate the probability of task success.
Second,we present a new robot-gated interactive IL algorithm, ThriftyDAgger (Figure 1), which employsthese measures jointly to solicit human interventions only when necessary.
Third, while prior robotgated algorithms [19, 18] require careful parameter tuning to modulate the timing and frequencyof human intervention requests, ThriftyDAgger only requires the supervisor to specify a desiredcontext switching rate and sets thresholds accordingly.
Fourth, experimental results demonstrate
ThriftyDAgger’s effectiveness for reducing supervisor burden while learning challenging tasks bothin simulation and in an image-based cable routing task on a physical robot.
Finally, the results ofa human user study applying ThriftyDAgger to control a ﬂeet of three simulated robots suggestthat ThriftyDAgger signiﬁcantly improves performance on both the robots’ task and an independenthuman task while imposing fewer context switches, fewer human intervention actions, and lowermental load and frustration than prior algorithms.
Related Work
Imitation Learning from Human Feedback: There has been signiﬁcant prior work in ofﬂineimitation learning, in which the agent leverages an ofﬂine dataset of expert demonstrations eitherto directly match the distribution of trajectories in the ofﬂine dataset [4, 26, 1, 3, 27, 28, 29], forinstance via Behavior Cloning [30, 31], or to learn a reward function that can then be optimizedvia reinforcement learning [32, 26, 33].
However, while these approaches have shown signiﬁcantsuccess in a number of domains [7, 10, 9, 31], learning from purely ofﬂine data leads to a trajectorydistribution mismatch which yields suboptimal performance both in theory and practice [12, 13].
Toaddress this problem, there have been a number of approaches that utilize online human feedbackwhile the agent acts in the environment, such as providing suggested actions [12, 34, 35, 17] orpreferences [36, 37, 38, 39, 40, 41].
However, many of these forms of human feedback may beunreliable if the robot visits states that signiﬁcantly differ from those the human supervisor wouldthemselves visit; in such situations, it is challenging for the supervisor to determine what correctbehavior should look like without directly interacting with the environment [16, 42].
Interactive Imitation Learning: A natural way to collect reliable online feedback for imitationlearning is to periodically cede control to a human supervisor, who then provides a correctiveintervention to illustrate desired behavior.
Human-gated interactive IL algorithms [15, 16, 23] suchas HG-DAgger require the human to determine when to engage in interventions.
However, thesealgorithms require a human to continuously monitor the robot to determine when to intervene, whichimposes signiﬁcant burden on the supervisor and is particularly impractical if a small number ofhumans must supervise a large number of robots.
Furthermore, it requires the human to determinewhen the robot needs help and when to cede control, which can be unintuitive and unreliable.
By contrast, robot-gated interactive IL algorithms, such as EnsembleDAgger [25], SafeDAgger [19],and LazyDAgger [18], allow the robot to actively query for human interventions.
In practice,these algorithms estimate various quantities correlated with task performance [19, 18, 43, 24] anduncertainty [25] and use them to determine when to solicit interventions.
Prior work has proposedintervention criteria which use the novelty of states visited by the robot [25] or the predicteddiscrepancy between the actions proposed by the robot policy and by the supervisor [19, 18].
However,while state novelty provides a valuable signal for soliciting interventions, we argue that this alone isinsufﬁcient, as a state’s novelty does not convey information about the level of precision with whichactions must be executed in that state.
In practice, many robotic tasks involve moving through critical“bottlenecks” [23], which, though not necessarily novel, still present challenges.
Examples includemoving an eating utensil close to a person’s mouth or placing an object on a shelf without disturbingnearby objects.
Similarly, even if predicted accurately, action discrepancy is often a ﬂawed riskmeasure, as high action discrepancy between the robot and the supervisor may be permissible whenﬁne-grained control is not necessary (e.g. a robot gripper moving in free space) but impermissiblewhen precision is critical (e.g. a robot gripper actively trying to grasp an object).
In contrast,
ThriftyDAgger presents an intervention criteria incorporating both state novelty and a novel riskmetric and automatically tunes key parameters, allowing efﬁcient use of human supervision.
Problem Statement
Given a robot, a task for the robot to accomplish, and a human supervisor with a speciﬁed contextswitching budget, the goal is to train the robot to imitate supervisor performance within the budget.
Wemodel the robot environment as a discrete-time Markov Decision Process (MDP) M with continuousstates s 2 S , continuous actions a 2 A , and time horizon T [44].
We consider the interactive 111 imitation learning (IL) setting [15], where the robot does not have access to a shaped reward functionor to the MDP’s transition dynamics but can temporarily cede control to a supervisor who uses policy⇡ h : S !
A .
We speciﬁcally focus on tasks where there is a goal set G which determines success, but 114 that can be challenging and long-horizon, making direct application of RL highly sample inefﬁcient.
We assume that the human and robot utilize the same action space (e.g. through a teleoperationinterface) and that task success can be speciﬁed by convergence to some goal set G ✓ S within the 117 time horizon (i.e., the task is successful if
G is reached within T timesteps).
We further assume accessto an indicator function 1 G : S ! { 0 , 1 } , which indicates whether a state belongs to the goal set G .
The IL objective is to minimize a surrogate loss function J ( ⇡ r ) to encourage the robot policy⇡ r : S !
A to match ⇡ h :
[Equation]
(1) where L ( ⇡ r ( s ) , ⇡ h ( s )) is an action discrepancy measure between ⇡ r ( s ) and ⇡ h ( s ) (e.g. MSE loss),and d ⇡ r t is the marginal state distribution at timestep t induced by the robot policy ⇡ r in M .
In the interactive IL setting, meanwhile, in addition to optimizing Equation (1), a key design goal isto minimize the imposed burden on the human supervisor.
To formalize this, we deﬁne a switchingpolicy ⇡ , which determines whether the system is under robot control ⇡ r (which we call autonomousmode ) or human supervisor control ⇡ h (which we call supervisor mode ).
Following prior work [18],we deﬁne C ( ⇡ ) , the expected number of context switches in an episode under policy ⇡ , as follows:
[Equation]
⇠
[Equation]
⇡
[Equation]
[ m
[Equation]
s t
[Equation]
⇡ )] , where m I ( s t ; ⇡ ) is an indicator for whether or not a contextswitch occurs from autonomous to supervisor control.
Similarly, we deﬁne I ( ⇡ ) as the expectednumber of supervisor actions in an intervention solicited by ⇡ .
We then deﬁne the total burden B ( ⇡ )imposed on the human supervisor as follows:
[Equation]
where L is the latency of a context switch between control modes (summed over both switchingdirections) in units of timesteps (one action per timestep).
The interactive IL objective is to minimizethe discrepancy from the supervisor policy while limiting supervisor burden within some  b :
[Equation]
Because optimizing policies explicitly to satisfy the supervisor burden constraint in Equation (3)is challenging, we present novel intervention criteria that enable reduction of supervisor burden bylimiting the total number of interventions to a user-speciﬁed budget.
Given sufﬁciently high latency
L , limiting the interventions C ( ⇡ ) directly corresponds to limiting supervisor burden B ( ⇡ ) .
ThriftyDAgger
ThriftyDAgger determines when to switch between autonomous and human supervisor controlmodes by leveraging estimates of both the novelty and risk of states.
Below, Sections 4.1 and4.2 discuss the estimation of state novelty and risk of task failure, respectively, while Section 4.3discusses ThriftyDAgger’s integration of these measures to determine when to switch control modes.
Section 4.4 then describes an online procedure to set thresholds for switching between control modes.
Finally, Section 4.5 describes the full control ﬂow of ThriftyDAgger.
4.1 Novelty Estimation
When the robot policy visits states that lie signiﬁcantly outside the distribution of those encountered inthe supervisor trajectories, it does not have any reference behavior to imitate.
This motivates initiatinginterventions to illustrate desired recovery behaviors in these states.
However, estimating the supportof the state distribution visited by the human supervisor is challenging in the high-dimensionalstate spaces common in robotics.
Following prior work [25], we train an ensemble of policies withbootstrapped samples of transitions from supervisor trajectories.
We then measure the novelty of agiven state s by calculating the variance of the policy outputs at state s across ensemble members.
In practice, the action a 2 A outputted by each policy is a vector; thus, we measure state noveltyby computing the variance of each component of the action vector a across the ensemble membersand then averaging over the components.
We denote this quantity by Novelty ( s ) .
Once in supervisormode, as noted in Hoque et al.
[18], we can obtain a more precise correlate of novelty by computingthe ground truth action discrepancy between actions suggested by the supervisor and the robot policy.
4.2 Risk Estimation
Interventions may be required not only in novel states outside the distribution of supervisor trajectories,but also in familiar states that are prone to result in task failure.
For example, a task might have a“bottleneck” region with low tolerance for error, which has low novelty but nevertheless requires moresupervision to learn a reliable robot policy.
To address this challenge, we propose a novel measure ofa state’s “riskiness,” capturing the likelihood that the robot cannot successfully converge to the goalset G .
We ﬁrst deﬁne a Q-function to quantify the discounted probability of successful convergenceto G from a given state and action under the robot policy:
[Equation]
where 1 G ( s t ) is equal to 1 if s t belongs to G .
We estimate Q ⇡ r G ( s t , a t ) via a function approximatorˆ Q ⇡ r  , G parameterized by  , and deﬁne a state’s riskiness in terms of this learned Q-function:
[Equation]
In practice, we train ˆ Q ⇡ r  , G on transitions ( s t , a t , s t +1 ) collected by the supervisor from both ofﬂinedata and online interventions by minimizing the following MSE loss inspired by [45]:
[Equation]
Note that since ˆ Q ⇡ r  , G is only used to solicit interventions, it must only be accurate enough todistinguish risky states from others, rather than be able to make the ﬁne-grained distinctions betweendifferent states required for accurate policy learning in reinforcement learning.
4.3 Regulating Switches in Control Modes
We now describe how ThriftyDAgger leverages the novelty estimator from Section 4.1 and the riskestimator from Section 4.2 to regulate switches between autonomous and supervisor control.
Whilein autonomous mode, the switching policy ⇡ initiates a switch to supervisor mode at timestep tif either (1) state s t is sufﬁciently unfamiliar or (2) the robot policy has a low probability of tasksuccess from s t .
Stated precisely, ⇡ initiates a switch to supervisor mode from autonomous modeat timestep t if the predicate Intervene ( s t ,  h ,  h ) evaluates to T RUE , where Intervene ( s t ,  h ,  h ) is
T RUE if (1) Novelty ( s t ) >
h or (2) Risk ⇡ r ( s t , ⇡ r ( s t ))
>  h and F ALSE otherwise.
Note that theproposed switching policy only depends on Risk ⇡ r for states which are not novel (as novel statesalready initiate switches to supervisor control regardless of risk), since the learned risk measureshould only be trusted on states in the neighborhood of those on which it has been trained.
In supervisor mode, ⇡ switches to autonomous mode if the action discrepancy between the human androbot policy and the robot’s task failure risk are both below threshold values (Section 4.4), indicatingthat the robot is in a familiar and safe region.
Stated precisely, ⇡ switches to autonomous mode fromsupervisor mode if the predicate Cede ( s t ,  r ,  r ) evaluates to T RUE , where Cede ( s t ,  r ,  r ) is T RUEif (1) ||
⇡ r ( s t )  ⇡ h ( s t )
||
22 <  r and (2) Risk ⇡ r ( s t , ⇡ r ( s t )) <
r , and F ALSE otherwise.
Here, therisk metric ensures that the robot has a high probability of autonomously completing the task, whilethe coarser 1-step action discrepancy metric veriﬁes that we are in a familiar region of the state spacewhere the ˆ Q ⇡ r  , G values can be trusted.
Motivated by prior work [18] and hysteresis control [46], weuse stricter switching criteria in supervisor mode (  r <  h ) to encourage lengthier interventions andreduce context switches experienced by the human supervisor.
4.4 Computing Risk and Novelty Thresholds from Data
One challenge of the control strategy presented in Section 4.3 lies in tuning the key parameters(  h ,  r ,  h ,  r ) governing when context switching occurs.
As noted in prior work [25], performanceand supervisor burden can be sensitive to these thresholds.
To address this difﬁculty, we assume thatthe user speciﬁes their availability in the form of a desired intervention budget ↵ h 2 [0 , 1] , indicatingthe desired proportion of timesteps in which interventions will be requested.
This desired contextswitching rate can be interpreted in the context of supervisor burden as deﬁned in Equation (2): if thelatency of a context switch dominates the time cost of the intervention itself, limiting the expectednumber of context switches to within some intervention budget directly limits supervisor burden.
Given ↵ h , we set  h to be the (1  ↵ h ) -quantile of Risk ⇡ r ( s, ⇡ r ( s )) for all states previously visitedby ⇡ r and set  h to be the (1  ↵ h ) -quantile of Novelty ( s ) for all states previously visited by ⇡ r .
Weset  r to be the mean action discrepancy on the states visited by the supervisor after ⇡ r is trained andset  r to be the median of Risk ⇡ r ( s, ⇡ r ( s )) for all states previously visited by ⇡ r .
(Note that  r caneasily be set to different quantiles to adjust mean intervention length if desired.)
We ﬁnd that thesesettings strike a balance between informative interventions and imposed supervisor burden.
4.5 ThriftyDAgger Overview
We now summarize the ThriftyDAgger procedure, with full pseudocode available in the supplement.
ThriftyDAgger
ﬁrst initializes ⇡ r via Behavior Cloning on ofﬂine transitions ( D h from the humansupervisor, ⇡ h ).
Then, ⇡ r collects an initial ofﬂine dataset D r from the resulting ⇡ r , initializes ˆ Q ⇡ r  , Gby optimizing Equation (5) on D r
[ D h , and initializes parameters  h ,  r ,  h , and  r as in Section 4.4.
We then collect data for N episodes, each with up to T timesteps.
In each timestep of each episode, wedetermine whether robot policy ⇡ r or human supervisor ⇡ h should be in control using the procedurein Section 4.3.
Transitions in autonomous mode are aggregated into D r while transitions in supervisormode are aggregated into D h .
After each episode, ⇡ r is updated via supervised learning on D h , whileˆ Q ⇡ r  , G is updated on D r
[ D h to reﬂect the probability of task success of the resulting ⇡ r .
Experiments
In the following experiments, we study whether ThriftyDAgger can balance task performance andsupervisor burden more effectively than prior IL algorithms in three contexts: (1) training a simulatedrobot to perform a peg insertion task (Section 5.3); (2) supervising a ﬂeet of three simulated robotsto perform the peg insertion task in a human user study (Section 5.4); and (3) training a physicalsurgical robot to perform a cable routing task (Section 5.5).
In the supplementary material, we alsoinclude results from an additional simulation experiment on a challenging block stacking task.
5.1 Evaluation Metrics
We consider ThriftyDAgger’s performance during training and execution.
For the latter, we evaluateboth the (1) autonomous success rate , or success rate when deployed after training without accessto a human supervisor, and (2) intervention-aided success rate , or success rate when deployed aftertraining with a human supervisor in the loop.
These metrics are reported in the Peg Insertion study(Section 5.3) and the Physical Cable Routing study (Section 5.5).
For all experiments, during bothtraining and intervention-aided execution, we evaluate the number of interventions, human actions,and robot actions per episode.
These metrics are computed over successful episodes only to preventbiasing the metrics by the maximum episode horizon length T ; such bias occurs, for instance, whenless successful policies appear to take more actions due to hitting the time boundary more often.
Additional metrics including cumulative statistics across all episodes are reported in the supplement.
In our user study (Section 5.4), we also report the following quantities: throughput (total number oftask successes across the three robots), performance on an independent human task, the idle time ofthe robots in the ﬂeet, and users’ qualitative ratings of mental load and frustration.
By comparingthe amount of human supervision and success rates across different algorithms, we are interested inevaluating how effectively each algorithm balances supervision with policy performance.
5.2 Comparisons
We compare ThriftyDAgger to the following algorithms: Behavior Cloning, which does not use interventions; HG-DAgger [15], which is human-gated and always requires supervision; SafeDAgger
[19],which is robot-gated and performs interventions based on estimated action discrepancy between thehuman supervisor and robot policy; and LazyDAgger [18], which builds on SafeDAgger by introducing an asymmetric switching criterion to encourage lengthier interventions.
We also implementtwo ablations: one that does not use a novelty measure to regulate context switches (ThriftyDAgger(-Novelty)) and one that does not use risk to regulate context switches (ThriftyDAgger (-Risk)).
5.3 Peg Insertion in Simulation
We ﬁrst evaluate ThriftyDAgger on a long-horizon peg insertion task (Figure 2) from the Robosuitesimulation environment [47].
The goal is to grasp a ring in a random initial pose and thread it overa cylinder at a ﬁxed target location.
This task has two bottlenecks which motivate learning frominterventions: (1) correctly grasping the ring and (2) correctly placing it over the cylinder (Figure 2).
A human teleoperates the robot through a keyboard interface to provide interventions.
The statesconsist of the robot’s joint angles and ring’s pose, while the actions specify 3D translation, 3D rotation,and opening or closing the gripper.
For ThriftyDAgger and its ablations, we use target interventionfrequency ↵ h = 0 .
01 and set other parameters via the automated tuning method (Section 4.4).
Wecollect 30 ofﬂine task demos (2,687 state-action pairs) from a human supervisor to initialize the robotpolicy for all compared algorithms.
Behavior Cloning is given additional state-action pairs roughlyequivalent to the average amount of supervisor actions solicited by the interactive algorithms (Table 4in the appendix).
For ThriftyDAgger and each interactive IL baseline, we perform 10,000 environmentsteps, during which each episode takes at most 175 timesteps and system control switches betweenthe human and robot.
Hyperparameter settings for all algorithms are detailed in the supplement.
Results (Table 1) suggest that ThriftyDAgger achieves a signiﬁcantly higher autonomous success ratethan prior robot-gated algorithms, although it does request more human actions due to its conservativeexit criterion for interventions (Cede ( s t ,  r ,  r ) ).
However, the number of interventions is similar toprior robot-gated algorithms, indicating that while ThriftyDAgger requires more human actions, itimposes a similar supervisor burden to SafeDAgger and LazyDAgger in settings in which contextswitches are expensive or time-consuming (e.g. high latency L in Equation 2).
We ﬁnd that allinteractive IL algorithms substantially outperform Behavior Cloning, which does not have access tosupervisor interventions.
Notably, ThriftyDAgger achieves a higher autonomous success rate thaneven HG-DAgger, in which the supervisor is able to decide the timing and length of interventions.
Thisindicates that ThriftyDAgger’s intervention criteria enable it to autonomously solicit interventions asinformative as those chosen by a human supervisor with expert knowledge of the task.
Furthermore,
ThriftyDAgger achieves a 100% intervention-aided success rate at execution time, suggesting that
ThriftyDAgger successfully identiﬁes the required states at which to solicit interventions.
Weﬁnd that both ablations of ThriftyDAgger (Ours (-Novelty) and Ours (-Risk)) achieve signiﬁcantlylower autonomous success rates, indicating that both the novelty and risk measures are critical to
ThriftyDAgger’s performance.
We calculate ThriftyDAgger’s context switching rate to be 1.15%novelty switches and 0.79% risk switches, both approximately within the budget of ↵ h = 0 .
[Table]
5.4 User Study: Controlling A Fleet of Three Robots in Simulation
We conduct a user study with 10 participants (7 male and 3 female, aged 18-37).
Participants supervisea ﬂeet of three simulated robots, each performing the peg insertion task from Section 5.3.
We evaluatehow different interactive IL algorithms affect the participants’ (1) ability to provide effective robotinterventions, (2) performance on a distractor task performed between robot interventions, and (3)levels of mental demand and frustration.
For the distractor task, we use the game Concentration (alsoknown as Memory or Matching Pairs), in which participants identify as many pairs of matching cardsas possible among a set of face-down cards.
This is intended to emulate tasks which require continualfocus, such as cooking a meal or writing a research paper, in which frequent context switches betweenperforming the task and helping the robots is frustrating and degrades performance.
The participants teleoperate the robots using three robot-gated interactive IL algorithms: SafeDAgger,
LazyDAgger, and ThriftyDAgger.
The participant is instructed to make progress on the distractortask only when no robot requests an intervention.
When an intervention is requested, the participantis instructed to pause the distractor task, provide an intervention from the requested state until therobot (or multiple robots queued after each other) no longer requires assistance, and then return to thedistractor task.
The participants also teleoperate with HG-DAgger, where they no longer perform thedistractor task and are instructed to continually monitor all three robots simultaneously and decide onthe length and timing of interventions themselves.
Each algorithm runs for 350 timesteps, where ineach timestep, all robots in autonomous mode execute one action and the human executes one actionon the currently-supervised robot (if applicable).
The supplement illustrates the user study interfaceand fully details the experiment protocol.
All algorithms are initialized as in Section 5.3.
Results (Table 2) suggest that ThriftyDAgger achieves signiﬁcantly higher throughput than all prioralgorithms while requiring fewer interventions and fewer human actions, indicating that ThriftyDAgger requests interventions more judiciously than prior algorithms.
Furthermore, ThriftyDAgger alsoenables a lower mean idle time for robots and higher performance on the distractor task.
Notably,
ThriftyDAgger solicits fewer interventions and total actions while achieving a higher throughput than
HG-DAgger, in which the participant chooses when to intervene.
We also report metrics of users’
[Table]
autonomous success rate than HG-DAgger, in which the human decides when to intervene during training.
[Table]
mental workload and frustration using the NASA-TLX scale [48] in the supplement.
Results suggest
[Table]
that users experience lower degrees of frustration and mental load when interacting with ThriftyDAg-
[Table]
ger and LazyDAgger compared to HG-DAgger and SafeDAgger.
We hypothesize that participants
[Table]
struggle with HG-DAgger due to the difﬁcultly of monitoring multiple robots simultaneously, while
[Table]
SafeDAgger’s frequent context switches lead to user frustration during experiments.
5.5 Physical Experiment: Visuomotor Cable Routing
Finally, we evaluate ThriftyDAgger on a long-horizon cable routing task with a da Vinci surgicalrobot [49].
Here, the objective is to route a red cable into a Figure-8 pattern around 4 pegs viateleoperation with the robot’s master controllers (see supplement).
The algorithm only observeshigh-dimensional 64 ⇥ 64 ⇥ 3 RGB images of the workspace and generates continuous actionsrepresenting delta-positions in ( x, y ) .
As in Section 5.3, ThriftyDAgger uses a target interventionfrequency of ↵ h = 0 .
We collect 25 ofﬂine task demonstrations (1,381 state-action pairs) from ahuman supervisor to initialize the robot policy for ThriftyDAgger and all comparisons.
We perform1,500 environment steps, where each episode has at most 100 timesteps and system control can switchbetween the human and robot.
The supplement details the hyperparameter settings for all algorithms.
Results (Table 3) suggest that both ThriftyDAgger and HG-DAgger achieve a signiﬁcantly higher autonomous success rate than Behavior Cloning, which is never able to complete the task.
Furthermore,
ThriftyDAgger achieves a comparable autonomous success rate to HG-DAgger while requesting fewerinterventions and a similar number of total human actions.
This again suggests that ThriftyDAgger’sintervention criteria enable it to solicit interventions equally as informative or more informativethan those chosen by a human supervisor.
Finally, at execution time ThriftyDAgger achieves a100% intervention-aided success rate with minimal supervision, again indicating that ThriftyDAggersuccessfully identiﬁes the timing and length of interventions to increase policy reliability.
Discussion and Future Work
We present ThriftyDAgger, a scalable robot-gated interactive imitation learning algorithm thatleverages learned estimates of state novelty and risk of task failure to reduce burden on a humansupervisor during training and execution.
Experiments suggest that ThriftyDAgger effectively enableslong-horizon robotic manipulation tasks in simulation, on a physical robot, and for a three-robotﬂeet while limiting burden on a human supervisor.
In future work, we hope to apply ideas from
ThriftyDAgger to interactive reinforcement learning and larger scale ﬂeets of physical robots.
We alsohope to study how ThriftyDAgger’s performance varies with the target supervisor burden (speciﬁedvia ↵ h ).
In practice, ↵ h could even be time-varying: for instance, ↵ h may be signiﬁcantly lower atnight, when human operators may have limited availability.
Similarly, ↵ h may be set to a highervalue during training than at deployment, when the robot policy is typically higher quality.