When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
A long-term goal of reinforcement learning is to design agents that can autonomously interact and learn in the world.
A critical challenge to such autonomyis the presence of irreversible states which require external assistance to recoverfrom, such as when a robot arm has pushed an object off of a table.
While standardagents require constant monitoring to decide when to intervene, we aim to designproactive agents that can request human intervention only when needed.
To thisend, we propose an algorithm that can efﬁciently learns to detect and avoid statesthat are irreversible, and proactively ask for help in case the agent does enter them.
On a suite of continuous control environments with unknown irreversible states,we ﬁnd that our algorithm exhibits both better sample- and intervention-efﬁciencycompared to existing methods.
Introduction
A reinforcement learning (RL) agent should be able to autonomously learn behavior by exploringin and interacting with its environment.
However, in most realistic learning environments, there areirreversible states from which the agent cannot recover on its own.
For example, a robot arm caninadvertently push an object off the table, such that an external supervisor must return it back to therobot’s workspace to continue the learning process.
Current agents demand constant monitoring todecide when the agent enters an irreversible state and therefore when to intervene.
In this work, weaim to build greater autonomy into RL agents by addressing this problem.
In particular, we envisionproactive agents that can instead detect irreversible states, proactively request interventions whenneeded, and otherwise learn autonomously.
While prior work has studied autonomy in RL with the objective of minimizing the number ofhuman-provided resets at the end of each episode, previous setups generally assume the environment is fully reversible [19, 51, 35, 34, 17].
A related desiderata, however, arises in the safe RL setting; safe
RL methods aim to learn policies that minimize visits to unsafe states, and the developed approachesare designed to avoid those particular parts of the state space [1, 10, 39, 37, 43, 41].
With this in mind,we design our setup to provide help in two ways, through an environment reset or reversibility label ofa state.
Prior safe RL algorithms assume that agent is given a safety label on demand for every state itvisits.
In contrast, an autonomous agent may not know when it has reached an irreversible state (suchas knocking an important object off a table), and an algorithm in this setting should instead learn toboth detect and avoid such states, while minimizing queries about whether a state is reversible.
We can reduce the labeling requirement with a simple observation: all states proceeding an irreversiblestate are irreversible, and all states preceding a reversible state will be reversible.
Based on thisobservation, we design a deterministic scheme based on binary search to generate reversibility labelsfor a trajectory of length T using at most O (log T ) label queries, compared to the O ( T ) queriesmade by safe RL methods.
We further reduce labeling burden by only querying labels in a large batchat the end of each extended episode, i.e. typically only after tens of thousands of steps.
By combiningthis label efﬁcient scheme with proactive requests for an intervention and batch of labels, we canenable agents to learn amidst irreversible states with a high degree of autonomy.
Concretely, we propose a framework for reversibility-aware autonomous RL, which we call proactiveagent interventions (PAINT) , that aims to minimize human monitoring and supervision requiredthroughout training.
First, we train a reversibility-aware Q -value function that penalizes visits toirreversible states.
Second, the reversibility labels are generated by our proposed label-efﬁcientbinary search routine, which makes at most logarithmic number of queries in the length of theinteraction with the environment.
Finally, the labeled states can be used to learn a classiﬁer forpredicting irreversible states, which can then be leveraged to proactively call for interventions.
Ourproposed framework PAINT can be used to adapt any value-based RL algorithm, in both episodicand non-episodic settings, to learn with minimal and proactive interventions.
We compare PAINT toprior methods for autonomous RL and safe RL on a suite of continuous control tasks, and ﬁnd that
PAINT exhibits both better sample- and intervention-efﬁciency compared to existing methods.
Onchallenging autonomous object manipulation tasks, PAINT only requires around 100 interventionswhile training for 3 million steps, which is upto 15 ⇥ fewer than those required by prior algorithms.
Related Work
Deployment of many RL algorithms in physical contexts is challenging, because they fail to avoidundesirable states in the environment and require human-provided resets between trials.
Safe RL,reversibility-aware RL, and autonomous RL, which we review next, address parts of these problems.
Safe RL.
The goal of our work is to learn to avoid irreversible states.
Algorithms for safe RL alsoneed to avoid regions of the state space, and achieve this by formulating a constrained optimizationproblem [10, 39, 37, 48] or by assigning low rewards to unsafe states [43, 41].
Another class ofalgorithms construct shielding-based policies that yield control to a backup policy if following thelearning policy leads to an unsafe state [1, 3, 42, 40, 7, 43, 6].
Critically, however, all of theseapproaches assume that safety labels for each state can be queried freely at every time-step of training,whereas our objective is to minimize labeling requirements over training.
Reversibility-aware RL.
Reversibility and reachability have been studied in the context of RL toavoid actions that lead to irreversible states [24, 23, 31, 15] or, conversely, to guide explorationtowards difﬁcult-to-reach states [33, 5].
Unlike prior work, our study of reversibility primarilyfocuses on the non-episodic setting to minimize the number of human interventions during learning.
While prior methods are self-supervised, our experiments also ﬁnd that our algorithm learns withsigniﬁcantly fewer interventions than prior methods by leveraging some binary reversibility labels.
Autonomous RL.
Multiple prior works have also studied autonomy in RL, motivated by the factthat deployments of RL algorithms on real robots often require human-provided resets betweenepisodes [13, 16, 14].
To avoid the supervision needed for episodic resets, prior work has proposed tolearn controllers to return to speciﬁc state distributions, such as the initial state [19, 11], the uniformdistribution over states [51] or demonstration states [36], adversarially learned distributions [47], or curriculum-based distributions [34].
However, most work in the reset-free setting assumes the agent’senvironment is reversible [30, 35, 34, 17], whereas we speciﬁcally tackle the setting where this isnot the case.
One notable exception is the Leave No Trace algorithm
[11], which checks whetherthe agent has successfully returned back to the initial state distribution and requests an interventionotherwise.
Our approach differs from Leave No trace by requesting a reset based on the estimatedreversibility of the state, which we ﬁnd requires signiﬁcantly fewer interventions in our evaluation.
Human-in-the-loop learning.
Learning from human feedback has enabled RL agents to acquirecomplex skills that are difﬁcult to encode in a reward function [22, 27, 45, 12, 4].
However, interactive
RL algorithms are often to difﬁcult to scale as they rely on feedback at every time-step of training.
More feedback-efﬁcient algorithms have learned reward models from human-provided preferences [2,38, 46, 32, 8, 26, 44], which removes the need for constant feedback.
Similarly, the interactiveimitation learning learning has seen more query-efﬁcient algorithms, which query expert actionsbased on the estimated risk or novelty of a visited state [49, 28, 21, 20].
While these algorithmsaugment the agent with human-provided preferences or expert actions, our approach leverages adifferent mode of feedback, that is, reversibility labels for visited states.
Reinforcement Learning in Irreversible Environments
Consider a Markov decision process M =
( S , A , P , r, ⇢ 0 ,  ) with state space S , action space A ,transition dynamics P : S ⇥
A ⇥ S 7!
[0 , 1] , bounded reward function r : S ⇥
A 7!
[ R min , R max ] , 92 initial state distribution ⇢ 0 : S 7!
[0 , 1] and discount factor  2 [0 , 1) .
In this work, we build on theformalism of autonomous RL [35], but we remove the assumption that the environment is reversible,i.e., the MDP is no longer strongly connected (see example in [25, Chapter 38] and descriptionbelow).
The environment is initialized at s 0 ⇠ ⇢ and an algorithm continually interacts with theenvironment till it requests the environment to be reset via an external intervention to state s 0 0 ⇠ ⇢ .
Speciﬁcally, an algorithm A : { s i , a i , s i +1 , r i } t  1 i =0 7!
( a t , ⇡ t ) generates a sequence ( s 0 , a 0 , s 1 , . . . )
in M , mapping the states, actions, and rewards seen till time t  1 to an action a t 2 A [ { a reset } , andthe current guess at the optimal policy ⇡ t : S ⇥
A 7!
Here, a reset is a special action the agentcan execute to reset the environment through extrinsic interventions, i.e. P ( · | s, a reset ) = ⇢ 0 ( · ) .
A MDP is strongly connected if for all pairs of states s i , s j 2 S , thereexists a policy ⇡ such that s j has a non-zero probability of being visitedwhen executing the policy ⇡ from state s i .
This assumption can easilybe violated in practice, for example, when a robot arm pushes an objectout of its reach.
At an abstract level, the agent has transitioned into acomponent of MDP that is not connected with the high reward states, andthus cannot continue making progress, as visualized in Figure 2.
The agentcan invoke an extrinsic agent (such as human) through a reset , and theextrinsic agent can reset the environment to a state from the initial statedistribution.
For example, the human supervisor can reset the object tothe initial state, which is within the reach of the robot arm.
For every states 2 S , deﬁne R ⇢ : S 7! { 0 , 1 } as the indicator whether the state s is in 113 the same component as the initial state distribution.
State s is deﬁned to be reversible if
R ⇢ ( s ) = 1 ,and irreversible if R ⇢ ( s ) = 0 .
We assume that the R ⇢ is unknown, but can be queried for a state s .
While we do not assume that the MDP is strongly connected, we assume that the states visited bythe optimal policy are in the same connected component as the initial state distribution.
Under thisassumption, we can design agents that can autonomously practice the task many times.
Otherwise, theenvironment would need to be reset after every successful trial of the task.
The objective is to learn anoptimal policy ⇡ ⇤ 2 arg max ⇡ J ( ⇡ ) =
arg max ⇡ E [
P 1 t =
t r ( s t , a t )] .
Note, J ( ⇡ ) is approximatedby computing the return when the policy is rolled out from s 0 ⇠ ⇢ 0 .
Algorithms are typicallyevaluated on the sample efﬁciency, that is minimizing D ( A ) =
P 1 t =0 J ( ⇡ ⇤ )
J ( ⇡ t ) .
However,since we care about minimizing the human supervision required and resetting the environment canentail expensive human supervision, we will primarily evaluate algorithms on intervention-efﬁciency ,deﬁned as I ( A ) =
P 1 k =
J ( ⇡ ⇤ )
J ( ⇡ k ) , where ⇡ k is the policy learned after k interventions.
Preliminaries
Episodic settings reset the environment to a state from the initial state distribution after every trial,typically after every few hundred steps of interaction with the environment.
Such frequent resettingof the environment entails an extensive amount of external interventions, typically from a human.
Prior works on autonomous RL have sought to reduce the supervision required for resetting theenvironments by learning a backward policy that resets the environment [11, 50, 34].
Meaningfullyimproving the autonomy of RL in irreversible environments requires us to curb the requirementof episodic resets ﬁrst.
While our proposed framework is compatible with any autonomous RLalgorithm, we describe MEDAL [36], which will be used in our experiments.
MEDAL learns a forward policy ⇡ f and a backward policy ⇡ b , alternately executed for a ﬁxednumber of steps in the environment.
The forward policy maximizes the conventional cumulativetask reward, that is E [
P
[Equation]
t ,
[Equation]
)] , and the backward policy minimizes the Jensen-Shannondivergence D JS
[Equation]
between the marginal state distribution of the backward policy ⇢ band the state distribution of the optimal forward policy ˆ ⇢ ⇤ , approximated by a small number of expertdemonstrations.
Thus, the backward policy keeps the agent close to the demonstration states, allowingthe forward agent to try the task from a mix of easy and hard initial states.
The proposed divergencecan be minimized via the objective min ⇡ b max C E s ⇠ ⇢ ⇤ ⇥
[Equation]
s
[Equation]
s ))
⇤ , where
C : S 7!
[0 , 1] is a classiﬁer that maximizes the log-probability of states visited in the forwarddemonstrations, and minimizes the probability of the states visited by the backward policy.
Theoptimization problem for the backward policy can be written as a RL problem:
[Equation] [Equation]
) where ⇡ b maximizes the reward function r ( s, a ) =
log (1  C ( s )) .
Correspondingly, C ( s ) istrained to discriminate between states visited by the backward policy and the demonstrations.
Proactive Agent Interventions for Autonomous Reinforcement Learning
To minimize human monitoring and supervision when an agent is learning in an environment withirreversible states, the agent needs to (a) learn to avoid irreversible states over the course of trainingand (b) learn to detect and request an intervention whenever the agent is stuck.
For the former,we ﬁrst describe a simple modiﬁcation to the reward function to explicitly penalize visitation ofirreversible states in Section 5.1.
However, such a modiﬁcation requires the knowledge of reversibilityof the visited states, which is not known apriori.
We learn a classiﬁer to estimate reversibility,proposing a label-efﬁcient algorithm to query reversibility labels of visited states in Section 5.2.
Sinceboth the dynamics and the set of irreversible states are unknown apriori, the agent will inevitablystill visit irreversible states as a part of the exploration.
To ensure that a human does not have tomonitor the agent throughout training, the agent should have a mechanism to decide and requestfor an intervention.
We discuss such a mechanism in Section 5.3.
Finally, we put together all thesecomponents in Section 5.4 for our proposed framework P roactive A gent INT erventions (PAINT), anoverview of which is given in Figure
5.1 Penalizing Visitation of Irreversible States
Our goal is to penalize visitation of irreversible states by ensuring all actions leading to irreversible states are ‘worse’ than those leading to reversible states.
To this end, we adapt thereward-penalty framework from safe RL [41] for learning in the presence of irreversible states.
Let S rev = { ( s, a ) | ( R ⇢ ( s 0 )
= 1 } denote the set of state-action pairs that lead to a reversible state,where R ⇢ indicates whether a state is reversible.
For a transition ( s, a, s 0 ) , consider a surrogatereward function ˜ r :
[Equation]
Whenever the next state s 0 is a reversible state, the agent gets the environment reward.
Otherwise if ithas entered an irreversible, it gets a constant reward R min  ✏ that is worse than any reward given out
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
by the environment.
Whenever an agent enters an irreversible state, it will continue to remain in anirreversible state and get a constant reward of R min  ✏ .
Therefore, for any ( s, a ) 62 S rev , the Q -valuecan be computed directly as:
[Equation]
This observation allows us to bypass the need to perform Bellman backups on irreversible states, andinstead directly regress to the Q -value.
More speciﬁcally, we can rewrite the loss function for the
Q -value function as, ` ( Q ) =
E ( s,a,s 0 ,r ) ⇠ D [ Q ( s, a )  B ⇡ Q ( s, a )] , where the application of Bellmanbackup operator B ⇡ Q ( s, a ) can be expanded as:
[Equation]
Here, D denotes the replay buffer, ˆ Q denotes the use of target networks commonly used in Q -learningalgorithms to stabilize training when using neural networks as function approximators [29].
Thissurrogate reward function and the modiﬁed Bellman operator can be used for any value-based
RL algorithm in both episodic and autonomous RL settings.
The hyperparameter ✏ controls howaggressively the agent is penalized for visiting irreversible states.
The following theorem shows thatthe surrogate reward function induces desirable behavior for ✏ > 0 :
Theorem 5.1.
Let ( s, a ) 2 S rev denote a state-action leading to a reversible state and let ( s _ , a _ ) 62 S rev denote a state-action pair leading to an irreversible state.
Then, for all such pairs
Q ⇡ ( s,
[Equation]
for all ✏ > 0 and for all policies ⇡ .
The proof can be found in Appendix A.1.
The result guarantees that actions leading to irreversiblestates indeed have worse Q -values than those keeping the agent amongst the reversible states,encouraging policies to visit irreversible states fewer times over the course of training.
Additionally,any ✏ > 0 should sufﬁce in theory, though practical considerations for optimization affect this choice.
5.2 Estimating Reversibility
In general, R ⇢ is not known apriori and will have to be estimated.
We deﬁne ˆ R ⇢ : S 7!
[0 , 1]as an estimator of the reversibility of a state s 2 S .
We can then deﬁne an empirical Bellman backup operator ˆ B ⇡ from equation 4 by replacing R ⇢ with the estimator
ˆ R ⇢ .
We propose to ﬁt a classiﬁer by minimizing the binary cross-entropy loss
[Equation]
⇤ , where the states s ⇠ D represent the states visited by the agent.
Minimizing the loss requires the reversibility labels R ⇢ ( s ) fors ⇠ D .
Since labeling requires supervision, it is critical to query R ⇢ efﬁciently.
Given a trajectory of states ⌧ = ( s 0 , s 1 , . . .
s T ) , a naïve approach would be to query the labels R ⇢ ( s i )for all states s i , leading to O ( T ) queries per trajectory.
However, observe that we have the followingproperties: (a) all states following an irreversible state will be irreversible and (b) all states precedinga reversible state will be reversible.
It follows from these properties that every trajectory can besplit into a reversible segment ⌧ r =
( s 0 , s 1 , . . .
s k ) and an irreversible segment ⌧ ⇠ r = ( s k +1 , . . .
s T ) ,where the irreversible segment ⌧ ⇠ r can be empty potentially.
Identifying s k
+1 , the ﬁrst irreversiblestate, generates the labels for the entire trajectory automatically.
Fortunately, we can construct ascheme based on binary search to identify s k +1 in O (log T ) queries: s k +1 occurs after the midpointof the trajectory if the midpoint is reversible, otherwise it occurs before it.
The pseudocode for thisbinary search inspired routine is given in Algorithm 1.
The total number of labels required would be O
N log | ⌧ | max  , where N is the number of trajectoriesin the replay buffer D and the | ⌧ | max denotes the maximum length of the trajectory.
This representsa reduction in label requirement of O
N | ⌧ | max  by prior safe RL methods.
Since the agent trainsto avoid irreversible states, there will be fewer and longer trajectories as training progresses.
Thus,labeling reduces over time because the labels required is linear in N and logarithmic in | ⌧ | max .
5.3 Proactive Interventions
Despite trying to avoid irreversible states via reward penalties, an agent will inevitably encountersome irreversible states due to exploratory behaviors.
It is critical that the agent proactively asks forhelp in such situations, rather than requiring a human to constantly monitor the training process.
Anagent should request an intervention when it is an irreversible state.
Since R ⇢ ( s ) is not available, theagent again needs to estimate the reversibility of the state.
It is natural to reuse the learned reversibilityestimator ˆ R ⇢ to proactively request interventions.
We propose the following rule: the agent executesa reset whenever the reversibility classiﬁer’s prediction falls below 0.5, i.e., ˆ R ⇢ ( s ) < 0 .
Putting it Together
Algorithm input: P ; // agent, params abstracted away initialize ˆ R ⇢ , D ; // rev classiﬁer, replay buffer while not done do s ⇠ ⇢ 0 ; // reset environment // continue till classiﬁer detects irreversibility while ˆ R ⇢ ( s ) > 0 .
5 do // step in the environment a ⇠ P ( s ) , s ⇠ P ( · | s, a ) ; // update replay buffer and agent update D , P ; // optionally explore environment for explore steps do a ⇠ unif ( A ) , s ⇠ P ( · | s, a ) ; update
D ; // reversibility labels via binary search update reversibility labels in D ; // train classiﬁer on all labeled data, new and old train ˆ R ⇢ ;
With the key components in place, we summarize our proposed framework.
High-level pseudocode is given in Alg. 2, and a more detailedpseudocode is deferred to Appendix A.2.
PAINT can modify any value-based RL algorithm, in both episodic and autonomous settings.
This description and Alg.
2 focus on the lattersetting, although adapting it to the episodic setting is straightforward.
The agent’s interactionwith the environment consists of a sequence oftrials that end whenever the environment is reset to a state s ⇠ ⇢ 0 .
During each trial, theagent operates autonomously, and the Bellmanupdate for the critic is modiﬁed according to theempirical Bellman backup ˆ B ⇡ .
Whenever thereversibility classiﬁer, parameterized as a neuralnetwork, predicts smaller than 0 . 5 , the agent requests an intervention.
The agent can execute a ﬁxednumber of exploration steps after requesting an intervention and before the intervention is performed.
Whenever the classiﬁer predicts an irreversible state correctly, these exploration steps can help theagent gather more information about irreversible states.
At the time of the intervention, all newstates visited since the previous intervention are labeled for reversibility via Algorithm 1.
Finally,the reversibility classiﬁer is trained on all the labeled data before the environment is reset to a states ⇠ ⇢ 0 for the next trial.
Full implementation details can be found in Appendix A.4.
The agent is provided reversibility labels only when the external reset is provided.
This simpliﬁessupervision as the human can reset the environment and provide labels at the same time.
This meansthe replay buffer D will contain states with and without reversibility labels, since states from thecurrent trial will not yet have labels.
We use Eq.
4 for states that have reversibility labels to avoiderrors from the classiﬁer affecting the critic update and use ˆ B ⇡ for those that do not have labels.
Experiments
We design several experiments to study the efﬁciency of our algorithm in terms of the requirednumber of reset interventions and number of queried reversibility labels.
Videos of our results are at:https://sites.google.com/view/proactive-interventions .
6.1 Experimental Setup
Environments.
To illustrate the wide applicability of our method, we design environments thatrepresent three distinct RL setups: episodic, forward-backward, and continuing.
• Maze (episodic).
A 2D continuous maze environment with trenches, which represent groups of
connected irreversible states.
The agent can fall into a trench, and once entered, it can roam freely
within the trench but cannot leave it without an environment reset.
• Tabletop Organization [35] (forward-backward).
The agent must grasp the mug and put it downon one of the four goal positions.
Dropping the mug outside of the red boundary is irreversible. • Peg Insertion [35] (forward-backward).
The agent must insert the peg into the goal but canpotentially drop it off the table, which is irreversible.• Half-Cheetah Vel [9] (continuing).
The agent must run at the speciﬁed target velocity, whichchanges every 500 steps, and can potentially ﬂip over onto its back, which is irreversible.
We visualize
and fully describe
each
environment in Fig.
4 and in Appendix A.3 respectively
.Comparisons.
In the episodic and continuing settings, we consider safe RL baselines that rely onreversibility labels at every time-step of training.• Safe Model-Based Policy Optimization (SMBPO)
This comparison implements the modiﬁed Bellman operator deﬁned in Eqn.
4 in Section 5.1, using the true reversibility labels.• Safety Q-functions for RL (SQRL) [37].
A safe RL method that trains a safety critic, whichestimates the future probability of entering an irreversible state for a safety-constrained policy.
In the forward-backward setting, we consider methods designed for the autonomous learning setup.
These methods do not require any reversibility labels.
Hence, our goal here is to compare thereset-efﬁciency of our method to prior work.•
Leave No Trace (LNT)
An autonomous RL method that jointly trains a forward policy andreset policy.
When the reset policy fails to return to the initial state, the agent requests a reset.• Matching Expert Distributions for Autonomous Learning (MEDAL) [36].
This method trains areset policy that returns to the distribution of demonstration states provided for the forward policy.
MEDAL does not have a built-in intervention rule.
[Figure]
In all tasks, we compare to a recently proposed reversibility-aware RL method, Reversibility-Aware
Exploration (RAE) [15], which does not require any reversibility labels.
It instead trains a selfsupervised reversibility estimator to predict whether a state transition ( s, ˜ s ) is more likely than thereverse (˜ s, s ) .
We augment RAE with an intervention rule, similar to our method, deﬁned in termsof predictions from its self-supervised classiﬁer.
In the forward-backward setting, we train both theforward and backward policies with RAE.
Finally, we also evaluate Episodic RL , which representsthe typical RL setup with frequent resets and thus provides an upper-bound on task success.
In
Appendix A.4, we provide full implementation details, and in Appendix A.5, we discuss the set ofassumptions made by each comparison.
6.2 Main Results
In Fig.
5 ( left ) and Fig.
7, we plot the task success versus the number of interventions in the 4 tasks.
For methods that use reversibility labels, we report the total number of labels queried in Table 6.
Maze.
In the Maze task, resets are provided to each agent every 500 time-steps.
While the safe RLmethods, SMBPO and SQRL, require reversibility labels at every time-step, our approach PAINTonly requires on average 3260 queries to label all 200 K states visited.
In Fig.
5 (right), we visualizepredictions from our reversibility classiﬁer at the end of training, where zero predicts ‘reversible’and one predicts ‘irreversible’.
The classiﬁer correctly identiﬁes the path that leads to the goal asreversible.
Interestingly, it classiﬁes all other regions as stuck states, including the states that arereversible.
Because these states are irrelevant to the task, however, classifying them as irreversible,and therefore to be avoided, is advantageous to our policy as it reduces its area of exploration.
More complex domains.
In the Tabletop Organization and Peg Insertion tasks, each agent is resetevery 200 K and 100 K time-steps, per the EARL benchmark [35].
However, we allow agents torequest earlier resets, and under this setting, we compare PAINT to other methods that implementintervention rules.
Compared to Leave No Trace and Reversibility-Aware Exploration, PAINTrequires signiﬁcantly fewer resets— 80 and 124 resets respectively, which corresponds to roughly oneintervention for every 25 K steps .
Importantly, the number of interventions plateaus as trainingprogresses, and the agent requires fewer and fewer resets over time (see Appendix A.6 for additionalplots of number of interventions versus time-steps).
The exception is MEDAL (green segment nearthe origin), which is not equipped with an early termination rule and so only uses 10 interventionstotal.
However, it also fails to make meaningful progress on the task with few resets.
On the continuing Half-Cheetah task, agents do not receive any resets, unless speciﬁcally requested.
Here, we compare PAINT to SMPBO with early termination, an oracle version of our method,which assumes that reversibility labels are available at every time-step and immediately requests anintervention if the agent is ﬂipped over.
PAINT converges to its ﬁnal performance after around 750resets, on par with the number of resets required by SMPBO with early termination.
On the otherhand, a standard episodic RL agent, which receives resets at every 2 K steps, and RAE, which trains aself-supervised classiﬁer, learn
6.3 Ablations and Sensitivity Analysis
Early termination .
In the episodic Maze setting, our algorithm switches to a uniform-random policyfor the remainder of the episode if the termination condition is met.
In Fig.
8 ( left ), we plot theperformance without early termination, i.e., running the agent policy for the full episode.
Takingrandom explorations, after the agent believes it has entered an irreversible state, signiﬁcantly helpsour method, as it increases the number and diversity of irreversible states the agent has seen.
Varying the number of demonstrations .
Our method leverages demonstrations in a subset ofenvironments.
While we provide these demonstrations to all comparisons as well, we want to studyhow much our method relies on them.
We plot the average task success during training versus numberof demonstrations in Fig.
8 ( right ).
While PAINT and SMBPO are robust to the amount, alternativemethods tend to achieve signiﬁcantly lower success when given fewer demonstrations.
Discussion
In this work, we sought to build greater autonomy into RL agents, particularly in irreversibleenvironments.
We proposed an algorithm, PAINT, that learns to detect and avoid irreversible states,and proactively requests an intervention when in an irreversible state.
PAINT leverages reversibilitylabels to learn to identify irreversible states more quickly, and improves upon existing methods on arange of learning setups in terms of task success, reset-efﬁciency, and label-efﬁciency.
Despite these improvements, PAINT has multiple important limitations.
In environments whereirreversible states are not encountered until further into training, the reversibility classiﬁer mayproduce false positives which would signiﬁcantly delay the next intervention.
Further, while PAINT isfar more label-efﬁcient than prior safe RL methods, it still requires around thousands of reversibilitylabels.
We expect that this limitation may be mitigated with more sophisticated querying strategies,e.g. that take into account the classiﬁer’s conﬁdence.
Finally, we hope that future work can validate theability for reversibility aware techniques to improve the autonomy of real robotic learning systems.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.1.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board ( the estimated hourly
paid to
participants and the total amount spent on participant compensation?
[N/