When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning
Abstract
A long-term goal of reinforcement learning is to design agents that can autonomously interact and learn in the world.
A critical challenge to such autonomy is the presence of irreversible states which require external assistance to recover from, such as when a robot arm has pushed an object off of a table.
While standard agents require constant monitoring to decide when to intervene, we aim to design proactive agents that can request human intervention only when needed.
To this end, we propose an algorithm that can efficiently learns to detect and avoid states that are irreversible, and proactively ask for help in case the agent does enter them.
On a suite of continuous control environments with unknown irreversible states, we find that our algorithm exhibits both better sample- and intervention-efficiency compared to existing methods.
Introduction
A reinforcement learning (RL) agent should be able to autonomously learn behavior by exploring in and interacting with its environment.
However, in most realistic learning environments, there are irreversible states from which the agent cannot recover on its own.
For example, a robot arm can inadvertently push an object off the table, such that an external supervisor must return it back to the robot’s workspace to continue the learning process.
Current agents demand constant monitoring to decide when the agent enters an irreversible state and therefore when to intervene.
In this work, we aim to build greater autonomy into RL agents by addressing this problem.
In particular, we envision proactive agents that can instead detect irreversible states, proactively request interventions when needed, and otherwise learn autonomously
Decision Awareness in Reinforcement Learning Workshop at the 39th International Conference on
While prior work has studied autonomy in RL with the objective of minimizing the number of human-provided resets at the end of each episode, previous setups generally assume the environment is fully reversible [19, 51, 35, 34, 17].
A related desiderata, however, arises in the safe RL setting; safe
RL methods aim to learn policies that minimize visits to unsafe states, and the developed approaches are designed to avoid those particular parts of the state space [1, 10, 39, 37, 43, 41].
Prior safe RL algorithms assume that agent is given a safety label on demand for every state it visits.
In contrast, an autonomous agent may not know when it has reached an irreversible state (such as knocking an important object off a table), and an algorithm in this setting should instead learn to both detect and avoid such states, while minimizing queries about whether a state is reversible.
We can reduce the labeling requirement with a simple observation: all states proceeding an irreversible state are irreversible, and all states preceding a reversible state will be reversible.
Based on this observation, we design a deterministic scheme based on binary search to generate reversibility labels for a trajectory of length T using at most O (log T ) label queries, compared to the O ( T ) queries made by safe RL methods.
We further reduce labeling burden by only querying labels in a large batch at the end of each extended episode, i.e. typically only after tens of thousands of steps.
By combining this label efficient scheme with proactive requests for an intervention and batch of labels, we can enable agents to learn amidst irreversible states with a high degree of autonomy.
Concretely, we propose a framework for reversibility-aware autonomous RL, which we call proactive agent interventions (PAINT) , that aims to minimize human monitoring and supervision required throughout training.
First, we train a reversibility-aware Q -value function that penalizes visits to irreversible states.
Second, the reversibility labels are generated by our proposed label-efficient binary search routine, which makes at most logarithmic number of queries in the length of the interaction with the environment.
Finally, the labeled states can be used to learn a classifier for predicting irreversible states, which can then be leveraged to proactively call for interventions.
Our proposed framework PAINT can be used to adapt any value-based RL algorithm, in both episodic and non-episodic settings, to learn with minimal and proactive interventions.
We compare PAINT to prior methods for autonomous RL and safe RL on a suite of continuous control tasks, and find that
PAINT exhibits both better sample- and intervention-efficiency compared to existing methods.
On challenging autonomous object manipulation tasks, PAINT only requires around 100 interventions while training for 3 million steps, which is upto 15 × fewer than those required by prior algorithms.
Related Work
Deployment of many RL algorithms in physical contexts is challenging, because they fail to avoid undesirable states in the environment and require human-provided resets between trials.
Safe RL, reversibility-aware RL, and autonomous RL, which we review next, address parts of these problems.
Safe RL.
The goal of our work is to learn to avoid irreversible states.
Algorithms for safe RL also need to avoid regions of the state space, and achieve this by formulating a constrained optimization problem [10, 39, 37, 48] or by assigning low rewards to unsafe states [43, 41].
Another class of algorithms construct shielding-based policies that yield control to a backup policy if following the learning policy leads to an unsafe state [1, 3, 42, 40, 7, 43, 6].
Critically, however, all of these approaches assume that safety labels for each state can be queried freely at every time-step of training, whereas our objective is to minimize labeling requirements over training.
Reversibility-aware RL.
Reversibility and reachability have been studied in the context of RL to avoid actions that lead to irreversible states [24, 23, 31, 15] or, conversely, to guide exploration towards difficult-to-reach states [33, 5].
Unlike prior work, our study of reversibility primarily focuses on the non-episodic setting to minimize the number of human interventions during learning.
While prior methods are self-supervised, our experiments also find that our algorithm learns with significantly fewer interventions than prior methods by leveraging some binary reversibility labels.
Autonomous RL.
Multiple prior works have also studied autonomy in RL, motivated by the fact that deployments of RL algorithms on real robots often require human-provided resets between episodes [13, 16, 14].
To avoid the supervision needed for episodic resets, prior work has proposed to learn controllers to return to specific state distributions, such as the initial state [19, 11], the uniform distribution over states [51] or demonstration states [36], adversarially learned distributions [47], or curriculum-based distributions [34].
However, most work in the reset-free setting assumes the agent’s environment is reversible [30, 35, 34, 17], whereas we specifically tackle the setting where this is not the case.
One notable exception is the Leave No Trace algorithm
[11], which checks whether the agent has successfully returned back to the initial state distribution and requests an intervention otherwise.
Our approach differs from Leave No trace by requesting a reset based on the estimated reversibility of the state, which we find requires significantly fewer interventions in our evaluation.
Human-in-the-loop learning.
Learning from human feedback has enabled RL agents to acquire complex skills that are difficult to encode in a reward function [22, 27, 45, 12, 4].
However, interactive
RL algorithms are often to difficult to scale as they rely on feedback at every time-step of training.
More feedback-efficient algorithms have learned reward models from human-provided preferences [2, 38, 46, 32, 8, 26, 44], which removes the need for constant feedback.
Similarly, the interactive imitation learning learning has seen more query-efficient algorithms, which query expert actions based on the estimated risk or novelty of a visited state [49, 28, 21, 20].
While these algorithms augment the agent with human-provided preferences or expert actions, our approach leverages a different mode of feedback, that is, reversibility labels for visited states.
Reinforcement Learning in Irreversible Environments
Consider a Markov decision process M =
( S , A , P , r, ρ 0 , γ ) with state space S , action space A , transition dynamics P : S ×
A × S (cid:55)→ [0 , 1] , bounded reward function r : S × A (cid:55)→ [ R min , R max ] , initial state distribution ρ 0 : S (cid:55)→ [0 , 1] and discount factor γ ∈ [0 , 1) .
In this work, we build on the formalism of autonomous RL [35], but we remove the assumption that the environment is reversible, i.e., the MDP is no longer strongly connected (see example in [25, Chapter 38] and description below).
The environment is initialized at s 0 ∼ ρ and an algorithm continually interacts with the environment till it requests the environment to be reset via an external intervention to state s ′
∼ ρ .
Specifically, an algorithm A : { s i , a i , s i +1 , r i } t − 1 i =
(cid:55)→ ( a t , π t ) generates a sequence ( s 0 , a 0 , s 1 , . . . )
in M , mapping the states, actions, and rewards seen till time t − 1 to an action a t ∈ A∪{ a reset } , and the current guess at the optimal policy π t : S × A (cid:55)→ [0 , ∞ ) .
Here, a reset is a special action the agent can execute to reset the environment through extrinsic interventions, i.e. P ( · | s, a reset ) =
ρ 0 ( · ) .
A MDP is strongly connected if for all pairs of states s i , s j ∈ S , there exists a policy π such that s j has a non-zero probability of being visited when executing the policy π from state s i .
This assumption can easily be violated in practice, for example, when a robot arm pushes an object out of its reach.
At an abstract level, the agent has transitioned into a component of MDP that is not connected with the high reward states, and thus cannot continue making progress, as visualized in Figure 2.
The agent can invoke an extrinsic agent (such as human) through a reset , and the extrinsic agent can reset the environment to a state from the initial state distribution.
For example, the human supervisor can reset the object to the initial state, which is within the reach of the robot arm.
For every state s ∈ S , define R ρ : S (cid:55)→ { 0 , 1 } as the indicator whether the state s is in the same component as the initial state distribution.
State s is defined to be reversible if R ρ ( s )
= 1 , and irreversible if R ρ ( s ) = 0 .
We assume that the R ρ is unknown, but can be queried for a state s .
While we do not assume that the MDP is strongly connected, we assume that the states visited by the optimal policy are in the same connected component as the initial state distribution.
Under this assumption, we can design agents that can autonomously practice the task many times.
Otherwise, the environment would need to be reset after every successful trial of the task.
The objective is to learn an optimal policy π ∗ ∈ arg max π J ( π ) =
arg max π E [ (cid:80) ∞ t =
γ t r ( s t , a t )] .
Note, J ( π ) is approximated by computing the return when the policy is rolled out from s 0 ∼ ρ 0 .
Algorithms are typically evaluated on the sample efficiency, that is minimizing D ( A ) =
(cid:80) ∞ t =
J ( π ∗ ) − J ( π t ) .
However, since we care about minimizing the human supervision required and resetting the environment can entail expensive human supervision, we will primarily evaluate algorithms on intervention-efficiency , defined as I ( A )
=
(cid:80) ∞ k =
J ( π ∗ ) − J ( π k ) , where π k is the policy learned after k interventions.
Preliminaries
Episodic settings reset the environment to a state from the initial state distribution after every trial, typically after every few hundred steps of interaction with the environment.
Such frequent resetting of the environment entails an extensive amount of external interventions, typically from a human.
Prior works on autonomous RL have sought to reduce the supervision required for resetting the environments by learning a backward policy that resets the environment [11, 50, 34].
Meaningfully improving the autonomy of RL in irreversible environments requires us to curb the requirement of episodic resets first.
While our proposed framework is compatible with any autonomous RL algorithm, we describe MEDAL [36], which will be used in our experiments.
MEDAL learns a forward policy π f and a backward policy π b , alternately executed for a fixed number of steps in the environment.
The forward policy maximizes the conventional cumulative task reward, that is E [ (cid:80) ∞ t =
γ t r ( s t , a t )] , and the backward policy minimizes the Jensen-Shannon divergence D JS (cid:0) ρ b ( s ) ||
ˆ ρ ∗ ( s ) (cid:1) between the marginal state distribution of the backward policy ρ b and the state distribution of the optimal forward policy ˆ ρ ∗ , approximated by a small number of expert demonstrations.
Thus, the backward policy keeps the agent close to the demonstration states, allowing the forward agent to try the task from a mix of easy and hard initial states.
The proposed divergence can be minimized via the objective min π b max C E s ∼ ρ ∗ (cid:2) log C ( s ) (cid:3) + E s ∼ ρ b (cid:2) log(1 − C ( s )) (cid:3) , where
C : S (cid:55)→ [0 , 1] is a classifier that maximizes the log-probability of states visited in the forward demonstrations, and minimizes the probability of the states visited by the backward policy.
The optimization problem for the backward policy can be written as a RL problem:
[Equation]
where π b maximizes the reward function r ( s, a ) = − log (1 − C ( s )) .
Correspondingly, C ( s ) is trained to discriminate between states visited by the backward policy and the demonstrations.
Proactive Agent Interventions for Autonomous Reinforcement Learning
To minimize human monitoring and supervision when an agent is learning in an environment with irreversible states, the agent needs to (a) learn to avoid irreversible states over the course of training and (b) learn to detect and request an intervention whenever the agent is stuck.
For the former, we first describe a simple modification to the reward function to explicitly penalize visitation of irreversible states in Section 5.1.
However, such a modification requires the knowledge of reversibility of the visited states, which is not known apriori.
We learn a classifier to estimate reversibility, proposing a label-efficient algorithm to query reversibility labels of visited states in Section 5.2.
Since both the dynamics and the set of irreversible states are unknown apriori, the agent will inevitably still visit irreversible states as a part of the exploration.
To ensure that a human does not have to monitor the agent throughout training, the agent should have a mechanism to decide and request for an intervention.
We discuss such a mechanism in Section 5.3.
Finally, we put together all these components in Section 5.4 for our proposed framework P roactive A gent INT erventions (PAINT), an overview of which is given in Figure[-
-]
5.1 Penalizing Visitation of Irreversible States
Our goal is to penalize visitation of irreversible states by ensuring all actions leading to irreversible states are ‘worse’ than those leading to reversible states.
To this end, we adapt the reward-penalty framework from safe RL [41] for learning in the presence of irreversible states.
Let S rev = { ( s, a ) | ( R ρ ( s ′ )
= 1 } denote the set of state-action pairs that lead to a reversible state, where R ρ indicates whether a state is reversible.
For a transition ( s, a, s ′ ) , consider a surrogate reward function ˜ r :
[Equation]
Whenever the next state s ′ is a reversible state, the agent gets the environment reward.
Otherwise if it has entered an irreversible, it gets a constant reward R min − ϵ that is worse than any reward given out by the environment.
Whenever an agent enters an irreversible state, it will continue to remain in an
[Figure]
irreversible state and get a constant reward of R min − ϵ .
Therefore, for any ( s, a ) ̸∈ S rev , the Q -value can be computed directly as:
[Equation]
This observation allows us to bypass the need to perform Bellman backups on irreversible states, and instead directly regress to the Q -value.
More specifically, we can rewrite the loss function for the
Q -value function as, ℓ ( Q ) =
E ( s,a,s ′ ,r ) ∼D
[ Q ( s, a ) − B π Q ( s, a )] , where the application of Bellman backup operator B π Q ( s, a ) can be expanded as:
[Equation]
Here, D denotes the replay buffer, ˆ Q denotes the use of target networks commonly used in Q -learning algorithms to stabilize training when using neural networks as function approximators [29].
This surrogate reward function and the modified Bellman operator can be used for any value-based
RL algorithm in both episodic and autonomous RL settings.
The hyperparameter ϵ controls how aggressively the agent is penalized for visiting irreversible states.
The following theorem shows that the surrogate reward function induces desirable behavior for ϵ > 0 :
Theorem 5.1.
Let ( s, a ) ∈ S rev denote a state-action leading to a reversible state and let ( s _ , a _ ) ̸∈ S rev denote a state-action pair leading to an irreversible state.
Then, for all such pairs
[Equation] for all ϵ > 0 and for all policies π .
The proof can be found in Appendix A.1.
The result guarantees that actions leading to irreversible states indeed have worse Q -values than those keeping the agent amongst the reversible states, encouraging policies to visit irreversible states fewer times over the course of training.
Additionally, any ϵ > 0 should suffice in theory, though practical considerations for optimization affect this choice.
5.2 Estimating Reversibility
In general, R ρ is not known apriori and will have to be estimated.
We define ˆ R ρ : S (cid:55)→ [0 , 1] as an estimator of the reversibility of a state s ∈
S .
We can then define an empirical Bellman backup operator ˆ B π from equation 4 by replacing R ρ with the estimator ˆ R ρ .
We propose to fit a classifier by minimizing the binary cross-entropy loss ℓ ( ˆ R ρ ) =
− E s ∼D (cid:2) R ρ ( s ) log ˆ R ρ ( s ) + (1 − R ρ ( s ))
log(1 − ˆ R ρ ( s )) (cid:3) , where the states s ∼ D represent the states visited by the agent.
Minimizing the loss requires the reversibility labels R ρ ( s ) for s ∼ D .
Since labeling requires supervision, it is critical to query R ρ efficiently.
Given a trajectory of states τ = ( s 0 , s 1 , . . .
s T ) , a naïve approach would be to query the labels R ρ
( s i ) for all states s i , leading to O ( T ) queries per trajectory.
However, observe that we have the following properties: (a) all states following an irreversible state will be irreversible and (b) all states preceding a reversible state will be reversible.
It follows from these properties that every trajectory can be split into a reversible segment τ r =
( s 0 , s 1 , . . .
s k ) and an irreversible segment τ ∼ r =
( s k +1 , . . .
s T ) , where the irreversible segment τ ∼ r can be empty potentially.
Identifying s k +1 , the first irreversible state, generates the labels for the entire trajectory automatically.
Fortunately, we can construct a scheme based on binary search to identify s k +1 in O (log T ) queries: s k +1 occurs after the midpoint of the trajectory if the midpoint is reversible, otherwise it occurs before it.
The pseudocode for this binary search inspired routine is given in Algorithm 1.
The total number of labels required would be O (cid:0) N log | τ | max (cid:1) , where N is the number of trajectories in the replay buffer D and the | τ | max denotes the maximum length of the trajectory.
This represents a reduction in label requirement of O (cid:0) N | τ | max (cid:1) by prior safe RL methods.
Since the agent trains to avoid irreversible states, there will be fewer and longer trajectories as training progresses.
Thus, labeling reduces over time because the labels required is linear in N and logarithmic in | τ | max .
5.3 Proactive Interventions
Despite trying to avoid irreversible states via reward penalties, an agent will inevitably encounter some irreversible states due to exploratory behaviors.
It is critical that the agent proactively asks for help in such situations, rather than requiring a human to constantly monitor the training process.
An agent should request an intervention when it is an irreversible state.
Since R ρ ( s ) is not available, the agent again needs to estimate the reversibility of the state.
It is natural to reuse the learned reversibility estimator ˆ R ρ to proactively request interventions.
We propose the following rule: the agent executes a reset whenever the reversibility classifier’s prediction falls below 0.5, i.e., ˆ R ρ ( s ) < 0 .
5.4 Putting it Together
input: P ; // agent, params abstracted away initialize ˆ R ρ , D ; // rev classifier, replay buffer while not done do s ∼ ρ 0 ; // reset environment // continue till classifier detects irreversibility while ˆ R ρ ( s ) > 0 .
5 do // step in the environment a ∼ P ( s ) , s ∼ P ( · | s, a ) ; // update replay buffer and agent update D , P ; // optionally explore environment for explore steps do a ∼ unif ( A ) , s ∼ P ( · | s, a ) ; update D ; // reversibility labels via binary search update reversibility labels in D ; // train classifier on all labeled data, new and old train ˆ R ρ ;
With the key components in place, we summarize our proposed framework.
High-level pseudocode is given in Alg. 2, and a more detailed pseudocode is deferred to Appendix A.2.
PAINT can modify any value-based RL algorithm, in both episodic and autonomous settings.
This description and Alg.
2 focus on the latter setting, although adapting it to the episodic setting is straightforward.
The agent’s interaction with the environment consists of a sequence of trials that end whenever the environment is reset to a state s ∼ ρ 0 .
During each trial, the agent operates autonomously, and the Bellman update for the critic is modified according to the empirical Bellman backup ˆ B π .
Whenever the reversibility classifier
ˆ R ρ < 0 .
5 , the agent requests an intervention.
The agent can execute a fixed number of exploration steps after requesting an intervention and before the intervention is performed.
Whenever the classifier predicts an irreversible state correctly, these exploration steps can help the agent gather more information about irreversible states.
At the time of the intervention, all new states visited since the previous intervention are labeled for reversibility via Algorithm 1.
Finally, the reversibility classifier is trained on all the labeled data before the environment is reset to a state s ∼ ρ 0 for the next trial.
The agent is provided reversibility labels only when the external reset is provided.
This simplifies supervision as the human can reset the environment and provide labels at the same time.
This means the replay buffer D will contain states with and without reversibility labels, since states from the current trial will not yet have labels.
We use Eq.
4 for states that have reversibility labels to avoid errors from the classifier affecting the critic update and use ˆ B π for those that do not have labels.
Experiments
We design several experiments to study the efficiency of our algorithm in terms of the required number of reset interventions and number of queried reversibility labels.
Videos of our results are at: https://sites.google.com/view/proactive-interventions .
6.1 Experimental Setup
Environments.
To illustrate the wide applicability of our method, we design environments that represent three distinct RL setups: episodic, forward-backward, and continuing.
• Maze (episodic).
A 2D continuous maze environment with trenches, which represent groups of connected irreversible states.
The agent can fall into a trench, and once entered, it can roam freely within the trench but cannot leave it without an environment reset.
• Tabletop Organization [35] (forward-backward).
The agent must grasp the mug and put it down on one of the four goal positions.
Dropping the mug outside of the red boundary is irreversible. • Peg Insertion [35] (forward-backward).
The agent must insert the peg into the goal but can potentially drop it off the table, which is irreversible.
• Half-Cheetah Vel [9] (continuing).
The agent must run at the specified target velocity, which changes every 500 steps, and can potentially flip over onto its back, which is irreversible.
We visualize and fully describe each environment in Fig.
4 and in Appendix A.3 respectively.
Comparisons.
In the episodic and continuing settings, we consider safe RL baselines that rely on reversibility labels at every time-step of training. • Safe Model-Based Policy Optimization (SMBPO)
This comparison implements the modified Bellman operator defined in Eqn.
4 in Section 5.1, using the true reversibility labels. • Safety Q-functions for RL (SQRL) [37].
A safe RL method that trains a safety critic, which estimates the future probability of entering an irreversible state for a safety-constrained policy.
In the forward-backward setting, we consider methods designed for the autonomous learning setup.
These methods do not require any reversibility labels.
Hence, our goal here is to compare the reset-efficiency of our method to prior work
• Leave No Trace (LNT)
An autonomous RL method that jointly trains a forward policy and reset policy.
When the reset policy fails to return to the initial state, the agent requests a reset. • Matching Expert Distributions for Autonomous Learning (MEDAL) [36].
This method trains a reset policy that returns to the distribution of demonstration states provided for the forward policy.
MEDAL does not have a built-in intervention rule.
In all tasks, we compare to a recently proposed reversibility-aware RL method, Reversibility-Aware
Exploration (RAE) [15], which does not require any reversibility labels.
It instead trains a selfsupervised reversibility estimator to predict whether a state transition ( s, ˜ s ) is more likely than the reverse (˜ s, s ) .
We augment RAE with an intervention rule, similar to our method, defined in terms
Task
Method
Labels
Maze
SMBPO/SQRL 200 K
PAINT (Ours) 3260 ±
Tabletop
PAINT (Ours) 1021 ±
Peg Insertion PAINT (Ours) 2083 ±
Cheetah
SMBPO w Term.
3 M
PAINT (Ours) 8748 ± of predictions from its self-supervised classifier.
In the forward-backward setting, we train both the forward and backward policies with RAE.
Finally, we also evaluate Episodic RL , which represents the typical RL setup with frequent resets and thus provides an upper-bound on task success.
In
Appendix A.4, we provide full implementation details of each comparison.
6.2 Main Results
In Fig.
5 ( left ) and Fig.
7, we plot the task success versus the number of interventions in the 4 tasks.
For methods that use reversibility labels, we report the total number of labels queried in Fig.
Maze.
In the Maze task, resets are provided to each agent every 500 time-steps.
While the safe RL methods, SMBPO and SQRL, require reversibility labels at every time-step, our approach PAINT only requires on average 3260 queries to label all 200 K states visited.
In Fig.
5 (right), we visualize predictions from our reversibility classifier at the end of training, where zero predicts ‘reversible’ and one predicts ‘irreversible’.
The classifier correctly identifies the path that leads to the goal as reversible.
Interestingly, it classifies all other regions as stuck states, including the states that are reversible.
Because these states are irrelevant to the task, however, classifying them as irreversible, and therefore to be avoided, is advantageous to our policy as it reduces its area of exploration.
More complex domains.
In the Tabletop Organization and Peg Insertion tasks, each agent is reset every 200 K and 100 K time-steps, per the EARL benchmark [35].
However, we allow agents to request earlier resets, and under this setting, we compare PAINT to other methods that implement intervention rules.
Compared to Leave No Trace and Reversibility-Aware Exploration, PAINT requires significantly fewer resets— 80 and 124 resets respectively, which corresponds to roughly one intervention for every 25 K steps .
Importantly, the number of interventions plateaus as training progresses, and the agent requires fewer and fewer resets over time (see Appendix A.5 for additional plots of number of interventions versus time-steps).
The exception is MEDAL (green segment near the origin), which is not equipped with an early termination rule and so only uses 10 interventions total.
However, it also fails to make meaningful progress on the task with few resets.
On the continuing Half-Cheetah task, agents do not receive any resets, unless specifically requested.
Here, we compare PAINT to SMPBO with early termination, an oracle version of our method, which assumes that reversibility labels are available at every time-step and immediately requests an intervention if the agent is flipped over.
PAINT converges to its final performance after aroundresets, on par with the number of resets required by SMPBO with early termination.
On the other hand, a standard episodic RL agent, which receives resets at every 2 K steps, and RAE, which trains a self-supervised classifier, learn significantly slower with respect to the number of interventions.
6.3 Ablations and Sensitivity Analysis
Early termination .
In the episodic Maze setting, our algorithm switches to a uniform-random policy for the remainder of the episode if the termination condition is met.
In Fig.
8 ( left ), we plot the performance without early termination, i.e., running the agent policy for the full episode.
Taking random explorations, after the agent believes it has entered an irreversible state, significantly helps our method, as it increases the number and diversity of irreversible states the agent has seen.
Varying the number of demonstrations .
Our method leverages demonstrations in a subset of environments.
While we provide these demonstrations to all comparisons as well, we want to study how much our method relies on them.
We plot the average task success during training versus number of demonstrations in Fig.
8 ( right ).
While PAINT and SMBPO are robust to the amount, alternative methods tend to achieve significantly lower success when given fewer demonstrations.
We study additional ablations on the termination condition and pseudo-labeling in Appendix A.5.
Discussion
In this work, we sought to build greater autonomy into RL agents, particularly in irreversible environments.
We proposed an algorithm, PAINT, that learns to detect and avoid irreversible states, and proactively requests an intervention when in an irreversible state.
PAINT leverages reversibility labels to learn to identify irreversible states more quickly, and improves upon existing methods on a range of learning setups in terms of task success, reset-efficiency, and label-efficiency.
Despite these improvements, PAINT has multiple important limitations.
In environments where irreversible states are not encountered until further into training, the reversibility classifier may produce false positives which would significantly delay the next intervention.
Further, while PAINT is far more label-efficient than prior safe RL methods, it still requires around thousands of reversibility labels.
We expect that this limitation may be mitigated with more sophisticated querying strategies, e.g. that take into account the classifier’s confidence.
Finally, we hope that future work can validate the ability for reversibility aware techniques to improve the autonomy of real robotic learning systems.
A Appendix A.1 Proofs A.1.1 Penalizing Visitation of Irreversible States
We give a simple proof for Theorem 5.1 assuming deterministic dynamics first, and then give a more restricted proof for stochastic dynamics under additional assumptions.
Theorem 5.1.
Let ( s, a ) ∈ S rev denote a state-action leading to a reversible state and let ( s _ , a _ ) ̸∈ S rev denote a state-action pair leading to an irreversible state.
Then, for all such pairs
[Equation] ) for all ϵ > 0 and for all policies π .
Proof.
By definition, the reward function is bounded, i.e., r ( s ′ , a ′ ) ∈
[ R min , R max ] for any ( s ′ ,
a ′ ) ∈ S × A .
For any ( s ′ , a ′ ) ∈ S rev , we have the following (using equation 2):
[Equation]
and for any ( s ′ , a ′ ) / ∈ S rev , we have:
[Equation]
This simplifies to ˜ r ( s ′ , a ′ ) ≥ R min − ϵ for any ( s ′ ,
a ′ ) ∈
S × A .
From the definition of Q π , we have
[Equation]
(5) where the lower bound of Q π ( s ′ , a ′ ) is achieved if ( s ′ , a ′ ) / ∈ S rev .
For ( s, a ) ∈ S rev ,
Q π ( s, a ) = ˜ r
( s, a ) + γ E a ′ ∼ π ( ·| s ′ )
[ Q π ( s ′ , a ′ )] ≥ ˜ r ( s
[Equation]
s, a ) =
r ( s, a ) ≥ R min , since ( s, a ) ∈ S rev (cid:3)
[Equation]
where ( s _ , a _ ) ̸∈ S rev , implying Q π ( s _ , a _ ) =
R min − ϵ 1 − γ .
This concludes the proof as
Q π ( s, a ) > Q π ( s _ , a _ ) for ϵ > 0 .
To extend the discussion to stochastic dynamics, we redefine the reversibility set as
S rev = { [Equation]
s, a ) | P
[Equation] s ′ )
= 1) ≥ η 1 } , i.e, state-action pairs leading to a reversible state with at least η 1 probability and let S irrev =
{ ( s, a ) | P ( R ρ ( s ′ )
= 1) ≤ η 2 } denote the set of state-action pairs leading to irreversible states with at most η 2 probability.
The goal is to show that actions leading to reversible states with high probability will have higher Q -values than those leading to irreversible states with high probability.
The surrogate reward function ˜ r is defined as:
[Equation]
Theorem A.1.
Let ( s, a ) ∈ S rev denote a state-action leading to a reversible state with at least ηprobability and let ( s _ , a _ ) ∈ S irrev denote a state-action pair leading to an irreversible state with at least (1 − η 2 ) probability.
Assuming η 1 > η 2 / (1 − γ ) ,
[Equation] for all ϵ > [Figure]
Proof.
For any ( s, a ) ∈ S rev , we have the following (using equation 6):
[Equation]
where P ( R ρ ( s ′ )
= 1) ≥ η 1 .
For any ( s, a ) ∈ S irrev ,
Q
[Equation]
s
[Equation]
= P
[Equation]
R
[Equation]
as P ( R ρ ( s ′ )
= 1) ≤ η 2 ) by definition of S irrev .
Under the assumption that η 1 > η 2 / (1 − γ ) , whenever
[Equation]
ensures that Q π ( s, a ) > Q π ( s _ , a _ ) for all ( s, a ) ∈ S rev and for all ( s _ , a _ ) ∈ S irrev , finishing the proof.
The above proof guarantees that for actions that lead to reversible states with high probability will have higher Q -values than actions leading to irreversible states with high probabilities for all policies , even under stochastic dynamics.
The restrictive assumption of η 1 > η 2 / (1 − γ ) is required to ensure that the Q -value for worst ( s, a ) ∈ S rev is better than the best ( s, a ) ∈ S irrev .
An alternate analysis can be found in [41, Appendix A.2], where the guarantees are only given for the optimal Q -functions but under less stringent assumptions.
Improved guarantees are deferred to future work.
A.1.2 On Empirical Bellman Backup Operator
The empirical Bellman backup operator was introduced in subsection 5.2.
In this section, we prove that it is a contraction, and analyze the convergence under empirical Bellman backup.
For ˆ R ρ : S (cid:55)→ [0 , 1] , the empirical Bellman backup operator can be written as:
[Equation]
Theorem A.2.
Empirical Bellman backup operator in equation 9 is a Contraction under the L ∞ norm.
Proof.
For all ( s, a ) ∈ S × A , we have the following:
[Equation]
ˆ R
[Equation]
[Equation]
∈
[Equation]
[Equation]
Since this holds for all ( s, a ) ∈ S × A , this implies:
[Equation]
The discount factor γ < 1 , proving our claim.
For a policy π , let Q π be the true Q -value function computed using conventional Bellman backup B π and ˆ Q π be the Q -values computed using the empirical Bellman backup ˆ B π .
Being fixed point of the operators, we have Q π = B π Q π and ˆ Q π
= ˆ
B π ˆ Q π .
The following theorem relates the two:
Theorem A.3.
Assuming that ∥R ρ − ˆ R ρ ∥ ∞ ≤ δ , the difference between true Q -values and empirical
Q -values for any policy π obeys the following inequality:
[Equation]
Proof.
Since Q π is the fixed point of B π and ˆ Q π is the fixed point of ˆ B π
[Equation]
we can write:
[Equation]
Consider the following identity:
[Equation]
Plugging Eq 11 in Eq 10, we get:
[Equation]
Taking the modulus inside the expectation and using the triangle inequality
[Equation]
we get:
[Equation]
Using the following inequalities: r ( s, a )+ γ E a ′ ∼ π ( ·| s ′ )
[ ˆ Q π ( s ′ , a ′ )] ≤ R max / (1 − γ ) as the maximum environment reward is R max , (cid:12)(cid:12)(cid:12) ˆ R ρ ( s ′ )
(cid:12)(cid:12)(cid:12) ≤ 1 as ˆ R ρ ∈ [0 , 1] and the assumption ∥R
ρ − ˆ R ρ ∥ ∞ ≤ δ , we can complete the proof:
[Equation]
Theorem A.3 gives us a bound on the difference between the true Q -values and Q -values computed using the reversibility estimator ˆ R ρ for any policy π .
The bound and the proof also suggest that closer ˆ R ρ ( s ) is to R ρ , closer the estimated Q -values are to true ones.
Similar to the empirical Bellman backup, we can define the empirical Bellman optimality operator:
[Equation]
The empirical Bellman optimality operator is also a contraction, following a proof similar as that for empirical Bellman backup.
Let ˆ Q ∗ denote the fixed point of Bellman optimality operator, that is ˆ B ∗ ˆ Q ∗ =
ˆ Q ∗ , and let ˆ π ∗ denote the greedy policy with respect to ˆ Q ∗ .
As a final result,
Theorem A.4.
Let π ∗ denote the optimal policy and Q ∗ denote the corresponding optimal Q -value function.
Let ˆ π ∗ denote the optimal policy returned by empirical Bellman optimality operator ˆ B ∗ .
Assuming ∥R
[Equation]
[Equation]
for all ( s, a ) ∈ S × A .
Proof.
For clarity of notation, we will use Q ( π ) denote Q π
( s, a ) and ˆ Q ( π ) denote ˆ Q π ( s, a ) for a policy π .
Now,
[Equation]
Using the fact ˆ π ∗ is the optimal policy with respect to ˆ B ∗ , we have ˆ Q ( π ∗ ) − ˆ Q (ˆ π ∗ ) ≤ 0 .
This implies that:
[Equation] [Equation]
Using theorem A.3, we have
[Equation]
Plugging these in Eq 13, we get
[Equation]
Rearranging the above bound gives us the statement in the theorem.
Theorem A.4 gives us the assurance that as long as the estimator ˆ R ρ is close to R ρ , the Q -values of the greedy policy obtained by value iteration using ˆ B ∗ , i.e. ˆ π ∗ will not be much worse than the optimal
Q -values.
The above result also suggests choosing a smaller ϵ for a smaller gap in performance.
A.2 Detailed Pseudocode
In Algorithms 3 and 4, we provide pseudo-code for the episodic and non-episodic variants of our method PAINT.
We build the non-episodic variant upon the MEDAL algorithm [36], which introduces a backward policy whose objective is to match the state distribution of the forward demonstrations (summarized in Section 4).
Algorithm 3: PAINT (Episodic) optional: forward demonstrations N initialize: π, Q, D ; // forward agent parameters initialize ˆ R ρ , D ρ ; // reversibility classifier and dataset of labels // add demonstrations to replay buffer
D ← D ∪ N ; while not done do s ∼ ρ 0 // reset environment
D new ← D new ∪ { s } ; aborted ← False; for t = 1 , 2 , . .
, H do if not aborted then a ∼ π ( · | s ) ; update π, Q ; // Eqelse a ∼ unif ( A ) ; s ′ ∼ P ( · | s, a ) , r ← r ( s, a ) ;
D ← D ∪ { ( s, a, s ′ , r ) } ; if not aborted and ˆ R ρ ( s ) ≤ 0 .
5 then aborted ← True;
D new ← D new ∪ { s ′ } ; s ← s ′ ; // query reversibility labels for newly collected states via Alg.label D new ;
D ρ ← D ρ ∪ D new ; D new ← ∅ ; // train classifier on all labeled data, new and old update ˆ R ρ ;
A.3 Environment Details
In this section, we provide details of each of the four irreversible environments, which are visualized in Fig.
4 and Fig.
Maze .
In this 2 -D continuous-control environment, the agent is a point mass particle that starts in the top left corner of the maze and must reach the bottom right corner.
Throughout the environment, there are trenches (marked in black) that the agent must avoid.
Entering them is irreversible: the agent can roam freely within the trench but cannot leave it without an environment reset.
The agent is placed back at the top left corner upon a reset, which is provided every 500 time-steps.
The agent’s state space consists of its xy -position, and two control inputs correspond to the change applied to its xy -position.
The reward function is defined as r t = 1 ( ∥ s t − g ∥ 2 < 0 .
1) , where g is the goal position.
We provide 10 demonstrations of the task to the agent at the beginning of training.
Tabletop Organization .
This environment modifies the Tabletop Organization task from Sharma et al.
The agent’s objective is to grasp the mug and move it to one of the four specified goal positions.
Grasping the mug and dropping it off beyond the red boundary is irreversible, i.e., the agent can no longer re-grasp the mug.
Upon a reset, which is provided every 200 K time-steps or when requested by the agent, the agent is placed back at the center of the table and the mug is placed on its right, just within the red boundary (see Fig. 4).
The agent’s state space consists of the its own xy -position, the mug’s xy -position, an indicator of whether the mug is grasped, the goal position of the mug, and finally, the goal position of the agent after putting the mug down at its goal.
There are three control inputs, which apply changes to the agent’s xy -position and toggle between grasping, if input: forward demonstrations N f ; optional: backward demonstrations N b ; initialize: π f , Q f , D f ; // forward agent parameters initialize: π b , Q b , D b ; // backward agent parameters initialize ˆ R ρ , D ρ ; // reversibility classifier and dataset of labels initialize C ( s ) ; // state-space discriminator for backward policy // add demonstrations to replay buffer
D f ← D f ∪ N f ;
D b ← D b ∪ N b ;
D new ← ∅ ; while not done do s ∼ ρ 0 ; // reset environment
D new ← D new ∪ { s } ; // continue till the reversibility classifier detects an irreversible state while ˆ R ρ ( s ) > 0 .
5 do // run forward policy for a fixed number of steps, switch to backward policy if forward then a ∼ π f ( · | s ) ; s ′ ∼ P ( · | s, a ) , r ← r ( s, a ) ;
D f ← D f ∪ { ( s, a, s ′ , r } ; update π f , Q f ; // Eqelse a ∼ π b ( · | s ) ; s ′ ∼ P ( · | s, a ) , r ← − log(1 − C ( s ′ )) ;
D b ← D b ∪ { ( s, a, s ′ , r ) } ; update π b , Q b ; //
Eq// train disriminator every K steps if train-discriminator then // sample a batch of positives S p from the forward demos N f , and a batch of negatives S n from backward replay buffer D b
S p ∼ N f , S n ∼ D b ; update C on S p ∪ S n ;
D new ← D new ∪ { s ′ } ; s ← s ′ ; // optionally explore environment for explore steps do a ∼ unif ( A ) , s ′ ∼ P ( · | s, a ) , r ← r ( s, a ) ; update D f , D b ; // use C ( s ) for the the object is nearby (i.e., within a distance of 0 . 4 ), and releasing, if the object is currently grasped.
The agent’s reward function is r t = 1 ( ∥ s t − g ∥ 2 < 0 . 1) , i.e., both the agent’s xy -position and the mug’s xy -position must be close to their targets.
We provide 50 forward demonstrations, 50 backward demonstrations, and 1000 examples of (randomly generated) irreversible states to the agent at the beginning of training.
Peg Insertion .
This environment modifies the Peg Insertion task from Sharma et al.
The objective of this task is to grasp and insert the peg into the hole in the box.
We modified the table so that the raised edges that stop the peg from rolling off the table are removed and the table is significantly narrower.
Hence, the peg may fall off the table, which cannot be reversed by the robot.
Instead, when the environment is reset, which automatically occurs every 100 K time-steps or when requested by the agent, the peg is placed back at one of 15 possible initial positions on the table.
The agent’s state space consists of the robot’s xyz -position, the distance between the robot’s gripper fingers, and the object’s xyz -position.
The agent’s action consists of 3D end-effector control and normalized gripper torque.
Let s peg t represent the state of the peg and g peg be its goal state, then the reward function is r t = 1 ( ∥ s peg t − g peg ∥ 2 < 0 . 05) .
We provide the agent with 12 forward demonstrations and 12 backward demonstrations.
Half-Cheetah .
We design this environment based on the version from Brockman et al.
In particular, the agent must run at one of six target velocities { 3 , 4 , 5 , 6 , 7 , 8 } , specified to the agent.
Every 500 time-steps, the target velocity switches to a different value selected at random.
The agent’s actions, which correspond to torque control of the cheetah’s six joints, are scaled up by a factor of 5 .
When the agent is flipped onto its back (i.e., its normalized orientation is greater than 2 π/ 3 ), we label these states irreversible.
When the agent is reset, which only occurs when requested, the agent is placed upright again at angle of 0 .
The agent’s observation space consists of the velocity of the agent’s center of mass, angular velocity of each of its six joints, and the target velocity.
Let v be the velocity of the agent, then the reward function, which is normalized to be between 0 and 1 , is r t = 0 .
95 ∗ (8 − v ) / 8 + 0 .
05 ∗ (6 − ∥ a t ∥ 22 ) / 6 .
There are no demonstrations for this task.
A.4 Implementation Details
Below, we provide implementation details of our algorithm PAINT and the baselines.
Every algorithm, including ours, has the following components.
Forward policy network .
The agent’s forward policy is represented by an MLP with 2 fully-connected layers of size 256 in all experimental domains, trained with the Soft Actor-Critic (SAC) [18] algorithm.
Forward critic network .
The agent’s forward critic is represented by an MLP with 2 fully-connected layers of size 256 in all experimental domains, trained with the Soft Actor-Critic (SAC) [18] algorithm.
Balanced batches .
In the Tabletop Organization and Peg Insertion tasks, the agent’s forward policy and critic networks are trained with batches that consist of demonstration tuples and of tuples sampled from the agent’s online replay buffer.
We control the ratio p of demonstration tuples to online tuples with a linearly decaying schedule,
[Equation]
In the Tabletop Organization and Peg Insertion tasks, p 0 = 0 .
5 , p T = 0 .
1 , and T =
K. We do not train with balanced batches in the Maze task, and simply populate the online replay buffer with the demonstrations at the beginning of training.
Episodic RL (Soft Actor-Critic) [18].
In addition to the policy and critic networks trained with balanced batches, the episodic RL comparison requests for resets every H ′ time-steps in the Tabletop
Organization ( H ′ = 2000 ), Peg Insertion ( H ′ = 1000 ), and Half-Cheetah ( H ′ = 2000 ) tasks.
Safe Model-Based Policy Optimization (SMBPO)
This comparison trains the forward critic with the modified Bellman update B π Q ( s, a ) defined in Eqn. 4, where ϵ = 0 in the Maze and
Half-Cheetah tasks and ϵ = − 0 .
1 in the Tabletop Organization and Peg Insertion tasks.
Safety Q-functions for RL (SQRL) [37].
This comparison trains an additional safety critic Q π safe , which estimates the future probability of entering an irreversible state, and the policy is updated with the following augmented objective
[Equation]
where ν is the Lagrange multiplier for the safety constraint and is updated via dual gradient descent.
We only evaluate this comparison in the Maze task, where ϵ safe = 10 .
A.4.1 Forward-Backward Algorithms
In the forward-backward setups, we train a backward policy and critic in addition to their forward counterparts.
The details of the backward components that are shared across all methods are described below.
Backward policy network .
The agent’s backward policy is represented by an MLP with 2 fullyconnected layers of size 256 in all experimental domains, trained with the SAC algorithm.
Backward critic network .
The agent’s backward critic is represented by an MLP with 2 fully-connected layers of size 256 in all experimental domains, trained with the SAC algorithm.
Backward balanced batches .
The agent’s backward policy and critic networks are trained with batches that consist of demonstration tuples and of tuples sampled from the agent’s online replay buffer.
We control the ratio p of demonstration tuples to online tuples with a linearly decaying schedule,
[Equation]
In the Tabletop Organization and Peg Insertion tasks, p 0 = 0 .
5 , p T = 0 .
1 , and T = 500 K.
Leave No Trace (LNT)
This comparison additionally trains backward policy and critic networks, whose reward function is the sparse indicator of whether the current state is within some threshold of the initial state.
The thresholds are the same as those used for the forward reward functions defined in A.3.
Policy switching .
Leave No Trace switches from the forward to backward policy if the backward critic’s Q -value is lower than ϵ LNT or after 300 time-steps, and switches from the backward to forward policy after 300 time-steps. In Tabletop, ϵ LNT = 0 . 1 , and in Peg Insertion, ϵ LNT = 0 .
Termination condition .
Leave No Trace additionally requests a reset if, after 300 time-steps, the backward policy fails to bring the environment within a distance of 0 .
1 of the initial state.
Matching Expert Distributions for Autonomous Learning (MEDAL) [36].
Like Leave No Trace,
MEDAL trains a backward policy and critic.
However, instead of returning to the initial state, the backward reward function is whether the current state matches the distribution of demonstration states, formally defined in Eqn. 1.
MEDAL classifier .
The classifier C in Eqn.
1 is represented by an MLP with 1 FC layer of size 128 .
Policy switching .
The algorithm switches policies (i.e., from forward to backward and from backward to forward) after every 300 time-steps.
Reversibility-Aware Exploration (RAE) [15].
RAE trains a self-supervised reversibility estimator, specifically to predict whether a state transition ( s, ˜ s ) is more likely than the reverse transition (˜ s, s ) .
RAE generates data for the binary classifier with a windowed approach.
For every state trajectory ( s t : t + w ) of length w collected by the agent, all state pairs ( s i , s j ) , where i < j , are labeled positive , and all pairs ( s j , s i ) , where i < j , are labeled negative .
For all experimental tasks, we use a window size of w = 10 time-steps.
With this estimator, the forward critic is trained with the modified Bellman update ˆ B π Q ( s, a ) , where ϵ = 0 in the Maze and Half-Cheetah tasks and ϵ = − 0 .
1 in the Tabletop
Organization and Peg Insertion tasks.
Reversibility classifier .
The classifier ˆ
R ρ is represented by an MLP with 1 FC layer of size 128 .
Termination condition .
In Maze and Half-Cheetah, the termination condition is ˆ R ρ > 0 .
In Tabletop
Organization and Peg Insertion, the condition is ˆ R ρ > 0 . 8 .
Exploration .
We augment RAE with uniform-random exploration after the termination condition is met as proposed in our method.
In the Maze environment, the agent takes uniform-random actions for the rest of the episode (of length 500 ).
In Tabletop Organization and Peg Insertion, N explore =time-steps.
In Half-Cheetah, N explore = 500 time-steps.
In the Tabletop Organization and Peg Insertion tasks, we train an additional backward policy and critic, whose reward functions are defined in terms of the MEDAL classifier.
The backward critic is also trained with the modified ˆ B π Q ( s, a ) , with the same hyperparameters as the forward critic.
MEDAL classifier .
The classifier C in Eqn.
1 is represented by an MLP with 1 FC layer of size 128 .
Policy switching .
The algorithm switches policies (i.e., from forward to backward and from backward to forward) after every 300 time-steps.
PAINT (Ours) .
PAINT trains a reversibility classifier ˆ R ρ and checks whether the current state is estimated to be irreversible.
If it is estimated to be irreversible, the agent takes uniform-random actions for H explore time-steps and requests a reset afterward.
The forward critic is also trained with the modified Bellman update ˆ B π Q ( s, a ) , where ϵ = 0 in the Maze and Half-Cheetah tasks and ϵ = − 0 .
1 in the Tabletop Organization and Peg Insertion tasks.
Reversibility classifier .
The classifier ˆ
R ρ is represented by an MLP with 1 FC layer of size 128 .
Termination condition .
In all tasks, the termination condition is ˆ R ρ > 0 .
Exploration .
In the Maze environment, the agent takes uniform-random actions for the rest of the episode (of length 500 ).
In Tabletop Organization and Peg Insertion, N explore = 300 time-steps.
In
Half-Cheetah, N explore = 500 time-steps.
In the Tabletop Organization and Peg Insertion tasks, we train an additional backward policy and critic.
The details for the backward policy and critic are the same as in RAE.
A.5 Additional Experimental Results
In this section, we present additional plots to accompany Section 6.2 and additional ablations to accompany Section 6.3.
A.5.1 Additional Plots
In Fig.
9, we plot the task success, number of reset interventions, and number of reversibility labels versus the number of time-steps in each of the four tasks.
A.5.2 Ablations and Sensitivity Analysis
Termination conditions .
To evaluate the importance of the reversibility classifier, we study an alternative choice for the termination condition, one based on the Q -value function.
Intuitively, the values for irreversible states will be low, as there is no path that leads to the goal from these states.
We define the value-based termination condition as V π ( s ) <
ϵ where ϵ is the threshold.
We approximate the value V π ( s ) =
E π
[ Q π ( s, a )] with the trained Q -value function evaluated at N =policy actions.
We plot the task success in the Tabletop Manipulation task in Fig.
10 (left).
After 1 M time-steps, the Q -value-based termination condition requires fewer interventions, approximately half of the interventions needed by PAINT, but does not converge to the same final performance as PAINT with the reversibility classifier.
Critically, PAINT with Q -value termination also trains a reversibility classifier to generate pseudo-labels for unlabeled states.
Pseudo-labels for unlabeled states .
Currently, when updating the agent with Eqn. 9, our method uses the predictions from the stuck classifier as pseudo-labels for the unlabeled states collected during a trial.
An alternative choice is labeling them one, i.e., treating them as reversible.
We evaluate this choice in the Tabletop Manipulation task, and plot the task success in Fig.
10 (right).
After 1 M time-steps, the number of reset interventions requested by both methods are similar.
However, our method succeeds at the task almost 100% of the time, while the agent trained with pseudo-labels of one only achieves success of around 60% .