Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates
Abstract: Long horizon robot learning tasks with sparse rewards pose a significant challenge for current reinforcement learning algorithms.
A key feature enabling humans to learn challenging control tasks is that they often receive expert intervention that enables them to understand the high-level structure of the task before mastering low-level control actions.
We propose a framework for leveraging expert intervention to solve long-horizon reinforcement learning tasks.
We consider option templates , which are speciﬁcations encoding a potential option that can be trained using reinforcement learning.
We formulate expert intervention as allowing the agent to execute option templates before learning an implementation.
This enables them to use an option, before committing costly resources to learning it.
We evaluate our approach on three challenging reinforcement learning problems, showing that it outperforms state-of-the-art approaches by two orders of magnitude.
Videos of trained agents and our code can be found at: https://sites.google.com/view/stickymittens
Introduction
Reinforcement learning is an effective tool to solve difﬁcult tasks such as robot planning and locomotion [1] but exploration is still a challenge.
Options are an RL tool to circumvent this problem [2].
Designed to achieve intermediate subgoals.
For instance in robot grasping tasks, an option might enable the robot to grasp a block which is a subgoal needed to build a tower out of blocks.
The goal is to learn a policy mapping each state to an option, instead of a concrete action to take.
When learning to perform complex visual-motor skills, humans often rely on expert interventions to help them escape these challenging reward plateaus.
For instance, the sticky-mittens experiment [3] considers infants who have not yet learned to grasp objects.
They give a subset of these infants mittens covered with Velcro hooks and allow them to play with toys ﬁtted with Velcro loops, making it signiﬁcantly easier for them to grasp these toys.
Even if the Velcro is taken away, these babies learn how to grasp objects signiﬁcantly faster than infants not exposed to this experience.
In other words, enabling infants to explore unreachable parts of the state space helps guide them towards skills that are worth learning.
This is a well known phenomenon in developmental psychology, which extends beyond ﬁne motor skills.
In this paper, we design an RL algorithm based on the idea from the sticky-mittens experiment.
The agent has access to an alternative Markov Decision Process (MDP) where the agent can leap multiple states without ﬁrst learning a policy to do so in the original MDP.
We term such a jumping mechanism an option template .
Option templates are described using a set of initial states and a set of ﬁnal states.
The idea of providing external help in the learning phase is referred to as primitives or skills in literature.
It offers a practical way to speed-up learning in a realistic setting.
For instance, using parameterized action spaces for RoboCup in [4], stitching independent behaviors in [5, 6] and providing action primitives in [7, 8, 9] to mention a few.
In the more typical RL framework the agent ﬁrst uses reinforcement learning to train an option to implement the speciﬁcation of each option template.
The issue with this strategy is that, in RL environments with large state spaces the options learnt need to be of a fairly generic nature, in order to be useful.
Which is hard without a knowledge of the state distribution, where the options will be invoked.
Using option templates we decouple the implementation of an option from its utility.
In more detail, our algorithm performs an alternating search over policies: at each iteration, it ﬁrst optimizes the high-level policy over the current option templates, and then learns options to implement the option templates.
In our experiments, we demonstrate that by leveraging option templates, our algorithm can achieve orders of magnitude reduction in sample complexity compared to the typical strategy of learning the options upfront and then planning with these learned options.
Contributions: Our contributions are: (1) an RL algorithm that leverages option-templates, and (2) an empirical comparison with state-of-the-art RL techniques on three challenging environments, demonstrating order-of-magnitudes reduction in sample complexity.
Background
We consider the RL setting where an agent interacts with an environment modeled as an MDP [10].
Deﬁnition 2.1 ( MDP ) .
A Markov Decision Process (MDP) is a tuple E =
( S , A , P , R ,  , I ) , where
S ✓ R n is the set of states, A ✓ R m is the set of actions, P ( s 0 | s, a ) is the probability of transitioning from state s to s 0 upon taking action a , R ( s, a ) is the reward accrued in state s upon taking action a ,  2 [0 , 1) is the discount factor, and I is the initial state distribution.
The value of a state for policy ⇡ is the expected return R ⇡ ( s 0 ) starting from state s 0 , while executing the policy ⇡ at every step, a t = ⇡
( s t ) .
The optimal policy ⇡ ⇤ maximizes the reward starting from the initial state distribution–i.e., ⇡ ⇤ = arg max V I ( ⇡ ) , where V I ( ⇡ ) =
E s 0
I [ R ⇡ ( s 0 )] .
Semi-MDPs (SMDPs) extend this framework by allowing temporally extended actions—i.e., policies executed over multiple steps—called options [2].
Deﬁnition 2.2 ( Option ) .
An option o is a tuple o :
= ( I, T, ⇡ ) , where I ✓ S are initial states, T : S !
[0 , 1] the termination condition, and ⇡ : S !
A is a policy.
For now, assume that we have a set O of options available for an MDP.
Intuitively, options constrain the search space over policies, thereby guiding exploration performed by the agent towards more promising action sequences.
We now consider policies ⇧ : S !
O that map a state s
I (i.e., s is a valid initial state for o ) to an option o = ⇧ ( s ) .
The agent uses option policy ⇡ until the termination condition holds.
More precisely, after taking action a = ⇡ ( s ) and transitioning to state s 0 ⇠ P ( ·| s, a ) , it stops using ⇡ with probability T ( s t +1 ) and chooses another option o
= ⇧ ( s 0 ) ; otherwise, it continues using ⇡ .
Suppose that an option o is invoked at time t in state s t , and the system follows a k step trajectory
Tr o ( s t , k ) =
( s t , a t , r t , s t +1 , . . .
, s t + k ) .
We consider policies such that for t  i < t + k , the action a
i depends on the entire history
Tr o ( s t , i ) from t until t
+ i .
This renders the behavior nonMarkovian.
Denoting the set of all such state-action trajectories as H , we can allow option policies ⇡ : H !
A , and termination conditions T : H !
[0 , 1] to be semi-Markovian.
Now, we recall the deﬁnition of Semi-MDPs as an extension of MDPs:
Deﬁnition 2.3 ( Semi-MDP , Sutton et al.
A semi-MDP is a tuple ( S , O , P , R ,  , I ) , where S is a set of states, O is a set of semi-Markovian options, P is the transition probability between states,
R is the reward function,  2 [0 , 1) is the discount factor, and I is the initial state distribution.
The state-prediction part of the model for taking an option o , in state s 2 I and transitioning to s 0 , is given by Sutton et al.
[2]: P ( s 0 | s, o ) =
P 1 k =1  k p ( s 0 , s, k ) , where p ( s 0 , s, k ) is the probability that the option terminates in s 0 after k steps from the time the option is invoked.
The optionvalue form of standard value iteration is, for s 2 S
and o 2 O , [2, eq. 12], [11] Q ⇤ O
( s, o ) =
R ( s, o )
+
[-s 0 2 S P ( s 
-]
P
[Equation]
O
[Equation]
) , where s 0 is the state the system reaches after exe- cuting option o starting from state s .
SMDP value learning updates the Q -values at the end of each option termination as [2, p. 195]: Q i +1 ( s, o )
Q i ( s, o )+ ↵ [ r c +  k max o 0 2 O Q
i ( s 0 , o 0 )  Q i ( s, o )] , for similar deﬁnitions of k , s , s 0 , with r c being the cumulative discounted reward over the time horizon, and ↵ 2 (0 , 1) the learning rate.
Choosing a function approximator parameterized by ✓ to represent Q ( s, o ; ✓ ) , with a loss given by : L ( ✓ i ) =
[ r +  k max o 0 2 O Q
i ( s 0 , o 0 ; ✓ i )  Q i ( s, o ; ✓ i )] 2 , allows one to use standard Q-learning [12] methods to train with options.
Learning with Option Templates 3.1 Option Templates
Motivated by the sticky-mittens experiment, we consider an environment where in certain states, the agent can call a “help switch”, called an option template , that immediately transitions it from a state s t to a different state s t + k ; this transition captures the desired result of executing an (unimplemented) option.
We denote a trajectory of this MDP starting from state s t under policy ⇡ by Tr ⇡ ( s t ) .
Deﬁnition 3.1 ( Option template ) .
Option template a o is a tuple ( I, T, P ) , where I ✓ S is the initial states, T : S !
[0 , 1] is the termination condition, and P is a distribution over states s such that T ( s )
= 1 .
An option template similar to an option, shifts control to the expert until the termination condition is reached; this termination condition T is intended to capture the satisfaction of some sub-goal along with a timeout mechanism.
Assume the termination condition at time t + k of an option template taken in state s t is of the form T ( s t , k )
=
F o ( s t + k ) _ ( k > k ⇤ ) , where the function F o : S ! { 0 , 1 } captures visitation of some key states, and k ⇤ is an upper limit on the number of steps.
We denote the set (or subset) of option templates by A O .
By the end of training, the agent must learn a policy to implement each option template that it uses (i.e., the ﬁnal policy cannot depend on option templates).
To ensure feasiblity, we assume there is an a priori unknown policy ⇡ o such that Tr ⇡ o ( s t ) satisﬁes T (denoted as Tr ⇡ o ( s t ) | = T ) with probability at least 1   , for some hyperparameter [0 , 1) .
Formally, the probability of success of a given policy ⇡ o can be expressed as P (( ⇡ o , I, E ) | =
T )  (1   ) —i.e., the trajectory starting from a state in I under the control of ⇡ o in environment
E satisﬁes T with probability at least 1   .
This success probability can be estimated by empirical rollouts of ⇡ o .
We organize option-templates in a hierarchical fashion depending on degrees of help.
Option templates that traverse longer sequences of transitions allow faster learning due to fewer decisions.
Thus, we consider a sequence of environments E 0 , E 1 , . . .
, E n  1 = E , equipped with option templates of varying capabilities organized in a hierarchical fashion.
Deﬁnition 3.2 ( Environment Level E l ) .
An environment E l at learning level l is the original environment E with the primitive actions A replaced by option templates A l O .
The goal of the levels is to gradually increase difﬁculty of the learning task while guiding the agent at each level in a way that is similar to curriculum learning [13].
In particular, learning longer timescale option templates typically precedes learning shorter ones.
Assumption 3.3 ( Realizability ) .
For any option template a o 2 A l O at level l , given by ( I, T, P ) , there is a policy ⇡ a o : S !
A l +1 O at the following level l + 1 such that for any s 0 2 I ,
Tr o ( s 0 ) | =
T with probability at least 1   .
That is, any option template in the current level is realizable by a policy in the following level.
3.2 Learning with Option Templates
Our algorithm for RL with option templates is presented in Algorithm 1.
Input: Environments [ E 0 , E 1 , . .
, E n  1 ]
Output:
A set of options D = { o q | q 2 Q } that provides an implementation of each option template a qo , where q 2 Q indexes the option templates.
1: Initialize D = {} 2: for level l 2 { 1 , . . . , n } do 3:
Let A l  1 o be the set of option templates on level l  1 4:
Build reward functions { R q | a qo
A l  1 o } 5: for a qo 2 A l  1 o do 6: ⇡ qo =
LearnOptionPolicy
( E l , R q , a qo , D ) 7: o q :=
( a qo , ⇡ qo ) 8:
D = D
[ { o q } 9: end for 10: end for output D
Algorithm
Input: Environment E , Reward Function R ,
Option template a o
= ( I, T, P ) , Set D of options
Parameter:
Threshold  , Max Episodes L
Output: Policy ⇡ o as an implementation of a o 1: Initialize policy ⇡ o 2: for episode = 1 , . . .
, L do 3:
ExecutionStack =
[] 4: o
Top-level option from D (or a o if
D = ? )
Push o onto ExecutionStack 6:
Sample initial state s ⇠ I 7: while not done do 8: o
Pull option from ExecutionStack 9: o⇡ o ( s ) 10: if IsOption ( o 0 , a o ) then 11: ( ⇡ o , s )
StepAndTrain ( E , R , ⇡ o , s ) 12: else if ¬ HasImplementation ( o 0 , D ) then 13: s
Teleport ( o 0 , s ) 14: else 15:
Push o 0 to ExecutionStack 16: end if 17: end while 18:
Exit loop if AvgReward  1   19: end for output ⇡ o
Intuitively, at each iteration, it “ﬂattens” the option templates used on the previous level l  1 based on the ones available at the current level l .
At each level l , we ﬁrst design reward functions R q for each option template a qo 2 A l  1 o (Line 4); this reward function is later used to learn a policy ⇡ qo implementing a qo .
For the base case l = 1 , we take the reward function R q to be the reward function for the original environment E ; thus, learning the policy for the option-template at this level amounts to accomplishing the goals of E using the (unique) option template a qo 2 A 0 o .
For subsequent iterations l > 1 , R q encodes the goal of achieving the termination condition of a qo , based on its termination condition (Line 4).
Given R q , our algorithm learns a policy that maximizes R q using the options available at the current level l (Line 6).
For l = 1 , we assume there is a single option template A 0 o = { a qo } ; the corresponding policy ⇡ qo aims to achieve the goal in the original environment E , so we refer to the resulting option
o q
=
( a qo , ⇡ qo ) as the top-level option .
For l > 1 , ⇡ qo implements an option template a qo [-
-]2 A l  1 o at level l  1 using the option templates available at the current level l .
Importantly, this process leverages policies learned so far to generate initial states from which to learn ⇡ qo .
Once we have learned ⇡ qo , we add the resulting option
o q
=
( a qo , ⇡ qo ) to D .
Recall that the termination condition for an option starting at state s 0 is T ( s 0 , k ⇤ )
=
F o ( s k ) _ ( k > k ⇤ ) .
The goal of training ⇡ qo is to satisfy this condition with high probability.
Hence, when training policy ⇡ qo for option a qo , we choose the reward function R q to be F o , and additionally restrict the length of each episode to k ⇤ time-steps.
Learning option policies.
Next, we describe LearnOptionPolicy, which Algorithm 2 uses to learn a policy for option template a qo .
For simplicity, we denote the current environment by E , the current reward function by R , the option template by a o , and the target policy by ⇡ o .
The goal of ⇡ o is to achieve the termination condition T for a o
= ( I, T, P ) from initial states s 2 I . [Figure]
One challenge is sampling an initial state s 2 I from which to train ⇡ o .
To do so, this subroutine leverages access to the previously learned options D ; it uses options in D until it arrives at a state s 2 I where a o is called.
In more detail, it samples s by executing the top-level option in D until it reaches a state where a o is called.
In general, executing an option o either relies on executing its policy ⇡ o in D if it exists, or by executing its “teleport” functionality.
Our algorithm keeps track of the execution of options for which ⇡ o exists using a stack, which is initialized with the top-level option (Line 3-5).
The structure is visualized in Figure 2.
Then, while executing the current option o , it obtains the next option o 0 to execute, which is processed in one of three ways:
• If o 0 is the option for a o (i.e., it has a matching initial state, termination condition, and a current policy ˜ ⇡
o , checked by IsOption on Line 10), then it takes steps to train ⇡ o (Line 11); as described
below. • If o 0 does not have an implementation (checked by HasImplementation on Line 12), then it teleports—i.e., we sample s ⇠ P (Line 13).
• Otherwise, o 0 has an implementation, so it pushes o 0 onto the stack (Line 15).
Finally, once we are at a state s where a o is called, the subroutine StepAndTrain takes control and trains ⇡
o using a standard reinforcement learning algorithm—i.e., by collecting rewards from s and taking a policy gradient step once it reaches a terminal state or hits the timeout of k ⇤ steps.
Then, the current episode continues from the state reached.
Once the current episode reaches a ﬁnal state, then the algorithm continues to the next episode.
This process continues until the average reward of ⇡ o exceeds a threshold 1   , or until a maximum number of episodes L is reached.
Experiments 4.1 Experiments on Planning tasks in the Craft Environment
Description of environment: The craft environment [14, 15] is a 2D world based on the Minecraft game, where an agent has to complete various hierarchical tasks with sparse rewards.
The environment is represented by a 12 ⇥ 12 grid with cells containing raw resources ( e.g., WOOD, IRON ), crafting areas ( e.g., WORKBENCH ), obstacles ( e.g., STONE, WATER ) and valuable items (GOLD or
GEM).
There are four move actions ( up, down, left, right ) and a special USE action.
To grab an item, the agent moves to a neighboring cell and applies the USE action.
Table 1 shows the hierarchical arrangement of the get gem task; the details for get gold are in the Appendix.
Option template at each environment: Our option templates (in Table 1) capture the hierarchy of task-dependencies in the environment.
For example, it is easier to get GEM when the agent has access to AXE (see Table 1).
Thus, at the topmost level the agent can “ask” for an AXE , and via primitive actions, use it to break stone and get GEM to realize the importance of an AXE .
Implementation:
In each task of each learning level, we use a vanilla actor-critic algorithm [16] with a network with one hidden layer of 100 neurons for both the actor and critic.
The input to the network is a feature vector consisting of one-hot encodings of the items in each cell in a 5 ⇥ 5 grid Task Policy Sketch get wood
[Table]
around the agent along with a one-hot encoding of its inventory.
At the topmost level, the agent is given a reward of 1 if and only if GOLD or GEM are obtained.
After training, we obtain a actor network for individual tasks at different learning levels.
Comparison with option-value iteration: We implement option value iteration [2] as a comparison with standard hierarchical learning.
In option value iteration, we learn options bottom-up, using options learnt at a lower level to accomplish sub-tasks of options at the level above.
That is the agent learns to implement lower level sub-tasks ﬁrst before it learns how to use them.
We plot average reward as a function of episodes for each sub-task of get gem in Figure 4; for get gold the
Figure is in Appendix A.2.
Figure 4 shows that option templates obtain higher average rewards than option-value iteration at all levels (detailed discussion in Appendix A.2).
[Table]
ten random seeds) to train an agent to solve the get gem task via option templates and curriculum learning [14].
Comparison with curriculum learning [14]:
Additionally, we compare our method with the curriculum learning algorithm employed in Andreas et al.
[14] and observe a 100 fold decrease in the number of episodes required to train an agent for the get gem and get gold tasks (see Table 2 for get gem and Appendix for get gold ).
For the proposed method, we report the total episodes it requires for average reward to stay consistently above 0 . 8 .
Averaged over ten runs for different random seeds.
The calculation of the total episodes for each task, includes all the sub-tasks in its hierarchy.
For instance, the episodes for get gem completion via option templates include episodes required for completing make axe , make stick , get wood and get iron .
The latter two tasks are straightforward, and not included in the table.
4.2 Experiments on Manipulation Tasks in the Fetch and Stack Environment
Description of environment:
This continuous action space environment, introduced in Lanier [17], consists of a robotic arm and a platform with blocks placed on it in random positions; see Figure 3b.
The goal is for the arm to move, lift, and release the colored blocks in the location and stacking order speciﬁed by the corresponding color-coded spheres.
The action consists of torques for 3 degrees-offreedom actuation, and an on-off control input opens and closes the gripper.
The environment offers a sparse reward for each block placed at the goal location. [Table] [Table]
[Table] [Table] [Table] Learning levels and option templates: We introduce two learning levels with tasks and option templates described in Table 3.
In this environment, the primitive actions are not continuous-space actions.
Instead, at the lowest level (level 2 ), we expose the agent to options which are implemented with proportional feedback controllers [18].
Such simple primitives have the ability to improve exploration and can be easily transferred among different learning scenarios.
Further, the hierarchical structure improves the agent’s learning speed as compared to a case where the agent has to learn to use the options at the bottom level ( 2 ) directly (see Appendix A.3 for more details).
Implementation: We use a standard DQN [12] for each level with four hidden layers ofneurons each.
The inputs to the agent are the 3D coordinates of the different blocks, their goals, and the states of the gripper arm.
The agent has 150 and 200 steps for stacking three and four blocks, respectively, in the correct order.
We also supply demonstrations to speed up learning.
Baseline: We consider a baseline that directly exposes the agent to all option templates, at levelof the hierarchy.
Figure 3b shows the average rewards as a function of episodes.
As can be seen, the baseline only recieves one-third reward since it only succeeds in pushing the bottom block to its location.
It does not learn to stack even after twice the number of learning steps as our method.
This demonstrates the challenge in learning longer duration tasks without a higher level guiding policy.
Comparison with Learning with Demonstrations [1]: In learning with demonstrations [1], the authors use demonstration traces, by combining behavioral cloning along with a Q-ﬁlter mechanism to speed-up the learning.
This allows them to evade an expensive exploration phase in the early stages of the learning.
This method takes upwards of 3 . 5 ⇥ 10 8 and 8 ⇥ 10 8 timesteps to learn stacking of three and four blocks respectively which is three orders of magnitude larger than our average learning time of 4 . 5 ⇥ 10 5 and 6 ⇥ 10 5 timesteps respectively (from 5 random seeds).
4.3 Experiments on Multi-Robot Tasks in the GFootball Environment
Description of environment: The Google football (gfootball) environment [19] is an 11 vs.game of soccer.
The opponents are controlled by an inbuilt game engine with game-play at three levels of difﬁculty (easy, medium, and hard).
An RL agent can control up to 11 players on the left team.
The agent provides each player with one of 19 actions such as a direction to move ( e.g. top, top-right, bottom-left, etc. )
, type of pass (short, long or high), shoot, or toggles for dribbling and sprinting.
We provide more details in Appendix A.4.
Learning levels and option templates: We create two learning levels with option templates as given in Table 4.
The primitive actions in this environment are defend , maintain ball possession , charge to the opponent’s goal and shoot which are options implemented with simple planers and open-loop controllers.
Similarly, we implement option templates at level 1 (see Appendix A.4).
Implementation: We train a standard DQN [12] for each level.
The input to every network is a 139 element vector consisting of left and right team states, ball state, score, and one-hot encodings of ball ownership and game mode.
For level 1, for all levels of difﬁculty, we use a network withhidden layers of 500 neurons each.
For level 2, we use a network with 5, 6, and 7 hidden layers for easy, medium and hard respectively where each hidden layer has 500 neurons.
Baseline: We consider a baseline where the agent is directly exposed to all the primitive options.
From Figure 5c, we ﬁnd that the baseline is unable to compete with option template learning even after twice the number of steps of our method (numerical values of Figure 5c are in Appendix A.4).
The error bars represent standard deviation over 5 random seeds.
Comparison with IMPALA and DQN from Kurach et al.
[19]: We compare our agent with the agents in Kurach et al.
As can be seen in Figure 5c, option template learning can achieve similar (in easy) or better (in medium and hard) performance than the IMAPALA and DQN agents in two orders of magnitude fewer steps.
Related Work
We present a detailed discussion on related work in Appendix A.1 and summarize closely relevant literature here.
On expert help via primitives or skills : Various recent works utilize expert help in the form of primitives or skills [4, 5, 6, 7, 8, 9, 20].
This takes place via parameterized action spaces [4], stitching together independent task schemas (or skills) [5, 6, 9, 20] or learning parameters of action primitves [7, 9, 8].
In all of these cases, learning takes place within the traditional hierarchical framework, i.e. , bottom-up (see Figure 2), where a sub-task is learnt before the policy that uses it.
Our method proposes a framework to learn top-down instead with large-improvements in sampleefﬁciency over traditional bottom-up learning.
With minimal changes in the implementation of their primitives and skills, the above diverse strategies can also beneﬁt with shorter learning times by utilizing our framework to learn top-down.
On expert help with humans-in-the-loop : Other approaches to expert intervention include explicit help with humans-in-the-loop required throughout training [21, 22].
In contrast, in our method, human effort is only required at the beginning to create the option template hierarchies.
On exploration in high-dimensional tasks : An alternative approach to counter the need for exploration in high dimensional tasks is through the use of demonstrations [1] for long horizon planning.
The intuition is that introducing a degree of behavioral cloning of expert demonstrations helps reduce the amount of exploration the agent has to perform.
A different approach known as Hindsight Experience Replay (HER) [23] incorporates the goal information into the state, using a failed terminal state as an alternative goal to reward the transitions leading to it.
These methods are orthogonal to our approach.
By themselves, they are unable to achieve the large degree of reduction in sample-complexity seen with our framework.
Yet, alongside expert help, they further improve sample-efﬁciency as seen in our experiments.
Limitations and Conclusion
We have proposed an approach that incorporates option templates into reinforcement learning.
Our experiments show that this strategy can drastically reduce sample complexity by implementing teleportation.
This can be a potential limitation.
In simulation, implementing teleportation is typically straightforward.
Policies may be trained in simulation before being deployed on a real robot.
For real-world environments, there are several strategies for implementing teleportation.
First, for challenging skills such as grasping an object, teleportation can be implemented via a temporary crutch that simpliﬁes the skill.
For example, in the “sticky-mittens” experiments (a key motivation for our work); the analog for a grasping robot would be to attach velcro to its grippers and to the objects to make them easy to pick up.
Second, teleportation can be implemented via a handcrafted policy that eventually achieves the goal but possibly in a suboptimal way.
For instance, teleportation in the
Fetch and GFootball environments are implemented using handcrafted policies.
While these policies are used to train the RL policy, the RL policy eventually signiﬁcantly outperforms them. Acknowledgements
This work was supported in part by ARO W911NF-20-1-0080 and AFRL and DARPA FA875018-C-0090.
Any opinions, ﬁndings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views of the Air Force Research Laboratory (AFRL), the Army Research Ofﬁce (ARO), the Defense Advanced Research Projects Agency (DARPA), the Department of Defense or the United States Government.