E FFICIENT META REINFORCEMENT LEARNING VIA META GOAL GENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Meta reinforcement learning (meta-RL) is able to accelerate the acquisition of new tasks by learning from past experience.
Current meta-RL methods usually learn to adapt to new tasks by directly optimizing the parameters of policies over primitive actions.
However, for complex tasks which requires sophisticated control strategies, it would be quite inefﬁcient to directly learn such a meta-policy.
Moreover, this problem can become more severe and even fail in spare reward settings, which is quite common in practice.
To this end, we propose a new meta-RL algorithm called meta goal-generation for hierarchical RL (MGHRL) by leveraging hierarchical actor-critic framework.
Instead of directly generate policies over primitive actions for new tasks, MGHRL learns to generate high-level meta strategies over subgoals given past experience and leaves the rest of how to achieve subgoals as independent RL subtasks.
Our empirical results on several challenging simulated robotics environments show that our method enables more efﬁcient and effective meta-learning from past experience and outperforms state-of-the-art meta-RL and Hierarchical-RL methods in sparse reward settings.
I NTRODUCTION
Deep Reinforcement Learning (DRL) has recently shown a great success on a wide range of tasks, ranging from games (Mnih et al., 2015) to robotics control (Levine et al., 2016; Bengio & LeCun, 2016).
However, for more complex problems with larger state and action spaces or sparse reward settings, traditional DRL methods barely works.
Hierarchical reinforcement learning (HRL) in which multiple layers of policies are trained to learn to operate on different levels of temporal abstraction, has long held the promise to learn such difﬁcult tasks (Dayan & Hinton, 1992; Parr & Russell, 1997;
Barto & Mahadevan, 2003).
By decomposing a complex problem into subproblems, HRL signiﬁcantly reduces the difﬁculty of solving speciﬁc task.
Learning multiple levels of policies in parallel is challenging due to non-stationary state transition functions.
Recent HRL approaches (Nachum et al., 2018; Levy et al., 2019) use states as goals directly, allowing simple and fast training of the lower layer.
Human intelligence is remarkable for their fast adaptation to many new situations using the knowledge learned from past experience.
However, agents trained by conventional DRL methods mentioned above can only learn one separate policy per task, failing to generalize to new tasks without additional large amount of training data.
Meta reinforcement learning (meta-RL) addresses such problems by learning how to learn.
Given a number of tasks with similar structures, meta-RL methods enable agents learn such structure from previous experience on many tasks.
Thus when encountering a new task, agents can quickly adapt to it with only a small amount of experience.
Most current meta-RL methods leverage experience from previous tasks to adapt to new tasks by directly learn the policy parameters over primitive action space.
(Finn et al., 2017; Rakelly et al., 2019).
Such approaches suffer from two problems: (i) For complex tasks which requires sophisticated control strategies, it would be quite inefﬁcient to directly learn such policy with one nonlinear function approximator and the adaptation to new tasks is prone to be inaccurate.
This problem can become more severe in spare reward settings.
(ii) When the task distribution is much wider (riding bicycle as meta-train task and riding motorcycle as meta-test task), these methods can hardly be effective since primitive action execution mechanism is entirely different although they may share a similar high-level strategy.
Moreover, existing current meta-RL methods perform badly in sparse reward settings, which are quite common in real world.
In this paper, we aim at tackling the problems mentioned above by proposing an efﬁcient hierarchical meta-RL method that realizes meta learning high-level goal generation and leaves the learning of low-level policy for independent RL.
Intuitively, this is quite similar to how a human being behaves: we usually transfer the overall understanding of similar tasks rather than remember speciﬁc actions.
Our meta goal-generation framework is built on top of the architecture of PEARL (Rakelly et al., 2019) and a two level hierarchy inspired by HAC (Levy et al., 2019).
Our evaluation on several simulated robotics environments (Plappert et al., 2018) shows the superiority of MGHRL to stateof-the-art meta-RL and hierarchical RL methods in sparse reward settings.
Generally, our contributions are as follows:
• We propose an algorithm that achieves efﬁcient meta reinforcement learning on challenging robotics environments with sparse reward settings and outperforms other leading methods.
• Similar to the way humans leverage past experience to learn new complex tasks, our algorithm focuses on meta learning the overall strategy for different tasks, which provides a much simpler and better way for meta RL comparing with directly learning the detailed solution.
Since we focus on meta goal-generation and leave the low level policy for independent learning, we believe our algorithm can still accelerate the acquisition of new tasks sampled from much wider task distributions.
For example, to learn tasks such as riding bicycles and riding a motorcycle, the two primitive action execution mechanism are entirely different but the two learning process still share similar high-level structures.
Through meta goal-generation learning, we expect our method can still accelerate the acquisition of such tasks.
We leave these for future work to explore.
R ELATED W ORK
Our algorithm is based on meta learning framework (Thrun & Pratt, 1998; Schmidhuber, 1987; Bengio et al., 1991), which aims to learn models that can adapt quickly to new tasks.
Meta learning algorithms for few-shot supervised learning problems have explored a wide variety of approaches and architectures (Santoro et al., 2016; Vinyals et al., 2016; Ravi & Larochelle, 2017).
In the context of reinforcement learning, recurrent (Duan et al., 2016; Wang et al., 2016) and recursive (Mishra et al., 2018) meta-RL methods adapt to new tasks by aggregating experience into a latent representation on which the policy is conditioned.
Another set of methods is gradient-based meta reinforcement learning (Finn et al., 2017; Stadie et al., 2018;
Rothfuss et al., 2019; Xu et al., 2018).
Its objective is to learn an initialization such that after one or few steps of policy gradients the agent attains full performance on a new task.
These methods focus on on-policy meta learning which are usually sample inefﬁcient.
Our algorithm is closely related to probabilistic embeddings for actor-critic RL (PEARL) (Rakelly et al., 2019), which is an off-policy meta RL algorithm.
PEARL leverages posterior sampling to decouple the problems of inferring the task and solving it, which greatly enhances meta-learning efﬁciency.
However, when facing complex tasks that require sophisticated control strategies, PEARL cannot effectively learn a proper meta-policy as we will show in Section 5.
Discovering meaningful and effective hierarchical policies is a longstanding research problem in
RL (Dayan & Hinton, 1992; Parr & Russell, 1997; Sutton et al., 1999; Bacon et al., 2017; Dietterich, 2000).
Schmidhuber 1987 proposed a HRL approach that can support multiple levels.
Multi-level hierarchies have the potential to accelerate learning in sparse reward tasks because they can divide a problem into a set of short-horizon subproblems.
Nachum et al.
2018 proposed HIRO, a 2-level
HRL approach that can learn off-policy and outperforms two other popular HRL techniques used in continuous domains: Option-Critic (Bacon et al., 2017) and FeUdal Networks (Vezhnevets et al., 2017).
Our algorithm is built on Hierarchical actor-critic (Levy et al., 2019), which is a framework that can learn multiple levels of policies in parallel.
Most current HRL works focus on the learning problem in a single task and few of them considers to take advantage of HRL for multi-task or metalearning tasks.
MLSH (Frans et al.) is such a work which also combines meta-RL with Hierarchical RL.
It focuses on meta learning on the low level policy and need to retrain its high level policy when facing new tasks.
In contrast, with the key insight that humans leverage abstracted prior knowledge obtained from past experience, our method focus on meta learning high level overall strategy using past experience and leave the detailed action execution for independent RL.
B ACKGROUND 3. META REINFORCEMENT LEARNING
In our meta learning scenario, we assume a distribution of tasks p ( τ ) that we want our model to adapt to.
Each task correspond to a different Markov decision process (MDP), M i = { S, A, T i , R i } , with state space S , action space A , transition distribution T i , and reward function R i .
We assume that the transitions and reward function vary across tasks.
Meta RL aims to learn a policy that can adapt to maximize the expected reward for novel tasks from p ( τ ) as efﬁciently as possible.
PEARL (Rakelly et al., 2019) is an off-policy meta-reinforcement learning method that drastically improves sample efﬁciency comparing to previous meta-RL algorithms.
The meta-training process of PEARL learns a policy that adapts to the task at hand by conditioning the history of past transitions, which we refer to as context c .
Speciﬁcally, for the i th transition in task τ , c τi = ( s i , a i , r i , s (cid:48) i ) .
PEARL leverages an inference network q φ ( z | c ) and outputs probabilistic latent variable z .
The parameters of q ( z | c ) are optimized jointly with the parameters of the actor π θ ( a | s, z ) and critic
Q θ ( s, a, z ) , using the reparametrization trick (Kingma & Welling, 2014) to compute gradients for parameters of q φ ( z | c ) through sampled z ’s.
3. H IERARCHICAL A CTOR - CRITIC
HAC (Levy et al., 2019) aims to accelerate learning by enabling hierarchical agents to jointly learn a multi-level hierarchy of policies in parallel.
HAC is comprised of two components: a particular hierarchical architecture and a method for learning the multiple levels of policies in parallel given sparse rewards.
The hierarchies produced by HAC have a speciﬁc architecture consisting of a set of nested, goal-conditioned policies that use the state space as the mechanism for breaking down a task into subtasks.
HAC extends the idea of Hindsight Experience Replay (Andrychowicz et al., 2017) by creating two types of hindsight transitions.
Hindsight action transition simulates a transition function that uses the optimal low level policy while hindsight goal transition use the ﬁnal states achieved as the goal state in each step’s transition.
They enable agents to learn multiple policies in parallel using only sparse reward functions.
A LGORITHM 4. T WO - LEVEL HIERARCHY
We set up a hierarchical two-layer RL structure similar to HAC.
The high level network uses policy µ h to generate goals for temporally extended periods in terms of desired observations.
In our task they correspond to the positional features of the gripper.
The low level policy µ l directly controls the agent and produces actions for moving towards the desired goals.
As shown in Figure 1 (a), the high level policy µ h observes the state and produces a high level action (or goal) g t .
Low level policy µ l has at most K attempts of primitive action to achieve g t .
Here, K which can be viewed as the maximum horizon of a subgoal action is a hyperparameter given by the user.
As long as the low level policy µ l run out of K attempts or g t is achieved, this high level [Figure] transition terminates.
The high level policy uses agent’s current state as the new observation and produced another goal for low level policy to achieve.
We use an intrinsic reward function in which a reward of 0 is granted only if the goal produced by high level policy is achieved and a reward of -1 otherwise.
Note that the environment’s return (i.e. whether the agent successfully accomplished the task) will not affect the reward received by the low level policy.
In our evaluation on simulated robotics environments, we use the positional features of the observations as the representation for g t .
A goal g t is judged to be achieved only if the distance between g t and the gripper’s current position s n +1 is less than threshold l .
4. M ETA GOAL - GENERATION FOR H IERARCHICAL REINFORCEMENT LEARNING
One primary motivation for our hierarchical meta reinforcement learning strategy is that, when people try to solve new tasks using prior experience, they usually focus on the overall strategy we used in previous tasks instead of the primitive action execution mechanism.
Most state-of-the-art meta learning methods (Rakelly et al., 2019; Finn et al., 2017) leverage experiences from previous tasks to quickly adapt to the new tasks and directly learn the policy parameters.
However, it can be difﬁcult to meta learn a proper policy that consider both the overall strategy and detailed action execution in some complex tasks.
Using only one level of non-linear function approximator may lead the agents to learning an inaccurate meta-policy when both the overall structure and primitive action execution mechanism are complex.
Moreover, in sparse reward settings which is a common situation in real-world problems, current meta learning algorithms do not perform well enough since their training methods are based on non-hierarchical RL methods like TRPO (Schulman et al., 2015),
SAC (Haarnoja et al., 2018), etc.
These methods suffer from the difﬁculty of effective exploration and the lack of positive update signals.
To address the problem mentioned above, we take advantage of our two-level hierarchy structure and propose a new meta reinforcement learning framework called meta goal-generation for hierarchical RL (MGHRL).
Instead of learning to generate detailed strategy for new tasks, MGHRL learns to generate overall strategy (goals) given past experience and leaves the detailed method of how to achieve the goals for independent RL.
We leverage PEARL framework (Rakelly et al., 2019) and independently train a high level meta-policy which is able to quickly adapt to new tasks and generate proper goals.
Note that off-policy RL method is indispensable in our structure when training high level policy due to its excellent sample efﬁciency during meta-training.
Good sample efﬁciency enables fast adaptation by accumulating experience online, and performs structured exploration by reasoning about uncertainty over tasks, which is crucial to hierarchical parallel training framework.
We leave the low level policy to be trained independently with non-meta RL algorithm using hindsight experience replay mechanism.
In our simulated robotics experiments, the low level policy aims to move the gripper to the desired goal position which can be reused when switching to other tasks.
Thus we only need to train a single set of low-level polices which can be shared and reused across different tasks.
On the other hand, in other situations where the tasks are from different domains, for example, when we use our experience of learning riding bicycle to help us learn how to ride a motorcycle, the primitive action execution mechanism are entirely different.
In this case, we can train low level policy independently on new tasks without using past experience.
Our main insight is that when dealing with entirely new tasks, the primitive action execution mechanism can be entirely different but the general strategy of how to accomplish the new tasks and prior tasks can be similar.
With meta learning on high level policy, our algorithm still greatly accelerate the acquisition of new tasks.
Our high level meta-RL network uses a probabilistic embedding actor-critic framework similar to
PEARL.
The network consists of two parts.
The ﬁrst part is a context encoder which leverages data from a variety of training tasks to learn to infer the value of z from a recent history of high-level experience in the new task, where z functions as a latent probabilistic context variable.
The encoder network parameterized by ϕ takes context (experience) c h as input and output posterior q ϕ
( z | c h ) as a permutation-invariant function (Rakelly et al., 2019) of prior high level experience.
The context c h consists of experience { s, g, r, s (cid:48) } collected using hindsight technique as we will introduce in
Section 4.3.
Then we can sample z from the posterior and compute policy output and Q value conditioned on it.
Through posterior sampling via latent contexts, the high level network can learn to infer new tasks efﬁciently using past experience.
The second part is built on top of soft actor-critic algorithm (Haarnoja et al., 2018).
As we mentioned before, samples from the posterior belief are passed to actor µ hφ ( g | s, z ) and critic Q θ ( s, g, z ) to make predictions of the sampled task.
Note that we treat z as part of the state when we implement with SAC.
The actor and critic are trained to predict optimally given z with batches of transitions drawn uniformly from the entire replay buffer.
The context encoder is optimized using gradients from the critic.
We summarize our meta-training procedure in Algorithm1 and Figure 1 (b).
Concretely, for each training task drawn from task distribution, we sample context and generate hindsight transitions for both levels of hierarchy ( line 4 ∼ 13 ) by performing current policy.
Then we train high level and low level networks with the collected data ( line 16 ∼ 22 ).
4. P ARALLEL TRAINING STRATEGY
Efﬁcient meta reinforcement learning requires parallel training for the two levels of our networks. To achieve parallel training paradigm, there exists two main issues in MGHRL framework. The ﬁrst issue for meta learning hierarchies is that agents need to act randomly to reach their goals and obtain the sparse reward which proves to be quite difﬁcult for both levels. We need other strategies to ensure each level learn effectively in sparse reward settings.
The second issue is the non-stationary problem when we do parallel training for the high level and low level networks.
Whenever low level policy π l changes, the high level transition function is likely to change as well.
Old off-policy experience may exhibit different transitions conditioned on the same goals, making the transition invalid for training.
The same problem occurs when the low level is exploring with some random noise.
Thus in our algorithm, we rewrite the past experience transitions as hindsight action transitions (Andrychowicz et al., 2017), and supplement both levels of hierarchy with additional sets of transitions as was done in HAC.
Hindsight action transition simulates a transition function that uses the optimal low level policy which enables our framework to train both levels in parallel.
It substitutes the action component in high level transition to the next state achieved in low level. If the original high level transition is Require: Batch of training tasks { τ i } i =1 ,...,T from p ( τ ) , maximum horizon K of subgoal action 1: Initialize replay buffers B ih , B il for each training task 2: while not done do 3: for each task τ i do 4:
Initialize high-level context c ih = {} 5: for m=1,...,M do 6:
Sample z ∼ q ϕ ( z | c ih ) 7: g i ← π h ( g | s, z ) 8: for K attempts or until g i achieved do 9:
Gather data using a i ← π l ( a | s, g ) 10:
Generate hindsight action transition, hindsight goal transition and add to B il 11: end for 12:
Generate hindsight transitions, subgoal test transitions and add to B ih 13:
Sample high level context c ih =
s j , g j , r j , s (cid:48)
j } j =1 ,...,N ∼ B ih 14: end for 15: end for 16: for steps in training steps do 17: for each task τ i do 18:
Sample high level context c ih ∼ B ih and RL batch b ih ∼ B ih , b il ∼ B il 19:
Sample z ∼ q ϕ ( z | c ih ) and calculate L hactor ( b ih , z ) , L hcritic ( b ih , z ) , L hKL 20:
Update low level actor and critic network with b il 21: end for 22:
Update high level networks with (cid:80) i
L hactor , (cid:80) i
L hcritic , (cid:80) i
L hKL 23: end for 24: end while [ s t , g t , r t , s t +1 ] , the hindsight action transition will be [ s t , s gt +1 , r t , s t +1 ] , where s gt +1 represents the component vector of next state that matches the goal vector.
The new transition we get is independent of changing or exploring low level policy since it’s always optimal.
We utilize hindsight goal transition and subgoal test transition to further address the problems mentioned before.
Hindsight goal transition is created for both levels.
After at most K attempts is executed, the ﬁnal states achieved is used as the goal state in each step’s transition instead of the original goal state.
And the reward will be updated to reﬂect the new goal state.
Subgoal test transition is meant to compensate for the drawbacks brought by hindsight action transition.
Hindsight action transition prefer the shortest path of goals that have been found but may ignore the range of goals that the low level policy is able to reach.
Thus, subgoal test transition add a penalty of − K to the reward if the goal is not achieved after K attempts by low level policy and set the discount rate to 0 to avoid non-stationary issues.
E XPERIMENTS
We evaluated our algorithm on several challenging continuous control robotics tasks (integrated with OpenAI Gym)
(Plappert et al., 2018), simulated via the MuJoCo physics simulator (Todorov et al., 2012).
Visualizations of these environments are shown in Figure 2.
More details on each environment can be found at https://openai.com/blog/ ingredients-for-robotics-research/ .
Fetch-Reach Fetch has to move the gripper to the desired goal position.
This task is very easy to learn and is therefore a suitable benchmark to ensure that a new idea works at all.
[Figure]
Fetch-Push A box is placed on a table in front of the robot and Fetch has to move a box by pushing it until it reaches a desired goal position.
The robot ﬁngers are locked to prevent grasping.
The learned behavior is usually a mixture of pushing and rolling.
Fetch-Slide A puck is placed on a long slippery table and the target position is outside of the robots reach so Fetch has to hit the puck across a long table such that it slides and comes to rest on the desired goal.
Fetch-PickandPlace Fetch has to pick up a box from a table using its gripper and move it to a desired goal located on the table.
5. E NVIRONMENTAL SETUP
In all our experiments, we compare our algorithm to baselines including PEARL with dense reward,
PEARL with sparse reward and HAC with shared policy.
The last one means we train a shared HAC policy jointly across all meta-train tasks sampled from the whole task distribution.
Note that Rakelly et al.
(2019) has already proved that PEARL greatly outperforms other existing meta RL methods like MAML (Finn et al., 2017), ProMP (Rothfuss et al., 2019) at both sample efﬁciency and ﬁnal performance.
Thus we mainly compare our results with PEARL using its public source code.
In addition, for a fair comparison, we modify the HAC source code with SAC algorithm which are considered to be much powerful than DDPG in the original implementation (Haarnoja et al., 2018), to ensure the consistence to PEARL and MGHRL.
We set the goal space to be the set of all possible positions of the gripper, in which a goal is a 3-d vector.
In the environments, the low level policy of our algorithm aims to move the gripper to the desired goal position.
Such policy won’t change at all when switching to other tasks since the mechanism of moving gripper keeps the same between different tasks.
Thus we use a shared policy trained jointly across all tasks for the low level of MGHRL.
In all four scenarios, we set the maximum low-level horizon K to be 10 and the distance threshold to be 0 .
The high level context data sampler S hc samples uniformly from the most recently collected batch of data, which is recollected every 1000 meta-training steps.
Unlike HAC, we use target networks for both levels, which updates with τ = 0 .
All context encoder, actor and critic neural networks had three hidden layers, with 300 nodes in each layer.
The discount factor was set to γ = 0 .
We use a sparse reward function in which a reward of 0 is granted only if the terminal goal given by the environment is achieved and a reward of -1 otherwise.
The dense reward used in our baseline is a value corresponding to the distance between current position of the box (or gripper in fetch-reach case) and desired goal position.
In all four scenarios, we do our experiments on 50 train tasks and 10 test tasks, where the difference between each task is in the terminal goal position we want the box or gripper to reach.
5. R ESULTS
We evaluate the performance of approaches in the term of the average success rate.
As shown in Figure 3, in Fetch-reach environment which is very easy to learn as we mentioned before, the tested methods except PEARL with sparse reward all reach a ﬁnal performance of 100% success [Figure]
rate. In other three scenarios, MGHRL signiﬁcantly outperforms the other three method in such sparse reward settings.
Our two-level hierarchy and hindsight transitions signiﬁcantly decrease the difﬁculty of meta learning with sparse reward.
As we expected, PEARL performs badly in sparse reward settings.
The original version of PEARL is based on SAC, such non-hierarchical RL method has been proved to perform badly before on challenging tasks with sparse reward settings.
Thus it is reasonable that PEARL, which can be viewed as a meta-version of SAC, performs badly as well in sparse reward settings.
HAC with shared policy generally performs better than PEARL in
Fetch-slide and Fetch-pickandplace environments.
We assume that it is because in our settings since we only change the terminal goals’ positions to create different tasks, thus it is possible that the policy learned from one task will work on other task whose terminal goal positions are very close to previous training ones.
But such method lacks generalization ability and cannot always achieve good performance when tested on varied tasks as shown in our results.
We also compare our method to PEARL with dense reward to demonstrate that MGHRL is able to more efﬁciently and accurately meta learn from past experience.
Shown in Figure 3, generally, our algorithm still outperforms PEARL and adapts to new tasks much more quickly.
In such environments with sophisticated control strategies, directly using PEARL to meta learn a policy that consider both overall strategy and detailed execution would decrease prediction accuracy and sample efﬁciency.
Thus it is better to decompose the meta-RL training process and focus on meta goalgeneration learning.
Moreover, under dense reward settings of these challenging tasks, the critic of
PEARL has to approximate a highly non-linear function that includes the Euclidean distance between positions and the difference between two quaternions for rotations (Plappert et al., 2018).
As our method use a hierarchical structure, learning with the sparse return is much simpler since the critic only has to differentiate between successful and failed states.
C ONCLUSION
In this paper, we have presented a hierarchical meta-RL algorithm, MGHRL, which realizes meta goal-generation and leave the low-level policy for independent RL.
MGHRL aims to more efﬁciently and accurately meta learn from past experience by focusing on learning the overall strategy of tasks instead of learning detailed action execution.
Our experimental results on a range of simulated robotics environments show the superiority of MGHRL over state-of-the-art meta RL and hierarchical RL methods in challenging and practical sparse reward settings.
We believe our work open up many directions in training agents that can quickly adapt to new tasks sampled from much wider distribution efﬁciently.
Currently, we have only conducted experiments on meta learning tasks with relatively narrow task distribution (e.g. different goal positions of the box).
As future work, we expect our algorithm can accelerate the acquisition of entirely new tasks (i.e. using fetch-push and fetch-slide as meta train tasks and using fetch-pickandplace as meta test task) by only meta learning overall strategy and leaving the details of primitive action execution mechanism for further separate low-level policy learning.
Moreover, we note our results on some tasks are still far from perfect.
There is still much work left for future research to improve meta-RL methods’ performance on those tasks.