Enhanced Meta Reinforcement Learning using Demonstrations in Sparse Reward Environments
Anonymous Author(s)
AffiliationAddress email
Abstract
Meta reinforcement learning (Meta-RL) is an approach wherein the experiencegained from solving a variety of tasks is distilled into a meta-policy.
The metapolicy, when adapted over only a small (or just a single) number of steps, is ableto perform near-optimally on a new, related task.
However, a major challenge toadopting this approach to solve real-world problems is that they are often associatedwith sparse reward functions that only indicate whether a task is completed partiallyor fully.
We consider the situation where some data, possibly generated by a suboptimal agent, is available for each task.
We then develop a class of algorithmsentitled Enhanced Meta-RL using Demonstrations (EMRLD) that exploit thisinformation—even if sub-optimal—to obtain guidance during training.
We showhow EMRLD jointly utilizes RL and supervised learning over the offline data togenerate a meta-policy that demonstrates monotone performance improvements.
We also develop a warm started variant called EMRLD-WS that is particularlyefficient for sub-optimal demonstration data.
Finally, we show that our EMRLDalgorithms significantly outperform existing approaches in a variety of sparsereward environments, including that of a mobile robot.
Introduction
Meta-Reinforcement Learning (meta-RL) is an approach towards quickly solving similar tasks whileonly gathering a few samples for each task.
The fundamental idea behind approaches such as thepopular Model Agnostic Meta-Learning (MAML) [5] is to determine a universal meta-policy bycombining the information gained over working on several tasks.
This meta-policy is optimized in amanner such that, when adapted using a small number of samples gathered for a given task, it is ableto perform near optimally on that specific task.
While the dependence of MAML on only a smallnumber of samples for task-specific adaptation is very attractive, its also means that these samplesmust be highly representative of that task for meaningful adaptation and meta-learning.
However,in many real-world problems, rewards provided are sparse in that they might only provide limitedfeedback.
For instance, there might be a reward only if a robot gets close to designated way point,with no reward otherwise.
Hence, MAML is likely to fail on both counts of task-specific adaptation and optimization of the meta-policy over tasks in sparse reward environments.
Making progress towards learning a viable meta-policy in such settings is challenging without additional information.
Many real-world tasks are associated with empirically determined polices used in practice.
Such polices could be inexpert, but even limited demonstration data gathered from applying these policiescould contain valuable information in the sparse reward context.
While the fact that the policy generating the data could be inexpert suggests that direct imitation might not be optimal, supervised learning over demonstration data could be used for enhancing adaptation and learning.
How best should we use demonstration data to enhance the process of meta-RL in the sparse reward setting
Our goal is a principled design of a class of meta-RL algorithms that can exploit demonstrations frominexpert policies in the sparse reward setting.
Our general approach follows two-step algorithmslike MAML that employ: (i) Task-specific Adaptation: execute the current meta-policy on a taskand adapt it based on the samples gathered to obtain a task-specific policy, and (ii) Meta-policy
Optimization: execute task-specific policies on an ensemble of tasks to which they are adapted, anduse the samples gathered to optimize the meta-policy from whence they were adapted.
Our key insightis that we can enhance RL-based policy adaptation with behavior cloning of the inexpert policy toguide task-specific adaptation in the right direction when rewards are sparse.
Furthermore, executionof such an enhanced adapted policy should yield an informative sample set that indicates how bestto obtain the sparse rewards for meta-policy optimization.
We aim at analytically and empiricallycapturing this progression of policy improvement starting from task-specific policy adaption to theultimate meta-policy optimization.
Thus, as long the inexpert policy has an advantage, we must beable to exploit it for meta-policy optimization.
Our main contributions are as follows.
(i)
We derive a policy improvement guarantee result for
MAML-like two-step meta-RL algorithms.
We show that the inclusion of demonstration data canfurther increase the policy improvement bound as long as the inexpert policy that generated the datahas an advantage.
(ii)
We propose an approach entitled Enhanced Meta-RL using Demonstrations(EMRLD) that combines RL-based policy improvement and behavior cloning from demonstrationsfor task-specific adaptation.
We further observe that directly applying the meta-policy to a newsparse-reward task sometimes does not yield informative samples, and a warm-start to the meta-policyusing the demonstrations significantly improves the quality of the samples, resulting in a variantthat we call EMRLD-WS.
(iii)
We show on standard MuJoCo and two-wheeled robot environmentsthat our algorithms work exceptionally well, even when only provided with just one trajectory ofsub-optimal demonstration data per task.
Additionally, the algorithms work well even when exposedonly to a small number of tasks for meta-policy optimization.
(iv) Our approach is amenable to avariety of meta-RL problems wherein tasks can be distinguished across rewards (e.g., whether forwardor backward motion yields a reward for a task) or across environment dynamics (e.g., the amount ofenvironmental drift that a wheeled robot experiences changes across tasks).
To illustrate the versatilityof EMRLD, we not only show simulations on different continuous control multi-task environments,but also demonstrate its excellent performance via real-world experiments on a TurtleBot robot [2].
We include videos of the robot experiments and code with the supplementary materials.
Related Work:
Here, we provide a brief overview of the related works.
We leave a more thoroughdiscussion on related works to the Appendix.
Meta learning: Basic ideas on the meta-learning framework are discussed in [11, 21, 4].
Modelagnostic meta-learning (MAML) [5] introduced the two-step approach described above, and canbe used in the supervised learning and RL contexts.
However, in its native form, the RL variant of
MAML can suffer from issues of inefficient gradient estimation, exploration, and dependence on arich reward function.
Among others, algorithms like ProMP [19] and DiCE [7] address the issue ofinefficient gradient estimation.
Similarly, E-MAML [1, 20] and MAESN [8] deal with the issue ofexploration in meta-RL.
PEARL [17] takes a different approach to meta-RL, wherein task specificcontexts are learnt during training, and interpreted from trajectories during testing to solve the task.
HTR [15] relabels the experience replay data of any off-policy algorithm such as PEARL [17] toovercome exploration difficulties in sparse reward goal reaching environments.
Different from thisapproach, we use demonstration data to aid learning and are not restricted to goal reaching tasks.
RL with demonstration: Leveraging demonstrations is an attractive approach to aid learning [9, 23,14].
Earlier work has incorporated data from both expert and inexpert policies to assist with policylearning in sparse reward environments [14, 10, 23, 12, 18].
In particular, [10] utilizes demonstrationdata by adding it to the replay buffer for Q-learning, [16] proposes an online fine-tuning algorithm bycombining policy gradient and behavior cloning, while [18] proposes a two-step guidance approachwhere demonstration data is used to guide the policy in the initial phase of learning.
Meta-RL with demonstration: Meta Imitation Learning [6] extends MAML [5] to imitationlearning from expert video demonstrations.
WTL [25] uses demonstrations to generate an explorationalgorithm, and uses the exploration data along with demonstration data to solve the task.
GMPS [13]extends MAML [5] to leverage expert demonstration data by performing meta-policy optimizationvia supervised learning.
Closest to our approach are GMPS [13] and Meta Imitation Learning [6], andwe will focus on comparisons with versions of these algorithms, along with the original MAML [5].
Our work differs from prior work on meta-RL with demonstration in several ways.
First, existingworks assume the availability of data generated by an expert policy, which severely limits their abilityto improve beyond the quality of the policy that generated the data.
This degrades their performancesignificantly when they are presented with sub-optimal data generated by an inexpert policy thatmight be used in practice.
Second, these works use demonstration data in a purely supervised learningmanner, without exploiting the RL structure.
We use a combination of loss functions associated with
RL and supervised learning to aid the RL policy gradient, which enables our approach to utilizeany reward information available.
This makes it superior to existing work in the sparse rewardenvironment, which we illustrate in several simulation settings and real-world robot experiments.
Preliminaries
A Markov Decision Processes (MDP) is typically represented as a tuple < S , A , R, P, γ, ρ > , where
S is the state space, A is the action space, R : S ×A → R is the reward function, P : S ×A → ∆( S )is the transition probability function, γ is the discount factor, and ρ ∈ ∆( S ) is the initial statedistribution.
The value function V π and the state-action value function Q π of a policy π are definedas
V π
( s ) =
E [ (cid:80) ∞ t =0 γ t R ( s t , a t ) | s 0 = s ] and Q π ( s, a ) =
E [ (cid:80) ∞ t =
γ t R ( s t , a t ) | s 0 =
s, a 0 = a ] .
The advantage function A π is defined as A π ( s, a ) =
Q π ( s, a ) − V π ( s ) .
A policy π generates atrajectory τ where τ = ( s 0 , a 0 , s 1 , a 1 , . . . )
, s 0 ∼ ρ, a t ∼ π ( s t , · ) , s t +1 ∼ P ( ·| s t , a t ) .
Since therandomness of τ is specified by P and π , we denote it as τ ∼ ( P, π ) .
The goal of a reinforcement
[Figure]
learning algorithm is to learn a policy that maximizes the expected infinite horizon discounted rewarddefined as J ( π ) =
E τ ∼ ( π,P ) [ (cid:80) ∞ t =
γ t R ( s t , a t )] .
It is easy to see that J ( π ) =
E s 0 ∼ ρ
[ V π ( s 0 )] .
The discounted state-action visitation frequency of policy π is defined as d π ( s, a ) =
(1 −γ ) (cid:80) ∞
t =0 γ t P ( s t = s, a t = a ) , s 0 ∼ ρ .
The discounted state visitation frequency is defined asthe marginal d π ( s ) =
(cid:80) a ∈A d π ( s, a ) .
It is straightforward to see that d π ( s, a )
=
π ( s, a ) d π ( s ) .
The total variation (TV) distance between two distributions p and q is defined as D TV ( p, q ) =(1 / 2) (cid:80) x | p ( x ) − q ( x ) | , the average TV distance between two policies π 1 and π 2 , averaged w.r.t.
thed π π is defined as D π TV ( π 1 , π 2 )
=
E s ∼ d π
[ D TV ( π 1 , π 2 )] .
Gradient-based meta-learning: The goal of a meta-learning algorithm
is to learn to performoptimally in a new (testing) task using only limited data, by leveraging the experience (data) fromsimilar (training) tasks seen during training.
Gradient-based meta-learning algorithms achieve thisgoal by learning a meta-parameter which will yield a good task specific parameter after performingonly a few gradient steps w.r.t.
the task specific loss function using the limited task specific data.
Meta-learning algorithms consider a set of tasks T with a distribution p over T .
Each task i ∼ p ( T )is also associated with a data set D i , which is typically divided into training data D tr i used for taskspecific adaptation and validation data D val i used for meta-parameter update.
The objective of thegradient-based meta-learning is typically formulated asmin
[Equation]
where L i is the loss function corresponding to task i
and α is the learning rate.
Here, θ i ( θ ) =
θ − α ∇ θ L i ( θ, D tr i ) is the task specific parameter obtained by one step gradient update starting fromthe meta-parameter θ , and the goal is to find the best meta-parameter which will minimize the metaloss function
L ( θ ) =
E i ∼ p ( T )
[ L i ( θ i ( θ ))]
Gradient-based meta-reinforcement learning: The gradient-based meta-learning framework isapplicable both in supervised learning and reinforcement learning.
In RL, each task i correspondsto an MDP with task specific model P i and reward function R i .
We assume that the state-actionspaces are uniform across the tasks, thus ensuring the first level of task similarity.
Task specific data
D i is the trajectories
τ i,m = ( s m 0 , a m 0 , s m 1 , a m 1 , . . . ) , s 0 ∼ ρ, a t ∼ π ( s t , · ) , s t +1 ∼ P
i ( ·| s t , a t ) for1 ≤ m ≤ M generated according to some policy π .
Since the randomness of the trajectory τ i,mis specified by P i and π , we denote it as τ i,m ∼ ( P i , π ) .
We consider the function approximationsetting where each policy π is represented by a function parameterized by θ ⊂ Θ and is denoted asπ θ .
The task specific loss in meta-RL is defined as L i RL ( θ )
= − J i ( π θ ) .
The gradient ∇ θ L i RL ( θ ) canthen be computed using policy gradient theorem.
The standard meta-RL training is done follows.
A task i ∼ p ( T ) (usually a batch of tasks) is sampledat each iterate k of the algorithm.
Now, starting with meta-parameter θ k , the training data D tr i for task adaptation is generated as the trajectories ( τ i,m )
Mm =1 , where τ i,m ∼ ( P i , π θ k ) , and the updatedparameter θ i,k for task i is computed by policy gradient evaluated on D tr i .
The validation data D val iis then collected as trajectories ( τ i,m )
Mm =1 , where τ i,m ∼ ( P i , π θ i,k ) , and the meta-parameter θ kis updated by policy gradient evaluated on D val i .
In the next section, we will introduce a modifiedapproach which will leverage the demonstration data for task adaptation and meta-parameter update.
Meta-RL using Demonstration Data
Most gradient-based meta-RL algorithms learn the optimal meta-parameter and the task-specificparameter from scratch using on-policy approaches.
These algorithms exclusively rely on the rewardfeedback obtained from the training and validation data trajectories collected through the on-policyroll-outs of the meta-policy and task-specific policy.
However, in RL problems with sparse rewards,a non-zero reward is typically achieved only when the task is completed or near-completed.
Insuch sparse rewards settings, trajectories generated according to a policy that is still learning maynot achieve any useful reward feedback, especially in the early phase of learning.
In other words,since the reward feedback is zero or near-zero, the policy gradient will also be similar, resulting innon-meaningful improvement in the policy.
Hence, standard meta-RL algorithms such as MAML,which rely crucially on reward feedback, will not be able to make much progress towards learning avaluable task-specific or meta-policy in sparse reward settings.
Learning the optimal control policy in sparse reward environments has been recognized as a challenging problem even in the standard RL setting, since most state-of-the-art RL algorithms fail tolearn any meaningful polices even after a large number of training episodes [16, 12, 18].
One widelyaccepted approach to overcome this challenge is known as learning from demonstration , whereindemonstration data obtained from an expert [16] or inexpert policy [12, 18] is used to aid onlinelearning.
The intuitive idea is that, even though the demonstration data does not contain any rewardfeedback, it can be used to guide the learning agent to reach non-zero reward regions of state-actionspaces.
This guidance, usually in the direction of the goal/target, is achieved by inferring somepseudo reward signal through supervised learning approaches using demonstration data.
Can we enhance the performance of meta-RL algorithms in sparse reward environments by usingdemonstration data from sub-optimal experts?
Meta-RL in sparse reward environments is significantlymore challenging than that of the standard RL setting.
This is because the reward feedback servesthe dual objectives of adapting the meta-parameter to specific tasks and for updating the metaparameter itself.
We note that demonstration data helps with both of these objectives.
Firstly, useof demonstration data to guide task-specific adaptation becomes important because adaptation isachieved in one or a few gradient steps, and policy resulting from each adaptation step might notachieve meaningful reward in a sparse reward setting.
Secondly, making use of demonstration datafor meta-parameter update is equally important because of the role of meta-policy as a rewardyielding exploratory policy.
Intuitively, the meta-policy should yield trajectories that reach in thevicinity of the reward-achieving region of the state-action spaces.
This does not happen in sparsereward environments.
However, using the guidance from demonstration data, the task-specific policyobtained after the task adaptation may be able to generate trajectories that will reach within thereward-achieving region resulting in performance acceleration of the meta-policy.
For meta-learning with demonstration, we assume that each task i is associated with demonstrationdata D dem i , which contains a trajectory generated according to a demonstration policy π dem in anenvironment with model P i .
We do not assume that π dem is the optimal policy for task i becausein many real-world applications D dem
i could be generated using an inexpert policy.
Our key ideais to enhance task adaptation using the demonstration data by introducing an additional gradientterm corresponding to the supervised learning guidance loss.
We define the supervised learningloss function for task i as L BC
[Equation] [Equation]
[Equation]
dem i log π θ ( a | s ) .
We note that, though this lossfunction is the same as in behavior cloning (BC), we use it directly in the gradient update insteadof performing a simple warm start.
This approach is known to achieve superior performance thannaive BC warm starting in standard RL problem under the sparse reward setting [16, 12, 18].
Thetask adaptation step at iteration k , starting with meta-parameter θ k is now obtained as
[Equation]
where w rl and w bc are hyperparameters that control the extant to which RL and demonstration datainfluence the gradient.
The next question is: how do we use demonstration data in the meta-parameter update?
One approachis to use only the demonstration data with a supervised learning loss function for updating themeta-parameter as done in [13].
We conjecture that such a reduction to supervised learning willseverely limit the learning capability of the algorithm.
Firstly, if demonstration data is obtainedfrom an inexpert policy, this approach will never be able to achieve the optimal performance.
Thisis because the role of the meta-policy as a reward-yielding exploratory policy will be limited bytrue performance of the inexpert policy.
Secondly, the task-specific policies obtained according to(2) may be able to reach within the reward-yielding region of state-action space as we mentionedbefore.
Hence, the validation data D val i collected through the roll-out of the policies obtained after taskadaptation might contain extremely valuable reward feedback.
Utilizing this data could potentiallyhave a significant impact on improving the learning of the meta-parameter.
Thus, in our approach, weupdate the meta-parameter using the RL loss with policy gradient as follows.
[Equation]
We note that the demonstration data is indeed used in the meta-parameter update implicitly, as itsimpact can be observed in (3).
We found empirically that the double use of the demonstration data,either by adding an additional gradient through a BC loss function, or by replacing D val i with D dem iresults in similar or worse performance than the approach described above.
We now formally present our algorithm called Enhanced Meta-RL using Demonstrations (EMRLD).
do 5: for i ∈ T do 6:
Execute the meta policy π θ k for task i to collect D tr i 7:
Task adaptation: Compute task adapted parameter θ k,i using D tr i and D dem
i according to8:
Collect data D val i by executing adapted policy π θ k,i for meta-policy update 9: end for 10:
Meta-parameter update: Update meta-policy according to 3 using D val i 11: end for
We now present a theoretical justification of why EMRLD should have a superior performance in thesparse reward setting as compared to other gradient based algorithms that do not use demonstrationdata.
First, we introduce some notation.
Let π k
=
π
θ k be the meta-policy used at iteration k of ouralgorithm.
Also, let π k,i be the policy obtained after task-specific adaptation for task i .
Recall that
J i ( π k,i ) is the value of the policy for the MDP corresponding to task i .
Similarly, we can definethe state-action value function and advantage function of policy π k,i for task i as Q π k,i i and A π k,i i ,respectively.
Also, let d π k,i i be the visitation frequency of policy π k,i for task i .
Now, we can definethe value of the meta-policy π k over the ensemble of all tasks as J meta ( π k ) =
E i ∼ p ( T )
[ J i ( π k,i )] .
If the demonstration data has to be useful, it should provide a reasonable amount of guidance.
Inparticular, we would like the task-specific policy adapted using this data to collect feedback thatwould ensure good meta-policy updates, particularly in the initial stages of meta-training.
Since thecapability of the demonstration data to guide adaptation will depend on the demonstration policyπ dem i according to which it is generated, we make the following assumption about π dem i .
Assumption 3.1.
During the initial stages of meta-training, E a ∼ π dem i ( s, · )
[ A π k,i i ( s, a )] ≥ ∆ , for alls ∈ S and i ∈ T .
Assumption 3.1 implies that during the initial stages of meta-training, the demonstration policycan provide a higher advantage on average than the current policy adapted to that task.
This is areasonable assumption, since any reasonable demonstration policy is likely to perform much betterthan an untrained policy in the initial phase of learning.
We also note that a similar assumption wasused in learning from demonstration literature [12, 18]
We now present the performance improvement result for EMRLD.
Theorem 3.2.
Let π k
=
π
θ k be the meta-policy used at iteration k of our algorithm and let π k,i bethe policy obtained after task adaptation in task i .
Let Assumption 3.1 holds for π k .
Then,
[Equation]
Theorem 3.2 presents a lower bound for the meta policy improvement as a sum of two groups ofterms.
Maximizing the first term in group one with a constraint on its second term will ensure a higherlower bound and hence an improvement in the meta-parameter training.
We notice that this is indeedachieved by the TRPO step used in the meta-parameter update.
Hence, this first group is the same forany MAML-type of algorithm.
The advantage of the demonstration data is revealed in the secondgroup of terms.
The term ∆ / (1 − γ ) adds a positive quantity to the lower bound, and this contributionfrom this second group of term can be maximized by minimizing
[Equation]
i
[Equation] [Equation]
However, this minimization is hard to perform in practice because estimating D π k +1 ,i TV requiressampling the data according to π k +1 ,i , and this is not feasible at iteration k .
Hence, in practice,we replace that term by E i
∼T
[ D π dem i TV (cid:0)
π k +1 ,i , π dem
i (cid:1) ] .
This can be easily achieved by including thestandard maximum likelihood objective in the adaptation step.
Thus, EMRLD both exploits theadvantage offered by an RL step, as well as that of behavior cloning for meta-policy optimization.
We can further improve the performance of EMRLD by including a behavior cloning warm startingstep before performing the update (2).
We simplify this warm start to a one step gradient as,θ ˜ k,i
= θ k −
α ∇ θ L BC
i ( θ, D dem i )
| θ = θ k , and then do the task adaptation as in (2) starting with θ ˜ k,i .
We call this version of our algorithm as EMRLD-WS.
Such a warm start is likely to provide moremeaningful samples than directly rolling out the meta-policy to obtain samples for task-specificadaptation.
In the next section, we will see empirically how our design choices for EMRLD and
EMRLD-WS enable them to learn policies that provide higher rewards using only a small amount of(even sub-optimal) demonstration data.
Experimental Evaluation
[Figure]
We evaluate the performance of EMRLD based on whether the meta-policyit generates is a good initial condition for task-specific adaptation in sparsereward environments over (i) Tasks already seen in training and (ii) Newunseen tasks.
We seek to validate the conjecture that in the sparse-rewardsetting, EMRLD should be able to leverage even demonstrations of inexpertpolicies to attain high test performance over previously unseen tasks.
Wedo so with regard to two classes of tasks, namely,
• Tasks that differ in their reward functions: Simulation experiments onPoint2D Navigation [5], TwoWheeled Locomotion [8], and HalfCheetah[24, 22]• Tasks that differ in
the environment dynamics: Real-world experimentsusing a TurtleBot, which is a two-wheeled differential drive robot [2].
4.1 Experiments on simulated environments
Sparse multi-task environments We present simulation results for threestandard environments shown in Figure 1 and described below.
We trainover a small number of tasks that differ in their reward functions.
Wegenerate unknown tasks for test by randomly modifying the reward function.
Point2D Navigation is a 2 dimensional goal-reaching environment.
Thestates are the ( x, y ) location of the agent on a 2D plane.
The actions are appropriate 2D displacements( dx, dy ) .
Training tasks are defined by a fixed set of 12 goal locations on a semi-circle of radius2 .
The agent is given a zero reward everywhere except when it is a certain distance near the goallocation, making the reward function highly sparse .
Within a single task, the objective of the agent is to reach the goal location in the least number of time steps starting from origin.
Test tasks aregenerated by sampling any point on the semicircle as the goal.
TwoWheeled Locomotion environment is a goal-reaching with sparse rewards, similar to Point2D
Navigation.
However, the robot is constrained by the permissible actions (limits on angular and linearvelocity) and trajectories feasible based on the turning radius of the robot.
Here, our training tasks area fixed set of 24 goal locations on a semi-circle of radius 2 , while test goals are sampled randomly.
Further details on state-space and dynamics are provided in the Appendix.
HalfCheetah Forward-Backward consists of two tasks in which the HalfCheetah agent learns to eithermove in the forward (task 1) or backward (task 2) directions with as high velocity as possible.
Theagent gets a reward only after it has moved a certain number of units along the x-axis in the correctdirection, making the rewards sparse.
Training and test are under the same two tasks.
2 X
Y  2 X
Y
Optimal data and sub-optimal data We provide a limited amount of demonstration data in the form of just onetrajectory per task for guidance .
Optimal data consistsof ( s, a ) transitions generated by an expert policy trainedusing TRPO.
Sub-optimal data is generated by an inexpert,partially trained TRPO policy with induced action noiseand truncated trajectories as shown in Figure 2.
Baselines We compare the performance of our algorithmagainst the following gradient based meta-reinforcement learning algorithms: (i) MAML: [5] The standard MAML algorithm for meta-RL (ii) Meta-BC: A variant of [6]; this is a supervisedlearning/behavior cloning version of MAML, where the maximum likelihood loss is used in theadaptation as well as the meta-optimization steps.
(iii) GMPS:
Guided meta policy search [13], whichuses RL for gradient based adaptation, and uses demonstration data for supervised meta-parameteroptimization.
The implemention of our algorithms and baselines is based on a publicly availablemeta-learning code base [3] licensed under the MIT License.
[Figure]
Performance with optimal demonstration data: We illustrate the training and testing performanceof the different algorithms trained and tested with optimal data in Figure 3.
The top row of Figure 3shows the average adapted return across training tasks of the meta-policy during training iterations.
The bottom row of Figure 3 shows the average return of the trained meta-policy adapted acrosstesting tasks over adaptation steps.
We see that our algorithms out perform the others by obtaining thehighest average return, and are able to quickly adapt to testing tasks with just one adaptation step andone trajectory of demonstration data.
Additionally, our algorithms demonstrate a nearly-monotone
[Figure]
improvement in average return demonstrating stable learning.
Meta-BC fails and has unstable training performance as the amount of demonstration data available per task is very small.
Training over onlya small number of tasks further hampers the performance of Meta-BC.
MAML and GMPS fail to learn due to sparsity of the environment as the purely RL adaptation step incurs almost zero reward,and hence, negligible learning signal.
Furthermore, GMPS is hampered in the meta update step dueto availability of only a small amount of demonstration data per-task.
Performance with sub-optimal demonstration data: EMRLD uses a combination of RL andimitation, which is valuable when presented with sub-optimal demonstrations.
For the Point2D
Navigation environment, we collect sub-optimal data for each task using a partially trained agentwith induced action noise, and truncate the trajectories short of the reward region.
Hence, pureimitation cannot reach the goal.
For the TwoWheeled Locomotion environment, we collect data in asimilar fashion for all tasks, but remove state-action pairs at the beginning of each trajectory.
Sincethe first few state-action pairs contain information on how to orient the two-wheeled agent towardsthe goal, this truncation eliminates the possibility of direct imitation being successful.
Similarly, in
HalfCheetah we use a partially trained policy and truncate trajectories before they reach the rewardbearing region.
Figure 4 illustrates that EMRLD outperforms all the baselines and is quickly ableto adapt to unseen tasks, emphasizing the benefit of its RL component.
Meta-BC and GMPS
failbecause they are restricted by the optimality of the data, and the absence of crucial informationgreatly impacts their performance.
MAML again fails due to the sparsity of the reward.
We conclude by presenting in Figure 5, sets of trajectories generated during testing in the TwoWheeledLococmotion environment when provided with optimal or sub-optimal demonstration data.
Thevariants of EMRLD clearly outperform the others, showing their strength in the sparse reward setting.
4.2 Real-world Experiments on TurtleBot
We demonstrate the ability of EMRLD variants to adapt to sparse-reward tasks when they differin environment dynamics and have sparse reward feedback.
We do so via performance evaluationin the real-world using a TurtleBot shown in Figure 6 (left).
We first we modify the TwoWheeled
Locomotion sparse reward environment by fixing the goal, and changing the dynamics by inducing aresidual angular velocity which mimics drifts in the environment.
This environmental drift is whatdifferentiates each task.
In other words, for a given task, the environment would cause the robot todrift in some specific unknown direction.
We train on a set of 9 tasks with different angular velocityvalues (i.e., 9 different driving environments).
We use one trajectory of demonstration data per taskcollected using an expert policy trained using TRPO.
Note that all the training and data collectionis done in simulation.
The results are shown in Figure 6 (middle), where we see that the EMRLDvariants clearly outperform the others.
2 X
Y  2 X Y  2 X
Y  2 X 11 2 Y  2 X
Y  2 X
Y (a) EMRLD  2 X
Y (b) EMRLD-WS  2 X
Y (c) MAML  2 X
Y (d) Meta-BC  2 X
Y (e) GMPS
For testing, we consider the environment where the Turtlebot experiences a fixed but unknownresidual angular velocity representing environmental drift.
Thus, we bias the angular velocity controlof the TurtleBot by some amount unknown to the algorithm under test.
We first execute the metapolicy on the TurtleBot in the real world to collect 5 trajectories.
We also provide one trajectoryof simulated demonstration data.
We use these samples to adapt the meta-policy, and execute theadapted task-specific policy on the TurtleBot.
The results are shown in Figure 6 (right), where theorigin is at (0 , 0) and the goal is indicated by a star.
It is clear that the variants of EMRLD are the bestat quickly adapting to the drift in the environment and are successful with just one step of adaptation.
[Figure]
Videos of real-world experiments: TurtleBot vidoes are included in the supplementary materials.
Limitations: We follow a gradient-based meta-RL approach, which requires data collection andgradient computation during testing, as opposed to purely context based meta-RL that only requiresdata.
Hence, our compute footprint for implementation is larger.
Furthermore, our on-policy approachimplies that we do not re-use data gathered over trajectories, and so are less data efficient thanoff-policy approaches.
Thus, we trade-off data re-usability for lower variance during training.
Conclusion
We studied the problem of meta-RL algorithm design for sparse-reward problems, in which demonstration data generated by a possibly inexpert policy is also provided.
Our key observation was thatsimple application of an untrained meta-policy in a sparse-reward environment might not providemeaningful samples, and guidance provided by imitating the inexpert policy can greatly assuage thiseffect.
We first showed analytically that this insight is accurate and that meta-policy improvementmight be feasible as long as the inexpert demonstration policy has an advantage.
We then developedtwo meta-RL algorithms, EMRLD and EMRLD-WS that are enhanced by using demonstration data.
We show through extensive simulations, as well as real world robot experiments that EMRLD is ableto offer a considerable advantage over existing approaches in sparse reward scenarios.
Ethics Statement and Societal Impacts
Our work considers the theory and instantiation of meta-RL algorithms that were trained and testedon simulation and robot environments.
No human subjects or human generated data were involved.
Thus, we do not perceive ethical concerns with our research approach.
While reinforcement learning shows much promise for application to societally valuable systems,applying it to environments that include human interaction must proceed with caution.
This is becauseguarantees are probabilistic, and ensuring that the risk is kept within acceptable limits is a must toensure safe deployments.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] Section
(b) Did you describe the limitations of your work?
[Yes] Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Section
(b) Did you include complete proofs of all theoretical results?
[Yes] Appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Supplementary
Material
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Appendix
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[Yes] Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Appendix
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Section
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Appendix
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]