Enhanced Meta Reinforcement Learning using Demonstrations in Sparse Reward Environments
Abstract
Meta reinforcement learning (Meta-RL) is an approach wherein the experience gained from solving a variety of tasks is distilled into a meta-policy.
The metapolicy, when adapted over only a small (or just a single) number of steps, is able to perform near-optimally on a new, related task.
However, a major challenge to adopting this approach to solve real-world problems is that they are often associated with sparse reward functions that only indicate whether a task is completed partially or fully.
We consider the situation where some data, possibly generated by a suboptimal agent, is available for each task.
We then develop a class of algorithms entitled Enhanced Meta-RL using Demonstrations (EMRLD) that exploit this information—even if sub-optimal—to obtain guidance during training.
We show how EMRLD jointly utilizes RL and supervised learning over the ofﬂine data to generate a meta-policy that demonstrates monotone performance improvements.
We also develop a warm started variant called EMRLD-WS that is particularly efﬁcient for sub-optimal demonstration data.
Finally, we show that our EMRLD algorithms signiﬁcantly outperform existing approaches in a variety of sparse reward environments, including that of a mobile robot.
Introduction
Meta-Reinforcement Learning (meta-RL) is an approach towards quickly solving similar tasks while only gathering a few samples for each task.
The fundamental idea behind approaches such as the popular Model Agnostic Meta-Learning (MAML) [9] is to determine a universal meta-policy by combining the information gained over working on several tasks.
This meta-policy is optimized in a manner such that, when adapted using a small number of samples gathered for a given task, it is able to perform near optimally on that speciﬁc task.
While the dependence of MAML on only a small number of samples for task-speciﬁc adaptation is very attractive, its also means that these samples must be highly representative of that task for meaningful adaptation and meta-learning.
However, in many real-world problems, rewards provided are sparse in that they might only provide limited feedback.
For instance, there might be a reward only if a robot gets close to designated way point, with no reward otherwise.
Hence, MAML is likely to fail on both counts of task-speciﬁc adaptation and optimization of the meta-policy over tasks in sparse reward environments.
Making progress towards learning a viable meta-policy in such settings is challenging without additional information.
Many real-world tasks are associated with empirically determined policies used in practice.
Such policies could be inexpert, but even limited demonstration data gathered from applying these policies could contain valuable information in the sparse reward context.
While the fact that the policy generating the data could be inexpert suggests that direct imitation might not be optimal, supervised ⇤ Equal contribution. 36th Conference on Neural Information Processing Systems (NeurIPS 2022). learning over demonstration data could be used for enhancing adaptation and learning.
How best should we use demonstration data to enhance the process of meta-RL in the sparse reward setting?
Our goal is a principled design of a class of meta-RL algorithms that can exploit demonstrations from inexpert policies in the sparse reward setting.
Our general approach follows two-step algorithms like MAML that employ: (i) Task-speciﬁc Adaptation: execute the current meta-policy on a task and adapt it based on the samples gathered to obtain a task-speciﬁc policy, and (ii) Meta-policy
Optimization: execute task-speciﬁc policies on an ensemble of tasks to which they are adapted, and use the samples gathered to optimize the meta-policy from whence they were adapted.
Our key insight is that we can enhance RL-based policy adaptation with behavior cloning of the inexpert policy to guide task-speciﬁc adaptation in the right direction when rewards are sparse.
Furthermore, execution of such an enhanced adapted policy should yield an informative sample set that indicates how best to obtain the sparse rewards for meta-policy optimization.
We aim at analytically and empirically capturing this progression of policy improvement starting from task-speciﬁc policy adaption to the ultimate meta-policy optimization.
Thus, as long the inexpert policy has an advantage, we must be able to exploit it for meta-policy optimization.
Our main contributions are as follows.
(i)
We derive a policy improvement guarantee result for
MAML-like two-step meta-RL algorithms.
We show that the inclusion of demonstration data can further increase the policy improvement bound as long as the inexpert policy that generated the data has an advantage.
(ii)
We propose an approach entitled Enhanced Meta-RL using Demonstrations (EMRLD) that combines RL-based policy improvement and behavior cloning from demonstrations for task-speciﬁc adaptation.
We further observe that directly applying the meta-policy to a new sparse-reward task sometimes does not yield informative samples, and a warm-start to the metapolicy using the demonstrations signiﬁcantly improves the quality of the samples, resulting in a variant that we call EMRLD-WS.
(iii)
We show on standard MuJoCo and two-wheeled robot environments that our algorithms work exceptionally well, even when only provided with just one trajectory of sub-optimal demonstration data per task.
Additionally, the algorithms work well even when exposed only to a small number of tasks for meta-policy optimization.
(iv) Our approach is amenable to a variety of meta-RL problems wherein tasks can be distinguished across rewards (e.g., whether forward or backward motion yields a reward for a task) or across environment dynamics (e.g., the amount of environmental drift that a wheeled robot experiences changes across tasks).
To illustrate the versatility of EMRLD, we not only show simulations on different continuous control multi-task environments, but also demonstrate its excellent performance via real-world experiments on a TurtleBot robot [2].
We provide videos of the robot experiments and code at https://github.com/DesikRengarajan/EMRLD .
Related Work:
Here, we provide a brief overview of the related works.
We leave a more thorough discussion on related works to the Appendix.
Meta learning: Basic ideas on the meta-learning framework are discussed in [15, 34, 8].
Modelagnostic meta-learning (MAML) [9] introduced the two-step approach described above, and can be used in the supervised learning and RL contexts.
However, in its native form, the RL variant of
MAML can suffer from issues of inefﬁcient gradient estimation, exploration, and dependence on a rich reward function.
Among others, algorithms like ProMP [28] and DiCE [11] address the issue of inefﬁcient gradient estimation.
Similarly, E-MAML [1, 33] and MAESN [12] deal with the issue of exploration in meta-RL.
PEARL [26] takes a different approach to meta-RL, wherein task speciﬁc contexts are learnt during training, and interpreted from trajectories during testing to solve the task.
HTR [23] relabels the experience replay data of any off-policy algorithm such as PEARL [26] to overcome exploration difﬁculties in sparse reward goal reaching environments.
Different from this approach, we use demonstration data to aid learning and are not restricted to goal reaching tasks.
RL with demonstration: Leveraging demonstrations is an attractive approach to aid learning [13, 36, 21].
Earlier work has incorporated data from both expert and inexpert policies to assist with policy learning in sparse reward environments [21, 14, 36, 17, 27].
In particular, [14] utilizes demonstration data by adding it to the replay buffer for Q-learning, [25] proposes an online ﬁne-tuning algorithm by combining policy gradient and behavior cloning, while [27] proposes a two-step guidance approach where demonstration data is used to guide the policy in the initial phase of learning.
Meta-RL with demonstration: Meta Imitation Learning [10] extends MAML [9] to imitation learning from expert video demonstrations.
WTL [40] uses demonstrations to generate an exploration algorithm, and uses the exploration data along with demonstration data to solve the task.
GMPS [20] extends MAML [9] to leverage expert demonstration data by performing meta-policy optimization via supervised learning.
Closest to our approach are GMPS [20] and Meta Imitation Learning [10], and we will focus on comparisons with versions of these algorithms, along with the original MAML [9].
Our work differs from prior work on meta-RL with demonstration in several ways.
First, existing works assume the availability of data generated by an expert policy, which severely limits their ability to improve beyond the quality of the policy that generated the data.
This degrades their performance signiﬁcantly when they are presented with sub-optimal data generated by an inexpert policy that might be used in practice.
Second, these works use demonstration data in a purely supervised learning manner, without exploiting the RL structure.
We use a combination of loss functions associated with
RL and supervised learning to aid the RL policy gradient, which enables our approach to utilize any reward information available.
This makes it superior to existing work in the sparse reward environment, which we illustrate in several simulation settings and real-world robot experiments.
Preliminaries
A Markov Decision Processes (MDP) is typically represented as a tuple < S , A , R, P,  , ⇢ > , where
S is the state space, A is the action space, R : S ⇥ A ! R is the reward function, P : S ⇥ A !  ( S ) is the transition probability function,  is the discount factor, and ⇢ 2  ( S ) is the initial state distribution.
The value function V ⇡ and the state-action value function Q ⇡ of a policy ⇡ are deﬁned as V ⇡ ( s ) =
E [ P 1 t {+=
 +} t R ( s t , a t ) | s 0 = s ] and Q ⇡ ( s, a ) =
E [ P 1 t =
t R ( s t , a t ) | s 0 =
s, a 0 = a ] .
The advantage function A ⇡ is deﬁned as A ⇡ ( s, a ) =
Q ⇡ ( s, a )  V ⇡ ( s ) .
A policy ⇡ generates a trajectory ⌧ where ⌧ = ( s 0 , a 0 , s 1 , a 1 , . . . )
, s 0 ⇠ ⇢ , a t ⇠ ⇡ ( s t , · ) , s t +1 ⇠ P ( ·| s t , a t ) .
Since the randomness of ⌧ is speciﬁed by P and ⇡ , we denote it as ⌧ ⇠ ( P, ⇡ ) .
The goal of a reinforcement learning algorithm is to learn a policy that maximizes the expected inﬁnite horizon discounted reward deﬁned as J ( ⇡ ) =
E ⌧ ⇠ ( ⇡ ,P ) [ P 1 t =
t R ( s t , a t )] .
It is easy to see that J ( ⇡ ) =
E s 0 ⇠ ⇢ [ V ⇡ ( s 0 )] .
The discounted state-action visitation frequency of policy ⇡ is deﬁned as d ⇡ ( s, a ) =
(1   )
P 1 t {+=
 +} t P ( s t = s, a t = a ) , s 0 ⇠ ⇢ .
The discounted state visitation frequency is deﬁned as the marginal d ⇡ ( s ) =
P a 2 A d ⇡ ( s, a ) .
It is straightforward to see that d ⇡ ( s, a ) =
⇡ ( s, a ) d ⇡ ( s ) .
The total variation (TV) distance between two distributions p and q is deﬁned as D TV ( p, q ) = P x | p [Equation]
x )  q [Equation] x ) | , the average TV distance between two policies ⇡ 1 and ⇡ 2 , averaged w.r.t.
the d ⇡ ⇡ is deﬁned as D ⇡ TV ( [Equation] 1 , ⇡
[Equation]
) = E s ⇠ d ⇡ [ D TV ( ⇡ 1 , ⇡ 2 )] .
Gradient-based meta-learning: The goal of a meta-learning algorithm
is to learn to perform optimally in a new (testing) task using only limited data, by leveraging the experience (data) from similar (training) tasks seen during training.
Gradient-based meta-learning algorithms achieve this goal by learning a meta-parameter which will yield a good task speciﬁc parameter after performing only a few gradient steps w.r.t.
the task speciﬁc loss function using the limited task speciﬁc data.
Meta-learning algorithms consider a set of tasks T with a distribution p over T .
Each task i ⇠ p ( T ) is also associated with a data set D i , which is typically divided into training data D tr i used for task speciﬁc adaptation and validation data D val i used for meta-parameter update.
The objective of the gradient-based meta-learning is typically formulated as
[Equation]
where L i is the loss function corresponding to task i and ↵ is the learning rate.
Here, ✓ i ( ✓ ) =
✓  ↵ r ✓ L i ( ✓ , D tr i ) is the task speciﬁc parameter obtained by one step gradient update starting from the meta-parameter ✓ , and the goal is to ﬁnd the best meta-parameter which will minimize the meta loss function L (
[Equation]
=
[Equation]
Gradient-based meta-reinforcement learning: The gradient-based meta-learning framework is applicable both in supervised learning and reinforcement learning.
In RL, each task i corresponds to an MDP with task speciﬁc model P i and reward function R i .
We assume that the state-action spaces are uniform across the tasks, thus ensuring the ﬁrst level of task similarity.
Task speciﬁc data
D i is the trajectories ⌧ i,m =
( s m 0 , a m 0 , s m 1 , a m 1 , . . . ) , s 0 ⇠ ⇢ , a t ⇠ ⇡ ( s t , · ) , s t {++
+}
⇠ P i ( ·| s t , a t ) for m  M generated according to some policy ⇡ .
Since the randomness of the trajectory ⌧ i,m is speciﬁed by P i and ⇡ , we denote it as ⌧ i,m ⇠ ( P i , ⇡ ) .
We consider the function approximation setting where each policy ⇡ is represented by a function parameterized by ✓ ⇢ ⇥ and is denoted as ⇡ ✓ .
The task speciﬁc loss in meta-RL is deﬁned as L i RL ( ✓ ) =
J i ( ⇡ ✓ ) .
The gradient r ✓ L i RL ( ✓ ) can then be computed using policy gradient theorem.
The standard meta-RL training is done follows.
A task i ⇠ p ( T ) (usually a batch of tasks) is sampled at each iterate k of the algorithm.
Now, starting with meta-parameter ✓ k , the training data D tr i for task adaptation is generated as the trajectories ( ⌧ i,m )
Mm =1 , where ⌧ i,m ⇠ ( P i , ⇡ ✓ k ) , and the updated parameter ✓ i,k for task i is computed by policy gradient evaluated on D tr i .
The validation data D val i is then collected as trajectories ( ⌧ i,m )
Mm =1 , where ⌧ i,m ⇠ ( P i , ⇡ ✓ i,k ) , and the meta-parameter ✓ k is updated by policy gradient evaluated on D val i .
In the next section, we will introduce a modiﬁed approach which will leverage the demonstration data for task adaptation and meta-parameter update.
Meta-RL using Demonstration Data
Most gradient-based meta-RL algorithms learn the optimal meta-parameter and the task-speciﬁc parameter from scratch using on-policy approaches.
These algorithms exclusively rely on the reward feedback obtained from the training and validation data trajectories collected through the on-policy roll-outs of the meta-policy and task-speciﬁc policy.
However, in RL problems with sparse rewards, a non-zero reward is typically achieved only when the task is completed or near-completed.
In such sparse rewards settings, trajectories generated according to a policy that is still learning may not achieve any useful reward feedback, especially in the early phase of learning.
In other words, since the reward feedback is zero or near-zero, the policy gradient will also be similar, resulting in non-meaningful improvement in the policy.
Hence, standard meta-RL algorithms such as MAML, which rely crucially on reward feedback, will not be able to make much progress towards learning a valuable task-speciﬁc or meta-policy in sparse reward settings.
Learning the optimal control policy in sparse reward environments has been recognized as a challenging problem even in the standard RL setting, since most state-of-the-art RL algorithms fail to learn any meaningful policies even after a large number of training episodes [25, 17, 27].
One widely accepted approach to overcome this challenge is known as learning from demonstration , wherein demonstration data obtained from an expert [25] or inexpert policy [17, 27] is used to aid online learning.
The intuitive idea is that, even though the demonstration data does not contain any reward feedback, it can be used to guide the learning agent to reach non-zero reward regions of state-action spaces.
This guidance, usually in the direction of the goal/target, is achieved by inferring some pseudo reward signal through supervised learning approaches using demonstration data.
Can we enhance the performance of meta-RL algorithms in sparse reward environments by using demonstration data from sub-optimal experts?
Meta-RL in sparse reward environments is signiﬁcantly more challenging than that of the standard RL setting.
This is because the reward feedback serves the dual objectives of adapting the meta-parameter to speciﬁc tasks and for updating the metaparameter itself.
We note that demonstration data helps with both of these objectives.
Firstly, use of demonstration data to guide task-speciﬁc adaptation becomes important because adaptation is achieved in one or a few gradient steps, and policy resulting from each adaptation step might not achieve meaningful reward in a sparse reward setting.
Secondly, making use of demonstration data for meta-parameter update is equally important because of the role of meta-policy as a rewardyielding exploratory policy.
Intuitively, the meta-policy should yield trajectories that reach in the vicinity of the reward-achieving region of the state-action spaces.
This does not happen in sparse reward environments.
However, using the guidance from demonstration data, the task-speciﬁc policy obtained after the task adaptation may be able to generate trajectories that will reach within the reward-achieving region resulting in performance acceleration of the meta-policy.
For meta-learning with demonstration, we assume that each task i is associated with demonstration data
D dem i , which contains a trajectory generated according to a demonstration policy ⇡ dem in an environment with model P i .
We do not assume that ⇡ dem is the optimal policy for task i because in many real-world applications D dem i could be generated using an inexpert policy.
Our key idea is to enhance task adaptation using the demonstration data by introducing an additional gradient term corresponding to the supervised learning guidance loss.
We deﬁne the supervised learning loss function for task i as L BC i ( ✓ , D dem i [Equation]
=
[Equation]
a [Equation] s ) .
We note that, though this loss function is the same as in behavior cloning (BC), we use it directly in the gradient update instead of performing a simple warm start.
This approach is known to achieve superior performance than naive BC warm starting in standard RL problem under the sparse reward setting [25, 17, 27].
The task adaptation step at iteration k , starting with meta-parameter ✓ k is now obtained as
[Equation]
where w rl and w bc are hyperparameters that control the extant to which RL and demonstration data inﬂuence the gradient.
The next question is: how do we use demonstration data in the meta-parameter update?
One approach is to use only the demonstration data with a supervised learning loss function for updating the meta-parameter as done in [20].
We conjecture that such a reduction to supervised learning will severely limit the learning capability of the algorithm.
Firstly, if demonstration data is obtained from an inexpert policy, this approach will never be able to achieve the optimal performance.
This is because the role of the meta-policy as a reward-yielding exploratory policy will be limited by true performance of the inexpert policy.
Secondly, the task-speciﬁc policies obtained according to (2) may be able to reach within the reward-yielding region of state-action space as we mentioned before.
Hence, the validation data D val i collected through the roll-out of the policies obtained after task adaptation might contain extremely valuable reward feedback.
Utilizing this data could potentially have a signiﬁcant impact on improving the learning of the meta-parameter.
Thus, in our approach, we update the meta-parameter using the RL loss with policy gradient as follows.
[Equation]
We note that the demonstration data is indeed used in the meta-parameter update implicitly, as its impact can be observed in (3).
We found empirically that the double use of the demonstration data, either by adding an additional gradient through a BC loss function, or by replacing D val i with D dem i results in similar or worse performance than the approach described above.
We now formally present our algorithm called Enhanced Meta-RL using Demonstrations (EMRLD).
We now present a theoretical justiﬁcation of why EMRLD should have a superior performance in the sparse reward setting as compared to other gradient based algorithms that do not use demonstration data.
First, we introduce some notation.
Let ⇡ k = ⇡ ✓ k be the meta-policy used at iteration k of our algorithm.
Also, let ⇡ k,i be the policy obtained after task-speciﬁc adaptation for task i .
Recall that
J i ( ⇡ k,i ) is the value of the policy for the MDP corresponding to task i .
Similarly, we can deﬁne the state-action value function and advantage function of policy ⇡ k,i for task i as Q ⇡ k,i i and A ⇡ k,i i , respectively.
Also, let d ⇡ k,i i be the visitation frequency of policy ⇡ k,i for task i .
Now, we can deﬁne the value of the meta-policy ⇡ k over the ensemble of all tasks as J meta ( ⇡ k ) =
E i ⇠ p ( T ) [ J i ( ⇡ k,i )] .
If the demonstration data has to be useful, it should provide a reasonable amount of guidance.
In particular, we would like the task-speciﬁc policy adapted using this data to collect feedback that would ensure good meta-policy updates, particularly in the initial stages of meta-training.
Since the capability of the demonstration data to guide adaptation will depend on the demonstration policy ⇡ dem i according to which it is generated, we make the following assumption about ⇡ dem i .
Assumption 3.1.
During the initial stages of meta-training, E a ⇠ ⇡ dem i ( s, · )
[ A ⇡ k,i i ( s, a )]   , for all s 2 S and i 2 T .
Assumption 3.1 implies that during the initial stages of meta-training, the demonstration policy can provide a higher advantage on average than the current policy adapted to that task.
This is a reasonable assumption, since any reasonable demonstration policy is likely to perform much better than an untrained policy in the initial phase of learning.
We also note that a similar assumption was used in learning from demonstration literature [17, 27]
We now present the performance improvement result for EMRLD.
Theorem 3.2.
Let ⇡ k = ⇡ ✓ k be the meta-policy used at iteration k of our algorithm and let ⇡ k,i be the policy obtained after task adaptation in task i .
Let Assumption 3.1 holds for ⇡ k .
Then,
[Equation]
⇤◆
Theorem 3.2 presents a lower bound for the meta policy improvement as a sum of two groups of terms.
Maximizing the ﬁrst term in group one with a constraint on its second term will ensure a higher lower bound and hence an improvement in the meta-parameter training.
We notice that this is indeed achieved by the TRPO step used in the meta-parameter update.
Hence, this ﬁrst group is the same for any MAML-type of algorithm.
The advantage of the demonstration data is revealed in the second group of terms.
The term  / (1   ) adds a positive quantity to the lower bound, and this contribution from this second group of term can be maximized by minimizing E i ⇠ p ( T )
[ D [Equation] [Equation]
i  ] .
However, this minimization is hard to perform in practice because estimating D ⇡ k +1 ,i TV requires sampling the data according to ⇡ k +1 ,i , and this is not feasible at iteration k .
Hence, in practice, we replace that term by E
i ⇠ T [ D ⇡ dem i TV  ⇡ k
+1 ,i , ⇡ dem i  ] .
This can be easily achieved by including the standard maximum likelihood objective in the adaptation step.
Thus, EMRLD both exploits the advantage offered by an RL step, as well as that of behavior cloning for meta-policy optimization.
We can further improve the performance of EMRLD by including a behavior cloning warm starting step before performing the update (2).
We simplify this warm start to a one step gradient as, ✓ ˜ k,i = ✓ k  ↵ r ✓ L BC i ( ✓ , D dem i ) | ✓ = ✓ k , and then do the task adaptation as in (2) starting with ✓ ˜ k,i .
We call this version of our algorithm as EMRLD-WS.
Such a warm start is likely to provide more meaningful samples than directly rolling out the meta-policy to obtain samples for task-speciﬁc adaptation.
In the next section, we will see empirically how our design choices for EMRLD and
EMRLD-WS enable them to learn policies that provide higher rewards using only a small amount of (even sub-optimal) demonstration data.
Experimental Evaluation
We evaluate the performance of EMRLD based on whether the meta-policy it generates is a good initial condition for task-speciﬁc adaptation in sparse-reward environments over (i) Tasks already seen in training and (ii) New unseen tasks.
We seek to validate the conjecture that in the sparse-reward setting, EMRLD should be able to leverage even demonstrations of inexpert policies to attain high test performance over previously unseen tasks.
We do so with regard to two classes of tasks, namely, Tasks
that differ in their
reward
functions: Simulation experiments on Point2D Navigation [9]
, TwoWheeled Locomotion [12], and HalfCheetah [38, 35] • Tasks that differ in the environment dynamics: Real-world experiments using a TurtleBot, which is a two-wheeled differential drive robot [2].
4.1 Experiments on simulated environments
Sparse multi-task environments We present simulation results for three standard environments shown in Figure 1 and described below.
We train over a small number of tasks that differ in their reward functions.
We generate unknown tasks for test by randomly modifying the reward function.
Point2D Navigation is a 2 dimensional goal-reaching environment.
The states are the ( x, y ) location of the agent on a 2D plane.
The actions are appropriate 2D displacements ( dx, dy ) .
Training tasks are deﬁned by a ﬁxed set of 12 goal locations on a semi-circle of radius 2 .
The agent is given a zero reward everywhere except when it is a certain distance near the goal location, making the reward function highly sparse .
Within a single task, the objective of the agent is to reach the goal location in the least number of time steps starting from origin.
Test tasks are generated by sampling any point on the semicircle as the goal.
TwoWheeled Locomotion environment is a goal-reaching with sparse rewards, similar to Point2D
Navigation.
However, the robot is constrained by the permissible actions (limits on angular and linear velocity) and trajectories feasible based on the turning radius of the robot.
Here, our training tasks are a ﬁxed set of 24 goal locations on a semi-circle of radius 2 , while test goals are sampled randomly.
Further details on state-space and dynamics are provided in the Appendix.
HalfCheetah Forward-Backward consists of two tasks in which the HalfCheetah agent learns to either move in the forward (task 1) or backward (task 2) directions with as high velocity as possible.
The agent gets a reward only after it has moved a certain number of units along the x-axis in the correct direction, making the rewards sparse.
Training and test are under the same two tasks.
Optimal data and sub-optimal data We provide a limited amount of demonstration data in the form of just one trajectory per task for guidance .
Optimal data consists of ( s, a ) transitions generated by an expert policy trained using TRPO.
Sub-optimal data is generated by an inexpert, partially trained TRPO policy with induced action noise and truncated trajectories as shown in Figure 2.
Baselines We compare the performance of our algorithm against the following gradient based meta-reinforcement learning algorithms: (i) MAML: [9] The standard MAML algorithm for meta-RL (ii) Meta-BC: A variant of [10]; this is a supervised learning/behavior cloning version of MAML, where the maximum likelihood loss is used in the adaptation as well as the meta-optimization steps.
(iii) GMPS:
Guided meta policy search [20], which uses RL for gradient based adaptation, and uses demonstration data for supervised meta-parameter optimization.
The implemention of our algorithms and baselines is based on a publicly available meta-learning code base [3] licensed under the MIT License.
Performance with optimal demonstration data: We illustrate the training and testing performance of the different algorithms trained and tested with optimal data in Figure 3.
The top row of Figureshows the average adapted return across training tasks of the meta-policy during training iterations.
The bottom row of Figure 3 shows the average return of the trained meta-policy adapted across testing tasks over adaptation steps.
We see that our algorithms out perform the others by obtaining the highest average return, and are able to quickly adapt to testing tasks with just one adaptation step and one trajectory of demonstration data.
Additionally, our algorithms demonstrate a nearly-monotone improvement in average return demonstrating stable learning.
Meta-BC fails and has unstable training performance as the amount of demonstration data available per task is very small.
Training over only a small number of tasks further hampers the performance of Meta-BC.
MAML and GMPS fail to learn due to sparsity of the environment as the purely RL adaptation step incurs almost zero reward, and hence, negligible learning signal.
Furthermore, GMPS is hampered in the meta update step due to availability of only a small amount of demonstration data per-task.
Performance with sub-optimal demonstration data: EMRLD uses a combination of RL and imitation, which is valuable when presented with sub-optimal demonstrations.
For the Point2D
Navigation environment, we collect sub-optimal data for each task using a partially trained agent with induced action noise, and truncate the trajectories short of the reward region.
Hence, pure imitation cannot reach the goal.
For the TwoWheeled Locomotion environment, we collect data in a
[Figure] [Figure]
similar fashion for all tasks, but remove state-action pairs at the beginning of each trajectory.
Since the ﬁrst few state-action pairs contain information on how to orient the two-wheeled agent towards the goal, this truncation eliminates the possibility of direct imitation being successful.
Similarly, in
HalfCheetah we use a partially trained policy and truncate trajectories before they reach the reward bearing region.
Figure 4 illustrates that EMRLD outperforms all the baselines and is quickly able to adapt to unseen tasks, emphasizing the beneﬁt of its RL component.
Meta-BC and GMPS fail because they are restricted by the optimality of the data, and the absence of crucial information greatly impacts their performance.
MAML again fails due to the sparsity of the reward.
We conclude by presenting in Figure 5, sets of trajectories generated during testing in the TwoWheeledLococmotion environment when provided with optimal or sub-optimal demonstration data.
The variants of EMRLD clearly outperform the others, showing their strength in the sparse reward setting.
4.2 Real-world Experiments on TurtleBot
We demonstrate the ability of EMRLD variants to adapt to sparse-reward tasks when they differ in environment dynamics and have sparse reward feedback.
We do so via performance evaluation in the real-world using a TurtleBot shown in Figure 6 (left).
We ﬁrst we modify the TwoWheeled
[Figure]
Locomotion sparse reward environment by ﬁxing the goal, and changing the dynamics by inducing a residual angular velocity which mimics drifts in the environment.
This environmental drift is what differentiates each task.
In other words, for a given task, the environment would cause the robot to drift in some speciﬁc unknown direction.
We train on a set of 9 tasks with different angular velocity values (i.e., 9 different driving environments).
We use one trajectory of demonstration data per task collected using an expert policy trained using TRPO.
Note that all the training and data collection is done in simulation.
The results are shown in Figure 6 (middle), where we see that the EMRLD variants clearly outperform the others.
For testing, we consider the environment where the Turtlebot experiences a ﬁxed but unknown residual angular velocity representing environmental drift.
Thus, we bias the angular velocity control of the TurtleBot by some amount unknown to the algorithm under test.
We ﬁrst execute the meta policy on the TurtleBot in the real world to collect 5 trajectories.
We also provide one trajectory of simulated demonstration data.
We use these samples to adapt the meta-policy, and execute the adapted task-speciﬁc policy on the TurtleBot.
The results are shown in Figure 6 (right), where the origin is at (0 , 0) and the goal is indicated by a star.
It is clear that the variants of EMRLD are the best at quickly adapting to the drift in the environment and are successful with just one step of adaptation.
[Figure]
Conclusion
We studied the problem of meta-RL algorithm design for sparse-reward problems, in which demonstration data generated by a possibly inexpert policy is also provided.
Our key observation was that simple application of an untrained meta-policy in a sparse-reward environment might not provide meaningful samples, and guidance provided by imitating the inexpert policy can greatly assuage this effect.
We ﬁrst showed analytically that this insight is accurate and that meta-policy improvement might be feasible as long as the inexpert demonstration policy has an advantage.
We then developed two meta-RL algorithms, EMRLD and EMRLD-WS that are enhanced by using demonstration data.
We show through extensive simulations, as well as real world robot experiments that EMRLD is able to offer a considerable advantage over existing approaches in sparse reward scenarios.
Limitations and Future Work
EMRLD inherits the limitations of the gradient-based meta-RL approaches like MAML namely on-policy training, and data collection and gradient computation during test adaptation.
A limitation speciﬁc to our proposed algorithms is the assumption on availability of task speciﬁc demonstration data.
However, we reiterate that for a small number of train tasks, this assumption is quite practical, further, our framework allows for this data to be sub-optimal.
A possible future direction to explore is the context based meta-RL (that does’t require gradient computation during testing) with demonstration data.
Another future work direction is to explore usage of demonstration data in off-policy meta-RL algorithms.
Acknowledgement
This work was supported in part by the National Science Foundation (NSF) grants NSF-CAREEREPCN-2045783 and NSF ECCS 2038963, and U
S. Army Research Ofﬁce (ARO) grant W911NF-191-0367.
Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views of the sponsoring agencies.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] Section(b) Did you describe the limitations of your work?
[Yes] Section(c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] Section(b) Did you include complete proofs of all theoretical results?
[Yes] Appendix 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Supplementary Material ( https://github.com/DesikRengarajan/EMRLD ) (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Appendix (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Section(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Appendix 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Section(b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Appendix
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[-+
-]
[-+
-]
[-t +1 − y g )if (cid:112) ( x t +1 − x g ) 2 + ( y t +1 − y g ) 2 ≤ 0 .100 − t −if (cid:112) ( x t +1 − x g ) 
-]
[-( x t +1 − x g ) 
-]
[-+ ( y t +1 − y g )if (cid:112) ( x t +1 − x g ) 
-]
[N/A]
[-We observe from figure 9, that setting w bc = 
-]
[-We also observe that setting w rl = 
-]