Robust Imitation of a Few Demonstrations with a Backwards Model
Anonymous Author(s)
AffiliationAddress email
Abstract
Behavior cloning of expert demonstrations can speed up learning optimal policiesin a more sample-efficient way over reinforcement learning.
However, the policycannot extrapolate well to unseen states outside of the demonstration data, creatingcovariate shift (agent drifting away from demonstrations) and compounding errors.
In this work, we tackle this issue by extending the region of attraction around thedemonstrations so that the agent can learn how to get back onto the demonstratedtrajectories if it veers off-course.
We train a generative backwards dynamics modeland generate short imagined trajectories from states in the demonstrations.
Byimitating both demonstrations and these model rollouts, the agent learns boththe demonstrated paths and how to get back on to these paths.
With optimal ornear-optimal demonstrations, the learned policy will be both optimal and robustto deviations, with a wider region of attraction.
On continuous control domains,we evaluate the robustness when starting from different initial states unseen in thedemonstration data.
While both our method and other imitation learning baselinescan successfully solve the tasks for initial states in the training distribution, ourmethod exhibits considerably more robustness to different initial states.
Introduction
While reinforcement learning (RL) has shown remarkable success in many challenging domains[19, 36, 34], tasks with sparse rewards and long horizons still remain extremely difficult to solve.
Insuch tasks, a positive reward is only encountered after the RL agent reaches the goal after a longsequence of actions, meaning that it cannot learn any useful signals until this occurs (typically theagent must reach the goal several times to learn reliably as well).
Furthermore, the learning signaldecreases exponentially with the horizon, which combined with slow gradient-based updates, canlead to catastrophic forgetting even after the agent learns to reach the goal.
Expert demonstrations can help RL agents solve difficult tasks [28, 40, 20].
These demonstrationscan be used in the supervised setting where the agent imitates the expert’s behavior, termed imitationlearning (IL).
However, naively cloning (BC) of the expert’s trajectories suffers from covariate shift:the agent’s policy drifts away from the expert’s, which leads to compounding errors due to RL’ssequential nature.
Furthermore, the distribution of states given in the demonstrations often haslow-dimensional support with respect to the entire state space.
As such, the agent cannot extrapolatecorrectly when outside of the demonstration data.
Many approaches to solve this issue have beenproposed [31, 37, 14], but such approaches require an interactive expert to query the correct actions.
Another way to use demonstrations is to combine imitation learning with reinforcement learning in aform of learning from demonstrations (LfD)
In this case, demonstrations do not simplyact as supervised labels and can guide the agent’s exploration, and also act as augmentations to gooddata samples.
These LfD approaches either use demonstrations to pretrain the policy [33, 8], use an
[Figure]
auxiliary imitation loss in conjunction with the policy update [28, 20], or modify the reward functionsuch that the agent is rewarded when it imitates the demonstrations [47, 29].
One primary concern of relying on demonstrations is that they are costly to obtain, especially inreal-world applications.
Requiring an operator providing corrections in-the-loop to handle covariateshift is often prohibitive as well.
Given only a few offline trajectories demonstrating successful taskcompletion, an agent ought to be able to replicate the behavior from similar starting conditions, evenif there are small perturbations along the way, and correct itself when necessary.
We are primarilyinterested in this setting.
In this work, we aim to minimize the number of demonstrations necessaryfor sparse-reward tasks, while preserving successful task completion.
To tackle covariate shift, weseek an approach that will be robust , in the sense of Figures 1a and 1b: the agent is only trained witha few demonstrations from a single start state, but at evaluation time, it must generalize its behaviorto a variety of unseen start states.
If the agent can successfully complete the task from states adjacentto the demonstrated trajectory, then it can recover from small deviations from the demonstration.
Inspired by the notion of “funnels” in robotics [18] and feedback control [2, 17], we introducea reverse-time generative model that can generate possible paths leading the agent back onto thedemonstrations.
These reverse rollouts provide useful information because every rollout ends withinthe support of the demonstration data.
Assuming that the demonstrations lead to the goal, imitatingthese generated reverse rollouts and the original demonstration data allow the agent to reach the goalfrom more starting conditions, including unseen ones.
As illustrated in Figures 1c and 1d, typicalbehavior-cloning (BC) methods focus learning on the small number of demonstrated states, whereasour proposed approach, Backwards Model-based Imitation Learning (BMIL), uses the reverse rolloutsto learn a wider region of attraction around the demonstration.
We validate our approach on a numberof long-horizon, sparse-reward continuous-control tasks.
Even from 5 – 20 demonstrations, BMILprovides a significant increase in the region of attraction and robustness on many domains comparedto BC, or to using a forward dynamics model.
We summarize our contributions as follows:
• We propose an imitation learning method that pairs a backwards dynamics model with a policyand train on both demonstrations and imagined model rollouts.
• In the restrictive setting of an offline expert and no access to environment interactions, we showthat a backwards model can improve robustness over behavior cloning.
• Our experiments on a variety of long-horizon, sparse-reward domains demonstrate that BMIL
can noticeably extend the region of attraction around the demonstration data, even when trained on very small subsets of the state space.
Related Work
Imitation learning has a long history [25, 33] and is well-studied, as documented in various surveys [1,23, 10].
The challenges of covariate shift and compounding errors are also well-known [25, 30].
Most solutions involve on-policy imitation learning, where environment interaction and interactivequerying of the expert allow for the agent’s distribution to match that of the expert [31, 37].
Moreclosely related to our approach are methods that modify or augment the demonstrations to increaserobustness.
Laskey et al.
[14] inject noise into the supervisor’s policy during training to force thedemonstrator to provide corrections.
Luo et al.
[16] learn value function and dynamics model fromdemonstrations, using this to conservatively extrapolate a value function that encourages the agent to return to the expert data distribution.
Generative approaches have also been used in imitation learning[9, 43], but their focus is not on robustly following a few goal-reaching demonstrations.
Time-reversibility has been explored in RL, often as a form of regularization [38, 21, 46, 27, 32].
Reverse-time dynamics models, also called predecessor models, have also been used in RL [3, 5,35, 13, 15, 45, 6].
However, in all cases, the reverse-time dynamics model is used as either analternative to the forward-time dynamics model or as an auxiliary model in addition to the standardforward-time dynamics model, in order to mitigate model-compounding errors.
The result is that thereverse-time dynamics model can accelerate RL and enable greater sample-efficiency.
In this work,we take a different perspective where the reverse-time dynamics model is used to generate possible,unseen paths that can lead back to the expert demonstration and thus to the goal, thereby improvingrobustness in following the demonstration.
Our work is also similar to [42], where a reverse-timedynamics model is used to generate possible trajectories; however, their focus is on offline RL, wherethe generated trajectories are used to connect distinct sets of states in the offline dataset.
Method3.1 Preliminaries
We model the setting as a Markov Decision Process (MDP) with a continuous state space S and acontinuous action space A .
T : S × A (cid:55)→ S is the transition function that defines the distributions ofthe next state s t +1 ∈ S , given the current state s t ∈ S and action a t ∈ A taken at timestep t .
The objective of imitation learning is to learn a policy π θ , parameterized by θ , that matches theexpert’s policy π E .
We assume that the expert generates demonstrations D demo by rolling out itspolicy π E in the environment.
Note that we consider the more restrictive case of where demonstrationsonly consist of transitions ( s, a, s ′ ) and not rewards, where s ′ is the next state and r is the reward.
Behavior cloning (BC) is a form of imitation learning where the policy π θ learns to imitate expertactions via supervised learning.
The policy learned by behavior cloning is found by minimizing thenegative log-likelihood over the demonstration data
[Equation]
Note that BC does not require environment interactions and can be considered to be offline.
Furthermore, expert behavior is inferred only from demonstrations without access to the expert policy.
Compounding errors in behavior cloning As pointed out in [30, 41], behavior cloning suffersfrom covariate shift, where errors in the policy can compound and lead the agent to states where itcannot recover.
Intuitively, this occurs as states in the training data D demo is a small subset of theentire state space S
and it is difficult for the policy π θ to learn the optimal action for states outside ofthe training data.
If during a rollout, once the policy π θ makes an error and leaves the D demo , it mayencounter completely new states, leading to the compounding of errors.
Furthermore, as the agentmoves farther away from D demo , there is very little hope for it to make the correct action and moveback onto the distribution of the training data.
3.2 Problem Setting
We consider the same setting as BC, where we do not assume access to the environment or the expertpolicy during training and only expert demonstrations with no rewards are given.
Furthermore, weassume that the expert demonstrations are given in the MDP where the set of initial states s 0 ∈ S 0and goals g ∈ G is restricted to a very small subset of the state space.
An example of this scenario isa maze environment where the agent starts from the same initial state and tries to reach a fixed goal.
Formally, we define the demonstration trajectories τ demo as coming from a probability density
[Equation]
where s 0 ∈ S 0 , and s T ∈ G .
In all of our experiments, S 0 and G consist of a single start or goal stateand the ε -ball of its neighborhood.
As such, only a few demonstrations are required to learn a stableoptimal policy.
Note that this is different from domains in previous work [28, 4] which consider random goal states, requiring many more demonstrations to learn optimal policies.
We also assumethat the expert policy is optimal in that all demonstrations successfully reach a goal.
While this isnot strictly necessary in our method, we assumed that there are no rewards in τ demo and thus wecannot discern whether demonstrations are optimal.
In order to use task completion success rates asan evaluation metric in our experiments, we consider optimal demonstrations.
Our objective is to learn a policy that is robust to policy errors when imitating expert behavior andcan learn to reach the goal from a variety of initial states.
This is different than the multi-goal ormulti-task setting, where the agent learns to solve multiple goals or tasks, usually from a smallnumber of initial states.
More formally, the robustness of the policy R ( π θ ) is defined as
[Equation]
where the expectation is taken over S R , where S R is a strict superset of S 0 .
Note that our measure ofrobustness is somewhat coarse, in that we do not consider the shortest path to reach the goal fromevery start state (which would probably require more information such as diverse trajectories orenvironment interactions).
Instead, we seek to extend the region of attraction around D demo such thatthe learned policy can still reach the goal.
As we consider continuous states in our work, we measure the robustness R using samples from S R ,either by sampling randomly or by using stratified random sampling over a discretized grid of thestate space.
For some domains like maze environments, we are primarily concerned with being ableto learn robustness in terms of spatial coordinates such that the agent can learn to go back onto theoptimal path.
Thus we restrict the expectation to only the spatial dimensions for such domains.
Inother domains, we are interested in the policy’s ability to recover from arbitrary initial states and sowe vary the initial joint positions by adding uniformly random noise.
Throughout, we assume that there exists an action a ∈ A that allows the policy to go towards D demowhen in a state s ̸∈ D demo .
This scenario is true for many navigation and physics-based domains ifwe ignore the rare circumstances such as irrecoverable unsafe states or the breakdown of the agent. We exclude such cases and assume that state transitions are reversible.
3.3 Backwards Model-based Imitation Learning
In our work, we use a backwards dynamics model to provide more synthetic training data to thepolicy and therefore increase the policy’s robustness.
We call our method backwards model-basedimitation learning or BMIL.
Backwards model The backwards model is a probabilistic generative model defined as B =p ( s t , a t | s t +1 ) .
This model estimates the conditional distribution of the reverse time dynamics.
It takes in the next state and outputs the previous state and previous action.
As we consider onlycontinuous state and action spaces, we implement B as a conditional Gaussian, parameterized by ϕ .
The backwards model is decomposed into two functions B = B A · B S = p ( a t | s t +1 ) · p ( s t | a t , s t +1 ) ,an action generator and previous state generator.
The action generator B A predicts which action wastaken in order to land in the next state.
There may be several such actions from different states thatcan lead to the next state.
Thus the action generator implicitly encodes a backwards policy.
It isimportant for this backwards policy to closely match the learned forward policy π but be differentenough to generate diverse new rollouts for the policy to train on.
The previous state generator B Spredicts the previous state given the next state and previous action taken.
The goal of this generator isto accurately predict the backwards dynamics, which are learned using rollouts from the policy π .
As we consider the setting with no access to the expert or the environment, B is trained only on D demo .
The action generator and previous state generator are jointly trained by maximum likelihood
[Equation]
(1) where s t , a t , s t
+1 is the state, action, and next state, respectively, at timestep t .
Model rollouts Given expert demonstrations τ E , we use the backwards model to generate possibleseveral short reverse rollouts or traces τ B , starting from every state in τ E .
As all of these traces end on states within the demonstration data s ∈ D demo and as the demonstration reaches the goal,following these traces will eventually lead to the goal.
For all s 1 , . . .
, s T in τ E , we generate K tracesτ B in a time-reversed manner, where we start from the last state action pair ( s H , a H ) ∈ D demo andthen predict ( s t , a t ) for timesteps t = H − 1 , . .
These traces are collected into a buffer D M .
Aswe assumed that there are no irrecoverable states in our setting, the rollouts reflect possible pathsthat the agent could have taken to reach s ∈ D demo .
If the reverse time model B is accurate and theprevious action generator B A gives sufficiently diverse actions, the traces τ B are then samples fromthe region of attraction or “funnels” around every state along the demonstration.
As we assume alldemonstrations reach the goal, these samples from the funnels can be used to learn a robust policy π θ ,as it can follow the traces onto the optimal path.
Action selection strategy for B A
As the backwards model B is trained only on a limited numberof expert demonstrations, it is likely that B can only learn accurate reverse-time dynamics for statescontained within or close to the demonstration data D demo [44].
Thus repeatedly rolling out B Awould only generate traces whose ( s, a ) pairs are contained within D demo and would not help withlearning robust policies.
However, we would like to generate diverse traces with new unseen stateaction pairs in order to robustify the policy. To balance the model misprediction accuracy withgenerating plausible state-action pairs, we slightly perturb the action generated from B A for the firstaction only and use short horizon lengths for the traces.
This amounts to choosing a good actionselection strategy for B A .
A good action selection strategy would then place more probability closerto the support of the D demo , providing a “cover” of p ( a E | s ) but with enough of a wide tail to providediverse rollouts.
As our backwards model B is probabilistic (implemented as a conditional Gaussianwith diagonal covariance), we can easily perturb p ( a E | s ) by increasing the distribution’s variance.
Let a ∼ N ( µ, σ 2 ) =
B Aϕ A ( a | s ′ ) .
We consider two ways to generate action a : 1) simple scaling ofthe σ and 2) resampling a noisy version of a by adding uniform noise, a ′ =
a + k, k = U [ − k, k ] ,where k is a fixed hyperparameter.
For the scaling strategy, we further scale σ by the entropy of theprobability density as we wish to make the distribution “wider” for more peaky distributions.
Algorithm Our method BMIL is outlined in Algorithm 1.
Given expert demonstrations of theform ( s t , a t ) , we train the backwards model using Eqn. 1 to estimate the reverse-time dynamicsp ( s t , a t | s t +1 ) .
We train our policy π θ on both the demonstration data D demo and the model tracesτ B by sampling from both at a fixed ratio and using maximum likelihood,
[Equation]
where p d is the probability of sampling from D demo .
As our aim is to learn a robust policy while stillsucceeding at the original start states and goals, we sample from the demonstrations a higher ratiothan the model traces.
As we consider the limited case of an offline expert and no-reward demonstrations, we pair behaviorcloning with the backwards model.
Note that BMIL does not depend on the type of imitation learningpolicy; any algorithm can be used as long as the policy can be trained with samples of the form ( s, a ) .
Experiment Design
4.1 Environments
We validate our approach on several continuous control domains: 1) the Fetch robotics environment[24], 2) maze navigation with two different agents, and 3) Adroit hand domain [28]. For the Fetch Algorithm 1 Backwards Model-based Imitation Learning (BMIL) 1: for N epochs do 2: Train backwards model parameters ϕ using Eqn.
Generate K model traces τ B from every state s
D and store them
M steps do 5: Sample mini Update policy 7: end for 8: end robotics environments, we consider the “Push” and “PickAndPlace” tasks where the objective isto control a Fetch end effector to either push an object to the goal or pick an object and place it atthe target location.
For the maze environments, we consider 3 mazes of increasing difficulty, wherean agent must learn to move to the goal.
We use both a simple point and 29-DoF ant agent.
Forthe Adroit environments, we use the ‘Relocate‘ task, where one must control a 24-DoF Adroit handto pick up a ball and move it to a target location.
All domains use the Mujoco simulator [39] fora total of 9 distinct domains.
All environments also have sparse reward structures, where eitheronly every step has a constant negative reward until the goal is reached (Fetch) or only the goalhas a non-zero reward (Maze, Adroit).
In particular, the maze and Adroit environments are quitechallenging as they both consist of controlling the agent’s joints to perform locomotion (maze) ordexterous manipulation (Adroit) over a long horizon.
More detailed descriptions of each environmentincluding its observation space are provided in Appendix A.
4.2 Demonstrations and Implementation Details
To generate demonstrations on the Fetch and Maze domains, we train an expert policy by adding thegoal position to the state, as in goal-oriented learning.
For the Adroit domain, we use a pre-trainedpolicy from [28].
We use 5 demonstrations on the Push task and 10 on the PickAndPlace task, and 20demonstrations for all maze and Adroit environments.
For the policy, we use neural networks with 3 fully connected hidden layers with 256 neurons and
ReLU activations.
For the backwards model, we use 4-layer MLPs with 256 hidden units for boththe action predictor B A and previous state predictor B S and use diagonal Gaussian distributions.
To train the policy, we use p d = 0 .
5 for the Fetch environments, p d = 0 .
8 ∼ 0 .
95 for the Mazeenvironments, and p d = 0 .
8 for the Adroit environments.
We find that higher demo ratios arenecessary for longer-horizon and more complex domains.
For the model rollouts, we use the variancescaling action selection strategy for the first action only and use increasing rollout lengths for alldomains, similar to [11].
For a more detailed discussion of experiment details, see Appendix B.
4.3 Evaluation
We evaluate BMIL against behavior cloning (BC) and VINS [16].
VINS specifically aims to learnvalue functions robust to perturbations using negative sampling and the induced policy learns selfcorrective behavior.
VINS was chosen as it is most relevant to our setting; other methods such as
DART [14] or SQIL [29] require either an online expert or environment interactions.
We use the same number of demonstrations for all methods and also keep the same policy networkarchitecture and the total number of policy gradient steps equal across all methods.
We train boththe policy and backwards model until the backwards model loss converges.
Note that our goal isnot to solve the training task faster but rather to robustify the policy using the backwards model.
Additionally, we wish to solve the task at various starting conditions while still being able to succeedat the original initial start-goal states.
To evaluate the robustness of the learned policy, we vary the initial states and compute task successrates.
For Fetch, we fix the initial gripper, object, and goal position during training and vary thegripper’s x and y position within the table boundaries during evaluation.
We use 10000 samplesduring evaluation.
For Maze, we initialize the agent to a random start position within a discretizedgrid of the maze.
We sample 100 for each discrete grid cell and compute the success rate.
Samplingpoints for each grid cell gives us an idea of which positions are easy for the agent to reach the goal.
Intuitively, such positions would be those near the goal and the demonstrated path.
For Adroit, wegenerate 10000 random initial states by adding uniform noise to the qpos of the hand (30-dim).
Results
Our experiments aim to answer the following questions: 1) how robust of a policy does BMIL learn?and 2) what components of BMIL are important to improve robustness? Robustness (%) [Table] [Figure]
5.1 Robustness evaluation
The robustness results are shown in Table 1.
We note that the absolute robustness percentages aregenerally low for all methods because it is very difficult to extrapolate from limited demonstrations( 5 − 20 ) from a single pair of initial start and goal states.
We therefore also include the relativeimprovement over BC.
In the Fetch environments, BMIL substantially increases robustness over BC and VINS.
In particular,our method has an approximately 20% and 330% improvement over BC on Push and PickAndPlace,respectively.
VINS on the other hand performs similarly to BC.
We see a similar pattern on the harder
Adroit environment, where BMIL improves robustness over BC by approximately 68% . For the Maze environments, BMIL generally outperforms BC for the Point agent, while the robustness isdecreased for the Ant agent.
Somewhat surprisingly, BC performs quite well on the long-horizon Maze domains.
It may be that BC has some built-in extrapolation capabilities or that the backwardsmodel may need better latent representations.
Empirically, we can see that having short reverse rollouts from the backwards model and using onlyslight perturbations still helps to increase robustness, even though the traces contain some modelmisprediction errors.
We hypothesize that this is these traces do not necessarily need to be accuratein order to benefit the policy and simply need to be plausible paths that lead to the demonstrations.
Itmay be that having the general correct direction contained in the traces is sufficient for the policy toeventually reach states in the demonstration data.
The success rates during training are shown in Table 5 in the Appendix.
BMIL achieves success ratesclose to 1 for most domains, suggesting that increased robustness does not necessarily come at thecost of decreased performance on demonstrated start-goal states.
On the other hand, VINS cannotconsistently succeed during training,
[Figure]
Visualization of robustness Figure 2 shows which starting positions succeed during the robustnessevaluation for Fetch. The green points correspond to successful episodes. We can see that both BCand BMIL succeed more frequently when starting from nearby the demonstration data (approximatelya straight line from the start (red) to goal (blue)).
However, we can see that BMIL learns a muchlarger region of attraction over BC and even succeeds at points that are much farther away from
D demo .
We hypothesize that instead of perturbing a single state within D demo as done in VINS,learning a short reverse rollout from this state allows BMIL to learn optimal paths from states muchfarther away from D demo , leading to higher robustness values.
Figure 3 shows a similar visualizationfor some Point maze environments where the agent’s position is discretized into a grid.
BMIL learnsa region of attraction similar in size to that of BC but has a higher rate of success at each cell.
5.2 Additional Experiments
Forward vs Backward Dynamics We first analyze the utility of a backwards vs a forward dynamicsmodel.
On the Fetch environments, we train BMIL with a forward dynamics model p ( s ′ | s, a ) andcompare it against the original backwards model p ( s, a | s ′ ) .
The forward model is implementednearly identically to other n-step model-based RL algorithms (e.g. [11]), with the exception of noenvironment interactions.
As with the backwards model, we generate rollouts from the forwardsmodel starting from demonstrated states and train the policy on both the demonstrations and traces.
To generate model rollouts from demonstration states, we use the action from the policy a = π θ ( s ) .
The total number of parameters is kept approximately constant for the forwards and backwardsmodels.
As shown in Table 2, the forwards model offers little to no benefit over BC for both Pushand PickAndPlace, suggesting that the backwards model is needed to produce a robust policy.
Action selection strategy We ablate different action selection strategies in trace generation in
Figure 4a (and Figure 9 in the Appendix).
Compared to no perturbation (None), we see that eitheraction selection strategy improves robustness as it can lead to more diverse trajectories unseenwithin the support of the demonstrations.
We use the variance scaling strategy SC(30) for all Fetchexperiments as it was more stable than SC(50).
Number of demonstrations We also studied our method with varying numbers of demonstrations.
As shown in Figure 4b, both BC and BMIL improve in robustness with more demonstrations, but BCplateaus at a much lower level.
On the other hand, BMIL requires slightly more ( 10 ) demonstrationsthan needed ( 3 demonstrations are sufficient for BC to succeed during training) in order to train thebackwards model (Figure 10 in the Appendix). [Table] [Figure]
Computation budget As BMIL trains both the policy and the backwards model, it requires moretotal gradient updates than BC.
On Fetch domains, BMIL uses approximately 5 x more computationthan BC.
We train BC for more steps to match or exceed BMIL’s computation budget, as shown in
Figure 5 (BC given 1 x 20 x computation budget).
However, more training for BC does not improve robustness and seems to have a harmful effect on robustness.
[Figure]
Training model first and then the policy As an offline method, BMIL does not require thebackwards model to be trained in a single loop along with the policy.
We can first train the model firstand then train the policy. We compare training the model first and then the policy with the processoutlined in Algorithm 1. We find there are no noticeable differences when using this model firstapproach on the Fetch domains, as seen in Table 6.
Discussion
This work proposed a method to tackle the issue of covariate shift in imitation learning.
Weconsidered the restrictive setting where the expert is offline, where its behavior can only be inferredfrom demonstrations, and no access to additional environment interactions.
Specifically, we showthat pairing a generative backwards model with behavior cloning can allow a policy to learn a widerregion of attraction around the demonstration data.
By rolling out imagined traces from states withinthe demonstration and perturbing actions to generate diverse traces, BMIL learns a wider funnel thannaive BC.
Through experiments on several long-horizon, sparse-reward, continuous control domains,
BMIL noticeably improves robustness when trained on a narrow set of initial start and goal states andevaluated at random starting positions.
Future work will include applying BMIL to more difficulttasks and extending BMIL to allow for environment interactions and pairing with other imitationlearning algorithms.
Checklist
For all authors...
(a Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](
b) Did you describe the limitations of your work
Yes
(c) Did you discuss any potential negative societal impacts of your work
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The code and
data will be provided in the supplementary material, and also released prior to final
publication.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Briefly in Section 4, and in full in the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 5.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]
[Figure]
Fetch These environments from [24] involve controlling the Fetch robotic arm.
We consider twotasks: “Push” and “PickAndPlace” as shown in Figure 6.
The objective of “Push” is to push the objecton a table towards a fixed goal position using a closed gripper.
The objective of “PickAndPlace” isto pick the object by controlling the gripper and move it towards the goal.
The observation spaceis 25-dimensional and consists of the end effector coordinates and its linear velocity, the gripper’sposition and velocity, and the object’s pose, velocities, and its relative position/velocity to the gripper.
All Fetch environments have a fixed episode length of 50 and an episode is considered successful ifthe object is at the goal at the end of the episode.
[Figure]
Maze There are 3 different maze environments of increasing difficulty and 2 different agents(Point, Ant) for each environment (c.f. Figure 7).
The initial start and goal position is fixed andthe objective is to control the agent to reach the goal (colored in red).
The observation space is theagent’s joint positions/velocities, the current timestep, and the agent’s current Cartesian position.
We
[Figure]
use dt = 0 .
02 within the Mujoco [39] simulator and set the gear ratio for Ant to 10 to prevent it fromfalling over as in [22].
[Figure]
Adroit In this domain from [28], the goal is to control a 24-DoF Adroit hand to pick up a ball fromthe table and move it to a target location.
The agent must manipulate each finger and wrist joint withdexterity to grasp the object correctly and learn the correct ball and target positions.
An observationconsists of hand joint angles, the object position/orientation, and the target position/orientation.
Wemodify the original domain by fixing the initial qpos of the hand and by terminating the episodewhen the agent correctly solves the task.
B Experiment Details
B.1 Expert demonstrations
To train an expert policy for the Fetch environments, we use pre-defined settings in [26] to train theexpert policy.
For the maze environments, we predefine a series of subgoals, and the position of thenext subgoal is concatenated to the state.
The reward is dense, using the negative Euclidean distanceto the next subgoal and subsequent subgoals, and we use soft actor-critic [7] as the expert policy.
On the Fetch domains, we use 5 and 10 demonstrations for the Push and PickAndPlace tasks,respectively.
As each episode is of length 50 , this amounts to a total of a couple of hundred trainingsamples.
We thus make 10 identical copies of each demonstration in order to stably train the policyand backwards model (we do the same for all other methods).
This has a similar effect of performingmore gradient updates, but we found it is computationally cheaper.
For both Maze and Adroitenvironments, we use 20 demonstrations and do not repeat the demonstrations as the episodes aresufficiently long. B.2 Hyperparameters All hyperparameter settings for BMIL are provided in Tables 3 and 4. For all experiments, we usedan internal cluster with single GPU compute nodes, with 10 virtual CPU cores and a GPU with eitheran NVIDIA P100 or V100.
For all methods, we use the same number of policy gradient steps for a fair comparison.
For VINS,we rely on the hyperparameter settings provided in the paper and run hyperparameter sweeps to findthe best settings specific to our environments.
As the Maze environments are new, we ran longerhyperparameter sweeps for VINS to find the best settings.
[Table]
[Table]
C Results
C.1 Main results
[Table]
Table 5 shows the success rates during training, on environments where the start and goal positionsare unchanged.
We see that both BC and BMIL achieve high success rates across all environments,while VINS does not for the Maze environments.
C.2 Additional Experiments
a similar in PickAndPlace, in that perturbing the action increases robustness over None. Figure 9 shows the robustness of different action selection strategies on Fetch-Push. Here we see thatall action selection strategies generally perform similarly, with possibly RS(0.3) and SC(50) a slightedge over the no strategy (None), though the error bars are fairly large. Figure 10 shows the success rates during training with a varying number of demonstrations on
Fetch-PickAndPlace. We see that 3 demonstrations are sufficient for BC to achieve a 100% successrate on the original start and goal positions. However, BMIL requires at least 5 demonstrations toattain the same level of performance, as the backwards model requires at least a certain number ofsamples to train in a stable manner. We use 10 demonstrations in our experiments.
[Table]
Table 6 shows the results of training the backwards model first compared to BMIL which trains boththe model and the policy in the same training loop.
We use 100 trials in the Fetch environments.
Wefind similar robustness values, where training the model first produces slightly lower values on Pushand slightly higher values on PickAndPlace.