G OAL -D RIVEN I MITATION L EARNING FROM O BSERVATION BY I NFERRING G OAL P ROXIMITY
Anonymous authors
Paper under double-blind review
A BSTRACT
Humans can effectively learn to estimate how close they are to completing a desired task simply by watching others fulﬁll the task.
To solve the task, they can then take actions towards states with higher estimated proximity to the goal.
From this intuition, we propose a simple yet effective method for imitation learning that learns a goal proximity function from expert demonstrations and online agent experience, and then uses the learned proximity to provide a dense reward signal for training a policy to solve the task.
By predicting task progress as the temporal distance to the goal, the goal proximity function improves generalization to unseen states over methods that aim to directly imitate expert behaviors.
We demonstrate that our proposed method efﬁciently learns a set of goal-driven tasks from state-only demonstrations in navigation, robotic arm manipulation, and locomotion tasks.
I NTRODUCTION
Humans are capable of effectively leveraging demonstrations from experts to solve a variety of tasks.
Speciﬁcally, by watching others performing a task, we can learn to infer how close we are to completing the task, and then take actions towards states closer to the goal of the task.
For example, after watching a few tutorial videos for chair assembly, we learn to infer how close an intermediate conﬁguration of a chair is to completion.
With the guidance of such a task progress estimate, we can efﬁciently learn to assemble the chair to progressively get closer to and eventually reach, the fully assembled chair.
Can machines likewise ﬁrst learn an estimate of progress towards a goal from demonstrations and then use this estimate as guidance to move closer to and eventually reach the goal?
Typical learning from demonstration (LfD) approaches (Pomerleau, 1989; Pathak et al., 2018;
Finn et al., 2016) greedily imitate the expert policy and therefore suffer from accumulated errors causing a drift away from states seen in the demonstrations.
On the other hand, adversarial imitation learning approaches (Ho & Ermon, 2016; Fu et al., 2018) encourage the agent to imitate expert trajectories with a learned reward that distinguishes agent and expert behaviors.
However, such adversarially learned reward functions often overﬁt to the expert demonstrations and do not generalize to states not covered in the demonstrations (Zolna et al., 2019), leading to unsuccessful policy learning.
Inspired by how humans leverage demonstrations to measure progress and complete tasks, we devise an imitation learning from observation (LfO) method which learns a task progress estimator and uses the task progress estimate as a dense reward signal for training a policy as illustrated in
Figure 1.
To measure the progress of a goal-driven task, we deﬁne goal proximity as an estimate of temporal distance to the goal (i.e., the number of actions required to reach the goal).
In contrast to prior adversarial imitation learning algorithms, by having additional supervision of task progress and learning to predict it, the goal proximity function can acquire more structured task-relevant information, and hence generalize better to unseen states and provide better reward signals.
However, the goal proximity function can still output inaccurate predictions on states not in demonstrations, which results in unstable policy training.
To improve the accuracy of the goal proximity function, we continually update the proximity function with trajectories both from expert and agent.
In addition, we penalize trajectories with the uncertainty of the goal proximity prediction, which prevents the policy from exploiting high proximity estimates with high uncertainty.
As a result, by leveraging the agent experience and predicting the proximity function uncertainty, our method can achieve more efﬁcient and stable policy learning.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
/Uvf6h5GoNCLpm1fc9NMMiYQcElzGp+aiFhfMJG0M+pYjHYIJufPKNnuTKkkTZ5KaRz9fdExmJrp3GYd8YMx3bZK8T/vH6K0XWQCZWkCIovFkWppKhp8T8dCgMc5TQnjBuR30r5mBnGMU+pCMFbfnmVdC+antv07i8brZsyjio5IafknHjkirTIHWmTDuFEk2fySt4cdF6cd+dj0Vpxyplj8gfO5w/gMZD4</latexit>
[Figure]
The main contributions of this paper include (1) an algorithm for imitation from observation that uses estimated goal proximity to inform an agent of the task progress; (2) modeling uncertainty of goal proximity estimation to prevent exploiting uncertain predictions; and (3) a joint training algorithm of the goal proximity function and policy.
We show that the policy learned with our proposed goal proximity function is more effective and generalizes better than the state-of-the-art LfO algorithms on various domains, such as navigation, robot manipulation, and locomotion.
Moreover, our method demonstrates comparable results with GAIL (Ho & Ermon, 2016), which learns from expert actions.
R ELATED W ORK
Imitation learning (Schaal, 1997) aims to leverage expert demonstrations to acquire skills.
While behavioral cloning (Pomerleau, 1989) is simple but effective with a large number of demonstrations, it suffers from compounding errors caused by the distributional drift (Ross et al., 2011).
On the other hand, inverse reinforcement learning (Ng & Russell, 2000; Abbeel & Ng, 2004; Ziebart et al., 2008) estimates the underlying reward from demonstrations and learns a policy through reinforcement learning with this reward, which can better handle the compounding errors.
Speciﬁcally, generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016) and its variants (Fu et al., 2018; Kostrikov et al., 2020) shows improved demonstration efﬁciency by training a discriminator to distinguish expert and agent transitions and using the discriminator output as a reward for policy training.
While most imitation learning algorithms require expert actions, imitation learning from observation (LfO) approaches learn from state-only demonstrations.
This enables the LfO methods to learn from diverse sources of demonstrations, such as human videos, demonstrations with different controllers, and other robots.
To imitate demonstrations without expert actions, inverse dynamics models (Niekum et al., 2015; Torabi et al., 2018a;
Pathak et al., 2018) or learned reward functions (Edwards et al., 2016; Sermanet et al., 2017; 2018;
Liu et al., 2018; Lee et al., 2019a) can be used to train the policy.
However, these methods require large amounts of data to train inverse dynamics models or representations.
On the other hand, state-only adversarial imitation learning (Torabi et al., 2018b;
Yang et al., 2019) can imitate an expert with few demonstrations, similar to GAIL.
In addition to discriminating expert and agent trajectories, our method proposes to also estimate the proximity to the goal, which can provide more informed reward signals and generalize better.
Closely related works to our approach are reinforcement learning algorithms that learn a value function or proximity estimator from successful trajectories and use them as an auxiliary reward (Mataric, 1994; Edwards & Isbell, 2019; Lee et al., 2019b).
While these value function and proximity estimator are similar to our proposed goal proximity function, these works require environment reward signals, and do not incorporate adversarial online training and uncertainty estimates.
Moreover, demonstrating the value of learning a proximity estimate for imitation learning, Angelov et al.
(2020) utilizes the learned proximity to choose a proper sub-policy but does not train a policy from the learned proximity.
Similar to our method, Burke et al.
(2020) proposes to learn a reward function using a ranking model and use it for policy optimization, demonstrating the advantage of using goal proximity as a reward for training a policy.
However, they learn the proximity function from demonstrations alone and solely provide proximity as a reward.
This hinders agent learning when the proximity function fails to generalize to agent experience, allowing the agent to exploit inaccurate proximity predictions for reward.
By incorporating the online update, uncertainty estimates, and difference-based proximity reward, our method can robustly imitate state-only demonstrations to solve goal-driven tasks without access to the true environment reward.
M ETHOD
In this paper, we address the problem of learning from observations for goal-driven tasks.
Adversarial imitation learning methods (Torabi et al., 2018b; Yang et al., 2019) suggest learning a reward function that penalizes an agent state transition off the expert trajectories.
However, these learned reward functions often overﬁt to expert demonstrations and do not generalize to states which are not covered in the demonstrations, leading to unsuccessful policy learning.
To acquire a more structured and generalizable reward function from demonstrations, we propose to learn a goal proximity function that estimates proximity to the goal distribution in terms of temporal distance (i.e., number of actions required to reach the goal).
Then, a policy learns to reach states with higher proximity (i.e., that are closer to the goal) predicted by the goal proximity function.
Moreover, during policy training, we propose to measure the uncertainty of the goal proximity function which prevents the policy from exploiting over-optimistic proximity predictions and yielding undesired behaviors.
In Section 3.2, we describe the goal proximity function in detail.
Then, in Section 3.3, we elaborate how the policy is jointly trained with the goal proximity function.
3. P RELIMINARIES
We formulate our learning problem as a Markov decision process (Sutton, 1984) deﬁned through a tuple ( S , A , R, P, ρ 0 , γ ) for the state space S , action space A , reward function R ( s, a ) , transition distribution P ( s (cid:48) | s, a ) , initial state distribution ρ 0 , and discounting factor γ .
We deﬁne a policy π ( a | s ) that maps from states to actions and correspondingly moves an agent to a new state according to the transition probabilities.
The policy is trained to maximize the expected sum of discounted rewards, E ( s,a ) ∼ π (cid:104)(cid:80) T i t =
γ t R ( s t , a t ) (cid:105) , where T i represents the variable length of episode i .
In imitation learning, the learner receives a ﬁxed set of expert demonstrations, D e = { τ e 1 , . .
, τ eN } .
In this paper, we speciﬁcally consider the learning from observation (LfO) setup where each demonstration τ ei is a sequence of states.
Moreover, we assume that all expert demonstrations are successful; therefore, the ﬁnal state of an expert trajectory reaches the task goal.
3. L EARNING G OAL P ROXIMITY F UNCTION
In goal-driven tasks, an estimate of how close an agent is to the goal can be utilized as a direct learning signal.
Therefore, instead of learning to discriminate agent and expert trajectories (Ho &
Ermon, 2016; Torabi et al., 2018b), we propose a goal proximity function , f : S → R , that learns how close states are to the goal distribution.
Speciﬁcally, we deﬁne goal proximity as a proximity that is discounted based on its temporal distance to the goal (i.e., inversely proportional to the number of actions required to reach the goal).
Note that the goal proximity function measures the temporal distance, not the spatial distance, between the current and goal states.
Therefore, a single proximity value can entail all information about the task, goal, and any roadblocks.
In this paper, we deﬁne goal proximity of a state s t as the linearly discounted proximity f
( s t ) = 1 − δ · ( T i − t ) , where δ ∈ (0 , 1) is a discounting factor and T i is the episode horizon.
In this paper, we set δ to 1 /H to evenly distribute the proximity between 0 and 1, where H is the maximum task horizon.
Note that we use the maximum episode length H , instead of the variable episode length T i , to deﬁne a ﬁxed δ for the temporal discounting to be consistent between episodes.
We use mean squared error as the objective for training the goal proximity function f φ parameterized by φ :
[Equation]
Require:
Expert demonstrations D e = { τ e 1 , . . .
, τ eN } 1: Initialize weights proximity function f φ policy π θ 2: for i = 0 , 1 , ..., M do 3:
Sample expert demonstration τ ∼ e
5: end for 6: for i = 0 , L do 7:
Rollout trajectories τ i =
( s 0 , . .
s T i ) π θ (cid:46) training 8:
Compute proximity reward R φ ( s t , s t +1 ) for ( s t , s t +1 ) ∼
τ i using Equation 5 9:
Update π θ using any RL algorithm 10:
Update f minimize Equation11: end for
There are alternative ways to represent and learn goal proximity, such as exponentially discounted proximity and ranking-based proximity (Brown et al., 2019).
But, in our experiments, linearly discounted proximity consistently performed better than alternatives; therefore, the linearly discounted proximity is used throughout this paper (see Figure 5b and Figure 11).
By learning to predict the goal proximity, the goal proximity function not only learns to discriminate agent and expert trajectories (i.e., predict 0 proximity for an agent trajectory and positive proximity for an expert trajectory with Equation 4), but also acquires the task information about temporal progress entailed in the trajectories.
From this additional supervision, the proximity function provides more informative learning signals to the policy and generalizes better to unseen states as empirically shown in Section 4.
3. T RAINING P OLICY WITH P ROXIMITY R EWARD
In a goal-driven task, a policy π θ aims to get close to and eventually reach the goal.
We can formalize this objective as maximizing the goal proximity at the ﬁnal state f φ ( s T i ) , which can be used as a sparse proximity reward.
In addition, to encourage the agent to make consistent progress towards the goal, we devise a dense proximity reward based on the increase in proximity, f φ ( s t +1 ) − f φ ( s t ) , at every timestep.
By combining the sparse and dense proximity rewards, our total proximity reward can be deﬁned as
[Equation]
Given the proximity reward, the policy is trained to maximize the expected discounted return:
[Equation]
However, a policy trained with the proximity reward can sometimes perform undesired behaviors by exploiting over-optimistic proximity predictions on states not seen in the expert demonstrations.
This becomes critical when the expert demonstrations are limited and cannot cover the state space sufﬁciently.
To avoid inaccurate predictions leading an agent to undesired states, we propose to (1) ﬁne-tune the goal proximity function with online agent experience to reduce optimistic proximity evaluations; and (2) penalize agent trajectories with high uncertainty in goal proximity predictions.
First, we set the target proximity of states in agent trajectories to 0 , similar to adversarial imitation learning methods (Ho & Ermon, 2016), and train the proximity function with both expert demonstrations and agent experience by minimizing the following loss:
[Equation]
Although successful agent experience is also used as negative examples for training the proximity function, in practice, this is not problematic since the proximity function ideally converges to the average of expert and agent labels (e.g., 1 / 2 − δ · ( T i − t ) / 2 for ours and 0.5 for GAIL).
Early stopping and learning rate decay can be used to further ease this problem (Zolna et al., 2019).
Also,
[Figure]
the online training of the goal proximity function can lower the goal proximity estimate in a local optimum, which helps the policy escape from such local optima.
To alleviate the effect of inaccurate proximity estimation in policy training, we discourage the policy from visiting states with uncertain proximity estimates.
Speciﬁcally, we model the uncertainty U ( s t ) as the disagreement in terms of standard deviation of an ensemble of proximity functions (Osband et al., 2016; Lakshminarayanan et al., 2017).
Then we use the estimated uncertainty to penalize exploration of states with uncertain proximity estimates.
The proximity estimate f
( s t ) is the average prediction of the ensemble.
With the uncertainty penalty, the modiﬁed reward can be written as:
[Equation]
(5) where λ is a tunable hyperparameter to balance the proximity reward and uncertainty penalty.
A larger λ results in more conservative exploration outside the states covered by the expert demonstrations
In summary, we propose to learn a goal proximity function that estimates how close the current state is to the goal distribution, and train a policy to maximize the goal proximity while avoiding states with inaccurate proximity estimates using the uncertainty measure.
We jointly train the proximity function and policy as described in Figure 1 and Algorithm 1.
E XPERIMENTS
In our experiments, we aim to answer the following questions: (1) How does our method’s efﬁciency and ﬁnal performance compare against prior work in imitation from observation and imitation learning with expert actions?
(2) Does our method lead to policies that generalize better to states unseen in the demonstrations?
(3) What factors contribute to the performance of our method?
To answer these questions we consider diverse goal-driven tasks: navigation, robot manipulation, and locomotion.
To demonstrate the improved generalization capabilities of policies trained with the proximity reward, we benchmark our method under two different setups: expert demonstrations are collected from (1) only a fraction of the possible initial states (e.g., 25%, 50%, 75% coverage) and (2) initial states with smaller amounts of noise.
(1) measures the ability of an agent to interpolate between states covered by the demonstrations while (2) evaluates extrapolating beyond the demonstrations to added noise during agent learning.
In both setups, our method shows superior generalization capability and thus, achieves higher ﬁnal rewards than LfO baselines.
Moreover, our method achieves comparable results with LfD methods that use expert actions.
These generalization experiments serve to mimic the reality that expert demonstrations may be collected in a different setting from agent learning.
For instance, due to the cost of demonstration collection, the demonstrations may poorly cover the state space.
An agent would then have to learn in an area of the state space not covered by the demonstrations.
We measure this in the experimental setup (1), where the demonstrations cover a fraction of the possible learner starting and goal states.
Likewise, demonstrations may be collected in controlled circumstances with little environment noise.
Then, an agent learning in an actual environment would encounter more noise than presented in the demonstrations.
We quantify this in the experimental setup (2), where less noise is applied to demonstration starting states.
4. B ASELINES
We compare our method to the state-of-the-art prior works in both imitation learning from observations and standard imitation learning with actions, which are listed below:
• BCO (Torabi et al., 2018a) learns an inverse model from environment interaction to provide action labels in demonstrations for behavioral cloning.
• GAIfO (Torabi et al., 2018b) is a variant of GAIL (Ho & Ermon, 2016) which trains a discriminator to discriminate state transitions ( s, s (cid:48) ) instead of state-action pairs ( s, a ) .
• GAIfO-s , as compared to in Yang et al.
(2019), learns a discriminator based off only a single state, not a state transition as with GAIfO. • BC (Pomerleau, 1989) ﬁts a policy to the demonstration actions with supervised learning.
This method requires expert action labels while our method does not.
• GAIL (Ho & Ermon, 2016) uses adversarial imitation learning with a discriminator trained on state-action pairs ( s, a ) .
This method also uses actions whereas ours does not.
Also, we study several variations of our method to evaluate the importance of different design choices:
• Ours (No Uncert) : Removes the uncertainty penalty from the reward function.
• Ours (No Online) :
Learns the proximity function ofﬂine from the demonstrations and does not reﬁne it using agent experience during policy learning.
This approach may fail as the proximity function will not learn outside of the demonstrations and thus provide a poor reward signal.
• Ours (No Ofﬂine) : Does not pre-train the proximity function.
This should be less efﬁcient than our method, which pre-trains the proximity function using the demonstrations.
• Ours (Exp) :
Uses the exponentially discounted goal proximity f
( s t ) =
δ ( T − t ) .
4. E XPERIMENTAL S ETUP
By default, our primary method uses the linearly discounted version of the proximity function as this empirically lead to the best results (see details in Figure 11) and set the discounting factor as δ = 1 /H , where H is the task horizon length.
For modeling uncertainty, we use an ensemble of sizeacross all tasks.
For all tasks, we pre-train the proximity function for 5 epochs on the demonstrations.
During online training (i.e., policy learning), we sample a batch of 128 elements from the expert and agent experience buffers.
The mean and standard deviation of outputs from the ensemble networks are used as the proximity prediction and uncertainty, respectively.
The same network architecture is used for proximity function, discriminator (for the baselines), and policy.
Details of the network architecture can be found in Section G.2.
Any reinforcement learning algorithm can be used for policy optimization, but we choose to use PPO (Schulman et al., 2017) and the hyperparameters of PPO are tuned appropriately for each method and task (see Table 2).
Each baseline implementation is veriﬁed against the results reported in its original paper.
We train each task with 5 different random seeds and report mean and standard deviation divided by 2.
4. N AVIGATION
In the ﬁrst set of experiments, we examine the N AVIGATION task between four rooms shown in
Figure 2a.
The purpose of this environment is to show the beneﬁts of our method in a simple setting where we can easily visualize and verify the learned goal proximity function.
The agent start and goal positions are randomly sampled and the agent has 100 steps to navigate to the goal.
We provideexpert demonstrations obtained using a shortest path algorithm.
During demonstration collection, we hold out 50% of the possible agent start and goal positions determined by uniform random sampling.
In contrast, during agent learning and evaluation, start and goal positions are sampled from all possible positions.
As can be seen in Figure 3a, our method achieves near 100% success rate in 3M environment steps, while all GAIL variants fail to learn the task.
Although BC and BCO could achieve the goal for about 60% and 30% cases respectively, they show limited generalization to unseen conﬁgurations.
This
[Figure] [Figure]
result proves that learning with the goal proximity reward is effective and the learned goal proximity function generalizes well to unseen conﬁgurations.
To verify whether the proximity function learns meaningful information of proximity to the goal, we visualize the proximity from all agent positions in Figure 4d.
Our proximity function predicts a qualitatively meaningful goal proximity: high estimated proximity near the goal and lower estimated proximity when the agent is farther away from the goal.
The corners of the rooms show low goal proximity since less expert trajectories pass over those regions compared to the center of each room.
Finally, we investigate our hypothesis that the goal proximity function allows for greater generalization, which results in better task performance with smaller demonstration coverage.
To test this hypothesis, we compare the cases where extreme (25% coverage), moderate, and no generalization (100% coverage) are required.
Figure 4 demonstrates that our method consistently achieves 100% success rates in 3M steps with 50%-100% demonstration coverages and is not as affected by increasingly difﬁcult generalization as baselines.
In contrast, GAIL and all LfO baselines fail to learn the N AVIGATION task when expert demonstrations do not cover all conﬁgurations.
This supports our hypothesis that the goal proximity function is able to capture the task structure and therefore, generalize better to unseen conﬁgurations.
4. R OBOT M ANIPULATION
We further evaluate our method in two continuous control tasks: F ETCH P ICK and F ETCH P USH from Plappert et al.
In the F ETCH P ICK task shown in Figure 2b, a Fetch robotic arm must grasp and move a block to a target position.
In F ETCH P USH , the Fetch arm pushes a block to a target position, as shown in Figure 2c.
Both the initial position of the block and target are randomly initialized.
For each, we provide 1k demonstrations, consisting of 33k and 28k transitions for F ETCH
P ICK and F ETCH P USH respectively, generated using a hand-engineered policy .
We create a 50% holdout set of starting states for agent learning by splitting the continuous state space into a 4 bygrid and holding out two cells per row to sample the block and target starting positions from.
In the F ETCH P ICK task, our method achieves more than 80% success rate while the success rates of
GAIfO and GAIfO-s are upper-bounded by 50% due to the limited coverage of expert demonstrations (see Figure 3b).
Our method learns slower than GAIL but achieves comparable ﬁnal performance even though GAIL learns from expert actions.
The F ETCH P USH task is more challenging than
F ETCH P ICK due to the more complicated contact dynamics for pushing interactions.
In Figure 3c, the demonstrations are collected with full coverage but the policy is trained in a version of the environment with 2 times larger noise applied to the starting state.
All methods fail to learn diagonal pushing movements but our method still learns horizontal pushing faster and achieves higher performance than all other baselines.
We evaluate both F ETCH tasks under two different generalization setups, different demonstration coverages (Figure 8) and different amounts of noise (Figure 9), and the results consistently show that our proximity function is able to accelerate policy learning in continuous control environments with superior generalization capability.
4. A NT L OCOMOTION
We used the A NT R EACH environment proposed in Ghosh et al.
(2018), simulated in the MuJoCo physics engine (Todorov et al., 2012).
In this task, the quadruped ant is tasked to reach a randomly generated goal, which is along the perimeter of a half circle of radius 5m centered around the ant (see Figure 2d).
We provide 1k demonstrations, which contain 25k transitions in total.
When demonstrations are collected, no noise is added to the initial pose of the ant whereas random noise is added during policy learning, which requires the reward functions to generalize to unseen states.
In Figure 3d, with 0.03 added noise, our method achieves 35% success rate while BCO, GAIfO, and
GAIfO-s achieve 1%, 2%, and 7%, respectively.
This result illustrates the importance of learning proximity over learning to discriminate expert and agent states for generalization to unseen states.
The performance of GAIfO and GAIfO-s drops drastically with larger joint angle randomness as shown in Figure 9.
As the A NT R EACH task is not as sensitive to noise in actions compared to other tasks, BC and GAIL show superior results but our method still achieves comparable performance.
We also ablate the various aspects of our method in Figure 5.
First, we verify the effect of the uncertainty penalty used in the proximity reward.
The learning curves with different λ are plotted in
Figure 5a and demonstrate that our method works the best with λ = 0 .
Both too low and too high uncertainty penalties degrade the performance.
Figure 5b shows the linearly discounted proximity function learns marginally faster than the exponentially discounted proximity function.
In Figure 5c, we test the importance of online and ofﬂine training of the proximity function.
The result shows that the agent fails to learn the task without online updates using agent trajectories.
Meanwhile, no proximity function pre-training lowers performance.
4. A BLATION S TUDY
Finally, we analyze the contribution of the proximity function, reward formulation, and uncertainty to our method’s performance in Figure 6.
Adding uncertainty to GAIfO-s (GAIfO-s+Uncert) produced a 5.8% boost in average success rate compared to regular GAIfO-s, which is not a signiﬁcant improvement.
Proximity supervision, without the uncertainty penalty, resulted in a 28.1% increase in average performance over GAIfO-s with the difference-based reward R ( s t , s t +1 )
=
f ( s t +1 ) − f ( s t ) (Prox+Diff) and 15.9% with the absolute reward R ( s t ) =
f ( s t ) (Prox+Abs).
This higher performance means modeling proximity is more important than using the uncertainty penalty for our method.
We also found that the uncertainty penalty and proximity function have a synergistic interaction.
Combining both the proximity and uncertainty gives a 43.3% increase with the difference-based reward (Prox+Diff+Uncert) and 33.0% increase with the absolute reward (Prox+Abs+Uncert).
We can observe that the difference-based reward consistently outperforms the absolute reward except on
A NT R EACH , where the bias of the absolute reward Kostrikov et al.
(2019) helps the agent survive
[Figure] [Figure]
o m p l e t i o n ( % )
[Figure]
n ( % ) (c
[Figure]
w/ noise 1M 2M 3M 4M 5M Step
G o a l C
o m p l e t
i o n ( % ) (
[Figure]
Ant w/ noise longer and reach the goal.
Firstly, this shows the uncertainty penalty is more important for the proximity function as it models ﬁne-grain temporal information where inaccuracies can be exploited, as opposed to the binary classiﬁcation given by other adversarial imitation learning discriminators.
Secondly, both with and without the uncertainty penalty, the difference-based proximity reward performs better than the absolute proximity reward.
In conclusion, all three components of proximity, uncertainty, and difference-based reward are crucial for our method.
In Figure 6, we also evaluate the advantage of the additional sparse proximity reward given at the ﬁnal time step.
Compared to our method without this ﬁnal reward, it results in a minor 0.9% average performance improvement, meaning this component is not critical to our method.
C ONCLUSION
In this work, we propose a learning from observation (LfO) method inspired by how humans acquire skills by watching others performing tasks.
Speciﬁcally, we propose to learn a goal proximity function from demonstrations which provides an estimate of temporal distance to the goal.
Then, we utilize this learned proximity to encourage the policy to progressively move to states with higher proximity and eventually reach the goal.
The experimental results on navigation, robotic manipulation, and locomotion tasks demonstrate that our goal proximity function improves the generalization capability to unseen states, which results in better learning efﬁciency and superior performance of our method compared to the state-of-the-art LfO approaches.
Moreover, our method achieves comparable performance with LfD approaches.
A O VERVIEW
As supplementary material for enhancing the main paper, we provide the following information:
• Further analysis : We ﬁrst provide analyses of how our model and baselines generalize to unseen states.
We also include qualitative results demonstrating what the proximity function learns.
Finally, we show ablation experiments on additional environments.
• Implementation details : We describe the environments and provide training hyperparameters and architectures for our models.
• Code : Complete code of our model, environments, and experiments can be found in the code directory.
The code/README.md ﬁle documents the installation process, how to download the expert datasets, and commands for running experiments.
• Video :
We provide the results.mp4 ﬁle to present qualitative results of our method and baseline methods.
B C OMPARISON WITH GAIL
Our method shares a similar adversarial training process with GAIL.
The following steps describe how to achieve our method starting from GAIL.
Firstly, like the discriminator in GAIfO-s, our proximity function takes only states as input.
Next, rather than training the discriminator to classify expert from agent, we train the proximity function to regress to the proximity labels which arefor agent and the time discounted value between 0 and 1 for expert.
Our reward formulation also differs from GAIL approaches which give a log probability reward based on the discriminator output.
We instead incorporate a proximity estimation uncertainty penalty, a difference-based reward, and a sparse reward given for the proximity of the ﬁnal state, as shown in Equation 2.
[Figure]
C A NALYSIS ON G ENERALIZATION OF O UR M ETHOD AND B ASELINES
By learning to predict the goal proximity, the proximity function not only learns to discriminate expert and agent states but also models task progress, which provides more information about the task.
With additional supervision on learning goal proximity, we expect the proximity function to provide a more informative learning signal to the policy and generalize better to unseen states than baselines which overﬁt the reward function to expert demonstrations.
To analyze how well our method and the baselines can generalize to unseen states, we vary the difference between the states encountered in expert demonstrations and agent learning.
One way, we vary the difference between expert demonstrations and agent learning is by restricting the expert demonstrations to only cover parts of the state space.
For N AVIGATION , F ETCH P ICK and
F ETCH P USH we show results for demonstrations that cover 100% 75%, 50% and 25% of the state space.
For the discrete state space in N AVIGATION we restricted expert demonstrations to the fraction of possible agent start and goal conﬁgurations.
For the two continuous state F ETCH tasks, we break the 2D continuous sampling region a 4 by 4 grid and hold out one cell per row for 75% coverage and three cells per row for 25% coverage to sample the block and target starting positions from.
[Figure]
G o a l C
o m p l e t
i o n ( % ) (g) F ETCH P USH 50% 1M 2M 3M 4M 5M Step
G o a l C
o m p l e t
i o n ( % ) (h) F ETCH P USH 25%
Likewise, we also measured generalization by varying the difference between expert demonstrations and agent learning by increasing the initial state noise during agent learning.
On F ETCH P ICK ,
F ETCH P USH and A NT R EACH we show results for four different noise settings for the two F ETCH tasks, the 2D sampling region is scaled by the noise factor.
For A NT R EACH , uniform noise, scaled by the noise factor, is added to the initial joint angles, whereas the demonstrations have no noise.
If our method allows for greater generalization from the expert demonstrations, our method should perform well even under states different than those in the expert demonstrations.
The results of our method and baselines across varying degrees of generalization are shown in
Figure 8 and Figure 9.
Note that the results in the main paper are for 50% coverage in F ETCH
P ICK , 2x noise in F ETCH P USH , and 0.05 noise in A NT R EACH .
Across both harder and easier generalization, our method demonstrates more consistent performance.
While GAIfO-s performs well on high coverage, which requires little generalization in agent learning, its performance deteriorates as the expert demonstration coverage decreases.
D Q UALITATIVE RESULTS
In this section, we aim to qualitatively verify if the learned goal proximity function gives a meaningful measure progress towards the goal for the agent.
It is important for agent learning, that this proximity function gives higher values for states which are temporally closer to the goal.
To verify these intuitions, we visualize the proximity values predicted by the proximity function in a successful episode from agent learning in Figure 10.
From the qualitative results of F ETCH P ICK and F ETCH P USH in Figure 10, we can observe that as the agent moves the block closer to the goal, the predicted proximity increases.
This provides an example of the proximity function generalizing to agent experience and providing a meaningful reward signal for agent learning.
We notice that while the predictions increase as the agent nears the goal, the proximity prediction values are often low ( < 0.1) as in Figure 10a.
We hypothesize this is due to the adversarial training which labels agent experience with 0 proximity and lowers the average proximity predicted across states.
For videos of more qualitative evaluations for our method and baselines, refer to results.mp4 , also included in the submission.
[Figure]
n ( % ) (h) F ETCH P USH 2x 1M 2M 3M 4M 5M Step
G o a l C
o m p l e t
i o n ( % ) (i)
A NT R EACH 0.00 noise 1M 2M 3M 4M 5M Step
G o a l C
o m p l e t
i o n ( % ) (j) A NT R EACH 0.
1M 2M 3M 4M 5M Step
G o a l C
o m p l e t
i o n ( % ) (k)
A NT R EACH 0.
1M 2M 3M 4M 5M Step
G o a l C
o m p l e t
i o n ( % ) (l) A NT R EACH 0.
[Figure]
E F URTHER A BLATIONS
We include additional ablations to further highlight the advantages of our main proposed method over its variants.
We evaluate against the same ablations proposed in the main paper, but across all environments.
We present all these experiments in Figure 11.
We also attempted to compare to an ablation which learns the proximity function through a ranking based loss from Brown et al.
However, we empirically found it to be ineffective and therefore did not include it.
This ranking based loss uses the criterion that for two states from an expert trajectory s t 1 , s t 2 , the proximities should obey f ( s t 1 ) <
f ( s t 2 ) if t 1 < t 2 .
We therefore train the proximity function with the cross entropy loss − (cid:80) t i <
t j log
[Equation]
s
[Equation]
exp f φ ( s ti
[Equation]
+exp f φ ( s tj ) .
We incorporate agent experience by adding an additional loss which ranks expert states above agent states for randomly sampled pairs of expert and agent states ( s e , s a ) , through the cross entropy loss −
[Equation]
Unlike the discounting factor in the discounting based proximity function, the ranking based training requires no hyperparameters.
However, the lack of supervision on ground truth proximity scores could result in less meaningful predicted proximities and a worse learning signal for the agent, which could explain the observed poor performance.
As seen from the additional ablation analysis in Figure 11, our method has the best performance in the majority of environments.
In each task, incorporating uncertainty and online updates is crucial.
Our main method using the linear proximity function always performs slightly better than the exponential proximity function.
Using ofﬂine updates is helpful in all environments except N AVIGATION .
[Figure]
F F URTHER B ASELINES
In Figure 12, we compare to SQIL (Reddy et al., 2020), an imitation learning approach which demonstrates higher sample efﬁciency with off-policy RL.
SQIL modiﬁes the replay buffer with the expert demonstrations which are assigned reward +1 while all agent experience is assigned reward 0 .
We use Soft Actor-Critic (Haarnoja et al., 2018) using the same hyperparameters as in Reddy et al.
Our method consistently outperforms SQIL in the Fetch tasks, despite SQIL using actions whereas our method does not.
This can be because the Fetch tasks require precise actions (i.e., it is difﬁcult to reverse moving the block in the wrong direction) which results in covariate shift that negatively impacts BC and SQIL (a form of regularized BC).
On the other hand, SQIL learns the Ant Reach task well and performs similar to the behavioral cloning (BC) baseline since Ant Reach is not sensitive to small errors (i.e., the agent can recover from bad actions and change heading).
We observed unstable
SQIL results in Figure 12c, which can also be observed in Figure 2,3 of (Reddy et al., 2020), further training or hyperparameter tuning could stabilize training.
G I MPLEMENTATION D ETAILS G. E NVIRONMENT D ETAILS
The implementations of N AVIGATION , F ETCH (which includes F ETCH P ICK and F ETCH P USH ), and
A NT R EACH environments are based on Chevalier-Boisvert et al.
(2018), Plappert et al.
(2018), and
Ghosh et al.
(2018), respectively.
The actions in the F ETCH experiments use end effector position control and continuous control for the gripper.
A 15-dimensional state in F ETCH tasks consists of [Figure] the relative position of the goal from the object, relative position of the end effector to the object, and robot state.
We found that not including the velocity information was beneﬁcial for all learning from observation approaches.
Meanwhile, in N AVIGATION the state consists of a one-hot vector for each grid cell encoding wall, empty space, agent or goal.
Finally, in A NT R EACH the state consists of velocity, force and the relative goal position with the action space consisting of joint control.
Each environment also randomly initializes the starting state and goal.
The details of observation spaces, action spaces, and episode lengths are described in Table 1.
All units in this section are in meters unless otherwise speciﬁed.
For N AVIGATION we collect 250 expert demonstrations and for all other environments we collect 1000 expert demonstrations.
In N AVIGATION we use BFS search to collect expert demonstrations.
In F ETCH P ICK we generate demonstrations by hard coding the Sawyer Robot to ﬁrst reach above the object, then reach down and grasp, and ﬁnally move to the target position.
Similarly, in F ETCH
P USH , we reach behind the object and then execute a planar push forward towards to the goal.
For
A NT R EACH , we collect demonstrations using an expert policy trained using PPO (Schulman et al., 2017) based on the reward function R ( s, a )
= 1 − 0 .
2 · || p ant − p goal || 2 − 0 .
005 · || a || 22 , where p ant and p goal are ( x, y ) -positions of the ant and goal, respectively, and a is an action.
Please refer to the code for more details.
To evaluate the generalization capability of our method and baselines, we constrain the coverage of expert demonstrations or add additional starting state noise during agent learning as discussed in
Section C.
[Table]
G. N ETWORK ARCHITECTURES
Actor and critic networks: We use the same architecture for actor and critic networks except for the output layer where the actor network outputs an action distribution while the critic network outputs a critic value.
For N AVIGATION , the actor and critic network consists of CONV (3 , 2 , 16) − ReLU −
MaxPool (2 , 2) − CONV (3 , 2 , 32) − ReLU − CONV (3 , 2 , 64) followed by two fully-connected layers with hidden layer size 64, where CONV ( k, s, c ) represents a c -channel convolutional layer with kernel size k and stride s .
For other tasks, we model the actor and critic networks as a two separate 3-layer MLP with hidden layer size 256 .
For the continuous control tasks, the layer of the actor MLP is two-headed to output the mean and standard deviation of the action distribution.
Goal proximity function and discriminator:
The goal proximity function and discriminator use a CNN encoder followed by a hidden layer of size 64 for N AVIGATION and a 3-layer MLP with a hidden layer size of size 64 for other tasks.
G. T RAINING DETAILS
For our method and all baselines except BC (Pomerleau, 1989) and BCO (Torabi et al., 2018a), we train policies using PPO.
The hyperparameters for policy training are shown in Table 2, while the hyperparameters for the proximity and discriminator function are shown in Table 3.
For our method, in some tasks, we also found it slightly helpful to scale the reward by a constant factor and these values are also included in Table 2.
In BC [Table] the demonstrations were split into 80% training data and 20% validation data.
The policy was trained on the training data until the validation loss stopped decreasing.
The policy is then evaluated for 1,000 episodes to get an average success rate.
In GAIfO-s and GAIL the AIRL reward from Fu et al.
(2018) is used.
[Table] [Table]