T ASK -R ELEVANT A DVERSARIAL I MITATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We show that a critical problem in adversarial imitation from high-dimensional sensory data is the tendency of discriminator networks to distinguish agent and expert behaviour using task-irrelevant features beyond the control of the agent.
We analyze this problem in detail and propose a solution as well as several baselines that outperform standard Generative Adversarial Imitation Learning (GAIL).
Our proposed solution, Task-Relevant Adversarial Imitation Learning (TRAIL), uses a constrained optimization objective to overcome task-irrelevant features.
Comprehensive experiments show that TRAIL can solve challenging manipulation tasks from pixels by imitating human operators, where other agents such as behaviour cloning (BC), standard GAIL, improved GAIL variants including our newly proposed baselines, and Deterministic Policy Gradients from Demonstrations (DPGfD) fail to ﬁnd solutions, even when the other agents have access to task reward.
I NTRODUCTION
Generative Adversarial Networks (GANs) have produced breath-taking conditional image synthesis results (Goodfellow et al., 2014; Brock et al., 2019), and have inspired adversarial learning approaches to imitating behavior.
In Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016), a discriminator network is trained to distinguish agent and expert behaviour through its observations, and is then used as a reward function.
GAIL agents can overcome the exploration challenge by taking advantage of expert demonstrations, while also achieving high asymptotic performance by learning from agent experience.
Despite the huge promise of GAIL, it has not yet had the same impact as GANs; in particular, robust
GAIL from pixels for control applications remains a challenge.
Here, we study a key shortcoming of GAIL: the tendency of the discriminator to mainly exploit task-irrelevant features.
For example, by focusing on slight background differences, a discriminator can achieve perfect generalization, assigning zero reward to all held-out agent observations.
However, this discriminator does not yield an informative reward function because it ignores behavior. [Figure]
[Figure] Assuming there is an expert policy π E that is optimal for an unknown reward function, here we refer to a feature as task-irrelevant if it does not affect that reward.
For example, if the task is to lift a red block, the positions of other blocks would be task-irrelevant; see Figures 1 and 2.
This paper makes the following contributions:
It reveals a fundamental limitation of GAIL by showing that discriminators do in practice exploit task-irrelevant information, thereby resulting in poor task performance.
It introduces powerful GAIL baselines.
In particular, it shows that standard regularization and data augmentation are generally useful and improve upon standard GAIL.
It shows that these improvements to GAIL, as well as other improvements proposed by Reed et al.
(2018), do not completely solve the problem, allowing GAIL agents to fail catastrophically with the addition of task-irrelevant distractors.
It introduces Task Relevant Adversarial Imitation Learning (TRAIL) , using constrained optimization to force the discriminator to focus on the relevant aspects of the task, which improves performance dramatically on manipulation tasks from pixels (see Figure 1).
R ELATED WORK
The use of demonstrations to help agent training has been studied extensively in robotics (Bakker & Kuniyoshi, 1996; Kawato et al., 1994; Miyamoto et al., 1996) with approaches ranging from
Q-learning (Schaal, 1997) to behavioral cloning (BC) (Pomerleau, 1989).
BC BC is effective in solving many control problems (Pomerleau, 1989;
Finn et al., 2017; Duan et al., 2017; Rahmatizadeh et al., 2018).
It has also been successfully applied to initialize RL training (Rajeswaran et al., 2017).
It, however, suffers from compounding errors as initial small deviations from the expert behaviors tend to cause bigger differences (Ross et al., 2011).
This often necessitates a large number of demonstrations for satisfactory performance.
Furthermore, BC typically does not lead to agents that are superior to their demonstrators.
Inverse RL : Ziebart et al.
(2008); Ng et al.
; Abbeel & Ng (2004) propose inverse reinforcement learning (IRL) as a way of learning reward functions from demonstrations.
Reinforcement learning can then be used to optimize that learned reward.
Recently, Finn et al.
(2016b) approached continuous robotic control problems with success by applying Maximum Entropy IRL algorithms which are very closely related to GAIL (Finn et al., 2016a) and have similar drawbacks.
Learning from Demonstrations : Hester et al.
(2018) developed deep Q-Learning from demonstration (DQfD), in which expert trajectories are added to experience replay and jointly used to train agents along with their own experiences.
This was later extended by Vecerik et al.
(2017) and Pohlen et al.
(2018) to better handle sparse-reward problems in control and Atari games respectively.
Despite their efﬁciency, this class of methods still requires access to rewards in order to learn.
GAIL : Following the success of Generative Adversarial Networks (Goodfellow et al., 2014) in image generation, GAIL (Ho & Ermon, 2016) applies adversarial learning to the problem of imitation.
Although many variants are introduced in the literature (Li et al., 2017; Fu et al., 2018; Merel et al., 2017; Zhu et al., 2018; Baram et al., 2017), making GAIL work for high-dimensional input spaces, particularly raw pixels, remains a challenge.
A few papers (Peng et al., 2018; Reed et al., 2018; Blondé & Kalousis, 2018) seek to address the problem of overﬁtting the discriminator.
Peng et al.
(2018) introduces the Variational Bottleneck to regularize the discriminator.
Reed et al.
(2018) proposes to not train the vision module of the discriminator (e.g. use the vision module of the critic network instead) and only train a tiny network on top of the vision module to discriminate.
Blondé & Kalousis (2018) follow a similar approach.
Unstructured regularization, however, cannot stop the discriminator from ﬁtting to features that are systematically different between the agents’ and the demonstration’s behavior, like those illustrated in Figure 2.
TRAIL, on the other hand, is much less prone to overﬁtting to these features.
Stadie et al.
(2017) extend GAIL to the setting of third person imitation, in which the demonstrator and agent observations come from different views.
To prevent the discriminator from discriminating based on viewpoint domain, they use gradient ﬂipping from an auxiliary classiﬁer to learn domain-invariant features.
Our approach is not to learn domain-invariant features, but instead learn domain-agnostic discriminators that only focus on behavior.
Several recent works have focused on improving the sample efﬁciency of GAIL (Blondé & Kalousis, 2018; Sasaki et al., 2018).
Common to these approaches and to this work, is the use of off-policy actor critic agents and experience replay, to improve the utilization of available experience.
R EINFORCEMENT L EARNING AND A DVERSARIAL I MITATION
Following the notation of Sutton & Barto (2018), a Markov Decision Process (MDP) is a tuple ( S , A , R, P, γ ) with states S , actions A , reward function R ( s, a ) , transition distribution P ( s (cid:48) | s, a ) , and discount γ .
An agent in state s ∈ S takes action a ∈ A according to its policy π and moves to state s (cid:48) ∈ S according to the transition distribution.
The goal of RL algorithms is to ﬁnd a policy that maximizes the expected sum of discounted rewards, represented by the action value function
Q π ( s, a ) =
E π [ (cid:80) ∞ t =
γ t R ( s t , a t )] , where E π is an expectation over trajectories starting from s 0 = s and taking action a 0 = a and thereafter running the policy π .
To apply RL, it is essential that we have access to the reward function which is often hard to design and evaluate (Singh et al., 2019).
In addition, sparse rewards can cause exploration difﬁculties that pose great challenges to RL algorithms.
We therefore look to imitation learning and particularly
GAIL to derive a reward function from expert demonstrations.
In GAIL, a reward function is learned by training a discriminator network D ( s, a ) to distinguish between agent and expert state-action pairs.
The GAIL objective is thus formulated as follows:
[Equation]
where π is the agent policy, π E the expert policy, and H ( π ) an (optional) entropy regularizer.
The reward function is deﬁned simply: R ( s, a )
=
− log(1 − D ( s, a )) .
GAIL is theoretically appealing and practically simple.
The discriminator, however, can focus on any features to discriminate, whether these features are task-relevant or not.
In the next subsection we describe a way to constrain the discriminator network in order to prevent it from using task-irrelevant details to distinguish agent and expert data.
T ASK -R ELEVANT A DVERSARIAL I MITATION L EARNING (TRAIL)
We want the discriminator to focus on task-relevant features.
Our proposed solution, TRAIL, prevents the discriminator from being able to distinguish expert and agent behaviour based on selected aspects of the data.
For instance, the discriminator should distinguish agent and expert frames only when meaningful behavior is present in those frames.
In the absence of behavior useful to solve the task, e.g. in initial frames prior to the execution of the behavior, the discriminator should be agnostic.
To this end, we propose a constraint for the discriminator, such that its accuracy must not be greater than chance on a particular set of observations I that we will call the invariant set.
I will include only observations that can be distinguished as agent or expert in task-irrelevant ways.
Precisely, we formulate TRAIL in terms of the following constrained optimization problem:
[Equation]
To apply the above constraint in practice, we can optimize the reverse of the objective function on I .
That is, given a batch of N examples s e ∼ π E , s θ ∼ π θ from the expert and agent, and ˆ s e ∼ π E
and ˆ s θ ∼ π θ both in the set
I , we maximize the following augmented objective function:
[Equation]
where accuracy ( · , · ) is deﬁned as the average of discriminator accuracies:
[Equation]
The scalar λ ≥ 0 is a tunable hyperparameter.
We used single frames as states but sequences can also be used, and we do not require expert actions.
Not requiring actions enables learning in very off-policy settings, where the action dimensions and distributions of the demonstrator (another robot or human) are different from those of the agent.
4. T HE SELECTION OF THE INVARIANT SET
The selection of the invariant set I is a design choice.
In general, we could always contrive nonstationary and adversarial ways of making this choice difﬁcult.
However, we argue that in many situations of great interest, including our robotic manipulation setup, it is easy to propose effective and very general invariant sets.
A straightforward way to collect robot data is to execute a random policy .
We can then use the resulting random episodes, for both expert and agent, to construct the invariant set I .
Another way to construct I is to use early frames from both expert and agent episodes.
Since in early frames little or no task behavior is apparent, this strategy turns out to be effective and no extra data has to be collected.
This strategy also improves robustness with respect to variation in the initial conditions of the task; see for example block insertion in Figure 1(d).
Importantly, if the set I captures some forms of irrelevance but not all forms, it will nonetheless always help in improving performance.
In this regard, TRAIL will dominate its GAIL predecessor whenever the designer has some prior on what aspects of the data might be task irrelevant.
E XPERIMENTS
We focus on solving robot manipulation tasks.
The environment implements two work-spaces: one with a Kinova Jaco arm ( Jaco ), and the other with a Sawyer arm ( Sawyer ).
See supplementary material A.1 for a detailed description.
Environment rewards, which are not used by GAIL-based methods, are sparse and equal to +1 for each step when a given task is solved and 0 otherwise.
The maximum reward for the episode is 200, since this is the length of a single evaluation episode.
Our agent is based on the off-policy D4PG algorithm (Barth-Maron et al., 2018) because of its stability and data-efﬁciency (see supplementary material A.6).
Following Vecerik et al.
, we add expert demonstrations into the agents’ experience replay, and refer to the resulting RL algorithm as D4PG from Demonstrations (D4PGfD).
For each task we collect 100 human demonstrations.
Data augmentation Traditional data augmentation has proved beneﬁcial in imitation learning (Berseth & Pal, 2019).
Surprisingly, to the best of our knowledge, this has not been explicitly studied in prior publications on GAIL.
However, we ﬁnd that data augmentation is a generally useful component to prevent discriminator overﬁtting.
It drastically improves the baseline GAIL agent, and is necessary to solve any of the harder manipulation tasks.
We distort images by randomly changing brightness, contrast and saturation; random cropping and rotation; adding Gaussian noise.
When multiple sensor inputs are available (e.g. multiple cameras), we also randomly drop out these inputs, but leaving at least one active.
All discriminator-based methods in this work (including all baselines) use data augmentation except as indicated in the relevant ablations (see Section 5.2).
We also considered regularizing the GAIL discriminator with spectral normalization (Miyato et al., 2018).
It performed slightly better than GAIL, but still failed in the presence of distractor objects, and we thus omit spectral normalization in the main experiments for simplicity.
Actor early stopping When the agent has learned the desired behavior, and the resulting data is used for training, the discriminator will become unable to distinguish expert and agent observations based only on behavior.
This forces the discriminator to rely on task-irrelevant information.
To avoid this scenario, we propose to restart each actor episode after a certain number of steps such that successful behavior is rarely represented in agent data.
This enables the discriminator to recognize the goal condition, which appears frequently at the end of demonstration episodes, as representative of expert behavior.
To avoid hand-tuning the stopping step number, we found that the discriminator score can be used to derive an adaptive stopping criterion.
Concretely, we restart an episode if the discriminator score at the current step exceeds the median score of the episode so far for T patience consecutive steps (in practice we set T patience = 10 ).
We set invariant set hyperparameter λ = 1 and use adaptive early stopping for TRAIL.
The ablation with λ = 0 , i.e. adaptive early stopping only, is referred to as TRAIL-0.
Hence, the only difference between TRAIL-0 and TRAIL is that the later uses invariant set constraints.
5. B LOCK LIFTING WITH DISTRACTORS
In this section, we consider two variants of the lift task in the Sawyer work space: a) lift alone , where only one red cube is present, b) lift distracted , with two extra blocks (blue and green, see Figure 9).
We show how adding these additional distractors affects the training procedure.
We ﬁrst compare our method to baselines.
In doing so, the invariant set is constructed using the ﬁrst 10 frames from every episode, as in the rest of the work.
This choice does not require us to collect any extra data, and hence the comparison with baselines is fair.
In the next section, we elaborate on the choice of the invariant set and provide additional experimental results.
As baselines, we run BC and GAIL (with data augmentation).
Baseline GAIL is the strongest we were able to implement.
The only difference between this baseline and TRAIL is the use of invariant set constrains and actor early stopping (code, agent conﬁgurations like number of actors, and network architecture are the same).
We additionally consider the approaches proposed by Reed et al.
(2018) as GAIL-based baselines; using either a randomly initialized convolutional network, or a convolutional critic network, to provide ﬁxed vision features on top of which a tiny discriminator network is trained.
We call these two baselines random and critic respectively.
Finally, to show the importance of actor early stopping, we run TRAIL-0 (Fig. 3).
All methods perform satisfactorily on lift alone , but the proposed methods TRAIL-0 and TRAIL do best.
As expected, the performance of BC on lift distracted is similar to its performance on lift alone , despite the two additional blocks.
The two additional blocks in lift distracted affect the GAIL-based baselines, despite being irrelevant to the task.
To understand this effect, we conducted an additional experiment ( lift distracted seeded ).
Here the initial block positions are randomly drawn from the expert demonstrations.
Therefore, it is impossible to discriminate between expert and actor episodes using the ﬁrst few frames of an episode.
Note this initialization procedure is not applied to the evaluation actor, keeping the evaluation scores comparable between lift distracted and lift distracted seeded .
This experiment exposes one major culprit behind the performance degradation of GAIL: memorization.
The discriminator can achieve perfect accuracy by memorizing all 100 initial positions from the demonstration set, making the reward function uninformative.
By constraining the discriminator,
TRAIL squeezes out this irrelevant information and succeeds in solving the task in the presence of distractions.
TRAIL is the only method that is able to handle the variety of initial cube positions during training, achieving better than expert performance on lift distracted .
Interestingly, random performs reasonably on lift distracted .
Given random ’s strong performance, we conducted additional experiments to evaluate its effectiveness when trained with adaptive early stopping and present the results in Figure 12 of the supplementary material.
C ONSTRUCTING THE INVARIANT SET I
In the previous subsection, early frames were used to construct the invariant set (TRAIL-early).
Here, we evaluate another previously mentioned approach for constructing I ; random policy (TRAILrandom).
The lift distracted task caused all baselines to fail, but was solved by TRAIL.
We introduce a harder version of the task, where the expert appearance is different, to tease out the differences between TRAIL-early and TRAIL-random.
The difference in appearance between the expert and imitator allows the GAIL discriminator to trivially distinguish them.
The results and the differences in the expert appearance are presented in Figure 4.
[Figure]
The new task is indeed harder and it takes longer for TRAIL methods to achieve performance better than BC baseline, which is not affected by the different body appearance.
GAIL is clearly outperformed and does not take off.
The difference between TRAIL methods is negligible.
We also tried to mix them but the differences remain imperceptible.
Hence, in the following experiments we simply use early frames.
This choice is pragmatic as it does not require that we collect any extra data, and hence the comparison with
GAIL and other baselines is fair.
It is also very simple to apply in practice, even if one does not have access to the expert setup anymore.
Finally, it is general and powerful enough to be successfully used across all robotic manipulation tasks considered in this work.
To decide how many initial frames should be used to construct I , we conducted an ablation study and found out that the method is not very sensitive to this choice (see supplementary material A.2).
Hence, we chose 10 initial frames, and intentionally used the same number for all tasks to further emphasize generality of this choice.
5. A BLATION STUDIES M EASURING DISCRIMINATOR MEMORIZATION OF TASK - IRRELEVANT FEATURES
In this section, we experimentally conﬁrm that memorization is a limiting factor of GAIL methods.
We equipped the discriminator with two extra heads whose inputs are the ﬁnal spatial layer of the
ResNet for the lift distracted task.
The ﬁrst head is trained on the ﬁrst frames only, and has the same target as the main head ( i.e. discriminating between agent and expert).
To train the second head, we randomly divide the expert demonstrations into two equi-numerous subsets, and the task is to predict to which of these randomly chosen sets the demonstration was assigned to.
Both heads are trained via backpropagation but their gradient is not propagated to the ResNet so they do not inﬂuence the training procedure directly.
If our claim is correct, we expect the extra heads to have higher accuracy for TRAIL-0 as compared to TRAIL, since TRAIL representations are penalized for having features triggering memorization, i.e. the features should not aid discriminating based on the ﬁrst frames or predicting a random label for each expert demonstration.
No reasonably performing method is able to force 50% accuracy for extra heads since some features are important to solve the task ( e.g. the position of the red cube).
We also collected 25 extra holdout demonstrations and visualize the average discriminator prediction on them, and compare with predictions on the training demonstrations.
In Fig.
5, we see the overall accuracy of the main head for TRAIL-0 is signiﬁcantly higher, and the difference is larger for early steps (when TRAIL achieves only 50%, as expected).
TRAIL representations are less helpful for extra heads (see Figure 5 C-D).
Finally, TRAIL-0 clearly overﬁts on training demonstrations, predicting almost the maximum score, while only ∼ 0 . 25 is predicted for holdout demonstrations. The TRAIL average predictions for both datasets are almost identical.
A CTOR EARLY STOPPING (TRAIL-0)
In this section, we analyze the importance of adaptive early stopping on 3 tasks in the Jaco workspace: lift red cube ( lift ), put red cube in box ( box ), and stack red cube on blue cube ( stack ).
We consider D4PGfD and three GAIL-based models with varying termination policies: a) ﬁxed step (50), b) based on ground truth task rewards, and c) based on adaptive early stopping (TRAIL-0).
Using ground truth task rewards, an episode is terminated if the reward at the current step exceeds the median reward of the episode so far for 10 consecutive steps.
Results are presented in Figure 6.
Termination based on task reward is clearly superior; although unrealistic in practice, it deﬁnes the performance upper-bound and clearly shows that early stopping is beneﬁcial.
TRAIL-0 is robust and reaches human performance on all tasks.
A ﬁxed termination policy, when tuned, can be very effective.
The same ﬁxed termination step, however, does not work for all tasks.
See Figure 13 in supplementary material for the effects of varying termination steps.
Finally, as can be inferred from stack results, the dense rewards provided by TRAIL-0 are helpful in solving this challenging problem which is unsolved with D4PGfD even though D4PGfD uses ground truth task rewards.
D ATA AUGMENTATION
In Table 5.2, we report the best reward obtained in the ﬁrst 12 hours of training (averaged for all seeds; see Figure 14 for full curves).
The results show that data augmentation is needed in lift distracted .
For the lift alone task, TRAIL-0 with data augmentation performs on par with TRAIL.
[Table]
The performance of TRAIL is not affected by the lack of data augmentation on the lift alone task, whereas the performance of TRAIL-0 is.
L EARNING WITH A FIXED , PERFECT DISCRIMINATOR
To assess whether learned discriminators are necessary, we compare TRAIL against agents using a ﬁxed reward function corresponding to R expert = 1 and R agent = 0 for the lift alone and lift distracted tasks.
This baseline simulates an oracle discriminator with perfect generalization, but which is agnostic to behavior.
On the lift alone task, agents using this ﬁxed reward achieve roughly half the reward of TRAIL asymptotically, and on lift distracted they do not solve the task (average rewards are less than 5).
See supplementary Figure 16 for learning curves.
5. L EARNING FROM OTHER EMBODIMENTS AND PROPS
Since the TRAIL discriminator is trained to ignore task-irrelevant features, it can learn from demonstrations with different embodiments and props.
Figure 7 shows that GAIL even with augmentation fails to learn block lifting from a different embodiment, and performs worse when the expert uses a different prop color.
TRAIL solves the task and achieves better performance in both cases.
[Figure]
TRAIL
5. E VALUATION ON DIVERSE MANIPULATION TASKS
To further demonstrate beneﬁts of using our proposed method we present results for TRAIL, TRAIL0, the baseline GAIL and BC on more challenging tasks.
Speciﬁcally we consider stack with the
Sawyer robot; and insertion and stack banana in the Jaco work-space.
The results are shown in
Figure 8.
The tasks we consider here are much harder as evidenced by the performance of BC agents.
These experiments suggest that TRAIL is generally useful as an improvement over GAIL, even when the tasks are not designed to include task-irrelevant information.
C ONCLUSIONS
To make adversarial imitation work on nontrivial tasks from pixels, it is crucial to prevent the discriminator from exploiting task-irrelevant information.
Data augmentation is a necessary component that prevents the discriminator from overﬁtting, but alone is not sufﬁcient to solve more challenging problems.
Our proposed method TRAIL proved to be a remedy which effectively focuses the discriminator on the task even when task-irrelevant features are present. We showed through ablations that TRAIL beneﬁts from data augmentation, actor early stopping, and the use of invariant set constraints. The last enables our agent to solve the full task suite in our experiments showing its decisive importance.
A S UPPLEMENTARY MATERIAL A. D ETAILED DESCRIPTION OF ENVIRONMENT
All our simulations are conducted using MuJoCo 1 (Todorov et al., 2012).
We test our proposed algorithms in a variety of different envrionments using simulated Kinova Jaco 2 , and Sawyer robot arms 3 ; see Figure 9.
We use the Robotiq 2F85 gripper 4 in conjuction with the Sawyer arm.
To provide demonstrations, we use the SpaceNavigator 3D motion controller 5 to set Cartesion velocities of robot arm.
The gripper actions are implemented via the buttons on the controller.
All demonstrations in our experiments are provided via human teleoperation and we collecteddemonstrations for each experiment.
Jaco : When using the Jaco arm, we use joint velocity control (9DOF) where we control all 6 joints of arm and all 3 joints of the hand.
The simulation is run with a numerical time step of 10 milliseconds, integrating 5 steps, to get a control frequency of 20HZ.
The agent uses a frontal camera of size 64 × 64 (see Figure 10(a)).
For a full list of observations the agent sees, please refer to Table 2(a).
[Figure]
Sawyer : When using the Sawyer arm, we use Cartesian velocity control (6DOF) for the robot arm and add one additional action for the gripper resulting in 7 degrees of freedom.
The simulation is run with a numerical time step of 10 milliseconds, integrating 10 steps, to get a control frequency of 10HZ.
The agent uses two frontal cameras of size 64 × 64 situated on the left and right side of the robot cage respectively (see Figure 10(b, c)).
For a full list of observations the agent sees, please refer to Table 2(b).
For all environments considered in this paper, we provide sparse rewards (i.e. if task is accomplished, the reward is 1 and 0 otherwise).
In experiments regards our proposed methods, rewards are only used for evaluation purposes and not for training the agent.
[Table]
A. E ARLY FRAMES ABLATION STUDY
Here
[Figure]
we vary the number of early frames of each episode used to form I .
We report that a range of values from 1 up to 20 works well across both tasks (Put in box and Stack banana)
[Figure]
with performance gradually degrading as the value increased beyond 20 (Fig. 11)
[Figure]
A. TRAIL-0 WITH random
We found out and mentioned in the subsection 5.1 that random beneﬁts from TRAIL-0.
Unfortunately, it is still prone to overﬁtting and hence, worse than our full method – TRAIL.
We present random + TRAIL-0 accompanied with our methods in Figure 12.
Our TRAIL and random + TRAIL-0 are the only methods exceeding BC performance on lift distracted .
However, TRAIL performance is clearly better (obtains higher rewards and never gets overﬁtted).
A. F IXED TERMINATION POLICY
As mentioned in the subsection 5.
[Figure]
the most basic early termination policy – ﬁxed step termination – may be very effective if tuned.
Since the tuning may be expensive in practice, we recommend using adaptive early stopping (TRAIL-0).
However, for the sake of completeness we provide results for ﬁxed step termination policy depending on the hyperparameter tuned.
The results for stack task are presented in Figure 13.
The Jaco work space is considered here because Sawyer requires TRAIL to obtain high rewards.
As can be inferred from the ﬁgure, the performance is very sensitive to the ﬁxed step hyperparameter.
We refer to subsection 5.2 for more comments on all methods.
A. D ATA AUGMENTATION
An extra set of experiments on lift alone and lift distracted tasks (described in subsection 5.1) has been performed to show importance of data augmentation.
Because in the subsection 5.2 only the peak performance is presented (Table 5.2), we present here the full curves in Figure 14.
The results shows that data augmentation is necessary to obtain high rewards in lift distracted .
For easier lift alone , TRAIL-0 with data augmentation perform at par with TRAIL.
A. D4PG
We use D4PG (Barth-Maron et al., 2018) as our main training algorithm.
Brieﬂy, D4PG is a distributed off-policy reinforcement learning algorithm for continuous control problems.
In a nutshell, D4PG uses Q-learning for policy evaluation and Deterministic Policy Gradients (DPG) (Silver et al., 2014) for policy optimization.
An important characteristic of D4PG is that it maintains a replay memory
M (possibility prioritized (Horgan et al., 2018)) that stores SARS tuples which allows for off-policy learning.
D4PG also adopts target networks for increased training stability.
In addition to these principles, D4PG utilized distributed training, distributional value functions, and multi-step returns to further increase efﬁciency and stability.
In this section, we explain the different ingredients of D4PG.
D4PG maintains an online value network Q ( s, a | θ ) and an online policy network π ( s | φ ) .
The target networks are of the same structures as the value and policy network, but are parameterized by different parameters θ (cid:48) and φ (cid:48) which are periodically updated to the current parameters of the online networks.
Given the Q function, we can update the policy using DPG:
[Equation]
Instead of using a scalar Q function, D4PG adopts a distributional value function such that
Q ( s t , a | θ ) =
E (cid:2) Z ( s t , a | θ ) (cid:3) where Z is a random variable such that Z
=
z i w.p. p
i ∝ exp( ω ( s t , a | θ )) .
The z i ’s take on V bins discrete values that ranges uniformly between V min and
V max such that z i
=
V min
+ i
[Equation]
max − V min V bins for i ∈ { 0 , · · · , V bins − 1 } .
To construct a bootstrap target, D4PG uses N-step returns.
Given a sampled tuple from the replay memory: s t , a t , { r t , r t +1 , · · · , r t + N − 1 } , s t + N , we construct a new random variable Z (cid:48) such that
Z
[Equation]
= z
[Equation]
) | θ
[Equation]
Notice, Z (cid:48) no longer has the same support.
We therefore adopt the same projection Φ employed by Bellemare et al.
The training loss for the value function
[Equation]
where H is the cross entropy.
D4PG is also distributed following Horgan et al.
Since all learning processes only rely on the replay memory, we can easily decouple the ‘actors’ from the ‘learners’.
D4PG therefore uses a large number of independent actor processes which act in the environment and write data to a central replay memory process.
The learners could then draw samples from the replay memory for learning.
The learner also serves as a parameter server to the actors which periodically update their policy parameters from the learner.
In our experiments, we always have access to expert demonstrations.
We, therefore adopt the practice from DQfD and DDPGfD and put the demonstrations into our replay buffers.
For more details see
Algorithms 1, 2.
Given: end
Algorithm 2 Learner
Given:
• an off-policy RL algorithm A • a replay buffer M • a replay buffer of expert demonstrations M e
t
A. N ETWORK ARCHITECTURE AND HYPERPARAMETERS
Actor and critic share a residual pixel encoder network with eight convolutional layers (3x3 convolutions, three 2-layer blocks with 16, 32, 32 channels), instance normalization (Ulyanov et al., 2016) and exponential linear units (Clevert et al., 2015) between layers.
The policy is a 3-layer MLP with ReLU activations with hidden layer sizes (300, 200).
The critic is a 3-layer MLP with ReLU activations with hidden layer sizes (400, 300).
For a illustration of the network.
Please see Figure 15.
[Figure]
The discriminator network uses a pixel encoder of the same architecture as the actor critic, followed by a 3-layer MLP with ReLU activations and hidden layer sizes (32,
[Table]
A. C OMPARING AGAINST LEARNING WITH FIXED REWARDS
To check whether learned discriminators are needed, we compare TRAIL against agents using a ﬁxed reward function corresponding to R expert = 1 and R agent = 0 for the lift alone and lift distracted tasks.
This baseline simulates an oracle discriminator with perfect generalization, but which is agnostic to behavior.
On the lift alone task, agents using this ﬁxed reward achieve roughly half the reward of TRAIL asymptotically, and on lift distracted they do not solve the task (average rewards are less than 5).
The learning curves are presented in Figure 16.