G ENERATIVE PREDECESSOR MODELS FOR SAMPLEEFFICIENT IMITATION LEARNING
A BSTRACT
We propose Generative Predecessor Models for Imitation Learning (GPRIL), a novel imitation learning algorithm that matches the state-action distribution to the distribution observed in expert demonstrations, using generative models to reason probabilistically about alternative histories of demonstrated states.
We show that this approach allows an agent to learn robust policies using only a small number of expert demonstrations and self-supervised interactions with the environment.
We derive this approach from ﬁrst principles and compare it empirically to a stateof-the-art imitation learning method, showing that it outperforms or matches its performance on two simulated robot manipulation tasks and demonstrate signiﬁcantly higher sample efﬁciency by applying the algorithm on a real robot
I NTRODUCTION
Training or programming agents to act intelligently in unstructured and sequential environments is a difﬁcult and central challenge in the ﬁeld of artiﬁcial intelligence.
Imitation learning provides an avenue to tackle this challenge by allowing agents to learn from human teachers, which constitutes a natural way for experts to describe the desired behavior and provides an efﬁcient learning signal for the agent.
It is thus no surprise that imitation learning has enabled great successes on robotic (Chernova and Thomaz, 2014) as well as software domains (e.g. Aytar et al.
Yet, key challenges in the ﬁeld are diverse and include questions such as how to learn from observations alone (e.g. Aytar et al.
(2018)), learning the correspondence between the expert’s demonstrations and the agent’s observations (e.g. Sermanet et al.
(2017)) as well as the question of how to integrate imitation learning with other approaches such as reinforcement learning (e.g. Vecerik et al.
However, at the core of the imitation learning problems lies the challenge of utilizing a given set of demonstrations to match the expert’s behavior as closely as possible.
In this paper, we approach this problem considering the setting where the set of expert demonstrations is given up-front and the dynamics of the environment can only be observed through interaction.
In principle, imitation learning could be seen as a supervised learning problem, where the demonstrations are used to learn a mapping from observed states to actions.
This solution approach is known as behavioral cloning.
However, it has long been known that the sequential structure of the task admits more effective solutions.
In particular, the assumptions made in supervised learning are restrictive and don’t allow the agent to reason about the effect of its actions on it’s future inputs.
As a result, errors and deviations from demonstrated behavior tend to accumulate over time as small mistakes lead the agent to parts of the observation space that the expert has not explored (Ross and
Bagnell, 2010).
In this work, we propose a novel imitation learning algorithm, Generative Predecessor Models for Imitation Learning (GPRIL), based on a simple core insight:
Augmenting the training set with state-action pairs that are likely to eventually lead the agent to states demonstrated by the expert is an effective way to train corrective behavior and to prevent accumulating errors.
Recent advances in generative modeling, such as Goodfellow et al.
; Kingma and Welling (2013); Van Den Oord et al.
(2016b;a);
Dinh et al.
(2016), have shown great promise at modeling complex distributions and can be used to reason probabilistically about such state-action pairs.
Speciﬁcally, we propose to utilize Masked Autoregressive Flows (Papamakarios et al., 2017) to model long-term predecessor distributions, i.e. distributions over state-action pairs which are conditioned on a state that the agent will see in the future.
Predecessor models have a long history in reinforcement learning (e.g. Peng and Williams (1993)) with recent approaches using deep networks to generate off-policy transitions (Edwards et al., 2018; Pan et al., 2018) or to reinforce behavior leading to high-value states (Goyal et al., 2018).
Here, we use predecessor models to derive a principled approach to state-distribution matching and propose the following imitation learning loop:
Interact with the environment and observe state, action as well as a future state.
To encode long-term corrective behavior, these states should be multiple steps apart.
Train a conditional generative model to produce samples like the observed state-action pair when conditioned on the observed future state.
Train the agent in a supervised way, augmenting the training set using data drawn from the model conditioned on demonstrated states.
The additional training data shows the agent how to reach demonstrated states, enabling it to recover after deviating from expert behavior.
In the above, we laid out the sketch of an algorithm that intuitively learns to reason about the states it will observe in the future.
In section 3, we derive this algorithm from ﬁrst principles as a maximum likelihood approach to matching the state-action distribution of the agent to the expert’s distribution.
In section 4, we compare our approach to a state-of-the-art imitation learning method (Ho and
Ermon, 2016) and show that it matches or outperforms this baseline on our domains while being signiﬁcantly more sample efﬁcient.
Furthermore, we show that GPRIL can learn using demonstrated states alone, allowing for a wider variety of methods to be used to record demonstrations.
Together these properties are sufﬁcient to allow GPRIL to be applied in real-world settings, which we demonstrate in section 4.3.
To our knowledge this is the ﬁrst instance of dynamic, contact-rich and adaptive behavior being taught solely using the kinesthetic-teaching interface of a collaborative robot, without resorting to tele-operation, auxiliary reward signals, or manual task-decomposition.
B ACKGROUND 2. M ARKOV DECISION PROCESSES WITHOUT REWARDS
As is usual, we model the problem as a Markov decision process without reward.
That is, given state and action sets S , A , the agent is observing states s ∈ S and taking actions a ∈ A .
In this work, we use s and a to refer to states and actions observed during self-supervision and s ∈ S and a ∈ A to refer to target and demonstration states and actions.
We furthermore use superscripts s ( i ) , a ( i ) to refer to speciﬁc instances, e.g. speciﬁc demonstrated state-action pairs, and subscripts, e.g. s t , a t , to indicate temporal sequences.
The observed transitions are guided by the Markovian dynamics of the environment and the probability of transitioning from state s to state s (cid:48) by taking action a is denoted as p ( s t +
= s (cid:48) | s t = s, a t = a ) .
The agent’s behavior is deﬁned by a stationary parametric policy π θ ( a | s ) while the expert’s behavior is modeled by a stationary distribution π ∗ ( a | s ) .
We denote as d πt ( s ) the probability of observing state s at time-step t when following policy π .
Under the usual ergodicity assumptions, each such policy induces a unique stationary distribution of observed states d π ( s ) =
lim t →∞ d πt ( s ) as well as a stationary joint state-action distribution ρ π ( s, a ) :
=
π ( a | s ) d π ( s ) .
Furthermore, we use q πt to refer to the dynamics of the time reversed Markov chain induced by a particular policy π at time-step t
[Equation]
t +1 =
s (cid:48) | s t = s, a t = a ) (1) and deﬁne q π ( s t
= s, a t = a | s t +1 =
s (cid:48) ) :
=
lim t →∞ q πt ( s t = s, a t = a | s t +
= s (cid:48) ) .
For the purposes of this work, we handle the episodic case with clear termination conditions by adding artiﬁcial transitions from terminal states to initial states.
This creates a modiﬁed, ergodic MDP with identical state-distribution and allows us to assume arbitrarily large t such that q π t = q π .
Finally, we extend this notation to multi-step transitions by writing q π ( s t = s, a t = a | s t
+
j = s (cid:48) ) .
2. I MITATION LEARNING
In this work, we are considering two settings of imitation learning.
In the ﬁrst setting, the agent is given a set of observed states s (1) , s (2) , · · · , s ( N ) and observed corresponding actions a (1) , a (2) , · · · , a ( N ) as expert demonstrations.
The goal in this setting is to learn a policy π θ that matches the expert’s behavior as closely as possible.
In the second setting, the agent is given the states observed by the expert but is not aware of the actions the expert has taken.
Recent years have seen heightened interest in a related setting where to goal is to track expert state trajectories (Zhu et al., 2018;
Peng et al., 2018; Pathak et al., 2018).
These approaches do not learn general policies that can adapt to unseen situations.
A straightforward approach to train general policies in the ﬁrst setting, usually referred to as behavioral cloning, is to treat the task as a supervised learning problem (e.g. Pomerleau (1989)).
However, as outlined in section 1, predictions made by π θ ( a | s ) inﬂuence future observations thus violating a key assumption of supervised learning, which states that inputs are drawn from an i.i.d. distribution.
This has formally been analyzed by Ross et al.
who introduce a family of algorithms (e.g. Ross and Bagnell (2010)
; Ross et al.
that provably avoid this issue.
However, these approaches require the expert to continuously provide demonstrations and thus are not applicable when the set of demonstrations is assumed to be ﬁxed.
A popular avenue of research that considers this setting is inverse reinforcement learning (IRL).
Inverse reinforcement learning (Ng and Russell, 2000) aims to learn a reward function for which π ∗ is optimal and thus captures the intent of the expert.
The arguably most successful approach to IRL aims to match the state-action distribution to that of the demonstrations (Ziebart et al., 2008) with recent approaches extending these ideas to the model free case (Boularias et al., 2011;
Finn et al., 2016; Fu et al., 2018).
However, inverse reinforcement learning is indirect and ill-deﬁned as many reward functions induce the same behavior.
Recently, methods have been proposed that aim to match state-action distributions directly and achieve state-of-the-art result without learning a reward function ﬁrst.
Generative
Adversarial Imitation Learning (GAIL) (Ho and Ermon, 2016) uses an adversarial objective, training a discriminator to identify demonstrations and using TRPO to train a policy that fools the discriminator.
While GAIL is able to achieve impressive results, the adversarial objective can make the learning procedure unstable and unpredictable.
This is especially true when parts of the state-space are not under the agent’s control, yielding a setting resembling a conditional GAN (Mirza and Osindero, 2014), which are prone to issues such mode collapse (Odena et al., 2017).
We compare our approach with GAIL in section 4.
State Aware Imitation Learning (SAIL) (Schroecker and Isbell, 2017) is an alternative method that aims to learn the gradient of the state-action distribution using a temporal-difference update rule.
This approach is able to avoid instabilities prompted by the adversarial learning rule but is only applicable to policies with a small number of parameters where learning a representation of the gradient is feasible.
In this work, we follow a gradient descent approach similar to SAIL but estimate the gradient without representing it explicitly by a neural network.
These methods can also be used to match state-distributions without actions.
While the order of states is unspeciﬁed and the induced policy is therefore not unique, including transition information such as velocities can allow the agent to learn solely from expert state trajectories nonetheless.
2. G ENERATIVE MODELS
Recent years have seen great advances in deep generative models.
A variety of approaches such as
Generative Adversarial Networks (Goodfellow et al., 2014), Variational Auto Encoders (Kingma and Welling, 2013), autoregressive networks (e.g. Germain et al. (2015); Van Den Oord et al.
(2016b;a) and normalizing ﬂows (Dinh et al., 2016) have been proposed which enable us to learn complex distributions and efﬁciently generate samples.
In this work, we use generative models to model the distribution of long-term predecessor state-action pairs.
While the approach we propose is model agnostic, we choose to model this distribution using masked autoregressive ﬂows (Papamakarios et al., 2017) (MAF).
MAFs are trained using a maximum likelihood objective, which allows for a stable and straight-forward training procedure.
Autoregressive models are capable of representing complex distributions p ( x ); x ∈ R n by factoring the distribution p ( x ) =
p 1 ( x 1 ) (cid:81) N − 1 i =1 p
i +
( x i +1 | x 1 , . . .
, x i ) and learning a model for each p i .
In this paper, we model each x i to be distributed by x i ∼ N ( ·| µ i , σ i ) where each µ i and σ i is a function of x 1: i − 1 .
Masked autoencoders (Germain et al., 2015) provide a straight-forward approach to parameter sharing and allow representing these functions using a single network.
MAFs stack multiple autoregressive models with different orderings and thus avoid the strong inductive bias imposed by the order of variables.
Using the reparameterization trick, the autoregressive model can be seen as a deterministic and invertible transformation of a random variable: x = f ( z ) :
[Equation]
+
[Equation]
; z
[Equation]
N ( ·| 0 , 1) .
The change of variable formula then allows us to calculate the density of x :
[Equation]
[Table]
Where the autoregressive nature of f in eq.
2 allows for tractable computation of the second term.
MAFs chain multiple such transformations to derive highly expressive explicit density models able to model complex dynamics between target states and long-term predecessor state-action pairs.
GPRIL
In section 1, we provided an intuitive framework for using predecessor models to augment our training set and achieve robust imitation learning from few samples.
In this section, we will derive this algorithm based on state-action distribution matching.
To this end, we ﬁrst derive the gradient of the logarithmic state distribution based on samples from a long-term predecessor distribution that we will deﬁne below.
In section 3.2, we describe how to train a generative model of the predecessor distribution, which will allow us to evaluate this gradient.
Ascending on this gradient evaluated at demonstrated states leads the agent to stick to those states and thus provides a corrective measure (Schroecker and Isbell, 2017).
Furthermore, reproducing the states of the expert can be sufﬁcient to achieve the correct behavior if the state-space is chosen appropriately as we will show in section 4.
We will show how to use this gradient to match state-action-distributions in section 3.3.
3. E STIMATING THE GRADIENT OF THE STATE DISTRIBUTION
Here, we will show that the samples drawn from a long-term predecessor distribution conditioned on s enable us to estimate the gradient of the logarithmic state distribution ∇ θ log d π θ ( s ) and, later, to match the agent’s state-action-distribution to that of the expert.
To achieve this goal, we can utilize the fact that the stationary state distribution of a policy can be deﬁned recursively in terms of the state distribution at the previous time step, similar to Morimura et al.
[Equation]
Taking the derivative shows that this notion extends to the gradient as well as its logarithm:
[Equation]
The recursive nature of this gradient then allows us to unroll the gradient indeﬁnitely.
However, this process is cumbersome and will be left for appendix A. We arrive at the following equality:
[Equation]
[Equation] The derivation of our approach now rests on two key insights: First, in ergodic Markov chains, such as the ones considered in our setting, decisions that are made at time t affect the probability of seeing state s at time t + j more strongly if j is small.
In the limit, as j → ∞ , the expectation of the gradient ∇ θ log π θ ( a t | s t ) vanishes and the decision at time t only adds variance to the gradient estimate.
Introducing a discount factor γ similar to common practice in reinforcement learning (Sutton and
Barto, 1998) places more emphasis on decisions that are closer in time and can thus greatly reduce variance.
We explore this interpretation further in appendix C. Second, by introducing a discount factor, the effective time-horizon is now ﬁnite.
This allows us to replace the sum over all states and actions in each trajectory with a scaled expectation over state-action pairs.
Formally, we can write this as follows and arrive at our main result:
[Equation]
where B π θ corresponds to the long-term predecessor distribution modeling the distribution of states and actions that, under the current policy π θ , will eventually lead to the given target state s :
[Equation]
3. L ONG - TERM GENERATIVE PREDECESSOR MODELS
In the previous section, we derived the gradient of the logarithm of the stationary state distribution as approximately proportional to the expected gradient of the log policy, evaluated at samples obtained from the long-term predecessor distribution B π θ .
In this work, we propose to train a model
B π θ ω to represent B π θ and use its samples to estimate ∇ θ log d π θ ( s ) .
However, rather than unrolling a time-reversed Markov model in time, which is prone to accumulated errors, we propose to use a generative model to directly generate jumpy predictions.
We have furthermore found that imposing a sensible order on autoregressive models achieves good results and thus propose to use two conditional MAFs (Papamakarios et al., 2017)
B sω s , B aω a in a factored representation:
[Equation]
To train this model, we collect training data using self-supervised roll-outs:
We sample states, actions and target-states where the separation in time between the state and target-state is selected randomly based on the geometric distribution parameterized by γ as a training set for B π θ ω .
Training data for B sω s , B aω
a are obtained by executing the current policy to obtain a sequence s 0 , a 0 , s 1 , a 1 , · · · , which we store in a replay buffer.
In practice, we store data from multiple it- erations in this buffer in order to decrease the variance of the gradient.
While our algorithm does not explicitly account for off-policy samples, we found empirically that a short replay buffer does not degrade ﬁnal performance while signiﬁcantly improving sample efﬁciency.
To obtain a training sample, we ﬁrst pick s = s t and a = a t for a random t .
We now select a future state s =
s t + j +1 from that sequence.
For any particular s t + j +1 we now have s, a ∼ q π θ t ( s t = · , a t = ·| s t
+ j +1 =
s )
≈ q π θ ( s t = · , a t = ·| s t + j +1 =
s ) .
Note that in the episodic case, we can add transitions from terminal to initial states and pick t to be arbitrarily large such that the approximate equality becomes exact (as outlined in section 2.1).
In non-episodic domains, we ﬁnd the approximation error to be small for most t .
Finally, we choose j at random according to a geometric distribution j ∼ Geom (1 − γ ) and have a training triple s, a, s that can be used to train B aω a and B sω s as it obeys [Equation]
[Equation]
3. M ATCHING STATE - ACTION DISTRIBUTIONS WITH GPRIL
State-action distribution matching has been a promising approach to sample-efﬁcient and robust imitation learning (see section 2.2).
While each policy induces a unique distribution of states and behavioral cloning would therefore be sufﬁcient in the limit, it is sub-optimal the case of limited
[Figure] cce [Figure]
t
[Figure]
data. Matching the joint-distribution directly ensures that we minimize discrepancies between everything we observed from the expert and the behavior the agent exhibits. In this work, we propose a maximum likelihood based approach, ascending on the estimated gradient of the joint distribution:
[Equation]
(11) where ∇ θ log π θ ( a | s ) can be computed directly by taking the gradient of the policy using the demonstrated state-action pairs and ∇ θ log d π θ ( s ) can be evaluated using samples drawn from B π θ ω ( · , ·| s ) according to equation 7.
We introduce scaling factors β π and β d to allow for ﬁner control, interpolating between matching states only ( β π = 0 ) and behavioral cloning ( β d = 0 ) and have:
[Equation]
Here, higher values of β π provide more supervised guidance while lower values aim to prevent accumulating errors.
This gives rise to the full-algorithm: We ﬁll the replay buffer by asynchronously collecting experience using the current policy.
Simultaneously, we repeatedly draw samples from the replay buffer to update the predecessor models and use expert samples in combination with an equal number of artiﬁcial samples to update the policy.
This procedure is described fully in algorithm 1.
E XPERIMENTS
To evaluate our approach, we use a range of robotic insertion tasks similar to the domains introduced by Vecerik et al.
but without access to a reward signal or, in some cases, expert actions.
We choose these domains both for their practical use, and because they highlight challenges faced when applying imitation learning to the real world.
Speciﬁcally, collecting experience using a robot arm is costly and demands efﬁcient use of both demonstrations and autonomously gathered data.
Furthermore, insertion tasks typically require complex searching behavior and cannot be solved by open-loop tracking of a given demonstration trajectory when the socket position is variable.
We ﬁrst compare against state-of-the-art imitation learning methods on a simulated clip insertion task, then explore the case of extremely sparse demonstrations on a simulated peg insertion task and ﬁnally, demonstrate real-world applicability on its physical counterpart.
[Figure]
IQR of trajectories that are successfully inserting the peg.
Providing only ﬁnal states is signiﬁcantly faster.
f)
Best seed performance on both variations of peg insertion on the real robot.
4. C LIP INSERTION
In the ﬁrst task, a simulated robot arm has to insert an elastic clip into a plug, which requires the robot to ﬁrst ﬂex the clip in order to be able to insert it (see ﬁgure 1a).
In real-world insertion tasks, the pose of the robot, the socket, or the grasped object may vary.
We capture this variability by mounting the socket on a pan-tilt unit, which is randomized by ± 0 .
8 (pan) and ± 0 .
2 radians (tilt).
To perform this behavior, the robot observes proprioceptive features, speciﬁcally joint position, velocity and torques as well as the position of the end-effector and the socket orientation as a unit quaternion.
The task terminates when the robot leaves the work-space, reaches the goal, or after 50 seconds.
For comparative evaluation, we train a policy network to predict mean and variance, modelling a multivariate normal distribution over target velocities and train it using GPRIL, GAIL as well as behavioral cloning.
We record expert demonstrations using tele-operation and normalize observations based on the recorded demonstrations.
We then train GPRIL using a single asynchronous simulation and compare against the open source implementation of GAIL 1 for which we use 16 parallel simulations.
We select the best hyper parameters for GAIL found on a grid around the hyperparameters used by Ho and Ermon (2016) but lower the batch size to 256 as it increases the learning speed and accounts for the signiﬁcantly slower simulation of the task.
We furthermore enable bootstrapping regardless of whether or not the episode terminated.
As all discriminator rewards are positive, handling terminal transitions explicitly can induce a bias towards longer episodes.
This is beneﬁcial in the domains used by Ho and Ermon but harmful in domains such as ours where the task terminates on success.
A detailed list of hyper-parameters can be found in appendix B.
We report ﬁnal results after convergence and can see in ﬁgure 1b that both GAIL and GPRIL outperform behavioral cloning, indicating that generalizing over state-action trajectories requires fewer demonstrations than generalizing over actions alone.
Furthermore, we observe a higher success rate using GPRIL and ﬁnd that policies trained using GPRIL are more likely to retry insertion if the robot slides the clip past the insertion point.
To compare sample efﬁciency of GPRIL to GAIL, we limit the rate at which the asynchronous actor is collecting data.
While sample efﬁciency of GAIL could be increased by decreasing batch size or increasing various learning rates, we found that this can lead to unstable learning performance while reducing the amount of samples required by only a small amount.
As can be seen in ﬁgure 1c, GPRIL requires several orders of magnitudes fewer environment interactions to learn this task.
Finally, we evaluate the case where the expert’s actions are unknown.
Since the state-space includes information about joint velocities as well as positions, we ﬁnd that matching the state-distribution is sufﬁcient to solve the task.
GPRIL can achieve this by setting β d = 1 and β π = 0 .
As can be seen in ﬁgure 1d, performance deteriorates only marginally with a similar difference in performance between both methods.
4. P EG INSERTION WITH PARTIAL DEMONSTRATIONS
The second task is a simulated version of the peg-insertion task depicted in ﬁgure 2a.
In this task, the robot has to insert the peg into the hole, which is again mounted on a pan-tilt that randomly assumes pan and tilt angles varying by 0.4 and 0.1 respectively.
Hyperparameters are largely identical and we report minor differences in appendix B. Observation and action space are identical with the exception of the omission of torques from the observation space as they are not necessary to solve this task.
We use this task to evaluate the performance of GAIL and GPRIL when learning from only a very limited set of demonstrated states.
To this end, we compare three different scenarios in which the demonstrations are sparsiﬁed to varying degrees: In the ﬁrst case, the agent has access to the full state-trajectories of the expert, in the second only every tenth state is available and in the third the agent sees only the ﬁnal state of each of the 25 trajectories.
Being able to learn from only partial demonstrations is a useful benchmark for the effectiveness of imitation learning methods but can also provide a convenient way of providing demonstrations and can free the agent to ﬁnd more optimal trajectories between states (see for example Akgun et al.
; Schroecker et al.
As can be seen in ﬁgures 2d and 2e, GPRIL achieves similar ﬁnal success rates in all three scenarios while being able to learn a signiﬁcantly faster insertion policy when learning from ﬁnal states alone.
We ﬁnd that in the ﬁrst two scenarios, this holds for GAIL as well as can been in ﬁgure 2b while in the third case, GAIL becomes highly unstable and the resulting performance can vary wildly, leading to a low average success rate.
We hypothesize that these instabilities are a result of the discriminator overﬁtting to the very small amount of negative samples in its training data.
4. P EG INSERTION ON A PHYSICAL SYSTEM
In previous sections we demonstrated sample-efﬁciency that indicates applicability of GPRIL to real-world physical systems.
To test this, we evaluate our approach on two variations of the physical peg-insertion task depicted in ﬁgure 2a, involving a physical pan-tilt unit which is ﬁxed in one scenario and has pan and tilt angles varying by 0 .
1 and 0 .
02 radians in the second scenario.
For each scenario, we provide 20 demonstrations using kinesthetic teaching, which constitutes a natural way of recording demonstrations but provides state-trajectories only (Chernova and Thomaz, 2014).
Hyper-parameters are altered from section 4.2 to trade off a small margin of accuracy for higher learning speeds and are reported in appendix B. Note, however, that tuning hyper-parameters precisely is very difﬁcult on a physical system.
As can be seen in ﬁgure 2f, GPRIL is able to learn a successful insertion policy that generalizes to unseen insertion angles using just a few hours of environment interactions 2 .
We report best-seed performance as we observe a high amount of variability due to factors outside the agent’s control, such as the pan-tilt unit not reporting accurate information after physical contact with the robot.
However, we wish to point out that the increased difﬁculty due to less predictable control is also likely to introduce additional variance that could be reduced further with careful design of exploration noise and other hyper-parameters.
We furthermore provide a video of the training procedure and ﬁnal policy to highlight the efﬁciency of our method 3 .
C ONCLUSION
We introduced GPRIL, a novel algorithm for imitation learning which uses generative models to model multi-step predecessor distributions and to perform state-action distribution matching.
We show that the algorithm compares favorably with state-of-the-art imitation learning methods, achieving higher or equivalent performance while requiring several orders of magnitude fewer environment samples.
Importantly, stability and sample-efﬁciency of GPRIL are sufﬁcient to enable experiments on a real robot, which we demonstrated on a peg-insertion task with a variable-position socket.
A D ERIVATION OF EQUATION
Here, we derive equation 6 which unrolls the recursive deﬁnition of ∇ θ log d π θ ( s ) and rewrites it such that it can be replaced by an expectation over states and actions along trajectories leading to the state s .
In section 3.
we derive a recursive deﬁnition of ∇ θ log d π θ ( s ) which we will restate in more detail:
[Equation]
We can now unroll this deﬁnition:
[Equation]
Note that lim T →∞
[Equation]
due to Markov chain mixing and, therefore, the second term of the above sum reduces to 0 as
[Equation]
By pulling out the sum
[Equation]
we can now marginalize out most variables and shift indices to arrive at the desired conclusion:
[Equation]
B H YPERPARAMETERS
General parameters
Total iterations 2 e
Batch size γ 0.
Replay memory size
N B
N π
B s and B
a
Stacked autoencoders
Hidden layers 500,
Optimizer
Adam
Learning rate 2 ·
Burnin 50000 iterations
L2-regularization 10 −min ( σ i ) 0.
Gradient clip, L 2 norm
Policy π θ Hidden layers 300,
Optimizer
Adam
Learning rate 10σ bounds (0.01, 0.1) (a) GPRIL parameters for clip insertion
General parameters
[Table]
Processes
[Table]
Discriminator
[Table]
γ
[Table]
Actor
[Table]
layers 300,
KL
[Table]
size
[Table]
01 σ bounds (
[Table] [Table]
Discriminator
Hidden layers 150,
Optimizer
Adam
Learning rate
Entropy regularization
Optimizer
Adam
Critic
Hidden layers 300,
Optimizer
Adam
Learning rate 5 · 10(b)
GAIL parameters for clip insertion
General parameters
Total iterations 5 e
Batch size
Hidden layers 300,σ bounds (0.01, 0.1)
Optimizer
Adam
Learning rate
L2-regularization 10 −(c) BC parameters for clip insertion
Simulation
Real robot
General parameters
Batch size γ 0.
Replay memory size
[Figure]
N B
[Figure]
N π
[Figure]
B s and B
a
Stacked autoencoders
Hidden layers
[Figure] [Figure]
Optimizer
Adam
Learning rate 10 −3 ·
[Figure]
−
Burnin
L2-regularization 10 −10 −min
( σ i ) 0.
Gradient clip ( L
[Table]
C R ELATION TO THE P OLICY G RADIENT T HEOREM
In this section, we outline the relation between the policy gradient theorem (Sutton et al., 1999) and the state-action-distribution gradient derived in section A and show equivalence of the discount factor used in reinforcement learning and the discount factor γ introduced in this work.
We ﬁrst show that the state-action distribution gradient is equal to the policy gradient in the average reward case using a speciﬁc reward function.
We then derive the γ -discounted approximation of the state-actiondistribution gradient presented in section 3.1 from the policy gradient in the discounted reward framework using γ as a discount factor.
While this derivation is more cumbersome than the one presented in the body of the paper, it allows us to gain a better understanding of the meaning of γ since discounting in the reinforcement learning setting is well understood.
For notational simplicity, we assume that the state space S and action space A are countable.
Note that this is a common assumption made in the ﬁeld (Tsitsiklis and Van Roy, 1997) and considering that any state-action space is countable and ﬁnite when states and actions are represented using a ﬁnite number of bits makes it apparent that this assumption does not constitute a simpliﬁcation in the practical sense.
In this work we propose to follow the gradient ∇ θ log ρ π θ evaluated at demonstrated states and actions, consider thus the gradient evaluated at s, a :
[Equation]
As we can see, the gradient is equivalent to the policy gradient using the reward function
R ( s, a ) :=
[Equation]
= s,
[Equation]
=
[Equation]
) ρ π
[Equation]
in the average reward framework where ˆ θ corresponds to the parameters of the policy at the current iteration.
This reward function is not practical as it can be inﬁnitely sparse and furthermore depends on the unknown stationary distribution.
However, it allows us to derive the policy gradient using this reward function in the discounted reward setting which constitutes a well understood approximation of the average-reward scenario.
Following the notation in Sutton et al.
(1999), we have:
[Equation]
We can now replace the expectation over the stationary distribution by an expectation over the path of the agent.
This yields a double sum whose order of summation can be changed:
[Equation]
After changing the order of summation we can replace the outer sum with the expectation over the stationary distributions and use the special nature of the chosen reward function to write the gradient as a conditional expectation:
[Equation]
Finally, we notice that this equation constitutes the discounted version of equation 15, thus we can immediately obtain our estimate of the state-action-distribution gradient:
[Equation]
While this derivation is less direct than the derivation used in section A, it draws a connection between the problem of matching state-action-distributions and the reinforcement learning problem with a reward that is positive for demonstrated state-action pairs and 0 otherwise.
This indicates that the role of the discount factor is similar in both settings: Lower values of γ trade-off accurate matching of distributions for lower variance and as expedience.
Agents with low γ will learn policies that recover and reach demonstrated states quicker over policies that are matching the experts state-action distribution more accurately long-term.
Furthermore, the long history of successes in reinforcement learning validates the use of γ as an approximation to the true objective which is often more accurately described by the average-reward objective.