D ETERMINISTIC P OLICY I MITATION G RADIENT A LGORITHM
Anonymous authors
Paper under double-blind review
A BSTRACT
The goal of imitation learning (IL) is to enable a learner to imitate an expert’s behavior given the expert’s demonstrations.
Recently, generative adversarial imitation learning (GAIL) has successfully achieved it even on complex continuous control tasks.
However, GAIL requires a huge number of interactions with environment during training.
We believe that IL algorithm could be more applicable to the real-world environments if the number of interactions could be reduced.
To this end, we propose a model free, off-policy IL algorithm for continuous control.
The keys of our algorithm are two folds: 1) adopting deterministic policy that allows us to derive a novel type of policy gradient which we call deterministic policy imitation gradient (DPIG), 2) introducing a function which we call state screening function (SSF) to avoid noisy policy updates with states that are not typical of those appeared on the expert’s demonstrations.
Experimental results show that our algorithm can achieve the goal of IL with at least tens of times less interactions than GAIL on a variety of continuous control tasks.
I NTRODUCTION
Recent advances in reinforcement learning (RL) have achieved super-human performance on several domains (Mnih et al., 2015; Silver et al., 2016; Mnih et al., 2016; Lillicrap et al., 2015).
However, on most of domains with the success of RL, the design of reward, that explains what agent’s behavior is favorable, is clear enough for humans.
Conversely, on domains where it is unclear how to design the reward, agents trained by RL algorithms often obtain poor policies and their behavior is far from what we want them to do.
Imitation learning (IL) comes in such cases.
The goal of IL is to enable the learner to imitate the expert’s behavior given the expert’s demonstrations but the reward signals.
We are interested in IL because we desire an algorithm that can be applied in real-world environments where it is often hard to design the reward.
Besides, since it is generally hard to model a variety of the real-world environments with an algorithm, and the state-action pairs in a vast majority of the real-world applications such as robotics control can be naturally represented in continuous spaces, we focus on model free IL for continuous control.
A widely used approach of existing model free IL methods is the combination of Inverse Reinforcement Learning (IRL) (Russell, 1998;
Ng et al., 2000; Abbeel & Ng, 2004;
Ziebart et al., 2008)
and
RL.
Recently, Ho & Ermon (2016) has proposed generative adversarial imitation learning (GAIL) on the line of those works.
GAIL has achieved state-of-the art performance on a variety of continuous control tasks.
However, as pointed out by Ho & Ermon (2016), a crucial drawback of GAIL is requirement of a huge number of interactions between the learner and the environments during training 1 .
Since the interactions with environment can be too much time-consuming especially in the real-world environments, we believe that model free IL could be more applicable to the real-world environments if the number could be reduced while keeping the imitation capability satisﬁed as well as GAIL.
To reduce the number of interactions, we propose a model free, off-policy IL algorithm for continuous control.
As opposed to GAIL and its variants (Baram et al., 2017; Wang et al., 2017; Hausman et al., 2017; Li et al., 2017) those of which adopt a stochastic policy as the learner’s policy, we adopt a deterministic policy while following adversarial training fashion as GAIL.
We show that combining the deterministic policy into the adversarial off-policy IL objective derives a novel type of policy gradient which we call deterministic policy imitation gradient (DPIG).
Because DPIG only integrates over the state space as deterministic policy gradient (DPG) algorithms (Silver et al., 2014), the number of the interactions is expected to be less than that for stochastic policy gradient (PG) which integrates over the state-action space.
Besides, we introduce a function which we call state screening function (SSF) to avoid noisy policy update with states that are not typical of those appeared on the experts demonstrations.
In order to evaluate our algorithm, we used 6 physics-based control tasks that were simulated with
MuJoCo physics simulator (Todorov et al., 2012).
The experimental results show that our algorithm enables the learner to achieve the same performance as the expert does with at least tens of times less interactions than GAIL.
It indicates that our algorithm is more applicable to the real-world environments than GAIL.
B ACKGROUND 2. P RELIMINARIES
We consider a Markov Decision Process (MDP) which is deﬁned as a tuple {S , A , T , r, d 0 , γ } , where
S is a set of states, A is a set of possible actions agents can take, T : S×A×S → [0 , 1] is a transition probability, r : S×A
→ R is a reward function, d 0 : S → [0 , 1] is a distribution over initial states, and γ ∈ (0 , 1] is a discount factor.
The agent’s behavior is deﬁned by a stochastic policy π : S×A
→
Performance measure of the policy is deﬁned as J ( π, r ) =
E (cid:2) (cid:80) ∞ t =0 γ t r ( s t , a t ) | d 0 , T , π (cid:3) where s t ∈ S is a state that the agent received at discrete time-step t , and a t ∈ A is an action taken by the agent after receiving s t .
Using discounted state visitation distribution (SVD) denoted by ρ π ( s )
= (cid:80) ∞ t =0 γ t P ( s t = s | d 0 , T , π ) where P is a probability that the agent receives the state s at time-step t , the performance measure can be rewritten as J ( π, r )
=
E s ∼ ρ π ,a ∼ π (cid:2) r
( s, a ) (cid:3) .
In IL literature, the agent indicates both the expert and the learner.
In this paper, we consider that the states and the actions are represented in continuous spaces S
=
R d s and A = R d a respectively.
2. R EINFORCEMENT L EARNING
The goal of RL is to ﬁnd an optimal policy that maximizes the performance measure.
In this paper, we consider the policy-based methods rather than the value-based method such as Q-learning.
Given the reward function r , the objective of RL with parameterized stochastic policies π θ is deﬁned as follows.
[Equation]
Typically, as do REINFORCE (Williams, 1992) family of algorithms, the update of θ is performed by gradient ascent with estimations of ∇ θ J ( π θ , r ) which is called PG (Sutton et al., 2000).
There are several different expressions of PG which can be represented as follows.
[Equation]
Where ζ denotes trajectories generated by the agent with current policy π θ , and Ψ t ( s t , a t ) can be chosen from a set of the expressions for expected discounted cumulative rewards
E
[Equation]
We refer the readers to (Schulman et al., 2015b) for the expressions in detail.
As mentioned in Section 1, we consider deterministic policy µ θ : S
→ A which is a special case of stochastic policy.
In the RL literature, gradient estimation of the performance measure for deterministic policy which is called DPG (Silver et al., 2014) can be represented as follows.
[Equation]
Where Q : S×A → R denotes Q-function.
Note that, both PG and DPG require calculation or approximation of the expected discounted cumulative rewards, and do not suppose that the reward function is accessible.
That is, as opposed to IRL algorithms, RL algorithms using PG or DPG do not suppose that the reward function is parameterized.
2. I NVERSE R EINFORCEMENT L EARNING
The goal of IRL is to ﬁnd a reasonable reward function based on an assumption that the cumulative rewards earned by the expert’s behavior are greater than or equal to those earned by any non-experts’ behavior.
The objective of IRL with parameterized reward functions r ω can be deﬁned as follows.
[Equation]
Where π E and π denote the expert’s and the non-expert’s policies respectively.
The update of ω is typically performed by gradient ascent with estimations of ∇ ω J ( π E , r ω ) − ∇ ω J ( π, r ω ) .
2. I MITATION L EARNING
Likewise IRL the reward function is unknown in IL setting.
Hence, the decision process turns to be characterized by MDP \ r = {S , A , T , d 0 , γ } and the performance measure becomes unclear unless the reward function can be found.
Given the parameterized reward functions r ω learned by IRL, we suppose the decision process becomes MDP ∪ r ω \ r = {S , A , T , r ω , d 0 , γ } .
We consider the widely used IL approach that combines IRL and RL, and its objective can be deﬁned as a composition of (1) and (4) as follows.
[Equation]
It is common to alternate IRL and RL using the gradient estimation of (5) with respect to ω and to θ respectively.
We can see this IL process as adversarial training as pointed out in (Ho & Ermon, 2016;
Finn et al., 2016a).
A LGORITHM 3. D ETERMINISTIC P OLICY I MITATION G RADIENT
We deﬁne the parameterized reward functions as r ω ( s, a ) = log R ω ( s, a ) , where R ω : S×A → [0 , 1] .
R ω ( s, a ) represents a probability that the state-action pair ( s, a ) belong to the trajectories demonstrated by the expert, and
thus r ω ( s, a ) represents its log probability.
Besides, we introduce a stochastic behavior policy (Precup et al., 2006; Degris et al., 2012) β : S
×
A → [0 , 1] and its discounted SVD ρ β ( s ) =
(cid:80) ∞ t =0 γ t P ( s t = s | d 0 , T , β ) .
We thereby deﬁne the objective of IRL in our algorithm as follows.
[Equation]
Although we adopt deterministic policy µ θ : S
→ A as mentioned in Section 1, a well-known concern for the deterministic policy is about its exploration in the state space.
To ensure the adequate exploration with the deterministic policy, we introduce an off-policy learning scheme with the stochastic behavior policy β introduced above.
We assume that the stochastic behavior policy β is a mixture of the past learner’s policies, and approximate the performance measure for µ θ as
[Equation]
as off-policy DPG algorithms (Silver et al., 2014).
Therefore, the objective of the learner to imitate the expert’s behavior can be approximated as follows.
[Equation]
We alternate the optimization for (6) and (7) as GAIL.
In practice, as do common off-policy RL methods (Mnih et al., 2015; Lillicrap et al., 2015), we perform the state sampling s ∼ ρ β using a replay buffer B β .
The replay buffer is a ﬁnite cache and stores the state-action pairs in ﬁrst-in ﬁrst-out manner while the learner interacts with the environment.
Whereas RL algorithms using DPG do not suppose that the reward function is parameterized as mentioned in Section 2.2, the reward function in (7) is parameterized.
Hence, our algorithm can obtain gradients of the reward function with respect to action executed by the deterministic policy, and apply the chain rule for the gradient estimation of (7) with respect to θ as follows.
[Equation]
[Equation]
Thus, combining the deterministic policy into the adversarial IL objective with the parameterized reward function enables us to derive this novel gradient estimation (8) which we call deterministic policy imitation gradient (DPIG).
3. S TATE S CREENING F UNCTION
Let S β ⊂ S and S E ⊂ S be subsets of S explored by the learner and the expert respectively, let
S β ∗ E = S β ∩ S E and S β ∗ E
=
S \ S β ∗ E be intersection of the two subsets and its complement, and let U ω : S → ∆ θ be updates of the learner’s policy with DPIG.
If there exist states s ∈ S β ∗ E , we would say the mapping U ω ( s ) for the states is reasonable to get r ω ( s, µ θ ( s )) close to r ω ( s, a ) where a ∼ π E ( ·| s ) .
In other words, since r ω can reason desirable actions a ∼ π E ( ·| s ) and undesirable actions µ θ ( s ) for the states s ∈ S β ∗ E through comparisons between ( s, a ) and ( s, µ θ ( s )) in IRL process, U ω ( s ) for the states could reasonably guide the leaner to right directions to imitate the expert’s behavior.
Conversely, if there exist states s ∈ S β ∗ E , the mapping U ω ( s ) for the states would be unreasonable, since the comparisons for the states are not performed in IRL process.
As a result, U ω ( s ) for the states s ∈ S β ∗ E , which we call noisy policy updates , often guide the leaner to somewhere wrong without any supports.
In IL setting, S β ∗ E is typically small due to a limited number of demonstrations, and S β ∗ E is greatly lager than S β ∗ E .
Thus, the noisy policy updates could frequently be performed in IL and make the learner’s policy poor.
From this observation, we assume that preventing the noisy policy updates with states that are not typical of those appeared on the expert’s demonstrations beneﬁts to the imitation.
Based on the assumption above, we introduce an additional function v η : S → R , which we call state screening function (SSF), parameterized by η .
We empirically found that SSF works with form v η ( s )
= 1 / ( − log V η ( s ) + (cid:15) )
where V η : S → [0 , 1] and (cid:15) > 0 .
V η ( s ) represents a probability that the state s belongs to S E .
Hence, the values of v η ( s ) are expected to be much smaller values for the states s ∈ S β ∗ E ∩ S β than those for s ∈ S β ∗ E ⊆ S E .
We show below the ﬁnal objective functions
K r ( ω ) , K v ( η ) , and K µ ( θ ) to be maximized in our algorithm.
[Equation]
The update of parameters ω , η and θ uses the gradient estimations ∇ ω K r ( ω ) , ∇ η K v ( η ) and ∇ θ K µ ( θ ) respectively, where ∇ θ K µ ( θ ) follows DPIG (7).
The overview of our algorithm is described in
Algorithm 1.
Note that, the parameter θ is updated while accessing states s ∈ S β
= ( S β ∗ E ∩ S β ) ∪
S β ∗ E , and SSF works as weighted sampling coefﬁcients of which amounts for the states in S β ∗ E are greater than those for the states in S β ∗ E ∩ S β .
Thus, effects of the noisy policy updates to the learner’s policy can be reduced.
One may think that the application of SSF in (11) makes policy updates more similar to applying generative adversarial training over the actions while sampling states from ρ E as true distribution.
However, a notable difference of IL from generative adversarial training such as GANs (Goodfellow et al., 2014) is that sampling states from the SVDs of current learner’s policy or behavioral policy are essential for estimating policy gradients that improve the learner’s behavior.
Whereas, sampling states from distributions, such as ρ E or uniform distribution on S , would not make sense for the improvement theoretically.
We think that sampling states from ρ β with SSF in equation (11) can also be interpreted as sampling states from SVDs of current learner’s policy only on S β ∗ E if V η was well approximated.
E XPERIMENTS 4. S ETUP
In order to evaluate our algorithm, we used 6 physics-based control tasks that were simulated with
MuJoCo physics simulator (Todorov et al., 2012).
We trained agents on each task by an RL algorithm, namely trust region policy optimization (TRPO) (Schulman et al., 2015a), using the rewards deﬁned in the OpenAI Gym (Brockman et al., 2016), then we used the resulting agents as the experts for IL algorithms.
Using a variety number of trajectories generated by the expert’s policy as datasets, we trained learners by IL algorithms.
See Appendix A for description of each task, an agents performance with random policy, and the performance of the experts.
The performance of the experts and the learners were measured by cumulative reward they earned in an episode.
To measure how well the learner imitates expert’s behavior, we introduced a criterion which we call performance recovery rate
[Table]
[Table] [Table]
[Table] [Table]
(PRR).
PRR was calculated by ( C − C 0 ) /
( C E − C 0 ) where C denotes the performance of the learner,
C 0 denotes the performance of agents with random policies, and C E denotes the performance of the experts.
We tested our algorithm against four algorithms: behavioral cloning (BC) (Pomerleau, 1991),
GAIL, MGAIL which is model based extension of GAIL, Ours \ SSF which denotes our algorithm without using SSF ( v η is always one), and Ours \ int which denotes our algorithm sampling states from ρ E instead of ρ β without using SSF in (11).
We ran three experiments on each task, and measure the
PRR during training the learner.
We implemented our algorithm with three neural networks with two hidden layers.
Each network represented the functions µ θ , R ω , and V η .
All neural networks were implemented using TensorFlow (Abadi et al., 2016).
See Appendix B for the implementation details.
For GAIL and MGAIL, we used publicly available code which are provided the authors 23 .
Note that number of hidden layers and number of hidden units in neural network for representing policy are the same over all methods tested.
All experiments were run on a PC with a 3.30 GHz Intel Core i7-5820k Processor, a GeForce
GTX Titan GPU, and 32GB of RAM.
4. R ESULTS
Figure 1 shows curves of PRR measured during training with BC, GAIL, Ours \ SSF, and Ours \ int.
The PRR of our algorithm and GAIL achieved were nearly 1.0 over all tasks.
It indicates that our algorithm enables the learner to achieve the same performance as the expert does over a variety of tasks as well as GAIL.
We also observe that the number of interactions our algorithm required to achieve the PPR was at least tens of times less than those for GAIL over all tasks.
It show that our algorithm is much more sample efﬁcient in terms of the number of interactions than GAIL, while keeping imitation capability satisﬁed as well as GAIL.
Table 1 shows that comparison of CPU-time required to achieve the expert’s performance between our algorithm and GAIL.
We see that our algorithm is more sample efﬁcient even in terms of computational cost.
We observe that SSF yielded better performance over all tasks compared to Ours \ SSF.
The difference of the PRR is obvious for Ant-v1 and Humanoid-v1 of which state spaces are comparatively greater than those of the other tasks.
We also observe that SSF made difference with relatively smaller number of trajectories given as dataset in Reacher-v1 and HalfCheetah-v1.
It shows that, as we noted in Section 3.2, SSF is able to reduce the effect of the noisy policy updates in the case where S β ∗ E is greatly lager than S β ∗ E .
[Table]
Over all tasks, both BC and Ours \ int performed worse than our algorithm.
It indicates that our algorithm is less affected by the compounding error as described in Section 5, and sampling states from any distribution other than ρ β does not make sense for improving the learner’s performance as mentioned in Section 3.2.
Figure 2 depicts the experimental results comparing our algorithm with MGAIL on Hopper-v1 4 .
As mentioned by Baram et al.
(2017), the number of the interactions MGAIL required to achieve the expert’s performance was relatively less than GAIL.
However, our algorithm is more sample efﬁcient in terms of the number of interactions than MGAIL.
R ELATED W ORK AND D ISCUSSION
A wide variety of IL methods have been proposed in these last few decades.
The simplest IL method among those is BC (Pomerleau, 1991) which learns a mapping from states to actions in the expert’s demonstrations using supervised learning.
Since the learner with the mapping learned by BC does not interacts with the environments, inaccuracies of the mapping are never corrected once the training has done, whereas our algorithm corrects the learner’s behavior through the interactions.
A noticeable point in common between BC and our algorithm is that the both just consider the relationship between single time-step state-action pairs but information over entire trajectories of the behavior in the optimizations.
In other words, the both assume that the reward structure is dense and the reasonable rewards for the states s ∈ S \ S E can not be deﬁned.
A drawback of BC due to ignorance of the information over the trajectories is referred to as the problem of compounding error (Ross & Bagnell, 2010) – the inaccuracies compounds over time and can lead the learner to encounter unseen states in th expert’s demonstrations.
For the state s ∈ S β ∗ E , it is assumed in our algorithm that the immediate reward is greater if the learner’s behavior is more likely to the expert’s behavior, and the expert’s behavior yields the greatest cumulative reward.
That is, maximizing the immediate reward for the state s ∈ S β ∗ E ⊂ S E implies maximizing the cumulative reward for trajectories stating from the state in our algorithm, and thus the information over the trajectories is implicitly incorporated in log R ω of the objective (11).
Therefore, our algorithm is less affected by the compounding error than
BC.
Another widely used approach for IL the combination of IRL and RL that we considered in this paper.
The concept of IRL was originally proposed by Russell (1998), and a variety of IRL algorithms have been proposed so far.
Early works on IRL (Ng et al., 2000; Abbeel & Ng, 2004; Ziebart et al., 2008) represented the parameterized reward function as a linear combination of hand-designed features.
Thus, its capabilities to represent the reward were limited in comparison with that of nonlinear functional representation.
Indeed, applications of the early works were often limited in small discrete domains.
The early works were extended to algorithms that enable to learn nonlinear functions for representing the reward (Levine et al., 2011; Finn et al., 2016b).
A variety of complex tasks including continuous control in the real-world environment have succeeded with the nonlinear functional representation (Finn et al., 2016b).
As well as those methods, our algorithm can utilize the nonlinear functions if it is differentiable with respect to the action.
In recent years, the connection between GANs and the IL approach has been pointed out (Ho &
Ermon, 2016; Finn et al., 2016a).
Ho & Ermon (2016) showed that IRL is a dual problem of RL which can be deemed as a problem to match the learner’s occupancy measure (Syed et al., 2008) to that of the expert, and found a choice of regularizer for the cost function yields an objective which is analogous to that of GANs.
After that, their algorithm, namely GAIL, has become a popular choice for IL and some extensions of GAIL have been proposed (Baram et al., 2017; Wang et al., 2017;
Hausman et al., 2017; Li et al., 2017).
However, those extensions have never addressed reducing the number of interactions during training whereas we address it, and our algorithm signiﬁcantly reduce the number while keeping the imitation capability as well as GAIL.
The way of deriving policy gradients using gradients of the parameterized reward function with respect to actions executed by the current learner’s policy is similar to DPG (Silver et al., 2014) and deep DPG (Lillicrap et al., 2015).
However, they require parameterized Q-function approximator with known reward function whereas our algorithm does not use Q-function besides the parameterized reward function learned by IRL.
In IL literature, MGAIL (Baram et al., 2017) uses the gradients derived from parameterized discriminator to update the learner’s policy.
However, MGAIL is modelbased method and requires to train parameterized forward-model to derive the gradients whereas our algorithm is model free.
Although model based methods have been thought to need less environment interactions than model free methods in general, the experimental results showed that MGAIL needs more interactions than our model free algorithm.
We think that the reasons are the need for training the forward model besides that for the policy, and lack of care for the noisy policy updates issue that
MGAIL essentially has.
C ONCLUSION
In this paper, we proposed a model free, off-policy IL algorithm for continuous control.
The experimental results showed that our algorithm enables the learner to achieve the same performance as the expert does with several tens of times less interactions than GAIL.
Although we implemented shallow neural networks to represent the parameterized functions in the experiment, deep neural networks can also be applied to represent the functions in our algorithm.
We expect that the that advanced techniques used in deep GANs enable us to apply our algorithm to more complex tasks.
A D ETAILED D ESCRIPTION OF E XPERIMENT
Table 2 summarizes the description of each task, an agents performance with random policy, and the performance of the experts.
[Table]
B I MPLEMENTATION D ETAILS
We implemented our algorithm with three neural networks each of which represents function µ θ ,
R ω , and V η .
For convenience, we call the networks as deterministic policy network (DPN) for µ θ , reward network (RN) for R ω
[Table]
and state screening network (SSN) for V η .
All networks were two layer perceptrons that used leaky rectiﬁed nonlinearity (Maas et al., 2013a) except for the ﬁnal output.
DPN had 100 hidden units in each hidden layer
[Table]
and its ﬁnal output was followed by hyperbolic tangent nonlinearity to bound its action range.
Note that the number of hidden units in DPN was the same as that of networks used in (Ho & Ermon, 2016).
RN had 1000 hidden units in each hidden layer and a single output was followed by sigmoid nonlinearity to bound the output between [0,1].
The input of RN was given by concatenated vector representations for the state-action pairs.
The SSN had the same architecture as RN had except for its input which was just vector representations for the state.
The choice of hidden layer width in both RN and SSN was based on preliminary experimental results.
We employed a sampling strategy which was almost the same as deep deterministic policy gradient (DDPG) algorithm (Lillicrap et al., 2015) does.
When updating RN and SSN, the state-action pairs were sampled from the replay buffers B β and a buffer B E that stores the datasets.
When updating
DPN, the states were sampled from the replay buffers B β .
We employed RMSProp (Hinton et al., 2012) for learning parameters of RN, DPN and SSN with a learning rate 10 − 4 , a decay rate 0.99, and epsilon 10 − 10 for all tasks.
The weights and biases in all layers were initialized by Xavier initialization (Maas et al., 2013b).
We used coefﬁcient (cid:15) =
10 − 2 for v η , mini-batch size N = 64 , size of the replay buffer |B β | = 5 × 10 5 .
Regarding to the maximum number of episodes for training, we used M = 10000 for Humanoid-v1, and M = 2000 for the other tasks.