D EMONSTRATION A CTOR C RITIC
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the problem of Reinforcement learning from demonstrations (RLfD) , where the learner is provided with both some expert demonstrations and reinforcement signals from the environment.
One approach leverages demonstration data in a supervised manner, which is simple and direct, but can only provide supervision signal over those states seen in the demonstrations.
Another approach uses demonstration data for reward shaping.
By contrast, the latter approach can provide guidance on how to take actions, even for those states are not seen in the demonstrations.
Speciﬁcally, such reward shaping approach trains an agent not only to imitate demonstrated actions when it encounters demonstrated states, but also to reach demonstrates states, when it confronts states that are not observed in the demonstration data.
However, existing algorithms in the latter one adopt shaping reward which is not directly dependent on current policy, limiting the algorithms to treat demonstrated states the same as other states, and fail to directly exploit supervision signal in demonstration data.
In this paper, we propose a novel objective function with policy-dependent shaping reward, so as to get the best of both worlds.
We present a convergence proof for policy iteration of the proposed objective, under the tabular setting.
Then we develop a new practical algorithm, termed as Demonstration Actor Critic (DAC).
Experiments on a range of popular benchmark sparse-reward tasks shows that our DAC method obtains a signiﬁcant performance gain over ﬁve strong and off-the-shelf baselines.
I NTRODUCTION
Reinforcement Learning (RL) aims at solving sequential decision-making problems by learning through interacting with environments in a trail-and-error way.
In many real scenarios, the existence of expert demonstrations has been well perceived as a critical value to enhance the capability of reinforcement learning algorithms.
Recent years have witnessed many studies exploring the paradigm of learning from demonstration (LfD), which provides the learner with some demonstration data generated by expert policies.
However, LfD yields a strong dependency on the assumption of demonstration optimality, which is usually inconsistent with the reality.
To better integrate LfD with reinforcement learning, increasing efforts turn to reinforcement learning from demonstrations (RLfD), with a relaxation to the demonstration optimality assumption, which can lead to signiﬁcantly boost sample efﬁciency of the RL process.
One major branch of RLfD proposes to leverage demonstration data in a supervised manner, by either using them to directly pretrain the policy (Silver et al., 2016) or supplement the learning target of the policy with a supervised objective when encountering the states in demonstration data (Rajeswaran et al., 2017a).
Although appealingly simple and direct, such branch of RLfD unfortunately fails to fully exploit demonstration data as it can only provide supervision signal over those states observed in the demonstrations (Brys et al., 2015; Rajeswaran et al., 2017a;
Reddy et al., 2019).
To deal with such problem, another major branch of RLfD takes advantage of the demonstrations in reward shaping, by either designing the demonstration-oriented potential-based reward shaping function (Brys et al., 2015; Sun et al., 2018), or inducing implicit dynamic reward shaping through learning a discriminator from demonstrations, which can distinguish between demonstrations and self-generated data (Zhu et al., 2018; Kang et al., 2018).
These methods can provide guidance on how to take actions, even for those states are not seen in the demonstrations.
Particularly, these methods train an RL agent not only to imitate demonstrated actions when it encounters the demonstrated states, but also to reach demonstrated states, when it confronts states that are not observed in the demonstration data 1 (Ho & Ermon, 2016; Reddy et al., 2019; Wang et al., 2019).
This is the core idea behind these reshaping reward based approaches.
However, since the new adopted shaping reward yields no direct dependence on the current policy, this branch of methods, updating policy over demonstrated states in the same way as others by the reshaped value function, overlook the validity of such direct supervision for demonstrated states when learning the policy.
In order to provide both guidance for all states as well as direct supervision for demonstrated states, we propose a new objective function with policy-dependent shaping reward.
To demonstrate the theoretic soundness of this approach, we ﬁrst present a convergence proof for policy iteration of the proposed objective, under the tabular setting given the assumption of the existence of an expert policy π E ( a | s ) .
Furthermore, to cope with the problem of missing explicit expression of π E ( a | s ) in reality, we develop a new practical algorithm, called Demonstration Actor Critic (DAC), by making several approximations that can be implemented using deep neural networks.
Intuitively, if the current state is not included in the demonstration, the agent will learn to update the policy merely relying on the reshaped Q-value function.
Otherwise, the agent will take advantage of both expert information and the reshaped Q-value function to update the policy.
To demonstrate the effectiveness of our algorithm, we conduct experiments on the continuous physical locomotion tasks based on Mujoco (Todorov et al., 2012) in sparse-reward environments.
In comparison with ﬁve strong and off-the-shelf baselines, the empirical results clearly show that our new
DAC approach can attain consistent and signiﬁcant improvements.
Considering the recent concerns on reproducibility (Henderson et al., 2017), all of our reported results are based on experiments run across a large number of seeds.
The main contributions of this paper are summarized as:
• We introduce a novel RLfD objective with policy-dependent shaping reward, which provide both guidance for all states as well as direct supervision signal over demonstrated states.
• We derive a Demonstration Policy Iteration method with guaranteed convergence, under the tabular setting, by assuming the existence of the expert policy π E .. •
We develop Demonstration Actor Critic (DAC), a new practical algorithm to learn the policy for the continuous setting, given the missing expert policy in reality.
• We conduct empirical experiments in a couple of popular continuous tasks in sparse-reward environments to demonstrate the advantage of DAC as it consistently outperforms state-ofthe-art baselines.
R ELATED W ORK
There is a growing interest in combining learning from demonstration (LfD) with reinforcement learning (RL).
Recently there are three popular approaches under this problem setting: 1) utilizing demonstration data by adopting value-based RL algorithms; 2) leveraging demonstration data in a supervised manner; 3) using the demonstrations to reshape the original reward function.
For the ﬁrst approach, they adopt value-based RL algorithms to utilize the demonstration data.
Kim et al.
(2013) proposes Approximate Policy Iteration with demonstrations (APID), which uses expert demonstrations to deﬁne linear constraints that guide the optimization of Approximate Policy
Iteration (API).
Piot et al.
(2014) builds on a similar idea but integrate expert constraint directly into the minimization of the optimal Bellman residual (OBR).
Following this line, Chemali &
Lazaric (2015) bases on classiﬁcation-based policy iteration and proposes Direct Policy Iteration with Demonstrations (DPID).
More recently, thanks for the development of deep learning, DQfD (Todd et al., 2018) introduces LfD into DQN (Mnih et al., 2015), using the same additional structured classiﬁcation loss than previous works.
Besides, DQfD also adds demonstration data into the replay buffer in the same way as self-generated data.
It employs a reﬁned priority replay mechanism (Schaul et al., 2016) and assigns additional priority to demonstration data.
However, these methods are limited by applications with discrete action spaces, due to the usage of max operator over the whole action space.
DDPGfD (Vecerik et al., 2017; Nair et al., 2017; Vecerik et al., 2019), which is built upon DDPG (Lillicrap et al., 2015), extends DQfD to continuous action domain.
Moreover,
NAC (Gao et al., 2018) uses a uniﬁed loss function to process both off-line demonstrations and on-line experience based on the maximum entropy reinforcement learning framework.
Nonetheless, treating demonstration data in the same way as self-generated experience usually requires a tremendous number of high-quality demonstration, which are difﬁcult to collect at scale, as discussed in (Kang et al., 2018).
For the second approach, they attempt to leverage the demonstration data in a supervised manner.
For instance, Silver et al.
(2016) proposes to pre-train the policy with the demonstration data as a policy initialization step for further reinforcement learning, and Rajeswaran et al.
(2017b) augments the original policy loss with a behavior cloning loss during the policy training.
Although appealingly simple and direct, such methods can only provide accurate supervision signal over those states that have been seen in the demonstrations.
For the third approach, they pursue to reshape the original reward function in order to align with the experience from the demonstrations.
Speciﬁcally, Brys et al.
(2015) introduces a reward reshaping mechanism by deﬁning a heuristic potential function based on non-normalized multi-variate Gaussian.
Besides, Sun et al.
(2018) uses expert’s value function as reward shaping, under the assumption of access to a reward-to-go oracle that provides an estimate of expert reward-to-go during training.
Furthermore, Kang et al.
(2018) introduces an implicit reward shaping via a parameterized discriminator, which aims to distinguish the demonstrated state-action pairs from self-generated pairs, and learn the policy with policy gradient methods.
These methods can encourage the agent not only to imitate demonstrated actions, but also to visit demonstrated states.
However, since the new adopted shaping reward yields no direct dependence on the current policy, this branch of methods pay rare attention to the validity of such direct supervision with respect to demonstrated states.
To address this problem in the following of this paper, we develop an algorithm that can both provide guidance on all states and directly exploit the supervision signal on demonstrated states.
As one of the most popular IL algorithms, Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) trains a discriminator to distinguish whether a state-action pair is from the expert or the learned policy.
Meanwhile, GAIL optimizes the policy by maximizing expected return with respect to the reward function, which is based on that discriminator.
Though effective for imitation learning, GAIL cannot leverage the valuable reward signal given by the environment and may suffer from declining performance when the demonstration data is imperfect.
By contrast, our algorithm can overcome such inherent limitation by introducing reward signals from the environment into the training process.
Regarding the usage of discriminator, GAIL trains a discriminator to distinguish expert state-action pairs from other state-action pairs, while our method uses a discriminator to distinguish expert actions from other actions given expert states, which is totally different.
Some of previous works (Peters et al., 2010; Azar et al., 2012; Schulman et al., 2015; Haarnoja et al., 2017; Neu et al., 2017; Abdolmaleki et al., 2018; Haarnoja et al., 2018; Geist et al., 2019) studied the entropy regularization MDPs.
Although there exist some similarities between our method and these works in terms of formulation, these methods are not very suitable for the RLfD problem studied in this paper.
Particularly, these methods do not encourage the agent to reach demonstrated states (states visited by the expert strategy) explicitly, but it is a very important unique property of
RLfD problem itself (Ho & Ermon, 2016; Kang et al., 2018; Reddy et al., 2019).
By contrast, in our work, we delicately design a new policy-dependent shaping reward, in order to not only imitate demonstrated actions over these demonstrated states, but also reach demonstrated states, speciﬁcally for the RLfD problem.
In addition, most of these works assume that the explicit expression of initial policy is available, but in our case, we can only access to expert demonstrations.
To this end, we use the GAN technique to replace the necessary of explicit expression of π E , and take advantage of support estimation techniques to estimate the indicator function of supp π E ( s ) , which also leads to an obvious difference between these entropy regularization works and our work.
B ACKGROUND 3. M ARKOV D ECISION P ROCESS
We consider the standard Markov Decision Process (MDP) (Sutton & Barto, 1998), deﬁned by the tuple (cid:104)S , A , P, r, γ (cid:105) , where S and A are the state space and the action space respectively, P ( s (cid:48) | s, a ) is the transition distribution, r ( s, a ) is the reward function, and γ ∈ (0 , 1) is the discount factor.
Given a stochastic policy π ( a | s ) that maps states to action probabilities, the performance of π is usually evaluated by its expected discounted return η ( π ) :
[Equation]
where τ = ( s 0 , a 0 , s 1 , ... )
denotes a trajectory generated by policy π .
Reinforcement Learning (RL) (Sutton & Barto, 1998) reﬂects the learning paradigm trying to infer a policy maximizing η ( π ) .
Deﬁnition 1.
( Occupancy measure) .
Let ρ π ( s ) :
S → R denote the unnormalized distribution of state visitation by following policy π in the environment:
[Equation]
The unnormalized distribution of state-action pairs ρ π ( s, a ) =
ρ π ( s ) π ( a | s ) is called occupancy measure of policy π .
Intuitively, the occupancy measure can be interpreted as the distribution of state-action pairs that an agent encounters when navigating the environment with policy π .
An important property of the occupancy measure is the one-to-one correspondence with the policy, as described in the theorem 2 of (Syed et al., 2008).
3. D EMONSTRATION D ATA S ETTING
We formalize the demonstration data setting considered in this paper.
The agent is provided with a few (and possibly imperfect) demonstrations as follows:
[Equation]
D E are sampled from executing an unknown expert policy π E in the environment.
For the follow-up convergence guarantee, we have the following necessary assumption on the expert policy π E :
Assumption 1.
The expert policy π E is a stochastic policy, and there exists a positive value δ satisfying that min a ∈A π E ( a | s ) ≥ δ , ∀ s ∈ S .
The point of this assumption is to ensure that D KL ( π, π E ) is bounded by a constant M for any π ∈ Π , under the tabular setting with |A| < ∞ .
2 Based on this, the augmented reward of our method (as shown later in Eq. 3) is also bounded, which can further lead to the convergence of demonstration policy evaluation (i.e. Lemma 1).
M ETHODOLOGY
In order to provide both guidance over all states and the supervision more directly on demonstrated states, we propose an objective function with policy dependent shaping reward:
[Equation]
where 1 s ∈ supp
ρ E ( s ) stands for the indicator function of supp ρ π E ( s ) 3 , and ¯ D KL ( π ( ·| s ) , π E ( ·| s ))
(cid:44)
M − D KL ( π ( ·| s ) , π E ( ·| s )) 4 .
More concretely, if state s is unseen in the demonstrated states, the
[Equation]
augmented reward equals zero; Otherwise, the augmented reward is a positive number, indicating that a current policy π closer to the expert policy π E will give rise to larger augmented reward.
In this way, we can encourage the agent to both reach the demonstrated states and take action in a way similar to expert.
Besides, we can prove that the optimal policy of our proposed objective is equal to that of the original RL objective, under the assumption that the expert policy π E is the optimal policy.
The detailed proof can be found in Appendix B.
Since our shaping reward depends on the current policy π , optimizing the objective (Eq. 3) w.r.t the policy π enables us to directly optimize the policy-dependent shaping reward itself.
In other words, we can directly minimize the KL divergence between π and π E over those demonstrated states.
Detailed optimization will be illustrated in the policy improvement part in Eq. 8.
Inspired by soft value function in SAC (Haarnoja et al., 2018), we further introduce demonstration value function V π ( s ) , by including the shaping reward at every time horizon:
[Equation]
In a similar way, we also deﬁne demonstration Q-value function Q π ( s, a ) by including shaping reward at every time horizon, except the initial time horizon:
[Equation]
In the remainder of this section, we will ﬁrst derive the Demonstration Policy Iteration method in
Section 4.1, with the convergence guarantee under the tabular setting given the assumption of known π E ( a | s ) .
However, the explicit expression of π E is usually missing in reality.
To tackle this challenge, we further develop the DAC algorithm in Section 4.2, which is more practical in real scenarios.
Finally, we summarize the whole DAC algorithm in Algorithm 1.
4. D EMONSTRATION P OLICY I TERATION
Given the assumption that π E ( a | s ) is known, we derive the demonstration policy iteration method, which alternates between policy evaluation and policy improvement.
Our derivation is based on a tabular setting, for the purpose of theoretical analysis and convergence guarantee.
The policy evaluation step aims at computing the demonstration Q-value function of a policy π , which includes both the extrinsic reward and the shaping reward from demonstrations.
Speciﬁcally, the demonstration Q-value function Q π ( s, a ) can be computed iteratively, starting from any function
Q : S ×
A → R and repeatedly applying a Bellman backup operator T π given by:
[Equation]
where
[Equation]
The detailed evaluation process is formalized below.
Lemma 1.
( Demonstration Policy Evaluation ).
Consider the demonstration Bellman backup operator
T π in Eq.
6 and a initial Q function Q 0 : S ×
A → R with |A| < ∞ , and deﬁne Q k
+1 =
T π Q k .
Then the sequence Q k will converge to the demonstration Q-value of π as k → ∞ .
In the policy improvement step, for each state, we update the policy according to:
[Equation]
This particular choice of update can be guaranteed to result in an improved policy in terms of its demonstration Q-value function.
This update rule consists of two different parts: the ﬁrst one refers to the expectation of Q π old ( s, a ) , which encourages the agent to obtain more cumulative rewards and visit the demonstrated states, and the other part signiﬁes the direct supervision signals over these demonstrated states.
The indicator function of supp
ρ E ( s ) determines whether the current state s belongs to the demonstrated states, and if it is true, the KL divergence term will enforce the matching between the learned policy π and the expert policy π E .
We formalize the detailed improvement result in Lemma 2.
Lemma 2.
( Demonstration Policy Improvement ).
Let π old ∈ Π and let π new be the optimizer of the maximization problem deﬁned in Eq.
Then Q new ( s, a ) ≥ Q old ( s, a ) , ∀ ( s, a ) ∈ S
× A with |A| < ∞ .
Overall, the complete demonstration policy iteration algorithm alternates between the policy evaluation and the policy improvement steps, and it will provably converge to the optimal policy, as demonstrated in Theorem 1.
We refer readers to Appendix C for the detailed proof.
Theorem 1.
( Demonstration Policy Iteration ).
Repeated application of demonstration policy evaluation and demonstration policy improvement from any π ∈ Π converges to a policy π ∗ such that
[Equation]
4. D EMONSTRATION A CTOR C RITIC
The derived demonstration policy iteration above is presumed to be under the tabular setting with known explicit expression of the expert policy π E ( a | s ) , which is, however, usually missing in reality.
Thus, a critical challenge remains as how to develop a practical DAC algorithm under the common real scenarios where only demonstration data D E exists.
First, we use function approximators (e.g. deep neural network), including value network V ϕ ( s ) ,
Q-value network Q φ ( s, a ) and policy network π θ ( a | s ) .
Computing the ratio between the current policy π ( a | s ) and the expert policy π E ( a | s ) is a necessary part of our derived theory, as shown in
Eq.
7 and Eq.
8 5 , especially when π E ( a | s ) is unknown.
To tackle this challenge, we borrow the idea of the discriminative modeling in Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), which is used to differentiate the real data from those created by the generator.
In our case, we construct a discriminator network D w ( s, a ) that can distinguish whether an action is from π ( a | s ) or π E ( a | s ) given state s .
More formally,
Theorem 2.
Given the policy π and expert policy π E , we deﬁne that D ∗ ( s, a ) (cid:44) arg max D ∈
[Equation] [Equation]
E
[Equation]
π [log D ( s, a )]
+ E a ∼ π E
[log(1 − D ( s, a ))] , then we have π ( a | s ) π E ( a | s )
=
D ∗ ( s,a ) 1 − D ∗ ( s,a ) .
Based on this theorem, the ratio of π ( a | s ) to π E ( a | s ) can be equivalently written as D ∗ ( s,a ) 1 − D ∗ ( s,a ) .
Then, we parameterize D ∗ ( s, a ) to the discriminator network D w ( s, a ) , and train the discriminator with the demonstration data.
We continue deriving update rules for the value functions and policy.
The value function V ϕ ( s ) is trained to minimize the squared residual error:
[Equation]
The Q-value function is trained to minimize the bellman residual:
[Equation] [Equation]
where ˆ Q ( s, a ) =
r ( s, a ) + γ E s (cid:48) ∼ p ( ·| s,a ) [ V ϕ ( s (cid:48) )] .
Finally, the policy parameter can be learned by applying the policy update rule from Eq. 8:
[Equation]
We can ﬁnd that the above objective includes both the Q-function Q φ ( s, a ) and discriminator
D w ( s, a ) , which are represented by neural networks and can be differentiated.
Hence, it is very convenient to apply the reparameterization trick, which can lead to a low-variance estimator.
To this end, we reparameterize the policy using a neural network transformation:
[Equation]
where (cid:15) is an input noise vector sampled from some ﬁxed distribution, such as multivariate Gaussian.
Then we can rewrite the objective in Eq.
11 as below:
[Equation]
Similarly, the above learning objective w.r.t the policy network π θ also consists of two different parts as Eq. 8.
In particular, the gradient from the discriminator D w ( s, a ) , which aims to distinguish whether an action is from the expert π E or the learned policy π θ given the current state, will guide the agent to take actions in accordance with the expert when it encounters the demonstrated states.
On the other hand, the gradient from the Q-value function Q φ ( s, a ) will encourage the agent to obtain more cumulative rewards and explore the demonstrated region.
Overall, our practical algorithm alternates between collecting experience from the environment, and updating the function approximators.
We use off-policy data from a replay buffer to train the value and policy networks, and use demonstration data to train the discriminator network.
Practical Expert Policy Support Estimation
The indicator function of supp ρ E is a key component in our DAC algorithm, which indicates whether current state s belongs to demonstrated states.
However, in practice, the expert policy is unknown and only a ﬁnite number of trajectories sampled according to π E are available.
Consequently, we consider taking advantage of support estimation techniques to estimate this indicator function.
Recently, Wang et al.
(2019) have established a connection between support estimation ideas and
Random Network Distillation (RND) (Burda et al., 2018) - a method to design intrinsic reward for
RL exploration based on the ”novelty” of states visited.
Their design of intrinsic reward is based on the observation that neural networks tend to have signiﬁcantly lower prediction errors on examples similar to those on which they have been trained, which also inspires us to use prediction errors of networks trained on the demonstration states to approximate the indicator function of supp
ρ E .
In particular, we introduce two neural networks: a label network representing the prediction task and a predictor network trained on demonstration states.
Note that, the label network is randomly initialized but ﬁxed then, and it takes a state as input with a scalar output, i.e., f : S → R , and the predictor network ˆ f : S → R is trained to minimize the expected MSE w.r.t its parameter ψ , as shown below:
[Equation]
This process distills a randomly initialized neural network into a trained one.
The prediction error is expected to be higher for the states that are outside demonstrated states.
Based on that, the approximate indicator function can be eventually deﬁned as follows:
[Equation]
where µ stands for the temperature parameter.
As the L2 norm is non-negative, the approximate indicator function ranges from 0 to 1.
We choose µ to make that from demonstrated states are mostly close to 1.
Overall, the complete DAC algorithm can be summarized in Algorithm 1 .
end
E XPERIMENTS
For the experiments below, we provide empirical results to answer the following questions:
Can our DAC algorithm achieve better performance than other counterparts, from the same RLfD setting or other settings?
What is the key ingredient in our algorithm that introduces better empirical results?
To answer the ﬁrst question, we evaluate our method against several baselines on ﬁve sparse physicsbased based control benchmarks.
Regarding the second question, we explore ablation analysis of the two major components in our algorithm (namely the KL shaped reward and direct KL policy loss).
Due to the space limit, we defer more detailed speciﬁcations into the appendix.
5. C OMPARATIVE E VALUATION
Experiment Settings We conduct experiments on the sparse version of ﬁve popular continuous control tasks ( Hopper-v1, HalfCheetah-v1, Walker2d-v1, Ant-v1, Humanoid-v1 ) from OpenAI Gym (Duan et al., 2016) .
Speciﬁcally, we use the delayed version of the Mujoco domains 6 as (Zheng et al., 2018;
Oh et al., 2018) did, where the reward is made sparse by accumulating the reward for N =timesteps before it to the agent.
Expert’s trajectories were collected from the expert policy released by the authors of the original GAIL 7 .
In particular, the maximum number of expert trajectories was chosen as (Ho & Ermon, 2016; Jeon et al., 2018), i.e. 240 for Humanoid-v1 and 25 for all other tasks.
For all tasks, feedforward neural networks with two hidden layers are used to represent the policy and value functions, where 256 hidden units for each hidden layer and relu activations are used.
For the policy, Gaussian policy is used with both mean and variance dependent on the state.
During training, we use the Adam optimizer (Kingma & Ba, 2015), with a learning rate of 3 × 10 −for all networks and set K = 1 to make the algorithm faster in terms of wall clock time.
We refer readers to Appendix D for more implementation details.
For comparative evaluation, We compare our DAC algorithm against ﬁve strong and off-the-shelf baselines including:
Policy Optimization with Demonstration (POfD): the algorithm of (Kang et al., 2018) leveraging demonstration to reshape the reward function.
Policy Optimization with Demonstration with Behavior Cloning (POfDBC): the simple combination of POfD algorithm and an augmented behavior cloning loss.
Deep Deterministic Policy Gradient from Demonstration (DDPGfD): the algorithm of Vecerik et al.
(2017) putting demonstrations into the replay buffer as self-generated data.
Generative Adversarial Imitation Learning (GAIL): the algorithm of (Ho & Ermon, 2016), a popular imitation learning method, mimicking the expert behaviour by matching the occupancy measure between the expert policy and the learned policy.
Soft Actor Critic (SAC): the algorithm of (Haarnoja et al., 2018), a state-of-the-art off-policy reinforcement learning method, building upon the maximum entropy reinforcement learning framework.
We report the average of the score of the agent over 10 episodes for every 10k steps performed in the environment, as shown in Fig.
The results show that DAC performs consistently across all tasks, and outperforms all strong baselines in terms of both sample efﬁciency and ﬁnal performance.
Besides, on most benchmarks, DAC displays smaller shaded region than other baselines, which implies that DAC can be more stable and robust across different random seeds.
Observing the learning curves of different methods, it is clear that SAC cannot learn very fast without the help of expert demonstrations, especially when the feedback is sparse.
On the other hand, GAIL can improve quickly in the early stage of training process of several tasks, e.g. Walker2d-v1 , but it tends to be limited by the quality of demonstration later.
Under the same RLfD setting, DAC also signiﬁcantly outperforms other counterparts, such as POfD, DDPGfD, and POfDBC, which is a simple combination of POfD
[Figure]
algorithm and an augmented behavior cloning loss, across all the benchmarks.
Furthermore, we can see that the more complex benchmarks, e.g. Ant-v1 and Humanoid-v1 , are exceptionally difﬁcult to be solved by other baselines.
In stark contrast, DAC can learn the policy fast and steadily.
5. A BLATION S TUDY
The previous results suggest that our proposed method can outperform other strong baselines on several challenging tasks.
Now we will further perform ablation study to investigate the inﬂuence of two major components inside our algorithm, i.e., the KL reward shaping and direct KL policy loss, on the overall performance of our DAC.
We conduct two ablation experiments on HalfCheetah-v1 by removing the KL augmented reward when computing Q-function, and the direct KL policy loss during policy improvement, respectively 8 .
The comparative results are shown in Fig.
We can observe that removing either of the two components will lead to the obvious degradation in learning performance.
This suggests that both the KL reward shaping and the direct KL policy loss effectively contribute to the overall performance of our DAC algorithm.
Furthermore, we ﬁnd the degradation of removing the direct KL policy loss is even larger than that of removing the KL reward shaping in this HalfCheetah-v1 task, which well demonstrates that the exploitation of direct supervision signal present in demonstration data may play an important role for better learning performance.
C ONCLUSIONS AND F UTURE W ORK
In this paper we studied reinforcement learning from demonstration (RLfD) and focused on developing a novel method that can not only provide guidance on all states but also pass supervision signal more directly on demonstrated states.
We propose a novel objective function with policy-dependent shaping reward and derive both theoretical guarantee (Demonstration Policy Interation) and practical algorithm (Demonstration Actor Critic) for our objective function.
Experiments on a range of popular benchmark sparse-reward tasks show that our method consistently achieves much higher performance than several strong and off-the-shelf baselines.
For future work, we will explore the direction for improving the robustness of DAC in terms of the demonstration quality, which is not particularly modeled in current algorithm
A D EFINITIONS
The deﬁnition of our proposed new objective is that:
[Equation]
And it also has another equivalent form as follows:
[Equation] [Equation]
Based on the above objective
[Equation]
we further introduce demonstration value function V π ( s ) , by including the shaping reward at every time horizon:
[Equation]
Similarly
[Equation]
we also deﬁne demonstration Q-value function Q π ( s, a ) by including the shaping reward at every time horizon, except the ﬁrst time horizon:
[Equation]
With these deﬁnitions
[Equation]
V π ( s ) and Q π ( s
[Equation]
a ) are connected by:
[Equation]
B O N THE OPTIMAL POLICY INVARIANCE OF OUR OBJECTIVE FUNCTION
Recall that our proposed objective is deﬁned as:
[Equation]
Proof.
Assume that π ∗ = arg max π ∈ Π E
( s,a )
∼ ρ π [ r ( s, a )] (in other words
[Equation]
π ∗ represents the optimal policy of original RL objective)
[Equation]
and the expert policy is perfect: π E = π ∗ .
We prove that π ∗ is also the optimal policy of our proposed objective J ( π ) as follows:
[Equation]
Therefore
[Equation]
we have that π ∗ is also the optimal policy of our proposed objective.
This implies that if a policy π is the optimal policy of original RL objective, then it is also the optimal policy of our proposed objective.
Next
[Equation]
we will continue to prove the inverse proposition: if a policy π is the optimal policy of our proposed objective function, then it is also the optimal policy of original RL objective.
Assume that π # is the optimal policy of our proposed policy, and π ∗ still denotes the optimal policy of original RL objective.
Due to the optimality of π # in terms of J , we have J ( π # ) ≥ J ( π ) , ∀ π ∈ Π .
In particular, π ∗ is a speciﬁc policy π ∈ Π .
Therefore, we have J ( π # ) ≥ J ( π ∗ ) .
From the above Eq.
23, we get that J ( π ∗ ) =
E ( s,a ) ∼ ρ π ∗
[ r ( s, a )] + M , so J ( π # ) ≥ E ( s,a ) ∼ ρ π ∗ [ r ( s, a )] + M .
However, we can easily ﬁnd that the upper bound of J ( π ) is also E ( s,a ) ∼ ρ π ∗ [ r ( s, a )] + M, ∀ π ∈ Π , according to
Eq. 22.
Here we come:
[Equation]
So we have J ( π # ) =
E ( s,a ) ∼ ρ π ∗
[ r ( s, a )] + M .
It is worth noticing that the necessary condition for J ( π # ) to reach the upper bound, is that E ( s,a ) ∼ ρ π
# [ r ( s, a )]
=
E ( s,a ) ∼ ρ π ∗
[ r ( s, a )] .
Therefore,
E ( s,a ) ∼ ρ π #
[ r ( s, a )] ≥ E ( s,a ) ∼ ρ π
[ r ( s, a )] , ∀ π ∈ Π .
That is, π # is the optimal policy of original RL objective.
Based on the above two proofs, we prove the optimal policy invariance of our proposed objective.
C
P ROOFS
C.
L EMMA
Lemma 1.
( Demonstration Policy Evaluation ).
Consider the demonstration Bellman backup operator
T π in Eq.
6, and a initial Q function Q 0 :
S ×
A → R with |A| < ∞ , and deﬁne Q k
+1 =
T π Q k .
Then the sequence Q k will converge to the demonstration Q-value of π as k → ∞ .
Proof.
Given Assumption 1, we ﬁrst prove that D KL ( π, π E ) is bounded for any π ∈ Π :
[Equation]
Given |A| < ∞ , we can get the ﬁrst term ||
(cid:80) a π ( a | s ) log π ( a | s ) || is bounded by log |A| , according to the principle of maximum entropy.
Given the existence of δ ∈ (0 , 1) satisfying that min a ∈A π E ( a | s ) ≥ δ, ∀ s ∈ S , so we can get the second term
||
(cid:80) a π ( a | s ) log π E ( a | s ) || is bounded by log 1 δ .
Therefore, we have that || D KL ( π, π E ) || ≤ log |A| + log 1 δ .
We introduce M (cid:44) log |A| + log 1 δ to represent the upper bound of D KL ( π, π E ) .
We use the notion ¯ r π ( s, a ) to denote the reshaped reward: ¯ r π ( s, a )
[Equation]
r ( s, a ) + 1 s ∈ supp ρ E ( s ) · ( M −
D KL ( π ( ·| s ) , π E ( ·| s ))) .
Then we prove that ¯ r π ( s, a ) is also bounded:
[Equation]
Supposing the extrinsic reward from the environment is bounded, the shaped reward will be also bounded from the above inequality.
We continue rewriting the update rule as
[Equation]
Then, we can ﬁnd that this bellman backup operator over Q can be viewed as a special case of the standard bellman backup operator over Q , by instantiating the reward function r ( s, a ) by the reshaped reward ¯ r π ( s, a ) .
After applying the standard convergence results for policy evaluation (Sutton &
Barto, 1998), the convergence of demonstration Q -value is eventually proved.
C.
L EMMA
Lemma 2.
( Demonstration Policy Improvement ).
Let π old ∈ Π and let π new be the optimizer of the maximization problem deﬁned in Eq.
Then Q new ( s, a ) ≥ Q old ( s, a ) , ∀ ( s, a ) ∈ S
× A with |A| < ∞ .
Proof.
Let π old ∈ Π and let Q π old and V π old be the corresponding demonstration Q-value function and demonstration value function, and let π new be deﬁned as π new
[Equation]
[Equation]
Given the maximum of π new , we can derive that F π old ( π new ) ≥ F π old ( π old ) .
Hence 9 ,
[Equation]
Next, we consider the demonstration bellman equation (i.e. Eq. 21):
[Equation]
where we have repeatedly expanded Q π old on the RHS by applying the demonstration bellman equation (i.e. Eq. 21) and the inequality in Eq. 29.
C.
T HEOREM
Theorem 1.
(Demonstration Policy Iteration).
Repeated application of demonstration policy evaluation and demonstration policy improvement from any π ∈ Π converges to a policy π ∗ such that
Q π ∗ ( s, a ) ≥ Q π ( s, a ) for all π ∈ Π and ( s, a ) ∈ S × A , assuming |A| < ∞ .
Proof.
Let π i be the policy at iteration i .
By Lemma 2, the sequence Q π i is monotonically increasing.
Since Q π is bounded above for π ∈ Π (the shaped reward ¯ r π ( s, a ) is bounded), the sequence converges to some π ∗ .
We will still need to show that π ∗ is indeed optimal.
At convergence, it must be case that F π ∗ ( ·| s ) ( π ∗ ( ·| s )) ≥ F π ∗ ( ·| s ) ( π ( ·| s )) , for all π ∈ Π , π (cid:54) =
π ∗ .
Using the same iterative argument as in the proof of Lemma 2, we get Q π ∗ ( s, a ) ≥ Q π ( s, a ) , ∀ ( s, a ) ∈ S × A .
That is, the demonstration Q-value of any other policy is lower than that of the converged policy.
Hence π ∗ is optimal in Π .
C. T HEOREM
Theorem 2.
Given any policy π and expert policy π E , the optimal discriminator D ∗ ( s, a ) =
[Equation]
Proof.
The criterion for training the discriminator D is to maximize the quantity V ( D ) :
[Equation]
For any ( a, b ) ∈ R 2 − { 0 , 0 } , the function y → a log( y ) +
b log(1 − y ) achieves its maximum in [0 , 1] at aa + b .
Therefore, we can infer that D ∗ ( s, a ) =
π ( a | s ) π ( a | s )+ π E ( a | s ) .
We further rewrite it in an equivalent form π ( a | s ) π E ( a | s ) =
D ∗ ( s,a ) 1 − D ∗ ( s,a ) .
Note that this proof can be seen as the extension of Proposition 1 of GANs(Goodfellow et al., 2014) in our context.
D I MPLEMENTATION D ETAILS
The hyperparameter setting in the experiment is provided in Table 1.
We use the GAIL code(implemented by Theano) released by the authors of the original GAIL paper as our GAIL baseline.
We also use the SAC code 11 (implemented by Tensorﬂow) released by the authors of the original
SAC paper as our SAC baseline.
The authors of POfD and DDPGfD have not publicly released their code by now.
Following the similar setting in (Kang et al., 2018; Vecerik et al., 2017), we implement
POfD and DDPGfD based on OpenAI Baseline 12 as our baselines.
For the seeds in the evaluation, we uniformly choose the values of {0, 1, 2, 3, 4, 5, 6, 7, 8, 9} and share them in all tasks we evaluated.
[Table]
The original hyper setting in the GAIL code is not that sample efﬁcient.
Following the practical suggestions from (Jeon et al., 2018), we reduce the batch size of state-action pairs sampled in each iteration from 50000 to 1000 for Hopper-v1, Walker2d-v1, HalfCheetah-v1, and from 50000 tofor Ant-v1 and Humanoid-v1 , to improve the sample efﬁciency of GAIL algorithm.
Besides, in our detailed implementation, we adopt the simple clip technique to ensure that the augmented reward is bounded in (0, M ), which keeps up with our derived theory and makes the training process more stable and effective.