L EARNING G AUSSIAN P OLICIES FROM S MOOTHED A CTION V ALUE F UNCTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
State-action value functions ( i e., Q-values) are ubiquitous in reinforcement learning (RL), giving rise to popular algorithms such as SARSA and Q-learning.
We propose a new notion of action value deﬁned by a Gaussian smoothed version of the expected Q-value.
We show that such smoothed Q-values still satisfy a
Bellman equation, making them learnable from experience sampled from an environment.
Moreover, the gradients of expected reward with respect to the mean and covariance of a parameterized Gaussian policy can be recovered from the gradient and Hessian of the smoothed Q-value function.
Based on these relationships we develop new algorithms for training a Gaussian policy directly from a learned smoothed Q-value approximator.
Our approach is amenable to proximal optimization techniques by augmenting the objective with a penalty on KLdivergence from a previous policy.
We ﬁnd that the ability to learn both a mean and covariance during training allows this approach to achieve much better results on standard continuous control benchmarks.
I NTRODUCTION
Model-free reinforcement learning algorithms often alternate between two concurrent but interacting processes: (1) policy evaluation , where an action value function ( i.e., a Q-value) is updated to obtain a better estimate of the return associated with taking a speciﬁc action, and (2) policy improvement , where the policy is updated aiming to maximize the current value function.
In the past, different notions of Q-value have led to distinct but important families of RL methods.
For example,
SARSA (Rummery & Niranjan, 1994; Sutton & Barto, 1998; Van Seijen et al., 2009) uses the expected Q-value, deﬁned as the expected return of following the current policy.
Q-learning (Watkins, 1989) exploits a hard-max notion of Q-value, deﬁned as the expected return of following an optimal policy.
Soft Q-learning (Haarnoja et al., 2017) and PCL (Nachum et al., 2017a)
both use a soft-max form of Q-value, deﬁned as the future return of following an optimal entropy regularized policy.
Clearly, the choice of Q-value function has a considerable effect on the resulting algorithm; for example, restricting the types of policies that can be expressed, and determining the type of exploration that can be naturally applied.
In this work we introduce a new notion of action value: the smoothed action value function ˜ Q π .
Unlike previous notions, which associate a value with a speciﬁc action at each state, the smoothed Qvalue associates a value with a speciﬁc distribution over actions.
In particular, the smoothed Q-value of a state-action pair ( s, a ) is deﬁned as the expected return of ﬁrst taking an action sampled from a normal distribution N ( a, Σ( s )) , centered at a , then following actions sampled from the current policy thereafter.
In this way, the smoothed Q-value can also be interpreted as a Gaussian-smoothed or noisy version of the expected Q-value.
We show that smoothed Q-values possess a number of interesting properties that make them attractive for use in RL algorithms.
For one, the smoothed Q-values satisfy a single-step Bellman consistency, which allows bootstrapping to be used to train a function approximator.
Secondly, for
Gaussian policies, the standard optimization objective (expected return) can be expressed in terms of smoothed Q-values.
Moreover, the gradient of this objective with respect to the mean and covariance of the Gaussian policy is equivalent to the gradient and the Hessian of the smoothed Q-value function, which allows one to derive updates to the policy parameters by having access to the derivatives of a sufﬁciently accurate smoothed Q-value function.
This observation leads us to propose an algorithm called Smoothie , which in the spirit of (Deep)
Deterministic Policy Gradient (DDPG) (Silver et al., 2014; Lillicrap et al., 2016), trains a policy using the derivatives of a trained (smoothed) Q-value function, thus avoiding the high-variance of stochastic updates used in standard policy gradient algorithms (Williams & Peng, 1991; Konda &
Tsitsiklis, 2000).
Unlike DDPG, which is well-known to have poor exploratory behavior (Haarnoja et al., 2017), the approach we develop is able to utilize a non-deterministic Gaussian policy parameterized by both a mean and a covariance, thus allowing the policy to be exploratory by default and alleviating the need for excessive hyperparameter tuning.
Furthermore, we show that Smoothie can be easily adapted to incorporate proximal policy optimization techniques by augmenting the objective with a penalty on KL-divergence from a previous version of the policy.
The inclusion of a KL-penalty is not feasible in the standard DDPG algorithm, but we show that it is possible with our formulation, and it signiﬁcantly improves stability and overall performance.
On standard continuous control benchmarks, our results are competitive with or exceed state-of-the-art, especially for more difﬁcult tasks in the low-data regime.
N OTATION & B ACKGROUND
We consider the standard model-free RL framework, where an agent interacts with a stochastic black-box environment by sequentially observing the state of the environment, emitting an action, and receiving a reward feedback; the goal is to ﬁnd an agent that achieves maximal cumulative discounted reward.
This problem can be expressed in terms of a Markov decision process (MDP) that consists of a state space S and an action space A , where at iteration t the agent encounters a state s t ∈ S and emits an action a t ∈ A , after which the environment returns a scalar reward r t ∼ R ( s t , a t ) and places the agent in a new state s t +1 ∼ P ( s t , a t ) .
We model the behavior of the agent using a stochastic policy π that produces a distribution over feasible actions at each state s as π ( a | s ) .
The optimization objective (expected discounted return), as a function of the policy, can then be expressed in terms of the expected action value function
Q π ( s, a ) by,
[Equation]
where ρ π ( s ) is the stationary distribution of the states under π , and Q π ( s, a ) is recursively deﬁned using the Bellman equation,
[Equation]
where γ ∈ [0 , 1] is the discount factor.
For brevity, we will often suppress explicit denotation of the sampling distribution R over immediate rewards and the distribution P over state transitions.
The policy gradient theorem (Sutton et al., 2000) expresses the gradient of O ER ( π θ ) w.r.t.
θ , the tunable parameters of a policy π θ , as,
[Equation]
Many reinforcement learning algorithms, including policy gradient and actor-critic variants, trade off variance and bias when estimating the random variable inside the expectation in (4); for example, by attempting to estimate Q π ( s, a ) accurately using function approximation.
In the simplest scenario, an unbiased estimate of Q π ( s, a ) is formed by accumulating discounted rewards from each state forward using a single Monte Carlo sample.
In this paper, we focus on multivariate Gaussian policies over continuous action spaces, A ≡ R d a .
We represent the observed state of the MDP as a d s -dimensional feature vector Φ( s ) ∈ R d s , and parametrize the Gaussian policy by a mean and covariance function, respectively µ ( s ) :
R d s → R d a and Σ( s ) : R d s → R d a × R d a .
These map the observed state of the environment to a Gaussian distribution,
[Equation]
where (cid:107) v (cid:107)
A = v T Av .
Below we develop new RL training methods for this family of parametric policies, but some of the ideas presented may generalize to other families of policies as well.
We begin the formulation by reviewing some prior work on learning Gaussian policies.
2. D ETERMINISTIC P OLICY G RADIENT
Silver et al.
(2014) present a new formulation of the policy gradient, called the deterministic policy gradient , for the family of Gaussian policies in the limit where the policy covariance approaches zero.
In such a scenario, the policy becomes deterministic because sampling from the policy always returns the Gaussian mean.
The key observation of (Silver et al., 2014) is that under a deterministic policy π ≡ ( µ, Σ → 0) , one can estimate the expected future return from a state s as,
[Equation] [Equation]
Then, one can express the gradient of the optimization objective (expected discounted return) for a parameterized π θ ≡ µ θ as,
[Equation]
This can be thought of as a characterization of the policy gradient theorem for deterministic policies.
In the limit of Σ → 0 , one can also re-express the Bellman equation (2) as,
[Equation]
Therefore, a value function approximator Q πw can be optimized by minimizing the Bellman error,
[Equation]
for transitions ( s, a, r, s (cid:48) ) sampled from a dataset D of interactions of the agent with the environment.
Algorithms like DDPG (Lillicrap et al., 2016) alternate between improving the value function by gradient descent on (9), and improving the policy based on (7).
In practice, to gain better sample efﬁciency, Degris et al.
(2012) and Silver et al.
(2014) replace the on-policy state distribution
ρ π ( s ) in (7) with an off-policy distribution ρ β ( s ) based on a replay buffer.
After this substitution, the policy gradient identity in (7) does not hold exactly, however, prior work ﬁnds that this works well in practice and improves sample efﬁciency.
We also adopt a similar approximation in our method to make use of off-policy data.
S MOOTHED A CTION V ALUE F UNCTIONS
In this paper, we introduce smoothed action value functions , the gradients of which provide an effective signal for optimizing the parameters of a Gaussian policy.
Our notion of smoothed Qvalues, denoted ˜ Q π ( s, a ) , differs from ordinary Q-values Q π ( s, a ) in that smoothed Q-values do not assume the ﬁrst action of the agent is fully speciﬁed, but rather they assume that only the mean of the distribution of the ﬁrst action is known.
Hence, to compute ˜ Q π ( s, a ) , one has to perform an expectation of Q π ( s, ˜ a ) for actions ˜ a drawn in the vicinity of a .
More formally, smoothed action values are deﬁned as,
[Equation]
With this deﬁnition of ˜ Q π , one can re-express the expected reward objective for a Gaussian policy π ≡ ( µ, Σ) as,
[Equation] [Equation]
The insight that differentiates this approach from prior work including Heess et al. (2015); Ciosek & Whiteson (2017) is that instead of learning a function approximator for Q π ( s, a ) and then drawing samples to approximate the expectation in (10) and its derivative, we directly learn a function approximator for ˜ Q
π ( s, a ) .
The key observation that enables direct bootstrapping of smoothed Q-values, ˜ Q π ( s, a ) , is that their form allows a notion of Bellman consistency.
First, note that for Gaussian policies π ≡ ( µ, Σ) we have
[Equation]
Then, combining (10) and (12), one can derive the following one-step Bellman equation for smoothed Q-values,
[Equation]
where ˜ r and ˜ s (cid:48) are sampled from R ( s, ˜ a ) and P ( s, ˜ a ) .
Below, we elaborate on how one can make use of the derivatives of ˜ Q π to learn µ and Σ , and how the Bellman equation in (13) enables direct optimization of ˜ Q π .
3. P OLICY I MPROVEMENT - O PTIMIZING ( µ θ , Σ φ ) G IVEN ˜ Q π
We parameterize a Gaussian policy π θ,φ ≡ ( µ θ , Σ φ ) in terms of two sets of parameters θ and φ for the mean and the covariance.
The gradient of the objective w.r.t.
mean parameters follows from the policy gradient theorem and is almost identical to (7),
[Equation]
[Equation]
Estimating the derivative of the objective w.r.t.
covariance parameters is not as straightforward, since ˜
Q π is not a direct function of Σ .
However, a key observation of this work is that the second derivative of ˜ Q π w.r.t.
actions is sufﬁcient to exactly compute the derivative of ˜ Q π w.r.t.
Σ ,
[Equation]
A proof of this identity is provided in the Appendix.
Then, the full derivative w.r.t.
φ takes the form,
[Equation]
[Equation]
3. P OLICY E VALUATION - O PTIMIZING ˜ Q πw G IVEN ( µ, Σ)
We can think of two ways to optimize ˜ Q πw .
The ﬁrst approach leverages (10) to update ˜ Q π based on expected Q-value function Q π .
In such an approach, one trains a parameterized Q πw to approximate the standard expected Q-value function Q π using standard methods (see e.g., Rummery & Niranjan (1994); Sutton & Barto (1998); Van Seijen et al.
Then, one ﬁts ˜ Q π w based on Q π w .
In particular, given transitions ( s, a, r, s (cid:48) ) sampled from interactions with the environment, one can train Q πw to minimize the Bellman error ( Q πw ( s, a ) − r − γQ πw ( s (cid:48) , a (cid:48) ))
2 where a (cid:48) ∼ N ( µ ( s (cid:48) ) , Σ( s (cid:48) )) .
Then, ˜ Q πw can be optimized to minimize the squared error ( ˜ Q πw ( s, a ) − E ˜ a Q πw ( s, ˜ a )) 2 where ˜ a ∼
N ( a, Σ( s )) , using several samples.
When the target values in these residuals are treated as ﬁxed ( i.e., using a target network), such a training procedure will achieve a ﬁxed point when ˜ Q πw ( s, a ) satisﬁes the recursion in the Bellman equation (10).
The second approach requires a single function approximator for ˜ Q πw ( s, a ) , resulting in a simpler implementation, and thus we use this approach in our experimental evaluation.
Suppose one has access to a tuple ( s, ˜ a, ˜ r, ˜ s (cid:48) ) sampled from a replay buffer with knowledge of the sampling probability q (˜ a | s ) (possibly unnormalized).
Then assuming that this sampling distribution has a full support, we draw a phantom action a ∼ N (˜ a, Σ( s )) and optimize ˜ Q πw ( s, a ) by minimizing a weighted Bellman error
[Equation]
For a speciﬁc pair of state and action ( s, a ) the expected value of the objective is,
[Equation]
Note that N ( a | ˜ a, Σ( s ))
=
N (˜ a | a, Σ( s )) .
Therefore, when the target value ˜ r + γ ˜ Q πw (˜ s (cid:48) , µ (˜ s (cid:48) )) is treated as ﬁxed ( e.g., when using target networks) this training procedure reaches an optimum when ˜
Q πw ( s, a ) satisﬁes the recursion in the Bellman equation (13).
In practice, we ﬁnd that it is unnecessary to keep track of the probabilities q (˜ a | s ) , and assume the replay buffer provides a near-uniform distribution of actions conditioned on states.
Other recent work has also beneﬁted from ignoring or heavily damping importance weights (Munos et al., 2016;
Wang et al., 2017; Schulman et al., 2017).
However, it is possible when interacting with the environment to save the probability of sampled actions along with their transitions, and thus have access to q (˜ a | s ) ≈ N (˜ a | µ old ( s ) , Σ old ( s )) .
3. C OMPATIBLE F UNCTION A PPROXIMATION
A function approximator ˜ Q πw of ˜ Q π should be sufﬁciently accurate so that updates for µ θ , Σ φ are not affected by substituting ∂ ˜ Q πw ( s,a ) ∂a and ∂ 2 ˜ Q πw ( s,a ) ∂afor ∂ ˜ Q
π ( s,a ) ∂a and ∂ 2 ˜ Q π ( s,a ) ∂a, respectively.
We claim that a ˜ Q πw is compatible with respect to µ θ if
1. ∇ a ˜ Q πw ( s, a ) (cid:12)(cid:12) a = µ θ ( s ) =
∇ θ µ θ ( s ) T w , 2.
∇ w (cid:82) S (cid:16) ∇ a ˜ Q πw ( s, a ) (cid:12)(cid:12)
a = µ θ ( s ) − ∇ a ˜ Q π ( s, a ) (cid:12)(cid:12) a = µ θ ( s ) (cid:17) 2 d ρ π ( s ) =
( i.e., w minimizes the expected squared error of the gradients).
Additionally, ˜ Q πw is compatible with respect to Σ φ if
[Equation] [Equation]
S
[Equation]
2 a ˜ Q πw ( s,
[Equation]
(cid:12)(cid:12) a = µ θ ( s )
[Equation]
2 a ˜ Q π ( s, a ) (cid:12)(cid:12) a = µ θ ( s ) (cid:17) 2 d ρ π ( s ) =
( i.e., w minimizes the expected squared error of the Hessians).
We provide a proof of these claims in the Appendix.
One possible parameterization of ˜ Q πw may be achieved by taking w =
[ w 0 , w 1 , w 2 ] and parameterizing
[Equation]
While it is reassuring to know that there exists a class of function approximators which are compatible, this fact is largely ignored in practice.
Not only is the class of compatible functions heavily restricted in terms of expressiveness, due to the ﬁrst set of conditions for µ θ , Σ φ , it is also impossible to satisfy the second set of conditions without access to derivative and Hessian information of the true ˜ Q π .
This problem is also present in DDPG, and we feel this issue merits additional investigation in future work.
3. P ROXIMAL P OLICY O PTIMIZATION
Policy gradient algorithms are notoriously unstable, particularly in continuous control problems.
Such instability has motivated the development of trust region methods that attempt to mitigate the issue by constraining each gradient step to lie within a trust region (Schulman et al., 2015), or augmenting the expected reward objective with a penalty on KL-divergence from a previous policy (Nachum et al., 2017b; Schulman et al., 2017).
These stabilizing techniques have thus far not been applicable to algorithms like DDPG, since the policy is deterministic.
The formulation we propose in this paper, however, is easily amenable to trust region optimization.
Speciﬁcally, we may augment the objective (11) with a penalty
[Equation]
where π old ≡ ( µ old , Σ old ) is a previous parameterization of the policy.
The optimization is straightforward, since the KL-divergence of two Gaussians can be expressed analytically.
R ELATED W ORK
This paper follows a long line of work that uses Q-value functions to stably learn a policy, which in the past has been used to either approximate expected (Rummery & Niranjan, 1994; Van Seijen et al., 2009; Gu et al., 2017) or optimal (Watkins, 1989; Silver et al., 2014; Nachum et al., 2017a;
Haarnoja et al., 2017;
Metz et al., 2017) future value.
Work that is most similar to what we present are methods that exploit gradient information from the Q-value function to train a policy.
Deterministic policy gradient (Silver et al., 2014) is perhaps the best known of these.
The method we propose can be interpreted as a generalization of the deterministic policy gradient.
Indeed, if one takes the limit of the policy covariance Σ( s ) as it goes to 0, the proposed Q-value function becomes the deterministic value function of DDPG, and the updates for training the Q-value approximator and the policy mean are identical.
Stochastic Value Gradient (SVG) (Heess et al., 2015) also trains stochastic policies using an update that is similar to DDPG ( i.e., SVG(0) with replay).
The key differences with our approach are that
SVG does not provide an update for the covariance, and the mean update in SVG estimates the gradient with a noisy Monte Carlo sample, which we avoid by estimating the smoothed Q-value function.
Although a covariance update could be derived using the same reparameterization trick as in the mean update, that would also require a noisy Monte Carlo estimate.
Methods for updating the covariance along the gradient of expected reward are essential for applying the subsequent trust region and proximal policy techniques.
More recently, Ciosek & Whiteson (2017) introduced expected policy gradients (EPG), a generalization of DDPG that provides updates for the mean and covariance of a stochastic Gaussian policy using gradients of an estimated Q-value function.
In that work, the expected Q-value used in standard policy gradient algorithms such as SARSA (Sutton & Barto, 1998; Rummery & Niranjan, 1994; Van Seijen et al., 2009) is estimated.
The updates in EPG therefore require approximating an integral of the expected Q-value function.
Our analogous process directly estimates an integral (via the smoothed Q-value function) and avoids approximate integrals, thereby making the updates simpler.
Moreover, while Ciosek & Whiteson (2017) rely on a quadratic Taylor expansion of the estimated Q-value function, we instead rely on the strength of neural network function approximators to directly estimate the smoothed Q-value function.
The novel training scheme we propose for learning the covariance of a Gaussian policy relies on properties of Gaussian integrals (Bonnet, 1964; Price, 1958).
Similar identities have been used in the past to derive updates for variational auto-encoders (Kingma & Welling, 2014) and
Gaussian back-propagation
(Rezende et al., 2014).
Finally, the perspective presented in this paper, where Q-values represent the averaged return of a distribution of actions rather than a single action, is distinct from recent advances in distributional
RL (Bellemare et al., 2017).
Those approaches focus on the distribution of returns of a single action, whereas we consider the single average return of a distribution of actions.
Although we restrict our attention in this paper to Gaussian policies, an interesting topic for further investigation is to study the applicability of this new perspective to a wider class of policy distributions.
E XPERIMENTS
We utilize the insights from Section 3 to introduce a new RL algorithm, Smoothie .
Smoothie maintains a parameterized ˜ Q πw trained via the procedure described in Section 3.2.
It then uses the gradient and Hessian of this approximation to train a Gaussian policy µ θ , Σ φ using the updates stated in (14) and (16).
We perform a number of evaluations of Smoothie compared to DDPG.
We choose DDPG as a baseline because it (1) utilizes gradient information of a Q-value approximator, much like our algorithm; and (2) is a standard algorithm well-known to have achieve good, sample-efﬁcient performance on continuous control benchmarks.
5. A S YNTHETIC T ASK
To evaluate Smoothie we begin with a simple synthetic task which allows us to study its behavior in a restricted setting.
We devised a simple single-action one-shot environment in which the reward function is a mixture of two Gaussians, one better than the other (see Figure 1 (Right)).
We initialize the policy mean to be centered on the worse of the two Gaussians.
We plot the learnable policy mean and standard deviation during training for Smoothie and DDPG in Figure 1 (Left).
Smoothie learns both the mean and variance, while DDPG learns only the mean and the variance plotted is the exploratory noise, whose scale is kept ﬁxed during training.
As expected we observe that DDPG cannot escape the local optimum.
At the beginning of training it exhibits some movement away from the local optimum (likely due to the initial noisy approximation given by Q πw ), it is unable to progress very far from the initial mean.
Note that this is not an issue of exploration.
The exploration scale is high enough that Q πw is aware of the better Gaussian.
The issue is in the update for µ θ , which is only with regard to the derivative of Q πw at the current mean.
On the other hand, we ﬁnd Smoothie is successfully able to solve the task.
This is because the smoothed reward function approximated by ˜ Q πw has a derivative which clearly points µ θ towards the better Gaussian.
We also observe that Smoothie is able to suitably adjust the covariance Σ φ during training.
Initially, Σ φ decreases due to the concavity of the smoothed reward function.
As a region of convexity is entered, it begins to increase, before again decreasing to near-zero as µ θ approaches the global optimum.
[Figure]
5. C ONTINUOUS C ONTROL
We now turn our attention to standard continuous control benchmarks available on OpenAI
Gym (Brockman et al., 2016) utilizing the MuJoCo environment (Todorov et al., 2012).
Our implementations utilize feed forward neural networks for policy and Q-values.
We parameterize the covariance Σ φ as a diagonal given by e φ .
The exploration for DDPG is determined by an Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein, 1930; Lillicrap et al., 2016).
Additional implementation details are provided in the Appendix.
We compare the results of Smoothie and DDPG in Figure 2.
For each task we performed a hyperparameter search over actor learning rate, critic learning rate and reward scale, and plot the average of six runs for the best hyperparameters.
For DDPG we extended the hyperparameter search to also consider the scale and damping of exploratory noise provided by the Ornstein-Uhlenbeck process.
Despite DDPG having the advantage of its exploration decided by a hyperparameter search while
Smoothie must learn its exploration without supervision, we ﬁnd that Smoothie performs competitively across all tasks, exhibiting a slight advantage in Swimmer, Ant, and Humanoid.
These results make it clear that on more difﬁcult continuous control tasks (Hopper, Walker2d, Ant,
Humanoid), both DDPG and Smoothie suffer from instability, showing high variance across runs and
[Figure]
a performance that is far behind trust region and proximal policy methods (Schulman et al., 2015;
Nachum et al., 2017b; Schulman et al., 2017).
Thus, we evaluated Smoothie with a KL-penalty and present these results in Figure 3.
We observe that Smoothie augmented with a KL-penalty consistently improves performance.
The improvement is especially dramatic for Hopper, where the average reward is doubled.
We also highlight the results for Humanoid, which also exhibits a signiﬁcant improvement.
As far as we know, these Humanoid results are the best published results for a method that only trains on the order of millions of environment steps.
In contrast, TRPO, which to the best of our knowledge is the only other algorithm which can achieve better performance, requires on the order of tens of millions of environment steps.
This gives added evidence to the beneﬁts of using a learnable covariance and not restricting a policy to be deterministic.
[Figure]
C ONCLUSION
We have presented a new Q-value function, ˜ Q π , that is a Gaussian-smoothed version of the standard expected Q-value, Q π .
The advantage of using ˜ Q π over Q π is that its gradient and Hessian possess an intimate relationship with the gradient of expected reward with respect to mean and covariance of a Gaussian policy.
The resulting algorithm, Smoothie, is able to successfully learn both mean and covariance during training, leading to performance that can match or surpass that of DDPG, especially when incorporating a penalty on divergence from a previous policy.
The success of ˜ Q π is encouraging.
Intuitively it may be argued that learning ˜ Q π is more sensible than learning Q π .
The smoothed Q-values by deﬁnition make the true reward surface smoother, thus possibly easier to learn; moreover the smoothed Q-values have a more direct relationship with the expected discounted return objective.
We encourage future work to further investigate these claims as well as techniques to apply the underlying motivations for ˜ Q π to other types of policies
A P ROOF OF E QUATION (15)
We note that similar identities for Gaussian integrals exist in the literature (Price, 1958; Rezende et al., 2014) and point the reader to these works for further information.
The speciﬁc identity we state may be derived using standard matrix calculus.
We make use of the fact that
[Equation]
and for symmetric A ,
[Equation]
We omit s from Σ( s ) in the following equations for succinctness.
The LHS of (15) is
[Equation]
Meanwhile, towards tackling the RHS of (15) we note that
[Equation]
Thus we have
[Equation]
B P ROOF OF THE C ONDITIONS FOR C OMPATIBILITY
We shall show how the conditions stated for compatibility with respect to Σ φ are sufﬁcient.
The reasoning for µ θ follows via a similar argument.
We also refer the reader to Silver et al.
(2014) which includes a similar procedure for showing compatibility.
From the second condition for compatibility with respect to Σ φ we have
[Equation]
We may combine this with the ﬁrst condition to ﬁnd
[Equation]
which is the desired property for compatibility.
[Equation]
C I MPLEMENTATION D ETAILS
We utilize feed forward networks for both policy and Q-value approximator.
For µ θ ( s ) we use two hidden layers of dimensions (400 , 300) and relu activation functions.
For ˜ Q πw ( s, a ) and Q πw ( s, a ) we ﬁrst embed the state into a 400 dimensional vector using a fully-connected layer and tanh nonlinearity.
We then concatenate the embedded state with a and pass the result through a 1-hidden
[Table]
layer neural network of dimension 300 with tanh activations.
We use a diagonal Σ φ ( s ) =
e φ for
Smoothie, with φ initialized to − 1 .
To ﬁnd optimal hyperparameters we perform a 100-trial random search over the hyperparameters speciﬁed in Table 1.
The OU exploration parameters only apply to DDPG.
The λ coefﬁcient on
KL-penalty only applies to Smoothie with a KL-penalty.