M AXIMUM A P OSTERIORI P OLICY O PTIMISATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relativeentropy objective.
We show that several existing methods can directly be related to our derivation.
We develop two off-policy algorithms and demonstrate that they are competitive with the state-of-the-art in deep reinforcement learning.
In particular, for continuous control, our method outperforms existing methods with respect to sample efﬁciency, premature convergence and robustness to hyperparameter settings.
I NTRODUCTION
Model free reinforcement learning algorithms can acquire sophisticated behaviours by interacting with the environment while receiving simple rewards.
Recent experiments (Mnih et al., 2015; Jaderberg et al., 2016; Heess et al., 2017) successfully combined these algorithms with powerful deep neural-network approximators while beneﬁting from the increase of compute capacity.
Unfortunately, the generality and ﬂexibility of these algorithms comes at a price: They can require a large number of samples and – especially in continuous action spaces – suffer from high gradient variance.
Taken together these issues can lead to unstable learning and/or slow convergence.
Nonetheless, recent years have seen signiﬁcant progress, with improvements to different aspects of learning algorithms including stability, data-efﬁciency and speed, enabling notable results on a variety of domains, including locomotion (Heess et al., 2017; Peng et al., 2016), multi-agent behaviour (Bansal et al., 2017) and classical control (Duan et al., 2016).
Two types of algorithms currently dominate scalable learning for continuous control problems: First,
Trust-Region Policy Optimisation (TRPO; Schulman et al. 2015) and the derivative family of Proximal Policy Optimisation algorithms (PPO; Schulman et al. 2017b).
These policy-gradient algorithms are on-policy by design, reducing gradient variance through large batches and limiting the allowed change in parameters.
They are robust, applicable to high-dimensional problems, and require moderate parameter tuning, making them a popular ﬁrst choice (Ho & Ermon, 2016).
However, as on-policy algorithms, they suffer from poor sample efﬁciency and are therefore limited to simulation.
In contrast, off-policy value-gradient algorithms such as the Deep Deterministic Policy Gradient (DDPG, Silver et al. 2014; Lillicrap et al. 2016), Stochastic Value Gradient (SVG, Heess et al.
, and the related Normalized Advantage Function formulation (NAF, Gu et al. 2016b) rely on experience replay and learned (action-)value functions.
These algorithms exhibit much better data efﬁciency, approaching the regime where experiments with real robots are possible (Gu et al., 2016a;
Andrychowicz et al., 2017).
While also popular, these algorithms can be difﬁcult to tune, especially for difﬁcult high-dimensional domains like general manipulation.
In this paper we propose a novel off-policy algorithm that beneﬁts from the best properties of both classes.
It exhibits the scalability, robustness and hyperparameter insensitivity of on-policy algorithms, while offering the data-efﬁciency of off-policy, value-based methods.
To derive our algorithm, we take advantage of the duality between control and estimation by using
Expectation Maximisation (EM), a powerful tool from the probabilistic estimation toolbox, in order to solve control problems.
This duality can be understood as replacing the question “what are the actions which maximise future rewards?”
with the question “assuming future success in maximising rewards, what are the actions most likely to have been taken?”.
By using this estimation objective we have more control over the policy change in both E and M steps yielding robust learning.
We show below that several algorithms, including TRPO, can be directly related to this perspective.
We leverage the fast convergence properties of EM-style coordinate ascent by alternating a nonparametric data-based E-step which re-weights state-action samples, with a supervised, parametric
M-step using deep neural networks.
This process is stable enough to allow us to use full covariance matrices, rather than just diagonal, in our policies.
We evaluate our algorithm on a broad spectrum of continuous control problems including a 56 DoF humanoid body, as well as a subset of games from the Atari Learning Environment (ALE).
All experiments used the same optimisation hyperparameters 1 .
Our algorithm shows remarkable data efﬁciency often solving the tasks we consider an order of magnitude faster than the state-of-the-art.
A video of some resulting behaviours can be found here dropbox.com/s/pgcmjst7t0zwm4y/MPO.mp4 .
B ACKGROUND AND N OTATION 2. R ELATED W ORK
Casting Reinforcement Learning (RL) as an inference problem has a long history dating back at least two decades (Dayan & Hinton, 1997).
The framework presented here is inspired by a variational inference perspective on RL that has previously been utilised in multiple studies; c.f. Dayan &
Hinton (1997); Neumann (2011); Deisenroth et al.
; Rawlik et al.
; Levine & Koltun (2013); Florensa et al.
Particular attention has been paid to obtaining maximum entropy policies as the solution to an inference problem.
The penalisation of determinism can be seen encouraging both robustness and simplicity.
Among these are methods that perform trajectory optimisation using either linearised dynamics (Todorov, 2008; Toussaint, 2009; Levine & Koltun, 2013) or general dynamics as in path integral control (Kappen, 2005; Theodorou et al., 2010).
In contrast to these algorithms, here we do not assume the availability of a transition model and avoid on-policy optimisation.
A number of other authors have considered the same perspective but in a model-free RL setting (Neumann, 2011;
Peters et al., 2010a; Florensa et al., 2017; Daniel et al., 2016) or inverse RL problems (Ziebart et al., 2008).
These algorithms are more directly related to our work and can be cast in the same (EM-like) alternating optimisation scheme on which we base our algorithm.
However, they typically lack the maximisation (M)-step – with the prominent exception of REPS (Peters et al., 2010a) to which our algorithm is closely related as outlined below.
An interesting recent addition to these approaches is an EM-perspective on the PoWER
algorithm
(Roux, 2016) which uses the same iterative policy improvement employed here, but commits to parametric inference distributions and avoids an exponential reward transformation, resulting in a harder to optimise lower bound.
As an alternative to these policy gradient inspired algorithms, the class of recent algorithms for soft
Q-learning (e.g. Rawlik et al.
(2012); Haarnoja et al.
Fox et al.
(2016) parameterise and estimate a so called “soft” Q-function directly, implicitly inducing a maximum entropy policy.
A perspective that can also be extended to hierarchical policies (Florensa et al., 2017), and has recently been used to establish connections between Q-learning and policy gradient methods (O’Donoghue et al., 2016; Schulman et al., 2017a).
In contrast, we here rely on a parametric policy, our bound and derivation is however closely related to the deﬁnition of the soft (entropy regularised) Q-function.
A line of work, that is directly related to the “RL as inference” perspective, has focused on using information theoretic regularisers such as the entropy of the policy or the Kullback-Leibler divergence (KL) between policies to stabilise standard RL objectives.
In fact, most state-of-the-art policy gradient algorithms fall into this category.
For example see the entropy regularization terms used in
Mnih et al.
or the KL constraints employed by work on trust-region based methods (Schulman et al., 2015; 2017b; Gu et al., 2017; Wang et al., 2017).
The latter methods introduce a trust region constraint, deﬁned by the KL divergence between the new policy and the old policy, so that the expected KL divergence over state space is bounded.
From the perspective of this paper these trust-region based methods can be seen as optimising a parametric E-step, as in our algorithm, but are “missing” an explicit M-step.
Finally, the connection between RL and inference has been invoked to motivate work on exploration.
The most prominent examples for this are formed by work on Boltzmann exploration such as Kaelbling et al.
; Perkins & Precup (2002); Sutton (1990); O’Donoghue et al. (2017), which can be connected back to soft Q-learning (and thus to our approach) as shown in Haarnoja et al.
2. M ARKOV DECISION P ROCESSES
We consider the problem of ﬁnding an optimal policy π for a discounted reinforcement learning (RL) problem; formally characterized by a Markov decision process (MDP).
The MDP consists of: continuous states s , actions a , transition probabilities p ( s t +1 | s t , a t ) – specifying the probability of transitioning from state s t to s t +1 under action a t –, a reward function r ( s, a ) ∈ R as well as the discounting factor γ ∈ [0 , 1) .
The policy π ( a | s, θ ) (with parameters θ ) is assumed to specify a probability distribution over action choices given any state and – together with the transition probabilities – gives rise to the stationary distribution µ π ( s ) .
Using these basic quantities we can now deﬁne the notion of a Markov sequence or trajectory τ π = { ( s 0 , a 0 ) . . .
( s T , a T ) } sampled by following the policy π ; i.e. τ π ∼ p π ( τ ) with p π ( τ ) = p ( s 0 ) (cid:81) t> 0 p ( s t +1 | s t , a t ) π ( a t
| s t ) ; and the expected return E τ π
[ (cid:80) ∞ t =
γ t r ( s t , s t )] .
We will use the shorthand r t = r
( s t , a t ) .
M AXIMUM A P OSTERIORI P OLICY O PTIMISATION
The derivation of our algorithm starts from the classical connection between RL and probabilistic inference.
Rather than estimating a single optimal trajectory methods in this space attempt to identify a distribution of plausible solutions by trading off expected return against the (relative) entropy of this distribution.
Building on classical work such as Dayan & Hinton (1997) and more recent work e.g. by Peters et al.
(2010b)
we cast policy search as a particular instance of the class of expectationmaximisation algorithms.
Our algorithm then combines properties of existing approaches in this family with properties of recent off-policy algorithms for neural networks.
Speciﬁcally we propose an alternating optimisation scheme that leads us to a novel, off-policy algorithm that is (a) data efﬁcient; (b) robust and effectively hyper-parameter free; (c) applicable to complex control problems that have so far been considered outside the realm of off-policy algorithms; (d) allows for effective parallelisation.
Our algorithm separates policy learning into two alternating phases which we refer to as E and M step in reference to the EM algorithm:
• E-step: In the E-step we obtain an estimate of the distribution of return-weighted trajectories.
We perform this step by re-weighting state action samples using a learned value function.
This is akin to posterior inference step when estimating the parameters of a probabilistic models with latent variables.
• M-step: In the M-step we update the parametric policy in a supervised learning step using the reweighted state-action samples from the E-step as targets.
This corresponds to the update of the model parameters given the complete data log-likelihood when performing EM for a probabilistic model.
These choices lead to the following desirable properties: (a) low-variance estimates of the expected return via function approximation; (b) low-sample complexity of value function estimate via robust off-policy learning; (c) minimal parametric assumption about the form of the trajectory distribution in the E-step; (d) policy updates via supervised learning in the M step; (e) robust updates via hard trust-region constraints in both the E and the M step.
3. P OLICY I MPROVEMENT
We start from the perspective of deriving an RL objective from a probabilistic inference problem.
To this end we assume the existence of an observed optimality event O whose probability is proportional to the exponentiated return 2 ; p ( O = 1 | τ ) ∝ (cid:81) ∞
t =0 exp( γ t r t α ) with temperature parameter α >
Intuitively, O can be interpreted as the event of obtaining maximum reward by choosing an action; or the event of succeeding at the RL task (Neumann, 2011; Toussaint, 2009).
We then aim to ﬁnd policy parameters θ such as to maximize the following lower bound on the maximum a-posteriori objective for the policy parameters θ ,
[Equation]
where q ( a | s ) is an auxilliary policy, aka variational distribution, that will be used as a “vehicle” for ﬁnding an improved policy.
We also deﬁne the regularized Q-value function
[Equation]
with KL (cid:0) q t || π t (cid:1) =
KL (cid:0) q ( a | s t ) (cid:1) (cid:107) π ( a | s t , θ ) (cid:1) .
Note that KL (cid:0) q 0 || p 0 (cid:1) and p ( θ ) are not part of the
Q-function as they are not a function of the action.
The objective from (1) can be motivated from different perspectives.
It arises naturally in the control as inference framework (Neumann, 2011; Toussaint, 2009), and it is closely related to the objective of maximum entropy RL where the relative entropy (KL) is replaced by a simple entropy term (Haarnoja et al., 2017; Schulman et al., 2017a).
We will consider a general framework for optimizing
J via alternate coordinate ascent in q and π θ , analogous to the expectation-maximization algorithm in the probabilistic modelling literature.
3. E-S TEP
In the E-step we optimize the lower bound J ( q, θ ) with respect to q .
We start by setting q =
π θ i and estimate the action-value function:
[Equation]
since KL( q || π t ) = 0 .
In practice we estimate Q θ i from off-policy data (we refer to Section 4 for details about the policy evaluation step).
This greatly increases the data efﬁciency of our algorithm.
Given Q θ i we improve the lower bound J w.r.t.
q by ﬁrst expanding Equation (3) via the Bellman operator and optimize the “one-step” KL regularised objective
[Equation]
This optimisation problem is equivalent to the inference problem:
[Equation]
where the likelihood of optimality for a state-action pair is p ( O | a, s )
∝ exp( Q θ i ( a, s ) /α ) .
Maximizing Equation (4), thus obtaining q i = arg max ¯ J ( q, θ i ) , does not fully optimize J since we treat Q
θ i as constant with respect to q .
This optimization thus implements a partial E-step.
In practice we also choose µ q to be the stationary distribution as given through samples from the replay buffer.
The reward and the KL terms are on an arbitray relative scale.
This can make it difﬁcult to choose a suitable α .
We therefore replace the soft KL regularization with a hard constraint with parameter (cid:15) , i.e,
[Equation]
If we choose to explicitly parameterize q ( a | s ) – option 1 below – the resulting optimisation is similar to that performed by the recent TRPO algorithm for continuous control (Schulman et al., 2015); only in an off-policy setting.
Analogously, the unconstrained objective (4) is similar to the objective used by PPO (Schulman et al., 2017b).
We note, however, that the KL is reversed when compared to the KL used by TRPO and PPO.
To implement (5) we need to choose a form for the variational policy q ( a | s ) .
Two options arise: 1.
We can use a parametric variational distribution q ( a | s, θ q ) , with parameters θ q , and optimise Equation (5) via the likelihood ratio or action-value gradients.
This leads to an algorithm similar to TRPO/PPO and an explicit M-step becomes unnecessary (see.
Alg. 3).
We can choose a non-parametric representation of q ( a | s ) given by one probability factor per sample.
To achieve generalization in state space we then ﬁt a parametric policy in the
M-step.
Fitting a parametric policy in the M-step is a supervised learning problem, allowing us to employ various regularization techniques at that point.
It also makes it easier to enforce the hard KL constraint.
N ON PARAMETRIC VARIATIONAL DISTRIBUTION
In the non-parametric case we can obtain the optimal sample based q distribution – the solution to
Equation (5) – in closed form (see the appendix for a full derivation), as,
[Equation]
where we can obtain η ∗ by minimising the following convex dual function,
[Equation]
after the optimisation of which we can evaluate q i ( a | s ) on given samples.
This optimization problem is similar to the one solved by relative entropy policy search (REPS) (Peters et al., 2010a) with the difference that we optimise only for the conditional variational distribution q ( a | s ) instead of a joint distribution q ( a, s ) – effectively ﬁxing µ q ( s ) to the stationary distribution given by previously collected experience – and we use the Q function of the old policy to evaluate the integral over a .
While this might seem unimportant it is crucial as it allows us to estimate the integral over actions with multiple samples without additional environment interaction.
This greatly reduces the variance of the estimate and allows for fully off-policy learning at the cost of performing only a partial optimization of J as described above.
3. M- STEP
Given q i from the E-step we can optimize the lower bound J with respect to θ to obtain an updated policy θ i +
= arg max θ J ( q i , θ ) .
Dropping terms independent of θ this entails solving for the solution of
[Equation]
which corresponds to a weighted maximum a-posteriroi estimation (MAP) problem where samples are weighted by the variational distribution from the E-step.
Since this is essentially a supervised learning step we can choose any policy representation in combination with any prior for regularisation.
In this paper we set p ( θ ) to a Gaussian prior around the current policy, i.e, p ( θ ) ≈ N (cid:16) µ = θ i , Σ
= F θ i λ (cid:17) , where θ i are the parameters of the current policy distribution, F θ i is the empirical Fisher information matrix and λ is a positive scalar.
As shown in the appendix this suggests the following generalized M-step:
[Equation]
which can be re-written as the hard constrained version:
[Equation]
This additional constraint minimises the risk of overﬁtting the samples, i.e. it helps us to obtain a policy that generalises beyond the state-action samples used for the optimisation.
In practice we have found the KL constraint in the M step to greatly increase stability of the algorithm.
We also note that in the E-step we are using the reverse, mode-seeking, KL while in the M-step we are using the forward, moment-matching, KL which reduces the tendency of the entropy of the parametric policy to collapse.
One caveat of the partial M-step is, however, that in combination with the partial E-step monotone improvement of the J cannot be guaranteed.
P OLICY E VALUATION
Our method is directly applicable in an off-policy setting.
For this, we have to rely on a stable policy evaluation operator to obtain a parametric representation of the Q-function Q
θ ( s, a ) .
We make use of the Retrace algorithm from Munos et al.
Concretely, we ﬁt the Q-function Q θ i ( s, a, φ ) as represented by a neural network, with parameters φ , by minimising the squared loss: min φ L ( φ ) = min φ E µ b ( s ) ,b ( a | s ) (cid:104)(cid:0) Q [Equation] i ( s t , a t , φ ) − Q ret t (cid:1)
[Equation]
, with
Q ret
[Equation]
= [Equation] (11) where Q φ (cid:48) ( s, a ) denotes the output of a target Q-network, with parameters φ (cid:48) , that we copy from the current parameters φ after each M-step.
We truncate the inﬁnite sum after N steps by bootstrapping with Q φ (cid:48)
(rather than considering a λ return).
Additionally, b ( a | s ) denotes the probabilities of an arbitrary behaviour policy.
In our case we use an experience replay buffer and hence b is given by the action probabilities stored in the buffer; which correspond to the action probabilities at the time of action selection.
E XPERIMENTS
For our experiments we evaluate our MPO algorithm across a wide range of tasks.
Speciﬁcally, we start by looking at a suite of continuous control tasks (see Figure 1) and then consider the challenging parkour environments recently published in Heess et al.
In both cases we use a Gaussian distribution for the policy whose mean and covariance are parameterized by a neural network (see appendix for details).
In addition, we present initial experiments for discrete control using ATARI environments using a categorical policy distribution (whose logits are again parameterized by a neural network) in the appendix.
5. E VALUATION ON CONTROL SUITE
The suite of continuous control tasks that we are evaluating against contains 18 tasks, comprising a wide range of domains including well known tasks from the literature.
For example, the classical cart-pole and acrobot dynamical systems, 2D and Humanoid walking as well as simple low-dimensional planar reaching and manipulation tasks.
This suite of tasks was built in python on top of mujoco and will also be open sourced to the public by the time of publication.
While we include plots depicting the performance of our algorithm on all tasks below; comparing it against the state-of-the-art algorithms in terms of data-efﬁciency.
We want to start by directing the attention of the reader to a more detailed evaluation on three of the harder tasks from the suite.
5.1. D ETAILED A NALYSIS ON W ALKER -2D, A CROBOT , H OPPER
We start by looking at the results for the classical Acrobot task (two degrees of freedom, one continuous action dimension) as well as the 2D walker (which has 12 degrees of freedom and thus a 12 dimensional action space and a 21 dimensional state space) and the hopper standing task.
The reward in the Acrobot task is the distance of the robots end-effector to an upright position of the underactuated system.
For the walker task it is given by the forward velocity, whereas in the hopper the requirement is to stand still.
Figure 2 shows the results for this task obtained by applying our algorithm MPO as well as several ablations – in which different parts were removed from the MPO optimization – and two baselines: our implementation of Proximal Policy Optimization (PPO) (Schulman et al., 2017b) and DDPG.
The hyperparameters for MPO were kept ﬁxed for all experiments in the paper (see the appendix for hyperparameter settings).
As a ﬁrst observation, we can see that MPO gives stable learning on all tasks and, thanks to its fully off-policy implementation, is signiﬁcantly more sample efﬁcient than the on-policy PPO baseline.
Furthermore, we can observe that changing from the non-parametric variational distribution to a parametric distribution 3 (which, as described above, can be related to PPO) results in only a minor asymptotic performance loss but slowed down optimisation and thus hampered sample efﬁciency; which can be attributed to the fact that the parametric q distribution required a stricter KL constraint.
Removing the automatically tuned KL constraint and replacing it with a manually set entropy regulariser then yields an off-policy actor-critic method with Retrace.
This policy gradient method still uses the idea of estimating the integral over actions – and thus, for a gradient based optimiser, its likelihood ratio derivative – via multiple action samples (as judged by a Q-Retrace critic).
This idea has previously been coined as using the expected policy gradient (EPG) (Ciosek & Whiteson, 2017) and we hence denote the corresponding algorithm with EPG + Retrace, which no-longer follows the intuitions of the MPO perspective.
EPG + Retrace performed well when the correct entropy regularisation scale is used.
This, however, required task speciﬁc tuning (c.f. Figure 4 where this hyperparameter was set to the one that performed best in average across tasks).
Finally using only a single sample to estimate the integral (and hence the likelihood ratio gradient) results in an actor-critic variant with Retrace that is the least performant off-policy algorithm in our comparison.
5.1. C OMPLETE RESULTS ON THE CONTROL SUITE
The results for MPO (non-parameteric) – and a comparison to an implementation of state-of-the-art algorithms from the literature in our framework – on all the environments from the control suite that we tested on are shown in Figure 4.
All tasks have rewards that are scaled to be between 0 and 1000.
We note that in order to ensure a fair comparison all algorithms ran with exactly the same network conﬁguration, used a single learner (no distributed computation), used the same optimizer and were tuned w.r.t.
their hyperparameters for best performance across all tasks.
We refer to the appendix for a complete description of the hyperparameters.
Our comparison is made in terms of data-efﬁciency.
From the plot a few trends are readily apparent: i)
We can clearly observe the advantage in terms of data-efﬁciency that methods relying on a Q-critic obtain over the PPO baseline.
This difference is so extreme that in several instances the PPO baseline converges an order of magnitude slower than the off-policy algorithms and we thus indicate the asymptotic performance of each algorithm of PPO and DDPG (which also improved signiﬁcantly later during training in some instances) with a colored star in the plot; ii) A quick look at the difference between the MPO results and the (expected) policy gradient (EPG) with entropy regularisation conﬁrm our suspicion from Section 5.1.1: ﬁnding a good setting for the entropy regulariser that transfers across environments without additional constraints on the policy distribution is very difﬁcult, leading to instabilities in the learning curves.
In contrast to this the MPO results appear to be stable across all environments; iii) Finally, in terms of data-efﬁciency the methods utilising Retrace obtain a clear advantage over DDPG.
The single learner vanilla DDPG implementation learns the lower dimensional environments quickly but suffers in terms of learning speed in environments with sparse rewards (ﬁnger, acrobot) and higher dimensional action spaces.
Overall, MPO is able to solve all environments using surprisingly moderate amounts of data.
On average less than 1000 trajectories (or 10 6 samples) are needed to reach the best performance.
5. H IGH - DIMENSIONAL CONTINUOUS CONTROL
Next we turn to evaluating our algorithm on two higher-dimensional continuous control problems; humanoid and walker.
To make computation time bearable in these more complicated domains we utilize a parallel variant of our algorithm: in this implementation K learners are all independently collecting data from an instance of the environment.
Updates are performed at the end of each collected trajectory using distributed synchronous gradient descent on a shared set of policy and
Q-function parameters (we refer to the appendix for an algorithm description).
The results of this experiment are depicted in Figure 3.
For the Humanoid running domain we can observe a similar trend to the experiments from the previous section: MPO quickly ﬁnds a stable running policy, outperforming all other algorithms in terms of sample efﬁciency also in this high-dimensional control problem.
The case for the Walker-2D parkour domain (where we compare against a PPO baseline) is even more striking: where standard PPO requires approximately 1M trajectories to ﬁnd a good policy
MPO ﬁnds a solution that is asymptotically no worse than the PPO solution in in about 70k trajectories (or 60M samples) , resulting in an order of magnitude improvement.
[Figure]
5. D ISCRETE CONTROL
As a proof of concept – showcasing the robustness of our algorithm and its hyperparameters – we performed an experiment on a subset of the games contained contained in the "Arcade Learning
Environment" (ALE) where we used the same hyperparameter settings for the KL constraints as for the continuous control experiments.
The results of this experiment can be found in the Appendix.
C ONCLUSION
We have presented a new off-policy reinforcement learning algorithm called Maximum a-posteriori
Policy Optimisation (MPO).
The algorithm is motivated by the connection between RL and inference and it consists of an alternating optimisation scheme that has a direct relation to several existing algorithms from the literature.
Overall, we arrive at a novel, off-policy algorithm that is highly data efﬁcient, robust to hyperparameter choices and applicable to complex control problems.
We demonstrated the effectiveness of MPO on a large set of continuous control problems
A A DDITIONAL E XPERIMENT : D ISCRETE CONTROL
As a proof of concept – showcasing the robustness of our algorithm and its hyperparameters – we performed an experiment on a subset of the games contained contained in the "Arcade Learning
Environment" (ALE).
For this experiment we used the same hyperparameter settings for the KL constraints as for the continuous control experiments as well as the same learning rate and merely altered the network architecture to the standard network structure used by DQN Mnih et al.
– and created a seperate network with the same architecture, but predicting the parameters of the policy distribution.
A comparison between our algorithm and well established baselines from the literature, in terms of the mean performance, is listed in Table 1.
While we do not obtain state-ofthe-art performance in this experiment, the fact that MPO is competitive, out-of-the-box in these domains suggests that combining the ideas presented in this paper with recent advances for RL with discrete actions (Bellemare et al., 2017) could be a fruitful avenue for future work.
B E XPERIMENT DETAILS
In this section we give the details on the hyper-parameters used for each experiment.
All the continuous control experiments use a feed-forward network except for Parkour-2d wjere we used the same network architecture as in Heess et al.
Other hyper parameters for MPO with non parametric variational distribution were set as follows,
[Table]
Q
[Table]
Hyperparameters for MPO with parametric variational distribution were as follows,
[Table]
C D ERIVATION OF UPDATE RULES FOR A G AUSSIAN P OLICY
For continuous control we assume that the policy is given by a Gaussian distribution with a full covariance matrix, i.e, π ( a | s , θ )
=
N ( µ, Σ ) .
Our neural network outputs the mean µ = µ ( s ) and
Cholesky factor
A =
A ( s ) , such that Σ = AA T .
The lower triagular factor A has positive diagonal elements enforced by the softplus transform A ii ← log(1 + exp( A ii )) .
C. N ON - PARAMETRIC VARIATIONAL DISTRIBUTION
In this section we provide the derivations and implementation details for the non-parametric variational distribution case for both E-step and M-step.
C. E-S TEP
The E-step with a non-parametric variational solves following program:
[Equation]
First we write the Lagrangian equation, i.e,
[Equation]
Next we maximise the Lagrangian L w.r.t the primal variable q .
The derivative w.r.t q reads,
[Equation]
Setting it to zero and rearranging terms we get
[Equation]
However the last exponential term is a normalisation constant for q .
Therefore we can write,
[Equation]
Note that we could write γ based on q and η .
At this point we can drive the dual function,
[Equation]
C. M- STEP
To obtain the KL constraint in the M step we set p ( θ ) to a Gaussian prior around the current policy, i.e,
[Equation]
where θ i are the parameters of the current policy distribution,
F θ i is the empirical Fisher information matrix and λ .
With this, and dropping constant terms our optimization program becomes
[Equation]
We can observe that ( θ − θ i ) T F − 1 θ i ( θ − θ i ) is the second order Taylor approximation of (cid:82) µ q ( s ) KL ( π ( a | s, θ i ) , π ( a | s, θ ))
ds which leads us to the generalized M-step objective:
[Equation]
(13) which we turn into constraint optimization problem as described in the main text:
[Equation]
After obtaining the non parametric variational distribution in the M step with a Gaussian policy we empirically observed that better results could be achieved by decoupling the KL constraint into two terms such that we can constrain the contribution of the mean and covariance separately i.e.
[Equation]
where
[Equation]
Where n is dimension of action space and for Gaussian policies we can verify,
[Equation]
This decoupling allows us to set different (cid:15) values for each component, i.e., (cid:15) µ , (cid:15) Σ for the mean, the covariance matrix respectively.
Different (cid:15) lead to different learning rates.
The effectivness of this decoupling has also been shown in Abdolmaleki et al.
We always set a much smaller epsilon for covariance than the mean.
The intuition is that while we would like the distribution moves fast in the action space, we also want to keep the exploration to avoid premature convergence.
In order to solve the constrained optimisation in the M-step, we ﬁrst write the generalised Lagrangian equation, i.e,
[Equation]
Where η µ and η Σ are Lagrangian multipliers.
Following prior work on constraint optimisation, we formulate the following primal problem,
[Equation]
In order to solve for θ we iteratively solve the inner and outer optimisation programs independently:
We ﬁx the Lagrangian multipliers to their current value and optimise for θ (outer maximisation) and then ﬁx the parameters θ to their current value and optimise for the Lagrangian multipliers (inner minimisation).
We continue this procedure untill policy parameters θ and Lagrangian multipliers converge.
Please note that the same approach can be employed to bound the KL explicitly instead of decoupling the contribution of mean and covariance matrix to the KL.
C. P ARAMETRIC VARIATIONAL DISTRIBUTION
In this case we assume our variational distribution also uses a Gaussian distribution over the action space and use the same structure as our policy π .
Similar to the non-parametric case for a Gaussian distribution in the M-step we also use a decoupled
KL but this time in the E-step for a Gaussian variational distribution, i.e,
[Equation] [Equation]
where
[Equation]
Where n is dimension of action space and for Gaussian policies we can verify,
[Equation]
For the parametric case we achieved better results using advantage function A ( a, s ) instead of Q function Q ( a, s ) .
Please note that the KL in the E-step is different than the one used in the M-step.
In order to solve the constrained optimisation program, we ﬁrst write the generalised Lagrangian equation, i.e,
[Equation]
Where η µ and η Σ are Lagrangian multipliers.
Following prior works on constraint optimisation, we formulate the following primal problem,
[Equation]
In order to solve for θ q we iteratively solve the inner and outer optimisation programs independently.
In order to that we ﬁx the Lagrangian multipliers to their current value and optimise for θ q (outer maximisation), in this case we use the likelihood ratio gradient to compute the gradient w.r.t θ q .
Subsequently we ﬁx the parameters θ q to their current value and optimise for Lagrangian multipliers (inner minimisation).
We iteratively continue this procedure untill the policy parameters θ q and the
Lagrangian multipliers converges.
Please note that the same approach can be used to bound the KL explicitly instead of decoupling the contribution of mean and covariance matrix to the KL.
As our policy has the same structure as the parametric variational distribution, the M step in this case reduce to set the policy parameters θ to the parameters θ q we obtained in E-step, i.e, θ i +
=
θ q
D I MPLEMENTATION DETAILS
While we ran most of our experiments using a single learner, we implemented a scalable variant of the presented method in which multiple workers collect data independently in an instance of the considered environment, compute gradients and send them to a chief (or parameter server) that performs parameter update by averaging gradients.
That is we use distributed synchronous gradient descent.
These procedures are described in Algorithms 1 and 2 for the non-parametric case andfor the parametric case.
7: s φ = s φ + [ δφ w ] 8: s φ = s θ + [ δθ w ] 9: s η = s η
+ [ δη w ] 10: s η µ = s η µ + [ δη wµ ] 11: s η θ =
s η θ
+ [ δη wθ ] 12: update
13: s φ
, s η µ , s s
14: send new parameters to workers
1: Input = (cid:15), (cid:15) Σ , (cid:15) µ , L max 2: i = 0 , L curr =3:
Initialise Q ω i ( a, s ) , π ( a | s, θ i ) , η, η µ , η Σ 4: for each worker do 5: while L curr > L max do 6: update replay buffer B with L trajectories from the environment 7: k =8: // Find better policy by gradient descent 9: while k < 1000 do 10: sample a mini-batch B of N ( s , a , r ) pairs from replay 11: sample M additional actions for each state from B , π ( a | s, θ i ) for estimating integrals 12: compute gradients, estimating integrals using samples 13: // Q-function gradient: 14: δ φ = ∂
φ L (cid:48) φ ( φ ) 15: //
E-Step gradient: 16: δη = ∂
η g ( η ) 17:
Let: q ( a | s ) ∝ π
( a | s, θ i )
exp( Q θt ( a,s,φ (cid:48) ) η ) 18: // M-Step gradient: 19: [ δ η µ , δ η Σ ] = α∂ η µ ,η Σ L ( θ k , η µ , η Σ ) 20: δ θ = ∂ θ
L ( θ , η µk +1 , η Σ k +1 ) 21: send gradients to chief worker 22: wait for gradient update by chief 23: fetch new parameters φ, θ, η, η µ , η Σ 24:
k = k +25: i
= i + 1 , L curr =
L curr + L 26: θ i = θ , φ (cid:48)
=
φ
1: Input = (cid:15) Σ , (cid:15) µ , L max 2: i = 0 , L curr =3:
Initialise Q ω i ( a, s ) , π ( a | s, θ i ) , η, η µ , η
[Equation]
4: for each worker do 5: while L curr > L max do 6: update replay buffer B with L trajectories from the environment 7: k =8: // Find better policy by gradient descent 9: while k < 1000 do 10: sample a mini-batch B of N ( s , a , r ) pairs from replay 11: sample M additional actions for each state from B , π ( a | s, θ k ) for estimating integrals 12: compute gradients, estimating integrals using samples 13: // Q-function gradient: 14:
[Equation]
15: // E-Step gradient: 16:
[Equation]
[Equation]
18: // M-Step gradient: In practice there is no M-step in this case as policy and variatinal distribution q use a same structure.
19: send gradients to chief worker 20: wait for gradient update by chief 21: fetch new parameters φ, θ, η, η µ , η Σ 22: k =
[Equation] [Equation]
L
[Equation]
24: θ
[Equation] [Equation]