NADPE X : A N ON - POLICY TEMPORALLY CONSISTENT EXPLORATION METHOD FOR DEEP REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Reinforcement learning agents need exploratory behaviors to escape from local optima.
These behaviors may include both immediate dithering perturbation and temporally consistent exploration.
To achieve these, a stochastic policy model that is inherently consistent through a period of time is in desire, especially for tasks with either sparse rewards or long term information.
In this work, we introduce a novel on-policy temporally consistent exploration strategy - Neural Adaptive
Dropout Policy Exploration (NADPEx) - for deep reinforcement learning agents.
Modeled as a global random variable for conditional distribution, dropout is incorporated to reinforcement learning policies, equipping them with inherent temporal consistency, even when the reward signals are sparse.
Two factors, gradients’ alignment with the objective and KL constraint in policy space, are discussed to guarantee NADPEx policy’s stable improvement.
Our experiments demonstrate that NADPEx solves tasks with sparse reward while naive exploration and parameter noise fail.
It yields as well or even faster convergence in the standard mujoco benchmark for continuous control.
I NTRODUCTION
Exploration remains a challenge in reinforcement learning, in spite of its recent successes in robotic manipulation and locomotion (Schulman et al., 2015b; Mnih et al., 2016; Duan et al., 2016; Schulman et al., 2017b).
Most reinforcement learning algorithms explore with stochasticity in stepwise action space and suffer from low learning efﬁciency in environments with sparse rewards (Florensa et al., 2017) or long information chain (Osband et al., 2017).
In these environments, temporally consistent exploration is needed to acquire useful learning signals.
Though in off-policy methods, autocorrelated noises (Lillicrap et al., 2015) or separate samplers (Xu et al., 2018) could be designed for consistent exploration, on-policy exploration strategies tend to be learnt alongside with the optimal policy, which is non-trivial.
A complementary objective term should be constructed because the purpose of exploration to optimize informational value of possible trajectories (Osband et al., 2016) is not contained directly in the reward of underlying Markov Decision Process (MDP).
This complementary objective is known as reward shaping e.g. optimistic towards uncertainty heuristic and intrinsic rewards.
However, most of them require complex additional structures and strong human priors when state and action spaces are intractable, and introduce unadjustable bias in end-to-end learning (Bellemare et al., 2016;
Ostrovski et al., 2017; Tang et al., 2017; Fu et al., 2017; Houthooft et al., 2016; Pathak et al., 2017).
It would not be necessary though, to teach agents to explore consistently through reward shaping, if the policy model inherits it as a prior.
This inspires ones to disentangle policy stochasticity, with a structure of time scales.
One possible solution is policy parameter perturbation in a large time scale.
Though previous attempts were restricted to linear function approximators (R ¨ uckstieß et al., 2008; Osband et al., 2014), progress has been made with neural networks, through either network section duplication (Osband et al., 2016) or adaptive-scale parameter noise injection (Plappert et al., 2017; Fortunato et al., 2017).
However, in Osband et al.
the episode-wise stochasticity is unadjustable, and the duplicated modules do not cooperate with each other.
Directly optimizing the mean of distribution of all parameters, Plappert et al.
(2017) adjusts the stochasticity to the learning progress heristically.
Besides, as all sampled parameters need to be stored in on-policy exploration, it is not directly salable to large neural networks, where the trade-off between large memory for multiple networks and high variance with sinlge network is non-trivial to make.
This work proposes Neural Adaptive Dropout Policy Exploration (NADPEx) as a simple, scalable and improvement-guaranteed method for on-policy temporally consistent exploration, which generalizes Osband et al.
(2016) and Plappert et al.
Policy stochasticity is disentangled into two time scales, step-wise action noise and episode-wise stochasticity modeled with a distribution of subnetworks.
With one single subnetwork in an episode, it achieves inherent temporal consistency with only a little computational overhead and no crafted reward, even in environments with sparse rewards.
And with a set of subnetworks in one sampling batch, agents experience diverse behavioral patterns.
This sampling of subnetworks is executed through dropout (Hinton et al., 2012;
Srivastava et al., 2014), which encourages composability of constituents and facilitates the emergence of diverse maneuvers.
As dropout could be excerted on connections, neurons, modules and paths, NADPEx naturally extends to neural networks with various sizes, exploiting modularity at various levels.
To align separately parametrized stochasticity to each other, this sampling is made differentiable for all possible dropout candidates.
We further discuss the effect of a ﬁrst-order KL regularizer on dropout policies to improve stability and guarantee policy improvement.
Our experiments demonstrate that NAPDEx solves challenging exploration tasks with sparse rewards while achieving as efﬁcient or even faster convergence in the standard mujoco benchmark for state-of-theart PPO agents, which we believe can be generalized to any deep reinforcement learning agents.
P RELIMINARIES
We consider a standard discrete-time ﬁnite-horizon discounted Markov Decision Process (MDP) setting for reinforcement learning, represented by a tuple M =
( S , A , P , r, ρ 0 , γ, T ) , with a state set S , an action set A , a transitional probability distribution P : S ×A×S
→ R + , a bounded reward function r : S ×
A → R + , an initial state distribution ρ 0 : S → R + , a discount factor γ ∈ [0 , 1] and horizon T .
We denote a policy parametrized by θ as π θ : S×A → R + .
The optimization objective of this policy is to maximize the expected discounted return η ( π θ ) =
E τ [ (cid:80) Tt =
γ t r ( s t , a t )] , where τ = ( s 0 , a 0 , ... ) denotes the whole trajectory, s 0 ∼ ρ 0 ( s 0 ) , a t ∼ π θ ( a t
| s t ) and s t +1 ∼ P ( s t +1 | s t , a t ) .
In experimental evaluation, we follow normal setting and use undiscounted return
E τ
[ (cid:80) Tt =
r ( s t , a t )] .
2. P ROXIMAL P OLICY O PTIMIZATION (PPO)
Theoretically speaking, NADPEx works with any policy gradient based algorithms.
In this work, we consider the recent advance in on-policy policy gradient method, proximal policy optimization (PPO) (Schulman et al., 2017b).
Policy gradient methods were ﬁrst proposed by Williams (1992) in REINFORCE algorithm to maximize the aforementioned objective in a gradient ascent manner.
The gradient of an expectation
∇ θ η ( π θ ) is approximated with a Monte Carlo average of policy gradient:
[Equation]
where N is the number of episodes in this batch, R it =
(cid:80)
Tt
[Equation]
=
t
[Equation]
− t r it (cid:48)
and b it is a baseline for variance reduction.
Schulman et al.
(2015b) proposes a policy iteration algorithm and proves its monotonicity in improvement.
With the π θ ’s corresponding discounted state-visitation frequency ρ θ , the sampling policy π θ old and the updated policy π θ , it solves the following constrained optimization problem with a line search for an appropriate step size within certain bound δ KL , called trust region :
[Equation]
[Equation]
where D KL ( ·||· ) is the Kullback-Leibler (KL) divergence,
A θ old ( s, a ) is calculated with Generalized
Advantage Estimation (GAE) (Schulman et al., 2015c).
Proximal Policy Optimization (PPO) transforms (2) to a unconstrained optimization and solves it with ﬁrst-order derivatives, embracing established stochastic gradient-based optimizer like Adam (Kingma & Ba, 2014).
Noting that D KL ( π old θ ( ·| s ) | π θ ( ·| s )) is actually a relaxation of total variational divergence according to Schulman et al.
(2015b), whose ﬁrst-order derivative is 0 when θ is close to θ old
, D KL ( π θ ( ·| s )
||
π old θ ( ·| s )) is a natural replacement.
Combining the ﬁrst-order derivative of D KL ( π θ ( ·| s )
||
π old θ ( ·| s ))
(Schulman et al., 2017a)
(check Appendix A for a proof):
[Equation]
PPO optimizes the following unconstrained objective, called KL PPO loss 1 :
[Equation]
where r θ ( s, a ) =
π θ ( a |
[Equation]
π θ old ( a | s ) .
Schulman et al.
(2017b) also proposes a clipping version PPO, as a lower bound to (4):
[Equation]
In this work, we try both KL PPO and clipping PPO.
2. D ROPOUT
Dropout is a technique used in deep learning to prevent features from co-adaptation and parameters from overﬁtting, by randomly dropping some hidden neuron units in each round of feed-forward and back-propagation (Hinton et al., 2012; Srivastava et al., 2014).
This is modeled by multiplying a Bernoulli random variable z kj to each hidden unit, i.e. neuron activation h kj , for j = 1 ...m , where m is the number of hidden units in k th layer.
Then the neuron activation of the k + 1 th layer is
[Equation] [Equation]
where D z = diag ( z ) ∈
R m × m , W ( k +1) and b ( k +1) are weights and biases at k + 1 th layer respectively
and we simply denote them with θ .
= <
W , b > .
σ ( x ) is the nonlinear neuron activation function.
The parameter of this Bernoulli random variable is p k j .
=
P ( z k j = 0) , aka the dropout rate of the j th hidden unit at k th layer.
In supervised learning, these dropout parameters are normally ﬁxed during training in some successful practice (Hinton et al., 2012; Wan et al., 2013).
And there are some variants to dropout connections, modules or paths (Wan et al., 2013).
M ETHODOLOGY 3. N EURAL A DAPTIVE D ROPOUT P OLICY E XPLORATION (NADPE X )
Designing an exploration strategy is to introduce a kind of stochasticity during reinforcement learning agents’ interaction with the environment to help them get rid of some local optima.
While action space noise (parametrized as a stochastic policy π θ ) might be sufﬁcient in environments where stepwise rewards are provided, they have limited effectiveness in more complex environments (Florensa et al., 2017; Plappert et al., 2017).
As their complement, an exploration strategy would be especially beneﬁcial if it can help either sparse reward signals (Florensa et al., 2017) or signiﬁcant long-term information (Osband et al., 2016) to be acquired and back-propagated through time (BPTT) (Sutton & Barto, 1998).
This motivates us to introduce a hierarchy of stochasticity and capture temporal consistency with a separate parametrization.
Our algorithm, Neural Adaptive Dropout Policy Exploration (NADPEx), models stochasticity at large time scales with a distribution of plausible subnetworks.
For each episode, one speciﬁc subnetworks is drawn from this distribution.
Inspected from a large time scale, this encourages exploration towards different directions among different episodes in one batch of sampling.
And from a small time scale, temporal correlation is enforced for consistent exploration.
Policies with a hierarchy of stochasticity is believed to represent a complex action distribution and larger-scale behavioral patterns (Florensa et al., 2017).
We achieve the drawing of subnetwork through dropout.
As introduced in Section 2.2, originally, dropout is modeled through multiplying a binary random variable z kj ∼ Bernoulli (1 − p kj ) to each neuron activation h kj .
In Srivastava et al.
(2014), this binary dropout is softened as continuous random variable dropout, modeled with a Gaussian random variable z ∼ N ( I , σ 2 ) , normally referred to as Gaussian multiplicative dropout .
Here we denote both distributions with q φ ( z ) .
Later we will introduce how both of them could be made differentibale and thus adaptative to learning progress.
During the sampling process, at the beginning of every episode τ i , a vector of dropout random variables z i is sampled from q φ ( z ) , giving us a dropout policy π θ |
z i for step-wise action distribution.
z i is fed to the stochastic computation graph (Schulman et al., 2015a) for the whole episode until termination, when a new round of drawing initiates.
Similar as observations, actions and rewards, this random variable is stored as sampled data, which will be fed back to the stochastic computation graph during training.
Policies with this hierarchy of stochasticity, i.e. NADPEx policies, can be represented as a joint distribution:
[Equation]
making the the objective:
[Equation]
If we only use the stochasticity of network architecture as a bootstrap, as in BootstrapDQN (Osband et al., 2016), the bootstrapped policy gradient training is to update the network parameters θ with the following gradients:
[Equation]
L ( θ , z i ) is the surrogate loss of dropout policy π θ , z i , variates according to the speciﬁc type of reinforcement learning algorithm.
In next subsection, we discuss some pitfalls of bootstrap and provide and our solution to it.
3. T RAINING NADPE X S TOCHASTICITY A LIGNMENT
One concern in separating the parametrization of stochasticity in different levels is whether they can adapt to each other elegantly to guarantee policy improvement in terms of objective (8).
Policy gradients in (9) alway reduces π θ |
z ’s entropy (i.e. stochasticity at small time scale) as the policy improves, which corresponds to the increase in agent’s certainty.
But this information is not propagated to q φ for stochasticity at large time scale in bootstrap.
As in this work q φ is a distribution of subnetworks, this concern could also be intuitively understood as component space composition may not guarantee the performance improvement in the resulting policy space, due to the complex non-linearity of reward function.
Gangwani & Peng (2017) observe that a naive crossover in the parameter space is expected to yield a low-performance composition, for which an extra policy distillation stage is designed.
We investigate likelihood ratio trick (Ranganath et al., 2014; Schulman et al., 2015a) for gradient back-propragation from the same objective (8) through discrete distribution e.g. binary dropout and reparametrization trick (Kingma & Welling, 2013; Rezende et al., 2014) for continuous distribution e.g. Gaussian multiplicative dropout , thus covering all possible dropout candidates (The proof is provided in Appendix B):
[Equation]
In (10) φ is the parameters for Bernoulli distributions, A
( s i 0 , a i 0 ) is the GAE (Schulman et al., 2015c) for the i th trajectory from the beginning, i.e. ( s i 0 , a i 0 ) .
In (11) φ = σ thus z
= I + φ (cid:12) (cid:15) i , (cid:12) is an element-wise multiplication, and (cid:15) i is the dummy random variable (cid:15) i ∼ N ( 0 , I ) .
P OLICY SPACE CONSTRAINT
However, simply making the dropout ditribution differentiable may not guarantee the policy improment.
As introduced in Section 2.1, Schulman et al.
(2015b) reduce the monotonic policy improvement theory to a contraint on KL divergence in policy space.
In this work, the analytical form of
NADPEx policy π θ,φ is obtainable as q φ is designed to be fully factorizable.
Thus ones can leverage the full-ﬂedged KL constraint to stablize the training of NADPEx.
Here we give an example of PPO-NADPEx.
Similar as (3), we leverage a ﬁrst-order approximated
KL divergence for NADPEx policies.
As credit assignment with likelihood ratio trick in (3) may suffer from curse of dimensionality for q φ ( z ) , we stop gradients w.r.t.
φ from the KL divergence to reduce variance with a little bias.
We further replace π θ , φ with a representative, the mean policy π θ = π θ | z , where z is the mean of dropout random variable vectors.
Then the objective is:
[Equation]
(12) where r θ | z ( s, a ) =
[Equation]
π θ old | z ( a | s ) .
A proof for the second term is in Appendix C. Intuitively, KL divergence between dropout polices π θ | z and mean policies π θ is added to remedy the omission of ∇ φ D KL .
Optimizing the lower bound of (12), clipping PPO could adjust the clip ratio accordingly.
3. R ELATIONS TO P ARAMETER N OISE
Episode-wise parameter noise injection (Plappert et al., 2017) is another way to introduce the hierarchy of stochasticity, which could be regarded as a special case of NADPEx, with Gaussian mulitplicative dropout on connection and a heurstic adaptation of variance.
That dropout at neurons is a local reparametrization of noisy networks is proved in Kingma et al.
A replication of the proof is provided in Appendix D. They also prove this local reparametrization trick has a lower variance in gradients.
And this reduction in variance could be enhanced if ones choose to drop modules or paths in NADPEx.
More importantly, as we scope our discussion down to on-policy exploration strategy, where z from q φ ( z ) need to be stored for training, NADPEx is more scalable to much larger neural networks with possibly larger mini-batch size for stochastic gradient-based optimizers.
As a base case, to dropout neurons rather than connections, NADPEx has a complexity of O ( Nm ) for both sampling and memory, comparing to O ( Nm 2 ) of parameter noise, with m denoting the number of neurons in one layer.
This reduction could also be enhanced if ones choose to drop modules or paths.
E XPERIMENTS
In this section we evaluate NADPEx by answering the following questions through experiments.
(i) Can NADPEx achieve state-of-the-art result in standard benchmarks?
(ii) Does NADPEx drive temporally consistent exploration in sparse reward environments?
(iii) Is the end-to-end training of the hierarchical stochasticity effective?
(iv) Is KL regularization effective in preventing divergence between dropout policies and the mean policy ?
We developed NADPEx based on openai/baselines (Dhariwal et al., 2017).
We run all experiments with PPO as reinforcement learning algorithm.
Especially, we developed NADAPEx based on the GPU optimized version PPO.
Details about network architecture and other hyper-parameters are available in Appendix E.
During our implemented parameter noise in the same framework as
NADPEx, we encountered out of memory error for a mini-batch size 64 and training batch size 2048.
This proves our claim of sampling/memory complexity reduction with NADPEx.
We had to make some trade-off between GPU parallelism and more iterations to survive.
For all experiment with NADPEx, we test the following conﬁgurations for the dropout hyperparameter, the initial dropout rate : (a) p kj = 0 .
01 , (b) p kj = 0 .
1 , (c) p kj = 0 .
For Gaussiasn dropout , we estimate the dropout rate as p kj =
σ kj / (1 + σ kj ) .
And for experiments with parameter noise, we set the initial variance accordingly: (a) σ kj = 0 .
001 , (b) σ kj = 0 .
01 , (c) σ kj = 0 .
05 , to ensure same level of stochasticity.
For experiments with ﬁxed KL version PPO, we run with β = 0 . 0005 .
All ﬁgures below show statistics of experiments with 5 randomly generated seeds.
4. NADPE X IN STANDARD ENVIRONMENTS
First, we evaluate NADPEx’s overall performance in standard environments on the continuous control environments implemented in OpenAI Gym (Brockman et al., 2016).
From Figure 2 we can see that they show similar pattern with Gaussian dropout and binary dropout , given identical dropout rates .
Between them, Gaussian dropout is slightly better at scores and consistency among initial dropout rates .
With the three initial dropout rates listed above, we ﬁnd p kj = 0 .
1 shows constistent advantage over others.
On the one hand, when the initial dropout rate is small ( p kj = 0 . 01 ), NADPEx agents learn to earn reward faster than agents with only action space noise.
It is even possible that these agents learn faster than ones with p kj = 0 .
1 in the beginning.
However, their higher variance between random seeds indicates that some of them are not exploring as efﬁciently and the NADPEx policies may not be optimal, therefore normally they will be surpassed by ones with p ki = 0 .
1 in the middle stage of training.
On the other hand, large initial dropout rate ( p kj = 0 . 3 ) tends to converge slowly, possibly due to the claim in Molchanov et al.
(2017) that a large dropout rate could induce large varaince in gradients, making a stable convergence more difﬁcult.
4. T EMPORALLY CONSISTENT EXPLORATION
We then evaluate how NADPEx with Gaussian dropout performs in environments where reward signals are sparse.
Comparing with environments with stepwise rewards, these environments are more challenging as they only yield non-zero reward signals after milestone states are reached.
Temporally consistent exploration therefore plays a crucial role in these environments.
As in Plappert et al.
(2017) we run experiments in rllab (Duan et al., 2016), modiﬁed according to Houthooft et al.
Speciﬁcally, we have: (a) SparseDoublePendulum , which only yields a reward if the agent reaches the upright position, and (b) SparseHalfCheetah , which only yields a reward if the agent crosses a target distance, (c) SparseMountainCar , which only yields a reward if the agent drives up the hill.
We use the same time horizon of T = 500 steps before resetting and gradually increment the difﬁculty during repetition until the performance of NADPEx and parameter noise differentiates.
We would like to refer readers to Appendix G for more details about the sparse reward environments.
As shown in Figure 3, we witness success with temporally consistent exploration through NADPEx, while action perturbation fails.
In all enviroments we examed, larger initial dropout rates can achieve faster or better convergence, revealing a stronger exploration capability at large time scales.
Comparing to parameter noise, NADPEx earns higher score in shorter time, possibly indicating a higher exploration efﬁciency.
4. E FFECTIVENESS OF STOCHASTICITY ADAPTATION
One hypothesis NADPEx builds upon is end-to-end training of separately parametrized stochasticity can appropriately adapt the temporally-extended exploration with the current learning progress.
In
Figure 4 we show our comparison between NADPEx and bootstrap, as introduced in Section 3.
And we can see that though the difference is subtle in simple task like Hopper , the advantage NADPEx has over bootstrap is obvious in Walker2d , which is a task with higher complexity.
In Humanoid , the task with highest dimension of actions, the empirical result is interesting.
Though bootstrap policy learns almost as fast as NADPEx in the begining, that the dropout is not adaptative drives it to over-explore when some dropout policies are close to converge.
Being trapped at a local optimum, bootstrap policy earns 500 scores less than NADPEx policy.
4. KL DIVERGENCE BETWEEN DROPOUT POLICIES
To evaluate the effect of the KL regularizer, we also run experiments with KL PPO.
Though in the original paper of Schulman et al.
(2017b) clipping PPO empirically performs better than KL PPO, we believe including this KL term explicitly in the objective makes our validation self-explanatory.
Figure 5 left shows the experiment result in Walker2d .
Different from clipping PPO, NADPEx with small initial dropout rate performs best, earning much higher score than action noise.
As shown in
Figure 5 right, the KL divergence between dropout polices and mean policies is bounded.
R ELATED WORKS
On-policy reinforcement learning methods have gained attention in recent years (Schulman et al., 2015b; Mnih et al., 2016; Schulman et al., 2017b), mainly due to their elegance with theoretical grounding and stability in policy iteration (Henderson et al., 2017b).
Despite of the effectiveness, to improve their data efﬁciency remains an active research topic.
In this work, we consider the exploration strategies for on-policy reinforcement learning methods.
Most of the aforementioned works employ naive exploration, with stochasticity only in action space.
However, they fail to tackle some tasks with either sparse rewards (Florensa et al., 2017) or longterm information (Osband et al., 2016), where temporally consistent exploration is needed.
One solution to this challenge is to shape the reward to encourage more directed exploration.
The speciﬁc direction has various foundations, including but not restricted to state visitation count (Jaksch et al., 2010; Tang et al., 2017), state density (Bellemare et al., 2016; Ostrovski et al., 2017;
Fu et al., 2017), self-supervised prediction error (Pathak et al., 2017) etc.
Some of them share the
Probably Approximately Correct (PAC) with discrete and tractable state space (Jaksch et al., 2010).
But when state space and action space are intractable, all of them need additional computational structures, which take non-trivial efforts to implement and non-negligible resources to execute.
Orthogonal to them, methods involving a hierarchy of stochasticity are proposed.
Based on hierarchical reinforcement learning, Florensa et al.
(2017) models the stochasticity at large time scales with a random variable - option - and model low-level policies with Stochastic Neural Networks.
However, authors employ human designed proxy reward and staged training.
Almost concurrently,
Osband et al.
(2016) and Plappert et al.
; Fortunato et al.
(2017) propose network section ensemble and parameter noise injection respetively to disentangle stochasticity.
Under the banner of Stochastic Neural Networks, NADPEx generalize them all.
BootstrapDQN is a special case of
NADPEx without stochasticity adapation and parameter noise is a special case with high variance and complexlity, as well as some heursitic approximation.
Details are discussed in Section 3.
In NADPEx, this stochasticity at large time scales is captured with a distribution of plausible neural subnetworks from the same complete network.
We achieve this through dropout (Srivastava et al., 2014).
In spite of its success in supervised deep learning literature and Bayesian deep learning literature, it is the ﬁrst time to combine dropout to reinforcement learning policies for exploration.
The closest ones are Gal & Ghahramani (2016);
Henderson et al.
(2017a), which use dropout in value network to capture agents’ uncertainty about the environment.
According to Osband et al.
Gal & Ghahramani (2016) even fails in environment requiring temporally consistent exploration.
There are also some attempts from the Evolutionary Strategies and Genetic Algorithms literature (Salimans et al., 2017;
Such et al., 2017; Gangwani & Peng, 2017) to continuous control tasks.
Though they model the problem much differently from ours, the relation could be an interesting topic for future research.
C ONCLUSION
Building a hierarchy of stochasticity for reinforcement learning policy is the ﬁrst step towards more structured exploration.
We presented a method, NADPEx, that models stochasticity at large time scale with a distribution of plausible subnetworks from the same complete network to achieve onpolicy temporally consistent exploration.
These subnetworks are sampled through dropout at the beginning of episodes, used to explore the environment with diverse and consistent behavioral patterns and updated through simultaneous gradient back-propagation.
A learning objective is provided such that this distribution is also updated in an end-to-end manner to adapt to the action-space policy.
Thanks to the fact that this dropout transformation is differentiable, KL regularizers on policy space can help to further stabilize it.
Our experiments exhibit that NADPEx successfully solves continuous control tasks, even with strong sparsity in rewards.
A KL DIVERGENCE FIRST ORDER APPROXIMATION
We derive the ﬁrst order derivatitve of KL divergence from a refernce policy π to a parametrized policy π θ to :
[Equation]
Note that line 3 derives line 4 with likelihood ratio trick, line 4 derives line 5 as (cid:82)
[Equation]
B G RADIENTS OF NADPE X
Here we provide a full derivation of NADPEx’s gradients.
Speciﬁcally, gradients for two types of dropout are discussed.
As shown in (7), a NADPEx policy π θ , z is a joint distribution, which could be factorized with a dropout distribution q φ ( z ) and a conditional distribution, i.e. the dropout policy , π θ | z , z ∼ q φ ( z ) is the dropout random variable vector.
In reinforcement learning with NADPEx, we use gradient based optimization to maximize the objective (8).
[Equation] [Equation]
B. D ISCRETE DROPOUT
Normally the dropout distribution q φ ( z ) is a discrete distribution, for example, z ∼ Bernoulli ( φ ) , ones can use likelihood ratio trick to calculate (14):
[Equation]
where L ( · ) is the surrogate loss, varying from reinforcement learning algorihtms,
A ( s i 0 , a i 0 ) is the
GAE (Schulman et al., 2015c) for the i th trajectory from the beginning i
[Equation]
e. ( s i 0 , a i 0 ) , such that the gradient has low variance.
B. G AUSSIAN MULTIPLICATIVE DROPOUT
In Srivastava et al.
(2014), Gaussian multiplicative dropout is proposed, where z ∼ N ( I , σ 2 ) is the multiplicative dropout random variable vector, with the same expectation of discrete dropout
P ( z = 1) = 11+ φ .
Reparametirzed with a dummy random variable (cid:15) ∼ N ( 0 , I ) (Kingma &
Welling, 2013; Rezende et al., 2014), we have z
= I + φ (cid:12) (cid:15) , where φ is used to denote σ for consistency of notation, (cid:12) is an element-wise multiplication, such that (14) could be calculated as:
[Equation]
C NADPE X KL DIVERGENCE FIRST ORDER APPROXIMATION
In NADPEx, the dropout distribution is designed to be fully factorizable, thus the gradient of KL divergence in the NADPEx policy space is obtainable:
[Equation]
And thus the same ﬁrst-order approximation could be done just as in Appendix A.
However, note that likelihood ratio trick is used during the derivation, which should be applied to the random variable vectors a and z here.
C. R EDUCING VARIANCE IN M ONTE C ARLO ESTIMATE WITH M U P ROP
As normally z could have a much higher dimension than a [Equation] making this likelihood ratio trick suffer from curse of dimensionality
[Equation]
we inspect into the second term with MuProp (Gu et al., 2016) for a low-variance estimate.
Let
[Equation]
[Equation]
then we have
[Equation]
z )
The idea of MuProp is to use a control variate that corresponds to the ﬁrst-order Taylor expansion of f around some ﬁxed value ¯ z , i.e. h ( z ) =
f (¯ z ) + f [Equation] ¯ z
[Equation]
¯ z ) , such that
[Equation]
z ∼ q
[Equation]
To further reduce the variance [Equation] the ﬁrst term is sometimes omitted with acceptable biased introduced (Raiko et al., 2015) [Equation]
And empirically [Equation] we ﬁnd it could be almost negligible comparing with the gradient from the reinforcement learning objective.
For Gaussian dropout , ∇ φ E z
[ z ] = ∇ φ 1 = 0 , g φ is thus eliminated.
With binary dropout , ∇ φ E z
[ z ] = ∇ φ φ = 1 , we have
[Equation]
Note that in line 3 we assume the probabilistic distribution function to be continuous in an inﬁnitesimal regions around { z | z i ∈ { 0 , 1 } for ∀ z i } to allow the existence of the ﬁrst order derivative for the ﬁrst term.
The ﬁrst term is then close to zero as φ is close to φ old .
Another possible relaxation method is to use Concrete distribution (Maddison et al., 2016):
[Equation]
where λ is the temperature of this relaxation.
Substituting it back to (21)
[Equation]
we have ˆ g M φ = f (cid:48) ( z ) | z =¯ z
[Equation]
We can reach the same claim as above that the ﬁrst term could be removed if we set z
= ¯ z → 0 .
And the second term could also be eliminated as ﬁrst order derivative of KL divergence between two different distributions parametrized with almost but not entirely null networks if ¯ z → [Equation]
C. R ELAXING ANALYTICAL KL DIVERGENCE WITH TRUST REGION
Ones may argue that the second term in (17) is decomposable to
[Equation]
[Equation]
[Equation] such that (24) has analytical form as q is either diagonal Gaussian or Bernoulli and thus the Monte Carlo gradient estimate and the MuProp approximation above need only to be exerted on (25). However, it can be proved that ∇ φ D KL ( q φ i ( z i )
||
q φ oldi ( z i )) is still omittable with trust region (Schulman et al., 2015b).
When ﬁrst proposed in Schulman et al.
(2015b), trust region is a relaxation on D KL to encourage efﬁcient learning, with the idea that step size will only be adapted if the trust region constraint is violated, i.e. D KL > δ .
The adaptation of step size is replaced in Wang et al.
with a mechanism to clip gradients from surrogate loss adaptively only if D KL > δ to further boost the efﬁciency.
We prove below that D kl will almost never violate the constraint δ = 1 proposed in Wang et al.
For Gaussian dropout we have:
[Equation]
where we let x = ∆ φ i φ oldi .
According to our experiment, as well as some conclusions in the literature for dropout (Kingma et al., 2015), φ i ∈ (0 . 005 , 0 . 5) .
Obviously x ∈ (0 , 1) ,
D KL
[Equation]
q φ i
[Equation]
z i
[Equation]
|| q φ oldi ( z i )) ∈ (0 , 1) .
For binary dropout we have:
[Equation]
We can thus discuss how D KL changes with ∆ φ i and φ oldi :
[Equation]
With h ( x ) = log x 1 − x , x ∈ (0 . 005 , 0 . 5) , it is easy to see that ∂D KL ∂ ∆ φ
i = 0 when ∆ φ i = 0 , ∂D KL ∂ ∆ φ i >when ∆ φ i > 0 and ∂D KL ∂ ∆ φ i < 0 when ∆ φ i < 0 .
Similarly, when φ old = 0 .
5 , ∂D KL ∂
φ old = 0 ; when φ old > 0 . 5 , ∂D KL ∂ φ old > 0 ; when φ old < 0 .
5 , ∂D KL ∂
φ old < 0 .
Hence D KL ( q φ i ( z i ) ||
q φ oldi ( z i )) reaches its maximum at | ∆ φ i | max and | φ oldi − 0 .
5 | max .
It is reasonable to set | ∆ φ i | max = 0 .
1 , leading us to D KL ( q φ i ( z i ) ||
q φ oldi ( z i )) max ≈ 0 .
225 <
δ = 1 .
We reach the same claim that ∇ φ D KL ( π θ , φ ( ·| s ) ||
π θ old , φ old ( ·| s )) could be stopped for the sake of learning efﬁciency at a cost of acceptable bias.
C. R EMEDY BIAS WITH MEAN POLICY
As derived above, the regularization term only back-propagates gradients to θ :
[Equation]
where N is the number of dropout policies in this batch.
Note that from line 3 to line 4 we simply remove log q φ ( z ) q
[Equation]
old ( z ) , which can be regarded as substracting it as a baseline b ( z ) to reduce variance.
Obviously, (30) has similar level of variance as (9), closing our discussion on variance reduction even though some other techniques could possibly go further.
Seeing the lack of regularization on φ due to the gradient omission, we further enforce the idea that dropout policy had better to be close to each other, which is the supposed role of regularizer on φ .
This could be intuitively understood as a remedy from θ to φ :
[Equation]
Noticing that this term has a complexity of O ( N 2 ) , we replace π θ , φ with a mean policy π θ =
π θ | z .
In the deep learning literature, it is fairly prevalent to use mean network for dropout training with some adjustment in gradient back-propagation, due to stability and efﬁency concern.
For instance, it is proved by Wang & Manning (2013) that the sampling process for dropout network with sigmoid or ReLU activators could be avoided by integrating a Gaussian approximation at each layer of a neural network.
Therefore, we have
[Equation]
Empirically, we found little difference between fast dropout back-propagation and normal backpropagation when (32) is combined into (12).
D V ARITAIONAL DROPOUT AND LOCAL REPARAMETRIZATION
As introduced in Section 2.1, multiplicative dropout at neuron activation of k th layer can also be viewed as a multiplicative noise for the input at k + 1 th layer:
[Equation]
After a simple rearrangement ˜ W ( k +1) T = W
( k +1) T D kz , we have:
[Equation]
That is, the stochastic neuron activation in networks with Gaussian multiplicatice dropout can also be regarded as stochastic neuron activation in Noisy Networks with correlated Gaussian parameter noise, whose means equal the corresponding ones in networks with dropout.
E H YPERPARAMETERS
For most of the hyperparamters, we follow the setting in original PPO paper.
Though, to emphasis the scalability of our method, we use 2 parallel enviroments and only 1 minibatch in each epoch.
[Table]
F E XPERIMENT RESULTS IN STANDARD ENVIRONMENTS
G E NVIRONMENTS WITH SPARSE REWARDS
We use the same sparse reward environments from rllab Duan et al.
(2016), modiﬁed by Houthooft et al.
• SparseHalfCheetah ( S ⊂ R 17 , A ⊂ R 6 ) , which only yields a reward if the agent crosses a distance threshold, • SparseMountainCar ( S ⊂ R 2 , A ⊂ R ) , which only yields a reward if the agent drives up the hill, • SparseDoublePendulum ( S ⊂ R 6 , A ⊂ R ) , which only yields a reward if the agent reaches the upright position.