N OISY N ETWORKS FOR E XPLORATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce NoisyNet, a deep reinforcement learning agent with parametric noise added to its weights, and show that the induced stochasticity of the agent’s policy can be used to aid efﬁcient exploration.
The parameters of the noise are learned with gradient descent along with the remaining network weights.
NoisyNet is straightforward to implement and adds little computational overhead.
We ﬁnd that replacing the conventional exploration heuristics for A3C, DQN and Dueling agents (entropy reward and (cid:15) -greedy respectively) with NoisyNet yields substantially higher scores for a wide range of Atari games, in some cases advancing the agent from sub to super-human performance.
I NTRODUCTION
Despite the wealth of research into efﬁcient methods for exploration in Reinforcement Learning (RL) (Kearns & Singh, 2002; Jaksch et al., 2010), most exploration heuristics rely on random perturbations of the agent’s policy, such as (cid:15) -greedy
(Sutton & Barto, 1998) or entropy regularisation (Williams, 1992), to induce novel behaviours.
However such local ‘dithering’ perturbations are unlikely to lead to the large-scale behavioural patterns needed for efﬁcient exploration in many environments (Osband et al., 2017).
Optimism in the face of uncertainty is a common exploration heuristic in reinforcement learning.
Various forms of this heuristic often come with theoretical guarantees on agent performance (Azar et al., 2017; Lattimore et al., 2013; Jaksch et al., 2010; Auer & Ortner, 2007; Kearns & Singh, 2002).
However, these methods are often limited to small state-action spaces or to linear function approximations and are not easily applied with more complicated function approximators such as neural networks (except from work by (Geist & Pietquin, 2010a;b) but it doesn’t come with convergence guarantees).
A more structured approach to exploration is to augment the environment’s reward signal with an additional intrinsic motivation term (Singh et al., 2004) that explicitly rewards novel discoveries.
Many such terms have been proposed, including learning progress (Oudeyer &
Kaplan, 2007), compression progress (Schmidhuber, 2010), variational information maximisation (Houthooft et al., 2016) and prediction gain (Bellemare et al., 2016).
One problem is that these methods separate the mechanism of generalisation from that of exploration; the metric for intrinsic reward, and–importantly–its weighting relative to the environment reward, must be chosen by the experimenter, rather than learned from interaction with the environment.
Without due care, the optimal policy can be altered or even completely obscured by the intrinsic rewards; furthermore, dithering perturbations are usually needed as well as intrinsic reward to ensure robust exploration (Ostrovski et al., 2017).
Exploration in the policy space itself, for example, with evolutionary or black box algorithms (Moriarty et al., 1999; Fix & Geist, 2012; Salimans et al., 2017), usually requires many prolonged interactions with the environment.
Although these algorithms are quite generic and can apply to any type of parametric policies (including neural networks), they are usually not data efﬁcient and require a simulator to allow many policy evaluations.
We propose a simple alternative approach, called NoisyNet, where learned perturbations of the network weights are used to drive exploration.
The key insight is that a single change to the weight vector can induce a consistent, and potentially very complex, state-dependent change in policy over multiple timesteps – unlike dithering approaches where decorrelated (and, in the case of (cid:15) -greedy, state-independent) noise is added to the policy at every step.
The perturbations are sampled from a noise distribution.
The variance of the perturbation is a parameter that can be considered as the energy of the injected noise.
These variance parameters are learned using gradients from the reinforcement learning loss function, along side the other parameters of the agent.
The approach differs from parameter compression schemes such as variational inference (Hinton & Van Camp, 1993; Bishop, 1995) and ﬂat minima search (Hochreiter & Schmidhuber, 1997) since we do not maintain an explicit distribution over weights during training but simply inject noise in the parameters and tune its intensity automatically.
It also differs from variational inference (Graves, 2011; Blundell et al., 2015) or
Thompson sampling (Thompson, 1933; Lipton et al., 2016) as the distribution on the parameters of our agents does not necessarily converge to an approximation of a posterior distribution.
At a high level our algorithm induces a randomised network for exploration, with care exploration via randomised value functions can be provably-efﬁcient with suitable linear basis (Osband et al., 2014).
Previous attempts to extend this approach to deep neural networks required many duplicates of sections of the network (Osband et al., 2016).
By contrast our NoisyNet approach requires only one extra parameter per weight and applies to policy gradient methods such as A3C out of the box (Mnih et al., 2016).
Most recently (and independently of our work), Plappert et al.
(Plappert et al., 2017) presented a similar technique where constant Gaussian noise is added to the parameters of the network.
Our method thus differs by the ability of the network to adapt the noise injection with time and it is not restricted to Gaussian noise distributions.
We need to emphasise that the idea of injecting noise to improve the optimisation process have been thoroughly studied in the literature of supervised learning and optimisation under different names (e.g., Neural deffusion process (Mobahi, 2016) and graduated optimisation (Hazan et al., 2016)).
Though these methods often rely on a non-trainable noise of vanishing size as opposed to NoisyNet which tunes the parameter of noise by gradient descent.
NoisyNet can also be adapted to any deep RL algorithm and we demonstrate this versatility by providing NoisyNet versions of DQN (Mnih et al., 2015),
Dueling (Wang et al., 2016) and A3C (Mnih et al., 2016) algorithms.
Experiments on 57 Atari games show that NoisyNet-DQN and NoisyNetDueling achieve striking gains when compared to the baseline algorithms without signiﬁcant extra computational cost, and with less hyper parameters to tune.
Also the noisy version of A3C provides some improvement over the baseline.
B ACKGROUND
This section provides mathematical background for Markov Decision Processes (MDPs) and deep
RL with Q-learning, dueling and actor-critic methods.
2. M ARKOV D ECISION P ROCESSES AND R EINFORCEMENT L EARNING
MDPs model stochastic, discrete-time and ﬁnite action space control problems (Bellman & Kalaba, 1965; Bertsekas, 1995; Puterman, 1994).
An MDP is a tuple M =
( X , A , R, P, γ ) where X is the state space, A the action space, R the reward function, γ ∈ ]0 , 1[ the discount factor and P a stochastic kernel modelling the one-step Markovian dynamics ( P ( y | x, a ) is the probability of transitioning to state y by choosing action a in state x ).
A stochastic policy π maps each state to a distribution over actions π ( ·| x ) and gives the probability π ( a | x ) of choosing action a in state x .
The quality of a policy π is assessed by the action-value function Q π deﬁned as:
[Equation]
where E π is the expectation over the distribution of the admissible trajectories ( x 0 , a 0 , x 1 , a 1 , . . . )
obtained by executing the policy π starting from x 0 = x and a 0 = a .
Therefore, the quantity Q π ( x, a ) represents the expected γ -discounted cumulative reward collected by executing the policy π starting from x and a .
A policy is optimal if no other policy yields a higher return.
The action-value function of the optimal policy is Q (cid:63) ( x, a ) = arg max π Q π ( x, a ) .
The value function V π for a policy is deﬁned as V π ( x ) =
E a ∼ π ( ·| x ) [ Q π ( x, a )] , and represents the expected γ
2. D EEP R EINFORCEMENT L EARNING
Deep Reinforcement Learning uses deep neural networks as function approximators for RL methods.
Deep Q-Networks (DQN) (Mnih et al., 2015), Dueling networks (Wang et al., 2016), Asynchronous
Advantage Actor-Critic (A3C) (Mnih et al., 2016), Trust Region Policy Optimisation (Schulman et al., 2015), Deep Deterministic Policy Gradient (Lillicrap et al., 2015) and distributional RL (c51)
(Bellemare et al., 2017) are examples of such algorithms.
They frame the RL problem as the minimisation of a loss function L ( θ ) , where θ represents the parameters of the network.
In our experiments we shall consider the DQN, Dueling and A3C algorithms.
DQN (Mnih et al., 2015) uses a neural network as an approximator for the action-value function of the optimal policy Q (cid:63) ( x, a ) .
DQN’s estimate of the optimal action-value function, Q ( x, a ) , is found by minimising the following loss with respect to the neural network parameters θ :
[Equation]
where D is a distribution over transitions e
= ( x, a, r = R ( x, a ) , y ∼ P ( ·| x, a )) drawn from a replay buffer of previously observed transitions.
Here θ − represents the parameters of a ﬁxed and separate target network which is updated ( θ − ← θ ) regularly to stabilise the learning.
An (cid:15) -greedy policy is used to pick actions greedily according to the action-value function Q or, with probability (cid:15) , a random action is taken.
The Dueling DQN (Wang et al., 2016) is an extension of the DQN architecture.
The main difference is in using Dueling network architecture as opposed to the Q network in DQN.
Dueling network estimates the action-value function using two parallel sub-networks, the value and advantage subnetwork, sharing a convolutional layer.
Let θ conv , θ V , and θ A be, respectively, the parameters of the convolutional encoder f , of the value network V , and of the advantage network A ; and θ = { θ conv , θ V , θ A } is their concatenation.
The output of these two networks are combined as follows for every ( x, a ) ∈
X ×
A :
[Equation]
The Dueling algorithm then makes use of the the double-DQN update rule (van Hasselt et al., 2016) to optimise θ :
[Equation]
where the deﬁnition distribution D and the target network parameter set θ − is identical to DQN.
In contrast to DQN and Dueling, A3C (Mnih et al., 2016) is a policy gradient algorithm.
A3C’s network directly learns a policy π and a value function V of its policy.
The gradient of the loss on the
A3C policy at step t for the roll-out ( x t + i , a t
+ i ∼ π ( ·| x t + i ; θ ) , r t + i ) ki
=0 is:
[Equation] [Equation]
H
[Equation]
π ( ·| x t ; θ )] denotes the entropy of the policy π and β is a hyperparameter that trades off between optimising the advantage function and the entropy of the policy.
The advantage function
A ( x t + i , a t + i ; θ ) is the difference between observed returns and estimates of the return produced by A3C’s value network:
A ( x t + i , a t + i ; θ ) = (cid:80) k − 1 j = i γ j − i r t + j + γ k − i V ( x t + k ; θ ) − V ( x t
+ i ; θ ) , r t + j being the reward at step t + j and V ( x ; θ ) being the agent’s estimate of value function of state x .
Parameters of the value function are found to match on-policy returns:
L V ( θ ) =
E π (cid:104)(cid:80) ki
=
( ˆ Q i − V
( x t + i ; θ )) 2 (cid:105) where ˆ Q
i is the return obtained by executing policy π starting in state x t
+ i : ˆ Q i =
(cid:80) k − 1 j = i γ j − i r t + j + γ k − i V
( x t + k ; θ ) .
The overall A3C loss is then L ( θ ) =
L π ( θ ) + λL V ( θ ) where λ balances optimising the policy loss relative to the baseline value function loss.
N OISY N ETS FOR R EINFORCEMENT L EARNING
NoisyNets are neural networks whose weights and biases are perturbed by a parametric function of the noise.
These parameters are adapted with gradient descent.
More precisely, let y =
f θ ( x ) be a neural network parameterised by the vector of noisy parameters θ which takes the input x and outputs y .
We represent the noisy parameters θ as θ def = µ + Σ (cid:12) ε , where ζ def = ( µ, Σ) is a set of vectors of learnable parameters, ε is a vector of zero-mean noise with ﬁxed statistics and (cid:12) represents element-wise multiplication.
The usual loss of the neural network is wrapped by expectation over the noise ε : ¯ L ( ζ ) def =
E [ L ( θ )] .
Optimisation now occurs with respect to the set of parameters ζ .
Consider a linear layer of a neural network with p inputs and q outputs, represented by
[Equation]
where x ∈ R p is the layer input, w ∈ R q × p the weight matrix, and b ∈ R q the bias.
The corresponding noisy linear layer is deﬁned as:
[Equation]
where µ w + σ w (cid:12) ε w and µ b + σ b (cid:12) ε b replace w and b in Eq.
(7), respectively.
The parameters µ w ∈ R q × p , µ b ∈ R q , σ w ∈ R q × p and σ b ∈ R q are learnable whereas ε w ∈ R q × p and ε b ∈ R q are noise random variables (the speciﬁc choices of this distribution are described below).
We now turn to explicit instances of the noise distributions for linear layers in a noisy network.
We explore two options:
(a) Independent Gaussian noise: the noise applied to each weight and bias is independent, where each entry ε wi,j (respectively each entry ε bj ) of the random matrix ε w (respectively of the random vector ε b ) is drawn from a unit Gaussian distribution.
This means that for each noisy linear layer, there are pq + q noise variables (for p inputs to the layer and q outputs).
(b) Factorised Gaussian noise reduces the number of random variables in the network from one per weight, to one per input and one per output of each noisy linear layer.
By factorising ε wi,j , we can use p unit Gaussian variables ε i for noise of the inputs and and q unit Gaussian variables ε j for noise of the outputs (thus p + q unit Gaussian variables in total).
Each ε wi,j and ε bj can then be written as:
[Equation]
where f is a real-valued function.
In our experiments we used f ( x ) = sgn( x ) (cid:112)
| x | .
Since the loss of a noisy network, ¯ L ( ζ ) =
E [ L ( θ )] , is an expectation over the noise, the gradients are straightforward to obtain:
[Equation]
We use a Monte Carlo approximation to the above gradients, taking a single sample ξ at each step of optimisation:
[Equation]
3. D EEP R EINFORCEMENT L EARNING WITH N OISY N ETS
We now turn to our application of noisy networks to exploration in deep reinforcement learning.
Noise drives exploration in many methods for reinforcement learning, providing a source of stochasticity external to the agent and the RL task at hand.
Either the scale of this noise is manually tuned across a wide range of tasks (as is the practice in general purpose agents such as DQN or A3C) or it can be manually scaled per task.
Here we propose automatically tuning the level of noise added to an agent for exploration, using the noisy networks training to drive down (or up) the level of noise injected into the parameters of a neural network, as needed.
A noisy network agent samples a new set of parameters after every step of optimisation.
Between optimisation steps, the agent acts according to a ﬁxed set of parameters (weights and biases).
This ensures that the agent always acts according to parameters that are drawn from the current noise distribution.
Deep Q-Networks (DQN) and Dueling.
We apply the following modiﬁcations to both DQN and
Dueling: ﬁrst, ε -greedy is no longer used, but instead the policy greedily optimises the (randomised) action-value function.
Secondly, the fully connected layers of the value network are parameterised as a noisy network, where the parameters are drawn from the noisy network parameter distribution after every replay step.
We used factorised Gaussian noise as explained in (b) from Sec. 3.
For replay, the current noisy network parameter sample is held ﬁxed across the batch.
Since DQN and Dueling take one step of optimisation for every action step, the noisy network parameters are re-sampled before every action.
We call the new adaptations of DQN and Dueling, NoisyNet-DQN and NoisyNet-Dueling, respectively.
We now provide the details of the loss function that our variant of DQN is minimising.
When replacing the linear layers by noisy layers in the network (respectively in the target network), the parameterised action-value function Q ( x, a, ε ; ζ ) (respectively Q ( x, a, ε (cid:48) ; ζ − ) ) can be seen as a random variable and the DQN loss becomes the NoisyNet-DQN loss:
[Equation]
where the outer expectation is with respect to distribution of the noise variables ε for the noisy value function Q ( x, a, ε ; ζ ) and the noise variable ε (cid:48) for the noisy target value function Q ( y, b, ε (cid:48) ; ζ − ) .
Computing an unbiased estimate of the loss is straightforward as we only need to compute, for each transition in the replay buffer, one instance of the target network and one instance of the online network.
Concerning the action choice, we sample from the online network and we act greedily with respect to the output action-value function.
The algorithm is provided in Sec.
C.1.
Similarly the loss function for NoisyNet-Dueling is deﬁned as:
[Equation]
The algorithm is provided in Sec.
C.2.
Asynchronous Advantage Actor Critic (A3C).
A3C is modiﬁed in a similar fashion to DQN: ﬁrstly, the entropy bonus of the policy loss is removed.
Secondly, the fully connected layers of the policy network are parameterised as a noisy network.
We used independent Gaussian noise as explained in (a) from Sec. 3.
In A3C, there is no explicit exploratory action selection scheme (such as (cid:15) -greedy); and the chosen action is always drawn from the current policy.
For this reason, an entropy bonus of the policy loss is often added to discourage updates leading to deterministic policies.
However, when adding noisy weights to the network, sampling these parameters corresponds to choosing a different current policy which naturally favours exploration.
As a consequence of direct exploration in the policy space, the artiﬁcial entropy loss on the policy can thus be omitted.
New parameters of the policy network are sampled after each step of optimisation, and since A3C uses n step returns, optimisation occurs every n steps.
We call this modiﬁcation of A3C, NoisyNet-A3C.
Indeed, when replacing the linear layers by noisy linear layers (the parameters of the noisy network are now noted ζ ), we obtain the following estimation of the return via a roll-out of size k :
[Equation]
As A3C is an on-policy algorithm the gradients are unbiased when noise of the network is consistent for the whole roll-out.
Consistency among action value functions ˆ Q
i is ensured by letting letting the noise be the same throughout each rollout, i.e., ∀ i, ε i = ε .
Additional details are provided in the
Appendix Sec.
A.2 and the algorithm is given in Sec.
C.3.
R ESULTS
We evaluated the performance of noisy network agents on 57 Atari games (Bellemare et al., 2015) and compared to baselines that, without noisy networks, rely upon the original exploration methods
[Figure]
( ε -greedy and entropy bonus).
We used the random start no-ops scheme for training and evaluation as described the original DQN paper (Mnih et al., 2015).
The mode of evaluation is identical to those of Mnih et al.
(2016) where randomised restarts of the games are used for evaluation after training has happened.
The raw average scores of the agents are evaluated during training, every 1M frames in the environment, by suspending learning and evaluating the latest agent for 500K frames.
Episodes are truncated at 108K frames (or 30 minutes of simulated play) (van Hasselt et al., 2016).
We consider three baseline agents: DQN (Mnih et al., 2015), duel clip variant of Dueling algorithm (Wang et al., 2016) and A3C (Mnih et al., 2016).
In each case, we used the neural network architecture from the corresponding original papers for both the baseline and NoisyNet variant.
For the NoisyNet variants we used the same hyper parameters as in the respective original paper for the baseline.
We compared absolute performance of agents using the human normalised score:
[Equation]
(17) where human and random scores are the same as those in Wang et al.
Note that the human normalised score is zero for a random agent and 100 for human level performance.
Per-game maximum scores are computed by taking the maximum raw scores of the agent and then averaging over three seeds.
However, for computing the human normalised scores in Figure 2, the raw scores are evaluated every 1M frames and averaged over three seeds.
The overall agent performance is measured by both mean and median of the human normalised score across all 57 Atari games.
The aggregated results across all 57 Atari games are reported in Table 1, while the individual scores for each game are in Table 2 from the Appendix.
The median human normalised score is improved in all agents by using NoisyNet, adding at least 18 (in the case of A3C) and at most 48 (in the case of
DQN) percentage points to the median human normalised score.
The mean human normalised score is also signiﬁcantly improved for all agents.
Interestingly the Dueling case, which relies on multiple modiﬁcations of DQN, demonstrates that NoisyNet is orthogonal to several other improvements made to DQN.
We also compared relative performance of NoisyNet agents to the respective baseline agent
[Table]
without noisy networks:
[Table]
As before, the per-game score is computed by taking the maximum performance for each game and then averaging over three seeds.
The relative human normalised scores are shown in Figure 1.
As can be seen, the performance of NoisyNet agents (DQN, Dueling and A3C) is better for the majority of games relative to the corresponding baseline, and in some cases by a considerable margin.
In some games, NoisyNet agents provide an order of magnitude improvement on the performance of the vanilla agent; as can be seen in Table 2 in the Appendix with detailed breakdown of individual game scores.
More information is provided in the learning curves plots from 4, 5 and Figs 6, for DQN,
Dueling and A3C respectively.
4. A NALYSIS OF L EARNING IN N OISY L AYERS
In this subsection, we try to provide some insight on how noisy networks affect the learning process and the exploratory behaviour of the agent.
In particular, we focus on analysing the evolution of the noise weights σ w and σ b throughout the learning process.
We ﬁrst note that, as L ( ζ ) is a positive and continuous function of ζ , there always exists a deterministic optimiser for the loss L ( ζ ) (deﬁned in
Eq.
Therefore, one may expect that, to obtain the deterministic optimal solution, the neural network may learn to discard the noise entries by eventually pushing σ w s and σ b towards 0 .
To test this hypothesis we track the changes in σ w s throughout the learning process.
Let σ wi denote the i th weight of a noisy layer.
We then deﬁne ¯Σ , the mean-absolute of the σ wi s of a noisy layer, as
[Equation]
Intuitively speaking ¯Σ provides some measure of the stochasticity of the Noisy layers.
We report the learning curves of ¯Σ in Fig.
3 for a selection of Atari games in NoisyNet-DQN agent.
We observe that ¯Σ of the last layer of the network decreases as the learning proceeds in all cases, whereas in the case of the penultimate layer this only happens for 2 games out of 5 (Pong and Beam rider) and in the remaining 3 games ¯Σ in fact increases.
This shows that in the case of NoisyNet-DQN the agent does not necessarily evolve towards a deterministic solution as one might have expected.
Another interesting observation is that the way ¯Σ evolves signiﬁcantly differs from one game to another.
This suggests that NoisyNet produce a problem-speciﬁc exploration strategy as opposed to ﬁxed exploration strategy used in standard DQN.
C ONCLUSION
We have presented a general method for exploration in deep reinforcement learning that shows signiﬁcant performance improvements across many Atari games in three different agent architectures.
In particular, we observe that in games such as Asterix, Astroid and Freeway that the standard
DQN, Dueling and A3C perform poorly compared with the human player, NoisyNet-DQN and
NoisyNet-A3C achieve super human performance.
Our method eliminates the need for (cid:15) -greedy and the entropy bonus commonly used in Q-learning-style and policy gradient methods, respectively.
Instead we show that better exploration is possible by relying on perturbations in weight space to drive exploration.
Another advantage of NoisyNet is that the amount of noise injected in the network is tuned automatically by the RL algorithm.
This alleviates the need for any hyper parameter tuning (required with standard entropy bonus and (cid:15) -greedy types of exploration).
This is also in contrast to many other methods that add intrinsic motivation signals that may destabilise learning or change the optimal policy.
Another interesting feature of the NoisyNet approach is that the degree of exploration is contextual and varies from state to state based upon per-weight variances.
While more gradients are needed, the gradients on the mean and variance parameters are related to one another by a computationally efﬁcient afﬁne function, thus the computational overhead is marginal.
Automatic differentiation makes implementation of our method a straightforward adaptation of many existing methods.
A similar randomisation technique can also be applied to LSTM units (Fortunato et al., 2017) and is easily extended to reinforcement learning, we leave this as future work.
Note NoisyNet exploration strategy is not restricted to the baselines considered in this paper.
In fact, this idea can be applied to any deep RL algorithms that can be trained with gradient descent, including DDPG (Lillicrap et al., 2015), TRPO (Schulman et al., 2015) or distributional RL (c51) (Bellemare et al., 2017).
As such we believe this work is a step towards the goal of developing a universal exploration strategy.
A A LGORITHM D ESCRIPTIONS A. N OISY N ET -DQN IMPLEMENTATION DETAILS
DQN (Mnih et al., 2015) is an approximate value iteration technique that uses a deep convolutional neural network to represent the Q-function as Q ( x, a ; θ ) , where θ are the parameters of the network.
We used the network architecture from (Mnih et al., 2015) and also the same set up for interacting with the Atari games.
We note, however, that we did not use (cid:15) -greedy technique, instead relied solely upon NoisyNet for exploration.
In DQN, the neural network is trained by minimising the following loss:
[Equation]
where D is a distribution over transitions e
= ( x, a, r = R ( x, a ) , y ∼ P ( ·| x, a )) .
Here θ − represents the parameters of the target network which is updated ( θ − ← θ ) regularly and stabilises the learning.
The distribution D is generated by constructing a replay buffer were training examples are uniformly sampled from previous experience tuples e t
= ( x t , a t , r t , x t +1 ) .
We now provide the details of the loss function that our variant of DQN with noisy network is minimising.
When replacing the linear layers by noisy layers (the parameters are now noted ζ ) in the network (respectively in the target network), the parameterised action-value function Q ( x, a, ε ; ζ ) (respectively Q ( x, a, ε (cid:48) ; ζ − ) ) can be seen as a random variable and the DQN loss becomes the NoisyNet-DQN loss:
[Equation]
The goal of NoisyNet-DQN is to optimise the loss ¯ L ( ζ ) in terms of ζ .
A. N OISY N ET -A3C IMPLEMENTATION DETAILS
In contrast with value-based algorithms, policy-based methods such as A3C (Mnih et al., 2016) parameterise the policy π ( a | x ; θ π ) directly and update the parameters θ π by performing a gradient ascent on the mean value-function E x ∼ D
[ V π ( ·|· ; θ π ) ( x )] (also called the expected return) (Sutton et al., 1999).
A3C uses a deep neural network with weights θ = θ π ∪ θ V to parameterise the policy π and the value V .
The network has one softmax output for the policy-head π ( ·|· ; θ π ) and one linear output for the value-head V ( · ; θ V ) , with all non-output layers shared.
The parameters θ π (resp.
θ V ) are relative to the shared layers and the policy head (resp.
the value head).
A3C is an asynchronous and online algorithm that uses roll-outs of size k + 1 of the current policy to perform a policy improvement step.
In A3C, there is no explicit exploratory action selection scheme (such as (cid:15) -greedy); and the chosen action is always drawn from the current policy.
For this reason, an entropy loss on the policy is often added to discourage updates leading to deterministic policies.
However, when using noisy weights in the network, sampling corresponds to choosing a different current policy which naturally favours exploration.
As a consequence of direct exploration in the policy space, the artiﬁcial entropy loss on the policy can thus be omitted.
For simplicity, here we present the A3C version with only one thread.
For a multi-thread implementation, refer to the pseudo-code C.3 or to the original A3C paper (Mnih et al., 2016).
In order to train the policy-head, an approximation of the policy-gradient is computed for each state of the roll-out ( x t
+ i , a t
+ i ∼ π ( ·| x t + i ; θ π ) , r t
+ i
[Equation]
where ˆ Q
i is an estimation of the return ˆ
Q i =
(cid:80) k − 1 j = i γ j − i r t + j + γ k − i V
( x t + k ; θ V ) .
The gradients are then added to obtain the cumulative gradient of the roll-out:
[Equation]
A3C trains the value-head by minimising the error between the estimated return and the value (cid:80) ki =
( ˆ Q i − V
( x t + i ; θ V )) 2 .
Therefore, the network parameters ( θ π , θ V ) are updated after each roll-out as follows:
[Equation]
where ( α π , α V ) are hyper-parameters.
As mentioned previously, in the original A3C algorithm, it is recommended to add an entropy term β (cid:80) ki =0 ∇ θ
π H
( π ( ·| x t
+ i ; θ π )) to the policy update, where
H ( π ( ·| x t + i ; θ π ))
= − β (cid:80)
a ∈
A π ( a | x t + i ; θ π ) log( π ( a | x t + i ; θ π )) .
Indeed, this term encourages exploration as it favours policies which are uniform over actions.
When replacing the linear layers in the value and policy heads by noisy layers (the parameters of the noisy network are now ζ π and ζ V ), we obtain the following estimation of the return via a roll-out of size k :
[Equation]
We would like ˆ Q i to be a consistent estimate of the return of the current policy.
To do so, we should force ∀ i, ε i = ε .
As A3C is an on-policy algorithm, this involves ﬁxing the noise of the network for the whole roll-out so that the policy produced by the network is also ﬁxed.
Hence, each update of the parameters ( ζ π , ζ V ) is done after each roll-out with the noise of the whole network held ﬁxed for the duration of the roll-out:
[Equation]
B I NITIALISATION OF N OISY N ETWORKS
In the case of an unfactorised noisy networks, the parameters µ and σ are initialised as follows.
Each element µ i,j is sampled from independent uniform distributions U [ − (cid:113) 3 p , + (cid:113) 3 p ] , where p is the number of inputs to the corresponding linear layer, and each element σ i,j is simply set to 0 .
017 for all parameters.
For factorised noisy networks, each element µ i,j was initialised by a sample from an independent uniform distributions
U [ − 1 √ p , + 1 √ p ] and each element σ i,j was initialised to a constant σ 0 √ p .
The hyperparameter σ 0 is set to 0 .
C
A LGORITHMS
C.
N OISY N ET -DQN
Input : Env Environment; N f maximum length of list x ; ε set of random variables of the network
Input : B empty replay buffer; ζ initial network parameters; ζ − initial target network parameters
Input : N B replay buffer size; N T training batch size; N − target network replacement frequency
Output : Q ( ., ε ; ζ ) action-value function 1 for episode e ∈ { 1 , . . .
, M } do /* x is a list of states */ x ← [ ]
Initialise state sequence x 0 ∼ Env x [0] ← x for t ∈ { 1 , . . .
} do / * l [ − 1] is the last element of the list l * /
Set x ← x [ − 1]
Sample a noisy network ξ ∼ ε
Select an action a ← argmax b ∈ A Q ( x, b, ξ ; ζ )
Sample next state y ∼ P ( ·| x, a ) , receive reward r ← R ( x, a ) and append x : x [ − 1] ← y if | x | > N f then
Delete oldest element from x end
Add transition ( x, a, r, y ) to the replay buffer B [ − 1] ← ( x, a, r, y ) if | B | > N B then
Delete oldest transition from B end /
* D is a distribution over the replay, it can be uniform or implementing prioritised replay */
Sample a minibatch of N T transitions (( x j , a j , r j , y j ) ∼ D ) N T
j
=
* Construction of the target values.
*/ for j ∈ { 1 , . .
. , N T } do
Sample a noisy network ξ j ∼ ε
Sample a noisy target network ξ (cid:48)
j ∼ ε if y j is a terminal state then (cid:98) Q
C.
N OISY N ET -D UELING
Input :
Environment; N f maximum length of x ; ε of random variables of the network
Input : B replay buffer; ζ initial network parameters; ζ − initial target network parameters
Input : N B replay buffer size; N T training batch size; N − target network replacement frequency
Output : Q ( ., ε ; ζ ) action-value function 1 for episode e ∈ { 1 , . . .
, M } do /* x is a list of states */ x ← [ ]
Initialise state sequence x 0 ∼ Env x [0] ← x for t ∈ { 1 , . . .
} do /* l [ − 1] is the last element of the list l */
Set x ← x [ − 1]
Sample a noisy network ξ ∼ ε
Select an action a ← argmax b ∈ A Q ( x, b, ξ ; ζ )
Sample next state y ∼ P ( ·| x, a ) , receive reward r ← R ( x, a ) and append x : x [ − 1] ← y if | x | > N f then
Delete oldest element from x end
Add transition ( x, a, r, y ) to the replay buffer B [ − 1] ← ( x, a, r, y ) if | B | > N B then
Delete oldest transition from B end /
* D is a distribution over the replay, it can be uniform or implementing prioritised replay */
Sample a minibatch of N T transitions (( x j , a j , r j , y j ) ∼ D ) N T
j
=
* Construction of the target values.
*/ for j ∈ { 1 , . .
. , N T } do
Sample a noisy network ξ j ∼ ε
Sample a noisy target network ξ (cid:48)
j ∼ ε if y j is a terminal state then (cid:98) Q ← r j else b ∗ ( y j ) =
arg max b ∈A Q ( y j , b, ξ (cid:48)
j ; ζ ) (cid:98) Q ← r j + γQ
y j , b ∗ ( y j ) , ξ (cid:48)
j ; ζ − )
Do a gradient step with loss ( (cid:98) Q − Q ( x j , a j , ξ j ; ζ )) end if t ≡ 0 (mod N − ) then
Update the target network: ζ − ← ζ end end 32 end
C.
N OISY N ET -A3C
Synchronise
Choice
∼ ε /* list
*/ r is list of actions */ a ←
] /* x is a list of states */ x ← [ ] and x [0] ← x t repeat
Policy choice: a t ∼ π t π
) a [
1] ← t
Receive reward r t and new state x t +  r [
1] ← t x [ t
t ← t
1 and T ← T +
=
until x t terminal or counter =
=
t max + if then
else
for do
Update Q
Accumulate π log( π ( a [ i ] | x [ i ]; ζ (cid:48) π , ξ ))
[ Q − V ( x [ i ]; ζ (cid:48) V , ξ )] .
Accumulate value-gradient: dζ V ← dζ V + ∇ ζ (cid:48)
V [ Q − V ( x [ i ]; ζ (cid:48) V , ξ )] 2 .
end
Perform asynchronous update of ζ π : ζ π ← ζ π + α π dζ π
Perform asynchronous update of ζ V : ζ V ← ζ V − α V dζ V 30 until T > T max
[Table]
jamesbond
[Table] [Table]
0.00 pong 14.
21.00 private eye 69571.
278.64 qbert 13455.
27120.72 riverraid 17118.
23133.61 road runner 7845.
234352.03 robotank 11.
63.75 seaquest 42054.
16753.99 skiing -4336.
-7549.53 solaris 12326.
6521.88 space invaders 1668.
5909.03 star gunner 10250.
75867.17 surround 6.
[Table]