E FFICIENT E XPLORATION THROUGH B AYESIAN D EEP Q-N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose Bayesian Deep Q-Network ( BDQN ), a practical Thompson sampling based Reinforcement Learning (RL) Algorithm.
Thompson sampling allows for targeted exploration in high dimensions through posterior sampling, but is usually computationally expensive.
We address this limitation by introducing uncertainty only at the output layer of the network through a Bayesian Linear Regression (BLR) model, which can be trained with fast closed-form updates and its samples can be drawn efﬁciently through the Gaussian distribution.
We apply our method to a wide range of Atari Arcade Learning Environments.
Since BDQN carries out more efﬁcient exploration, it is able to reach higher rewards substantially faster than a key baseline, DDQN .
I NTRODUCTION
Designing algorithms that achieve an optimal trade-off between exploration and exploitation is the primary goal of reinforcement learning (RL).
However, targeted exploration in high dimensional spaces is a challenging problem in RL.
Recent works on deep RL mostly employ simple exploration strategies such as ε -greedy, where the agent chooses the optimistic action with probability (1 − ε ) , otherwise, uniformly at random picks one of the available actions.
Due to this uniform sampling, the ε -greedy method scales poorly with the dimensionality of state and action spaces.
Recent work has considered scaling exploration strategies to large domains (Tang et al., 2016;
Bellemare et al., 2017; Osband et al., 2016).
Several of these papers have focused on employing optimism-under-uncertainty approaches, which essentially rely on computing conﬁdence bounds over different actions, and acting optimistically with respect to that uncertainty.
An alternative to optimism-under-uncertainty is Thompson Sampling ( TS ) (Thompson, 1933), where we draw samples from a posterior distribution over models.
It is a Bayesian framework where we start with a prior distribution and compute the posterior beliefs based on the collected data through interaction with the environment.
We then maximize the expected reward under the sampled belief.
Posterior sampling can provide more targeted exploration since it can trade off uncertainty with the expected return of actions.
In contrast, the ε -greedy strategy is indifferent to uncertainty of the actions and the expected rewards of sub-optimistic ones (set of actions excluding the optimistic action).
There is compelling evidence that Thompson Sampling can provide better results than optimismunder-uncertainty approaches, but so far this has only been demonstrated on smaller scale domains (Osband et al., 2013).
There has been relatively little work on scaling Thompson Sampling to large state spaces.
The primary difﬁculty in implementing Thompson sampling is the difﬁculty of sampling from general posterior distributions.
Prior efforts in this space have generally required extremely expensive computations (e.g.(Ghavamzadeh et al., 2015)).
In this paper , we derive a practical Thompson sampling framework, termed as Bayesian deep
Q-networks ( BDQN ), where we sample from the posterior on the set of Q-functions.
BDQN is computationally efﬁcient since it incorporates uncertainty only at the output layer, in the form of a Bayesian linear regression model.
Due to linearity and by choosing a Gaussian prior, we derive a closed-form analytical update to the posterior distribution over Q functions.
We can also draw samples efﬁciently from the Gaussian distribution.
As addressed in Mnih et al. (2015), one of the major beneﬁts of deploying function approximation methods in deep RL is that the estimation of the Q-value given a state-action pair can generalize well to other state-action pairs, even if they are visited rarely.
We expect this to hold in BDQN as well, but additionally, we also expect the uncertainty of state-action pairs to generalize well.
We test BDQN on a wide range of Arcade Learning Environment Atari games (Bellemare et al., 2013)(Brockman et al., 2016) against a strong baseline DDQN (Van Hasselt et al., 2016).
Aside from simplicity and popularity of DDQN , BDQN and DDQN share the same architecture (except for the last layer), and follow same target objective.
These are the main reasons we chose DDQN for our comparisons.
In table.
1 we see signiﬁcant gains for BDQN over DDQN .
BDQN is able to learn signiﬁcantly faster and reach higher rewards due to more efﬁcient exploration.
The evidence of this is further seen from the fact that we are able to train BDQN with much higher learning rates compared to DDQN .
This suggests that BDQN is able to learn faster and reach better scores.
These promising results suggest that BDQN can further beneﬁt from additional modiﬁcations that were done to DQN , e.g. Prioritized Experience Replay (Schaul et al., 2015), Dueling approach (Wang et al., 2015),
A3C (Mnih et al., 2016) etc.
This is because BDQN only changes that exploration strategy of DQN , and can easily accommodate additional improvements to DQN .
[Table]
[Equation] [Table] ORK
The complexity of the exploration-exploitation trade-off has been vastly investigated in RL.
Auer (2002) addresses this question for multi-armed bandit problems where regret guarantees are provided.
Jaksch et al.
(2010) investigates the regret analyses in MDPs where exploration happens through the optimal policy of optimistic model, a well-known Optimism in Face of Uncertainty ( OFU ) principle where a high probability regret upper bound is guaranteed
OFU in order to propose the high probability regret upper bound for Partially Observable MDPs (POMDPs) and ﬁnally, Bartók et al.
(2014) tackles a general case of partial monitoring games and provides minimax regret guarantee which is polynomial in certain dimensions of the problem.
In the OFU based methods, the exploration is under a policy which is optimal for the most optimistic setting while the OFU based methods might fail to provide an efﬁcient exploration (Lattimore &
Szepesvari, 2016).
Agrawal & Goyal (2012); Gopalan & Mannor (2015); Agrawal & Jia (2017) deploy TS for the exploration where the frequentist guarantees are provided while the Bayesian analyses are studied by Osband et al.
(2014; 2013); Abbasi-Yadkori & Szepesvári (2015).
Even through theoretical RL addresses the complexity of these trade-offs, these problems are still prominent in empirical reinforcement learning research (Mnih et al., 2015;
Jiang et al., 2016; Abel et al., 2016).
On the empirical side, the prominent recent success in the video games has sparked a ﬂurry of research interest.
Following the success of Deep RL on Atari games (Mnih et al., 2015) and the board game Go (Silver et al., 2016), many researchers have begun exploring practical applications of deep reinforcement learning (DRL).
Some investigated applications include, robotics (Levine et al., 2016), energy management (Night, 2016), and self-driving cars (Shalev-Shwartz et al., 2016).
Among the mentioned literature, the prominent exploration strategy for Deep RL agent has been ε -greedy.
For the ﬁrst time in large scale practical applications, Lipton et al.
(2016) used posterior sampling based exploration in dialogue systems, inspired by variational based methods in order to estimate the posterior distribution over the weights of the network using back-propagation.
Since they do not investigate the Atari games we are not able to have their method as an additional baseline.
Recently Osband et al.
(2016) introduced using multiple heads of a deep neural network to represent an approximate posterior computed by bootstrapping.
However, here, they did not estimate a true posterior since all collected data was used to update all estimated Q-functions, which meant that the diversity between the estimated Q-functions is mainly due to differences in the initialization weights.
Also, the explorations are still combined with uniform ε exploration.
Moreover, the authors were not able to show superior performance over uniformly sampled DQN S and DDQN .
Drop-out, as a randomized exploration method is proposed by Gal & Ghahramani (2016) but
Osband et al.
(2016) argues about its non-informative uncertainty estimate.
These results motivate us to investigate TS for practical classes of RL problems.
P RELIMINARIES
An inﬁnite horizon λ -discounted MDP M is a tuple (cid:104)X , A , T, R (cid:105) , with state space X , action space A , and the transition kernel T , accompanied with reward function of R
where 0 < λ ≤ 1 .
At each time step t , the environment is at a state x t , called current state, where the agent needs to make a decision a t under its policy.
Given the current state and action, the environment stochastically proceed to a successor state x t +1 under probability distribution T ( X t +1 | x t , a t ) :
= P ( X t +1 | x t , a t ) and provides a stochastic reward r t with mean of E [ r | x = x t , a = a t ] =
R ( x t , a t ) .
The agent objective is to optimize the overall expected discounted reward over its policy π :=
X → A , a stochastic mapping from states to actions, π ( a | x ) :=
P ( a | x ) .
[Equation]
The expectation in Eq.
1 is with respect to the randomness in the distribution of initial state, transition probabilities, stochastic rewards, and policy where η ∗ , π ∗ are optimal return and optimal policy, respectively.
Let Q π ( x, a ) denote the average discounted reward under policy π starting off from state x and taking action a in the ﬁrst place.
[Equation]
For a given policy π and Markovian assumption of the model, we can rewrite the equation for the Q functions as follows:
[Equation]
To ﬁnd the optimal policy, one can solve the Linear Programing problem in Eq.
1 or follow the corresponding Bellman equation Eq.
2 where both of optimization methods turn to
[Equation]
where Q ∗ ( x, a ) =
Q π ∗ ( x, a ) and the optimal policy is a deterministic mapping from state to actions in A , i.e. x → arg max a Q ∗ ( x, a ) .
Since in RL, we do not know the transition kernel and the reward function in advance and we can not solve the posed Bellman equation directly.
In order to tackle this problem, (Antos et al., 2008) studies the property of minimizing the Bellman residual of a given
Q-function
[Equation]
Where the tuple ( x, a, r, x (cid:48) , a (cid:48) ) consists of consecutive samples under behavioral policy π .
Furthermore, (Mnih et al., 2015) carries the same idea, and introduce Deep Q-Network ( DQN )
where the
Q-functions are parameterized by a DNN.
To improve the quality of Q function, they use back propagation on loss L ( Q ) .
In the following we descibe the setting used in DDQN .
In order to reduce the bias of the estimator, they introduce target network Q target and target value y
= r + λQ target ( x (cid:48) , ˆ a ) where ˆ a = arg max a (cid:48) Q ( x (cid:48) , a (cid:48) )
with a new loss L ( Q, Q target )
[Equation]
Minimizing this regression loss, and respectably its estimation (cid:98) L ( Q, Q target ) , matches the Q to the target y and provides an unbiased estimator of the target.
Once in a while, the algorithm sets Q target network to Q network and peruse the regression with the new target value.
3: for episode do 4:
Initialize x 1 to the initial state of the environment 5: for t = to the end of episode do 6: if count mod T sample =
then 7: sample W ∼ N ( W target , Cov ) 8: end if 9:
Select action a t = argmax a (cid:48) (cid:2) W (cid:62) φ θ ( x t ) (cid:3) a (cid:48) 10:
Execute action a t in environment, observing reward r t and successor state x t + 11:
Store transition ( x t , a t , r t , x t +1 ) in replay buffer 12:
Sample a random minibatch of transitions ( x τ , a τ , r τ , x τ +1 ) from replay buffer 13: y τ ← (cid:40) r τ , for terminal x τ + r τ + (cid:104) W target (cid:62) φ θ target ( x τ +1 ) (cid:105) ˆ a ; ˆ a = argmax a (cid:48) (cid:2) W (cid:62) φ θ ( x τ +1 ) (cid:3) a (cid:48) for non-terminal x τ + (cid:41) 14: θ ← θ − η · ∇ θ ( y τ − (cid:2) W (cid:62) φ θ ( x τ ) (cid:3) a τ )15: if count mod T target =
then 16: set θ target ← θ 17: end if 18: if count mod T Bayes target =
then 19:
Update W target and Cov 20: end if 21: count = count +22: end for 23: end for
B AYESIAN D EEP Q-N ETWORKS
In this work, we propose a Bayesian method to approximate the Q -function and match it to the target value.
We utilize the DQN architecture and remove its last layer and replace it with Bayesian linear regression (BLR) (Rasmussen & Williams, 2006) to BDQN network.
We utilize BLR to efﬁciently approximate the distortion over the Q-values where the uncertainty over the values is captured.
A common assumption in DNN is that the feature representation of the layer before the last layer of the network is suitable for linear classiﬁcation or linear regression methods (same assumption in DQN ).
Let the output feature vector of the layer before the last be represented as φ θ ( x ) ∈ R d , parametrized by θ .
Let φ targetθ ( x ) ∈ R
d be the feature representation for target value.
The Q-functions can be approximated as a linear combination of features in φ , i.e for a given pair of state-action Q ( x, a ) =
φ θ ( x ) (cid:62) w a .
Therefore, by deploying BLR, we can write the generative model of the Q-function to its corresponding target value:
y =
r + λφ target w target ˆ a , for any given state and action pair ( x, a ) as follows
[Equation]
where scalar (cid:15) is an independent noise drawn from a Gaussian iid distribution with mean zero and variance σ 2 (cid:15) , i.e., (cid:15) ∼ N (0 , σ 2 (cid:15) ) .
Furthermore, we assume that, for each action a ∈ A , the vector w a ∈ R d has a Gaussian prior N (0 , σ 2 ) .
Therefore, y | x, a, w a ∼ N ( φ ( x ) (cid:62) w a , σ 2 (cid:15) ) .
Moreover, the distribution of the target value y is as follows:
[Equation]
Given a dataset D = { x i , a i , y i } Di =1 , we construct |A| disjoint datasets for each action, D = ∪ a ∈A D a where D a is a set of tuples x i , a i , y i with the action a i = a and size D a .
Given the data set D a , we are interested in P ( w a |D a ) and P ( Q ( x, a ) |D a ) , ∀ x ∈ X .
Let us construct a matrix Φ a ∈ R d × D a , a concatenation of feature column vectors { φ ( x i ) } D a i =1 , and y a ∈ R D a , a concatenation of target values in set D a .
Therefore the posterior distribution is as follows:
[Equation]
where Ξ a
[Equation]
and since the prior and likelihood are conjugate of each other we have
[Equation]
where 1 ∈ R d is a identity matrix.
The expression in Eqs.
6, 7 give the posterior distribution over weights w a and the function Q respectively.
As TS suggests, for the exploration, we exploit the expression in Eq. 6.
For all actions, we set w targeta as the mean of posterior distribution over w a .
For each action, we sample a wight vector w a in order to have samples of mean Q-value.
Then we act optimally with respect the sampled means
[Equation]
Let W = { w a } |A| a =1 , respectively W target = { w targeta } |A| a =1 , and Cov = { Ξ a } |A| a =1 .
In BDQN , the agent interacts with the environment through applying the actions proposed by TS , i.e. a TS .
We utilize a notion of replay buffer where the agent stores its recent experiences.
The agent draws
W ∼ N ( W target , Cov ) (abbreviation for sampling of each action separately)
every T sample steps and act optimally with respect to drawn weights.
During the inner loop of the algorithm, we draw a minibatch of data from replay buffer and use loss
[Equation]
where
[Equation]
and update the weights of network: θ ← θ − η · ∇ θ
[Equation]
y
[Equation]
−
[Equation]
W
[Equation]
φ θ
[Equation]
x τ ) (cid:3) a τ ) 2 .
We update the target value every T target steps and set θ target to θ .
With the period of T Bayes target the agent updates its posterior distribution using larger minibatch of data drawn from replay buffer and sample W with respect to updated posterior.
Thus, we train BDQN end-to-end from scratch.
Algorithm 1 gives the full description of Bayes DQN .
TS vs ε -greedy:
The beneﬁts of TS can be explained via Eq.
Let us assume at a given time point, the environment is at state x and the empirical estimate of Q-function for an action a , Q ( x, a ) , is relatively lower than many of other actions.
Assume the agent is certain about the value of that action, as in Fig.
1 (a) (actions a = 5 or 6 ).
In this conﬁguration, exploring the action a may not advance the exploration-exploitation trade-off since the agent is quite sure about the low return of the action a and chooses actions with higher rewards with relatively higher uncertainties.
Under TS the chance that the action a = 5 or 6 gets explored is low and TS randomizes mostly on a set of actions with higher rewards with relatively higher uncertainties (e.g. actions 1,2,3,4).
The TS does not pick the action 5 or 6 as frequent as other sub-optimistic action until its uncertainty due to the lack of samples increases.
On the other hand, ε -greedy based strategy chooses the most optimistic action with the probability of 1 − ε (most of the time) and for exploration, chooses the other actions uniformly.
This is wasteful compared to TS .
Thus, the exploration phase of the ε -greedy strategy does not use all the valuable information about the uncertainty levels of actions, while TS uses this to its beneﬁt.
Another superiority of TS over ε -greedy can be described using Fig.
1 (c) .
Consider an episodic maze-inspired deterministic game, where the agent is placed to the green arrow at the beginning of each round of the game and the goal state is exiting form red arrow.
Let us further consider that the shortest path from starting point to the end point has a length of H and the duration of each round is exactly H as well.
If the agent reaches the goal state in the H time step, it wins the games and receives a score of 1 , otherwise, it receives 0 .
After each H times step, the agent’s location is reset to the starting point.
Consider an agent, which is given a set of Q-functions where the true
[Figure]
(a) : Even though the agent is certain about low return of action 6 , the ε -greedy strategy chooses action 6 as frequent as other sub-optimistic ones while TS based algorithm, randomizing over all the promising actions (most likely actions 1 , 2 , 3 , 4 ).
Furthermore, it does not pick this action frequently, until its uncertainty, due to the lack of samples, increases.
(b) : If an action with the low estimated expected return is accompanied with a high uncertainty, the TS based algorithm starts to choose that action until the algorithm reaches the “mean-value and uncertainty” trade-off.
On the other hand, ε -greedy strategy does same as (a) .
(c) : a maze-like deterministic game Q-function is within the set and is the most optimistic Q-function in the set as well.
In this situation, the optimism based strategy, with no exploration, chooses the true Q-function at ﬁrst place, reaches to the endpoint for all the rounds.
On the other hand, TS randomizes over the Q-functions with high promising returns and relatively low uncertainty, including the true Q-function.
When it picks the true
Q-function, it increases the posterior probability of this Q-function, since it matches the likelihood.
When TS chooses other functions since they predict deterministically wrong values, the posterior update of those functions set zero posteriors to them, therefore, the model will not choose them again, i.e. TS ﬁnds the true Q-function very fast.
For ε -greedy based agent, even though it chooses the true function right away from the begging (it is the optimistic one) and aims to follow the suggested path by true Q-function, at each time step it randomizes its action whit probity ε .
Therefore, it takes exponentially many trials in order to get to the target in this game.
E XPERIMENTS
We apply BDQN on a variety of games in the OpenAI-Gym 1 (Brockman et al., 2016).
As a baseline 2 , we run DDQN algorithm and evaluate BDQN on the measures of sample complexity and score.
Network architecture:
The input x to the network part of BDQN is 4 × 84 × 84 tensor with a rescaled, gray-scale version of the last four observations.
The ﬁrst convolution layer has 32 ﬁlters of size 8 with a stride of 4 .
The second convolution layer has 64 ﬁlters of size 4 with stride 2 .
The last convolution layer has 64 ﬁlters of size 3 followed by a fully connected layers with size 512 .
We add a BLR layer on top of this.
Choice of hyperparameters:
For BDQN , we set the values of W target to mean of the prior distribution over the weights of BLR and draw W from that prior.
For the ﬁxed W and W target , we randomly initialize the parameters of network part of BDQN , θ , and trained it using RMSProp, a learning rate of 0 . 0025 , and a momentum of 0 .
95 , inspired by (Mnih et al., 2015) where the discount factor is γ = 0 .
99 , the number of steps between target updates T target steps, and weights W are re-sampled from their posterior distribution with mean every T sample step.
We update the network part of BDQN every 4 steps by randomly sampling a mini-batch of size 32 samples from the replay buffer.
We update the posterior distribution of the weight set W every T Bayes target using mini-batch of size B (if size of replay buffer is less than B at the current step, we choose the minimum of these two ), with entries sampled uniformly form replay buffer.
We set W target to the posterior mean and the covariances to Cov .
The experience replay contains the 1 M most recent transitions.
Further hyperparameters, e.g. no-opt, are equivalent to ones in DQN setting.
BDQN has a few more hyperparameters to tune.
For the BLR, we have noise variance σ (cid:15) , variance of prior over weights σ , sample size B , posterior update period T Bayes target , and the posterior sampling period T sample .
To ﬁne the ﬁrst three, we set up a simple hyper parameter search.
We used a pretrained DQN model for game of Assault , and removed the last fully connected layer in order to have access to its already pretrained feature representation.
Then we tried combination of
B = { T target , 10 · T target } , σ = { 1 , 0 . 1 , 0 . 001 } , and σ (cid:15) = { 1 , 10 } and test for 1000 episode of the game.
We set these parameters to their best B = 10 · T target , σ = 0 .
001 , σ = 1 .
The above hyperparameter tuning is cheap since it requires only O ( B ) number of forward passes.
For the remaining parameter, we ran BDQN ( with weights randomly initialized) on the same game,
Assault , for 5 M time steps, with a set of T Bayes target =
{ T target , 10 · T target } and T sample =
{ T target, T target 100 } where BDQN performed better with choice of T Bayes target = 10 · T target .
For both choices of T sample , it performed almost equal where we chose the higher one.
In addition to
BLR parameters, we use bigger learning compared to DDQN .
We just started off with that learning rate and did not tune for that.
We guess, due to the efﬁcient exploration of BDQN , it can learn a better policy in even shorter time, in contrast, we experienced that changing the learning of DDQN may cause a major degradation in in its performance.
The proposed hyperparameter search is very simple where the exhaustive hyperparameter search is likely to provide even better performance.
Computational and sample cost comparison:
For a given period of game time, the number of backward pass in both BDQN and DQN are the same where for BDQN it is cheaper since it has one layer (the last layer) less than DQN .
In the sense of fairness in sample usage, for example in duration of 10 · T Bayes target = 100 k , all the layers of both BDQN and DQN , except the last layer, see the same number of samples, but the last layer of BDQN sees 16 times less samples compared to the last layer of DQN .
The last layer of DQN for a duration of 100 k , observes 25 k = 100 k/ 4 ( is backprob period) minibatchs of size 32 , which is 16 · 100 k , where the last layer of BDQN just observes samples size of B = 100 k .
As it is mentioned in Alg. 1, to update the posterior distribution,
BDQN draws B samples from the replay buffer and needs to compute the feature vector of them.
This step of BDQN , gives a superiority to DQN in the sense of speed which is almost 50% faster than
BDQN .
One can easily relax this limitation by parallelizing this step with the main body of BDQN or deploying on-line posterior update methods.
All the implementations were performed using the
MXNet framework (Chen et al., 2015).
Results:
The results are in Fig.
2 and Table.
Mostly the focus of the experiments are on sample complexity in Deep-RL, even though, BDQN provides much larger score compared to based line.
For example, for game Atlantis , DDQN † 3 gives score of 767 k after 200 M samples, while BDQN reaches 2 .
46 M after 20 M samples.
We can observe that BDQN can immediately learn a signiﬁcantly better policy due to its targeted exploration in much shorter period of time.
Since BDQN on game Atlantis show a big jump around time step 20 M , we ran it two more times in order to make sure it was not just a coincidence.
We did the same additional two experiments for the game Amidar as well.
We observed that the improvements are consistent among the different runs.
[Figure]
[Table]
C ONCLUSION
In this work we proposed BDQN , a practical TS based RL algorithm which provides targeted exploration in a computationally efﬁcient manner.
It involved making simple modiﬁcations to the DDQN architecture by replacing the last layer with a Bayesian linear regression.
Under the
Gaussian prior, we obtained fast closed-form updates for the posterior distribution.
We demonstrated signiﬁcantly faster training and enormously better rewards over a strong baseline DDQN .
This suggests that BDQN can beneﬁt even more from further modiﬁcations to DQN such as e.g. Prioritized Experience Replay (Schaul et al., 2015), Dueling approach (Wang et al., 2015), A3C (Mnih et al., 2016) etc.
We plan to explore the beneﬁt of modiﬁcations in future.
We also plan to combine strategies that incorporate uncertainty over model parameters with BDQN .