E FFICIENT E XPLORATION THROUGH B AYESIAN D EEP Q-N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose Bayesian Deep Q-Network ( BDQN ), a practical Thompson sampling based Reinforcement Learning (RL) Algorithm.
Thompson sampling allows for targeted exploration in high dimensions through posterior sampling, but is usually computationally expensive.
We address this limitation by introducing uncertainty only at the output layer of the network through a Bayesian Linear Regression (BLR) model, which can be trained with fast closed-form updates and its samples can be drawn efﬁciently through the Gaussian distribution.
We apply our method to a wide range of Atari Arcade Learning Environments.
Since BDQN carries out more efﬁcient exploration, it is able to reach higher rewards substantially faster than a key baseline, DDQN .
I NTRODUCTION
Designing algorithms that achieve an optimal trade-off between exploration and exploitation is the primary goal of reinforcement learning (RL).
However, targeted exploration in high dimensional spaces is a challenging problem in RL.
Recent works on deep RL mostly employ simple exploration strategies such as ε -greedy, where the agent chooses the optimistic action, the action with highest promising return, with probability (1 − ε ) , otherwise, uniformly at random picks one of the available actions.
Due to this uniform sampling, the ε -greedy method scales poorly with the dimensionality of state and action spaces.
Recent work has considered scaling exploration strategies to large domains (Bellemare et al., 2016; Tang et al., 2016).
Several of these papers have focused on employing optimism-under-uncertainty approaches, which essentially rely on computing conﬁdence bounds over different actions, and acting optimistically with respect to that uncertainty.
An alternative to optimism-under-uncertainty is Thompson Sampling ( TS ) (Thompson, 1933), where we draw samples from a posterior distribution over beliefs.
It is a Bayesian framework where we start with a prior distribution and compute the posterior beliefs based on the collected data through interaction with the environment.
We then maximize the expected reward under the sampled belief.
Posterior sampling can provide more targeted exploration since it can trade off uncertainty with the expected return of actions.
In contrast, the ε -greedy strategy is indifferent to uncertainty of the actions and the expected rewards of sub-optimistic ones (set of actions excluding the optimistic action).
There is compelling empirical evidence that Thompson Sampling can provide better results than optimism-under-uncertainty approaches, but so far this has only been demonstrated on smaller scale domains (Osband et al., 2013).
There has been relatively little work on scaling Thompson Sampling to large state spaces.
The primary difﬁculty in implementing Thompson sampling is the difﬁculty of sampling from general posterior distributions.
Prior efforts in this space have generally required extremely expensive computations (e.g. (Ghavamzadeh et al., 2015)
In this paper , we derive a practical Thompson sampling framework, termed as Bayesian deep
Q-networks ( BDQN ), where we sample from the posterior on the set of Q-functions.
BDQN is computationally efﬁcient since it incorporates uncertainty only at the output layer, in the form of a Bayesian linear regression model.
Due to linearity and by choosing a Gaussian prior, we derive a closed-form analytical update to the approximated posterior distribution over Q functions.
We can also draw samples efﬁciently from the Gaussian distribution.
As addressed in Mnih et al. (2015), one of the major beneﬁts of deploying function approximation methods in deep RL is that the estimation of the Q-value given a state-action pair can generalize well to other state-action pairs, even if they are visited rarely.
We expect this to hold in BDQN as well, but additionally, we also expect the uncertainty of state-action pairs to generalize well.
We test BDQN on a wide range of Arcade Learning Environment Atari games (Bellemare et al., 2013)(Brockman et al., 2016) against a strong baseline DDQN (Van Hasselt et al., 2016).
Aside from simplicity and popularity of DDQN , BDQN and DDQN share the same architecture, and follow same target objective.
These are the main reasons we chose DDQN for our comparisons.
In table.
1 we see signiﬁcant gains for BDQN over DDQN .
BDQN is able to learn signiﬁcantly faster and reach higher rewards due to more efﬁcient exploration.
The evidence of this is further seen from the fact that we are able to train BDQN with much higher learning rates compared to DDQN .
This suggests that BDQN is able to learn faster and reach better scores.
These promising results suggest that BDQN can further beneﬁt from additional modiﬁcations that were done to DQN , e.g. Prioritized Experience Replay (Schaul et al., 2015), Dueling approach (Wang et al., 2015),
A3C (Mnih et al., 2016) etc.
This is because BDQN only changes that exploration strategy of DQN , and can easily accommodate additional improvements to DQN .
[Table]
R ELATED W ORK
The complexity of the exploration-exploitation trade-off has been vastly investigated in RL.
Auer (2002) addresses this question for multi-armed bandit problems where regret guarantees are provided.
Jaksch et al.
(2010) investigates the regret analyses in MDPs where exploration happens through the optimal policy of optimistic model, a well-known Optimism in Face of Uncertainty ( OFU ) principle where a high probability regret upper bound is guaranteed. Azizzadenesheli et al.
deploys OFU in order to propose the high probability regret upper bound for Partially Observable MDPs (POMDPs) and ﬁnally, Bartók et al.
(2014) tackles a general case of partial monitoring games and provides minimax regret guarantee which is polynomial in certain dimensions of the problem [Table]
In the OFU based methods, the exploration is under a policy which is optimal for the most optimistic setting while the OFU based methods might fail to provide an efﬁcient exploration (Lattimore &
Szepesvari, 2016).
Agrawal & Goyal (2012); Gopalan & Mannor (2015); Agrawal & Jia (2017) deploy TS for the exploration where the frequentist guarantees are provided while the Bayesian analyses are studied by Osband et al.
(2014; 2013); Abbasi-Yadkori & Szepesvári (2015).
Even through theoretical RL addresses the complexity of these trade-offs, these problems are still prominent in empirical reinforcement learning research (Mnih et al., 2015;
Jiang et al., 2016; Abel et al., 2016).
On the empirical side, the prominent recent success in the video games has sparked a ﬂurry of research interest.
Following the success of Deep RL on Atari games (Mnih et al., 2015) and the board game Go (Silver et al., 2016), many researchers have begun exploring practical applications of deep reinforcement learning (DRL).
Some investigated applications include, robotics (Levine et al., 2016), energy management (Night, 2016), and self-driving cars (Shalev-Shwartz et al., 2016).
Among the mentioned literature, the prominent exploration strategy for Deep RL agent has been ε -greedy.
For the ﬁrst time in large scale practical applications, Lipton et al.
(2016) deployed posterior sampling of network weights, trained via back-propagation thorough variational distribution, for dialogue systems.
Recently Osband et al.
(2016) introduced using multiple heads of a deep neural network to represent an approximate posterior computed by bootstrapping, followed by randomized network for deep exploration
Osband et al.
However, here, they did not estimate a true posterior since all collected data was used to update all estimated Q-functions, which meant that the diversity between the estimated Q-functions is mainly due to differences in the initialization weights.
Also, the explorations are still combined with uniform ε exploration.
Moreover, the authors were not able to show superior performance over uniformly sampled DQN S and DDQN .
Drop-out, as a randomized exploration method is proposed by Gal & Ghahramani (2016) but
Osband et al.
(2016) argues about its non-informative uncertainty estimate.
These results motivate us to investigate TS for practical classes of RL problems.
T HOMPSON S AMPLING VS ε − GREEDY
In this section, we enumerate a few beneﬁts of TS over ε -greedy strategies.
We show how TS strategies exploit the uncertainty and expected return to design a randomized exploration while ε − greedy strategies disregard all these useful information for the exploration.
Fig.
1 (a) expresses the agent’s estimated values and uncertainties for the available actions at a given state x .
While ε − greedy strategy mostly focus on the action 1 , the optimistic action, the TS based strategy randomizes over actions 1 through 4 , utilizing their approximated returns and uncertainties, and with low frequency tries actions 5 and 6 . Not only the ε − greedy strategy explores actionsand 6 , where the RL agent is almost sure about low return of these actions, as frequent as other sub-optimistic actions, but also spend the network capacity to accurately estimate their values. A common and key technique in deep RL is a moving window of replay buffer. The TS based agent, after a few tries of actions 5 and 6 builds a belief in low return of these actions given the current target values.
Since the replay buffer is bounded moving window, lack of samples of these actions pushes the posterior belief to the prior over time and the agent tries these actions again in order to update its belief Fig.
1 (b) .
In general, TS based strategy advances the exploration-exploitation trade-off by making trade-off between the expected returns and the uncertainties, while ε − greedy strategy ignores all of this information.
Another superiority of TS over ε -greedy can be described using Fig.
1 (c) .
Consider an episodic maze-inspired deterministic game, with episode length H of shortest pass from the start point to the destination.
The agent is placed to the start point at the beginning of each episode and the goal state is reaching to the destination and receive a reward of 1 otherwise reward of 0 .
Consider an agent, which is given a set of Q-functions where the true Q-function is within the set and is the most optimistic function in the set.
In this situation, TS randomizes over the Q-functions with high promising returns and relatively high uncertainty, including the true Q-function.
When it picks the true Q-function, it increases the posterior probability of this Q-function because it matches the likelihood.
When
TS chooses other functions, they predict deterministically wrong values and the posterior update of those functions set to zero, therefore, the agent will not choose them again, i.e. TS ﬁnds the true
[Figure]  P RELIMINARIES An inﬁnite horizon γ -discounted MDP M is a tuple (cid:104)X , A , T, R (cid:105) , with state space X , action space A , and the transition kernel T , accompanied with reward function of R
where 0 < γ ≤ 1 .
At each time step t , the environment is at a state x t , called current state, where the agent needs to make a decision a t under its policy.
Given the current state and action, the environment stochastically proceed to a successor state x t +1 under probability distribution T ( X t +1 | x t , a t ) :
= P ( X t +1 | x t , a t ) and provides a stochastic reward r t with mean of E [ r | x = x t , a = a t ] =
R ( x t , a t ) .
The agent objective is to optimize the overall expected discounted reward over its policy π :=
X → A , a stochastic mapping from states to actions, π ( a | x ) :=
P ( a | x ) .
[Equation]
The expectation in Eq.
1 is with respect to the randomness in the distribution of initial state, transition probabilities, stochastic rewards, and policy where η ∗ , π ∗ are optimal return and optimal policy, respectively.
Let Q π ( x, a ) denote the average discounted reward under policy π starting off from state x and taking action a in the ﬁrst place.
[Equation]
For a given policy π and Markovian assumption of the model, we can rewrite the equation for the Q functions as follows:
[Equation]
To ﬁnd the optimal policy, one can solve the Linear Programing problem in Eq.
1 or follow the corresponding Bellman equation Eq.
2 where both of optimization methods turn to
[Equation]
where Q ∗ ( x, a ) =
Q π ∗ ( x, a ) and the optimal policy is a deterministic mapping from state to actions in A , i.e. x → arg max a Q ∗ ( x, a ) .
Since in RL, we do not know the transition kernel and the reward function in advance and we can not solve the posed Bellman equation directly.
In order to tackle this problem, (Antos et al., 2008) studies the property of minimizing the Bellman residual of a given
Q-function
[Equation]
Where the tuple ( x, a, r, x (cid:48) , a (cid:48) ) consists of consecutive samples under behavioral policy π .
Furthermore, (Mnih et al., 2015) carries the same idea, and introduce Deep Q-Network ( DQN ) where do 4:
5: for end do 6: if count mod T sample
0 then 7: sample W ∼ N ( W target Cov ) 8: end if 9: Select action a t = argmax a (cid:48) (cid:2) W (cid:62) φ θ ( x t ) (cid:3) a (cid:48) 10: action t successor state Store transition
Sample a random minibatch of transitions ( x τ , a τ , r τ , x
1 ) from
13: y τ ← (cid:40) r
, for terminal x
{+r τ + (cid:104) W target (cid:62) φ θ target ( x τ +1 ) (cid:105) ˆ a ; ˆ a = argmax a (cid:48) (cid:2) W (cid:62) φ θ ( x τ +1 ) (cid:3) a (cid:48) for non terminal x τ + (cid:41) 14: θ ← θ − η · ∇ θ ( y τ − (cid:2) W (cid:62) φ θ ( x τ ) (cid:3) a τ )15: if count mod T target = 
 then 16: set θ target ← θ 17: end if 18: if count mod T Bayes target = 
 then 19: Update W target and Cov 20: end if 21: count = count +22: end for 23: end for the+} Q-functions are parameterized by a DNN.
To improve the quality of Q function, they use back propagation on loss L ( Q ) using TD update.
In the following we describe the setting used in DDQN .
In order to reduce the bias of the estimator, they introduce target network Q target and target value y
= r + γQ target ( x (cid:48) , ˆ a ) where ˆ a = arg max a (cid:48) Q ( x (cid:48) , a (cid:48) )
with a new loss L ( Q, Q target )
[Equation]
Minimizing this regression loss, and respectably its estimation (cid:98) L ( Q, Q target ) , matches the Q to the target y and provides an unbiased estimator of the target.
Once in a while, the algorithm sets Q target network to Q network and peruse the regression with the new target value.
[-Initialize x 1 to the initial state of the environment 5: for t = to the end of episode do 6: if count mod T sample = 
-]
[-Sample a random minibatch of transitions ( x τ , a τ , r τ , x τ +1 ) from replay buffer 13: y τ ← (cid:40) r τ , for terminal x τ + r τ + (cid:104) W target (cid:62) φ θ target ( x τ +1 ) (cid:105) ˆ a ; ˆ a = argmax a (cid:48) (cid:2) W (cid:62) φ θ ( x τ +1 ) (cid:3) a (cid:48) for non-terminal x τ + (cid:41) 14: θ ← θ − η · ∇ θ ( y τ − (cid:2) W (cid:62) φ θ ( x τ ) (cid:3) a τ )15: if count mod T target = 
-]
[-then 16: set θ target ← θ 17: end if 18: if count mod T Bayes target = 
-]
B AYESIAN D EEP Q-N ETWORKS
In this work, we propose a Bayesian method to approximate the Q -function and match it to the target value.
We utilize the DQN architecture and remove its last layer and replace it with Bayesian linear regression (BLR) (Rasmussen & Williams, 2006) to BDQN network.
We utilize BLR to efﬁciently approximate the distribution over the Q-values where the uncertainty over the values is captured.
A common assumption in DNN is that the feature representation of the layer before the last layer of the network is suitable for linear classiﬁcation or linear regression methods (same assumption in DQN ).
Let the output feature vector of the layer before the last be represented as φ θ ( x ) ∈ R d , parametrized by θ .
Let φ targetθ ( x ) ∈ R
d be the feature representation for target value.
The Q-functions can be approximated as a linear combination of features in φ , i.e for a given pair of state-action Q ( x, a ) =
φ θ ( x ) (cid:62) w a .
Therefore, by deploying BLR, we can write the generative model of the Q-function to its corresponding target value:
y =
r + γφ target w target ˆ a , for any given state and action pair ( x, a ) as follows
[Equation]
where scalar (cid:15) is an independent noise drawn from a Gaussian iid distribution with mean zero and variance σ 2 (cid:15) , i.e., (cid:15) ∼ N (0 , σ 2 (cid:15) ) .
Furthermore, we assume that, for each action a ∈ A , the vector w a ∈ R d has a Gaussian prior N (0 , σ 2 ) .
Therefore, y | x, a, w a ∼ N ( φ ( x ) (cid:62) w a , σ [Equation]
Moreover, the distribution of the target value y is as follows:
[Equation]
Given a dataset D = { x i , a i , y i } Di =1 , we construct |A| disjoint datasets for each action, D = ∪ a ∈A D a where D a is a set of tuples x i , a i , y i with the action a i = a and size D a .
Given the data set D a , we are interested in P ( w a |D a ) and P ( Q ( x, a ) |D a ) , ∀ x ∈ X .
Let us construct a matrix Φ a ∈ R d × D a , a concatenation of feature column vectors { φ ( x i ) } D a i =1 , and y a ∈ R D a , a concatenation of target values in set D a .
Therefore the posterior distribution is as follows:
[Equation]
where Ξ a =
[Equation]
and Q ( x, a ) |D a = w (cid:62) a φ ( x ) .
Since the prior and likelihood are conjugate of each other we have the posterior distribution over the discounted return N
[Equation]
where 1 ∈ R d is a identity matrix.
The expression in Eqs.
6, 7 give the posterior distribution over weights w a , the function Q and the stochastic return.
As TS suggests, for the exploration, we exploit the expression in Eq. 6.
For all actions, we set w targeta as the mean of posterior distribution over w a .
For each action, we sample a wight vector w a in order to have samples of mean Q-value.
Then we act optimally with respect the sampled means
[Equation]
Let W = { w a } |A| a =1 , respectively W target = { w targeta } |A| a =1 , and Cov = { Ξ a } |A| a =1 .
In BDQN , the agent interacts with the environment through applying the actions proposed by TS , i.e. a TS .
We utilize a notion of replay buffer where the agent stores its recent experiences.
The agent draws
W ∼ N ( W target , Cov ) (abbreviation for sampling of each action separately)
every T sample steps and act optimally with respect to drawn weights.
During the inner loop of the algorithm, we draw a minibatch of data from replay buffer and use loss
[Equation]
(9) where
y
[Equation]
) and update the weights of network: θ ← θ − η · ∇
[Equation]
We update the target value every T target steps and set θ target to θ .
With the period of T Bayes target the agent updates its posterior distribution using larger minibatch of data drawn from replay buffer and sample W with respect to updated posterior.
Thus, we train BDQN end-to-end from scratch.
Algorithm 1 gives the full description of Bayes DQN .
It is worth mentioning that, in tabular setting with ﬁx-horizon episodic conﬁguration, the BDQN on ﬁxed linear feature representation (no feature learning) is equivalent to RLSVI (Osband et al., 2014).
E XPERIMENTS
We apply BDQN on a variety of games in the OpenAI-Gym 1 (Brockman et al., 2016).
As a baseline 2 , we run DDQN algorithm and evaluate BDQN on the measures of sample complexity and score.
Network architecture:
The input to the network part of BDQN is 4 × 84 × 84 tensor with a rescaled and averaged over channels of the last four observations.
The ﬁrst convolution layer hasﬁlters of size 8 with a stride of 4 .
The second convolution layer has 64 ﬁlters of size 4 with stride 2 .
The last convolution layer has 64 ﬁlters of size 3 followed by a fully connected layers with size 512 .
We add a BLR layer on top of this.
Choice of hyperparameters:
For BDQN , we set the values of W target to mean of the posterior distribution over the weights of BLR and draw W from that posterior.
For the ﬁxed W and W target , we randomly initialize the parameters of network part of BDQN , θ , and train it using RMSProp, with learning rate of 0 . 0025 , and a momentum of 0 .
95 , inspired by (Mnih et al., 2015) where the discount factor is γ = 0 .
99 , the number of steps between target updates T target steps, and weights W are re-sampled from their posterior distribution every T sample = 10 k step.
We update the network part of BDQN every 4 steps by uniformly at random sampling a mini-batch of size 32 samples from the replay buffer.
We update the posterior distribution of the weight set W every T Bayes target using mini-batch of size B (if size of replay buffer is less than B at the current step, we choose the minimum of these two ), with entries sampled uniformly form replay buffer.
We set W target to the posterior mean and the covariances to Cov .
The experience replay contains the 1 M most recent transitions.
Further hyperparameters, are equivalent to ones in DQN setting.
BDQN has a few more hyperparameters to tune.
For the BLR, we have noise variance σ (cid:15) , variance of prior over weights σ , sample size B , posterior update period T Bayes target , and the posterior sampling period T sample .
To optimize for this set of hyper parameters we set up a very simple, fast, and cheap hyper parameter tunning procedure which proves the robustness of BDQN .
To ﬁne the ﬁrst three, we set up a simple hyper parameter search.
We used a pretrained DQN model for the game of Assault , and removed the last fully connected layer in order to have access to its already pretrained feature representation.
Then we tried combination of B = { T target , 10 · T target } , σ = { 1 , 0 . 1 , 0 . 001 } , and σ (cid:15) = { 1 , 10 } and test for 1000 episode of the game.
We set these parameters to their best B = 10 · T target , σ = 0 .
001 , σ = 1 .
The above hyperparameter tuning is cheap and fast since it requires only a few times B number of forward passes.
For the remaining parameter, we ran BDQN ( with weights randomly initialized) on the same game, Assault , for 5 M time steps, with a set of T Bayes target =
{ T target , 10 · T target } and T sample =
{ T target, T target 100 } where BDQN performed better with choice of T Bayes target = 10 · T target .
For both choices of T sample , it performed almost equal where we chose the higher one.
In addition to BLR parameters, we use bigger learning rate compared to DDQN .
We just started off with that learning rate and did not tune for that.
Due to the efﬁcient exploration of BDQN and closed from BLR, it can learn a better policy in even shorter time, in contrast, it is well known for DQN based methods changing learing rate cause a major degradation in in its performance Apx. A.
The proposed hyperparameter search is very simple where the exhaustive hyperparameter search is likely to provide even better performance.
In order to compare the fairness in sample usage, we argue in
Apx.
A, that the network part of BDQN and its corresponding part in DDQN use the same number of samples but the BLR part of BDQN uses 16 times less samples compared to its corresponding last layer in DDQN .
All the implementations are coded in MXNet framework (Chen et al., 2015).
Results:
The results are provided in Fig.
2 and Table.
Mostly the focus of the experiments are on sample complexity in Deep-RL, even though, BDQN provides much larger score compared to based line.
For example, for game Atlantis , DDQN † gives score of 64 . 67 k after 200 M samples, while BDQN reaches 3 .
24 M after 40 M samples.
We can observe that BDQN can immediately learn a signiﬁcantly better policy due to its targeted exploration in much shorter period of time.
Since BDQN on game Atlantis show a big jump around time step 20 M , we ran it ﬁve more times in order to make sure it was not just a coincidence.
We did the same additional ﬁve experiments for the game Amidar as well.
We observed that the improvements are consistent among the different runs.
As it is been shown in Fig.
2, BDQN saturates after 20M samples, we check the reason and realized that BDQN reaches the internal OpenAIGym limit of max _ episode _ steps . We removed this limitation and the score after 15 M steps gets to 62 M .
[Table]
C ONCLUSION
In this work we proposed BDQN , a practical TS based RL algorithm which provides targeted exploration in a computationally efﬁcient manner.
It involved making simple modiﬁcations to the DDQN architecture by replacing the last layer with a Bayesian linear regression.
Under the
Gaussian prior, we obtained fast closed-form updates for the posterior distribution.
We demonstrated signiﬁcantly faster training and enormously better rewards over a strong baseline DDQN .
This suggests that BDQN can beneﬁt even more from further modiﬁcations to DQN such as e.g.
Prioritized Experience Replay (Schaul et al., 2015), Dueling approach (Wang et al., 2015), A3C (Mnih et al., 2016) etc.
We plan to explore the beneﬁt of modiﬁcations up to small changes in future.
We also plan to combine strategies that incorporate uncertainty over model parameters with BDQN .
In RL, policy gradient (Sutton et al., 2000; Kakade, 2002; Schulman et al., 2015) is another approach which directly learn the policy.
In practical policy gradient, even though the optimal policy, given
Markovian assumption, needs to be deterministic, the policy regularization is a dominant approach to make the policy stochastic and preserve the exploration thorough the stochasticity of the policy (Neu et al., 2017; Schulman et al., 2017).
We plan to explore the advantage of TS based exploration instead of regularizing the policy and make it stochastic.
A A PPENDIX
Learning rate: It is well known that DQN and DDQN are sensitive to the learning rate and change of learning rate can degrade the performance to even worse than random policy.
We tried the same learning rate as BDQN , 0.0025, for DDQN and observed that its performance drops.
Fig.
3 shows that the DDQN with higher learning rates learns as good as BDQN at the very beginning but it can not maintain the rate of improvement and degrade even worse than the original DDQN .
Computational and sample cost comparison:
For a given period of game time, the number of the backward pass in both BDQN and DQN are the same where for BDQN it is cheaper since it has one layer (the last layer) less than DQN .
In the sense of fairness in sample usage, for example in duration of 10 · T Bayes target = 100 k , all the layers of both BDQN and DQN , except the last layer, see the same number of samples, but the last layer of BDQN sees 16 times fewer samples compared to the last layer of DQN .
The last layer of DQN for a duration of 100 k , observes 25 k = 100 k/ 4 ( is back prob period)
mini batches of size 32 , which is 16 · 100 k , where the last layer of BDQN just observes samples size of B = 100 k .
As it is mentioned in Alg. 1, to update the posterior distribution,
BDQN draws B samples from the replay buffer and needs to compute the feature vector of them.
This step of BDQN gives a superiority to DQN in the sense of speed which is almost 60% faster than
BDQN .
One can easily relax this limitation by parallelizing this step with the main body of BDQN or deploying on-line posterior update methods.
Thompson sampling frequency:
The choice of TS update frequency can be crucial from domain to domain.
If one chooses T sample too short, then computed gradient for backpropagation of the feature representation is not going to be useful since the gradient get noisier and the loss function is changing too frequently.
On the other hand, the network tries to ﬁnd a feature representation which is suitable for a wide range of different weights of the last layer, results in improper use of model capacity.
If the TS update frequency is too low, then it is far from being TS and losses randomized exploration property.
The current choice of T sample is suitable for a variety of Atari games since the length of each episode is in range of O ( T sample ) and is infrequent enough to make the feature representation robust to big changes.
For the RL problems with shorter horizon we suggest to introduce two more parameters, ˜ T sample and ˜ W where ˜ T sample , the period that of ˜ W is sampled our of posterior, is much smaller than
T sample and ˜ W is being used just for making TS actions while W is used for backpropagation of feature representation.
For game Assault, we tried using ˜ T sample and ˜ W but did not observe much a difference, and set them to T sample and W .
But for RL setting with a shorter horizon, we suggest using them.
Further investigation in Atlantis: After removing the maximum episode length limit for the game
Atlantis, BDQN gets the score of 62M.
This episode is long enough to ﬁll half of the replay buffer and make the model perfect for the later part of the game but losing the crafted skill for the beginning of the game.
We observe in Fig.
4 that after losing the game in a long episode, the agent forgets a bit of its skill and loses few games but wraps up immediately and gets to score of 30 M .
To overcome this issue, one can expand the replay buffer size, stochastically store samples in the reply buffer where the later samples get stored with lowers chance, or train new models for the later parts of the episode.
There are many possible cures for this interesting observation and while we are comparing against
DDQN, we do not want to advance BDQN structure-wise.