H YPER DQN: A R ANDOMIZED E XPLORATION
M
ETHOD FOR D EEP R EINFORCEMENT L
EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Randomized least-square value iteration (RLSVI) is a provably efﬁcient exploration method.
However, it is limited to the case where 1) a good feature is known in advance and 2) this feature is ﬁxed during the training: if otherwise, RLSVI suffers an unbearable computational burden to obtain the posterior samples of the parameter in the Q -value function.
In this work, we present a practical algorithm named
HyperDQN , addressing these two issues under the context of deep reinforcement learning, where the feature changes over iterations.
HyperDQN is built on two parametric models: in addition to a non-linear neural network (i e., base model) that predicts Q -values, our method employs a probabilistic hypermodel (i e., meta model), which outputs the parameter of the base model.
When both models are jointly optimized under a speciﬁcally designed objective, three purposes can be achieved.
First, the hypermodel can generate approximate posterior samples regarding the parameter of the Q -value function.
As a result, diverse Q -value functions are sampled to select exploratory action sequences.
This retains the punchline of
RLSVI for efﬁcient exploration.
Second, a good feature is learned to approximate
Q -value functions.
This addresses limitation 1.
Third, the posterior samples of the Q -value function can be obtained in a more efﬁcient way than the existing methods and the changing feature does not affect the efﬁciency.
This deals with limitation 2.
On the Atari 2600 suite, after 20 M samples, HyperDQN achieves about 2 × improvements over (double) DQN, the advanced method Bootstrapped
DQN, and the SOTA exploration bonus method OB2I. For another challenging task
SuperMarioBros, HyperDQN outperforms baselines on 7 out of 9 games.
I NTRODUCTION
Reinforcement learning (RL) (Sutton & Barto, 2018) involves an agent that interacts with an unknown environment to maximize cumulative reward.
The trade-off between exploration and exploitation is a fundamental problem in RL (Kakade, 2003).
On the one hand, the agent needs to explore highly uncertain states and actions, which may sacriﬁce immediate reward.
On the other hand, in the long term, the agent should take the best-known action; however, this action may be sub-optimal due to partial information.
To this end, sample-efﬁcient RL agents should qualify the epistemic uncertainty (i.e., subjective uncertainty due to limited samples (Osband, 2016a)) to address the trade-off.
Bayesian approaches like Thompson sampling (Russo et al., 2018) provide a nice way of encoding the epistemic uncertainty by posterior distribution.
For instance, randomized least-square value iteration (RLSVI) 1 (Osband et al., 2016b; 2019) is a celebrated Bayesian algorithm.
Speciﬁcally, RLSVI takes three steps to address the exploration and exploitation trade-off.
First, conditioned on observed data,
RLSVI solves a Bayesian linear regression problem and updates the posterior distribution over (the parameter of) the optimal Q -value functions by the Bayes update rule.
Second, RLSVI samples a speciﬁc Q -value function from the posterior distribution.
Third, to collect more data, greedy actions are selected based on this Q -value function.
In theory, the randomness in posterior sampling could yield positive bias, which boosts optimistic behaviors (Osband et al., 2019; Russo, 2019; Zanette et al., 2020).
However, RLSVI is restricted to the following case
• A good feature is known in advance.
Over this feature, the Q -values can be approximated as a linear function.
• This feature is ﬁxed so that the posterior distribution is easy to compute in an incremental update way (see Remark 2 in Appendix B.1 or (1.1) for an illustration).
Unfortunately, these two requirements do not hold in the deep RL scenario.
The challenges of extending RLSVI to deep RL are listed below.
1) If the provided feature is not good and ﬁxed, RLSVI is rarely competent.
Under the deep RL setting, only a raw feature outputted by a randomly initialized neural network is available.
Such an unpolished feature has limited representation power, so the Q -value function cannot be approximated accurately.
Further, the mechanism of RLSVI only involves updating the model parameter, while the feature is left to be ﬁxed (as a raw feature in this setting); see Line 10 in Algorithm 1 in Appendix.
As a result, the empirical performance of RLSVI is often poor; see Appendix F.1 for the evidence.
2) When we update the feature over iterations in deep RL, the computational complexity of RLSVI becomes unbearable.
As the data accumulates over iterations, RLSVI needs to repeatedly compute the feature covariance matrix to update the posterior distribution.
With the ﬁxed feature mapping, this process can be efﬁciently implemented in an incremental way (see (1.1), where x K denotes the state-action pair ( s K , a K ) at iteration K and φ : S ×
A → R d is a feature mapping.).
However, when the feature mapping φ K is changing over iterations
we need to repeatedly calculate the matrix Φ K using all historical data as in
e
g. in Atari, this calculation could involve more than 1M samples with dimension 512)
and this process results in a huge computation burden.
ﬁxed
[Equation] [Equation]
[Equation]
) changing φ K : Φ K :=
[Equation]
[Table]
To address these two issues in deep RL, Bootstrapped DQN (BootDQN) (Osband et al., 2016a; 2018) takes a remarkable ﬁrst step.
To bypass the hurdle of issues 1 and 2, BootDQN simultaneously trains tens of ensembles (i.e., differently initialized neural networks) and views them as approximate posterior samples of Q -value functions.
However, with limited computation resources, BootDQN typically uses ensembles with a strictly limited number of elements.
As such, the quality of the learned posterior distribution could be poor (Lu & Van Roy, 2017).
In this work, we present a principled approach named HyperDQN that addresses the above issues under the context of deep RL.
Speciﬁcally, HyperDQN is built on two parametric models: in addition to a non-linear neural network (i.e., base model) that predicts Q -values, it employs a hypermodel (Dwaracherla et al., 2020)
(i.e., meta model).
This hypermodel maps a vector z (from the Gaussian distribution) to a parameter instance of the Q -value function.
As shown in Figure 1, this hypermodel aims to serve as a proxy for generating the posterior samples of θ predict .
To achieve this goal, both models are jointly optimized by a specially designed temporal difference (TD) objective (see (4.2)).
[Equation]
[Table]
/oH+hXfGFNQiOiHJmXPvOTP3Xi/yZaIc5zVnTUxOTc/kZwtz8wuLS8XllXoSpjEXNR76Ydz0WCJ8GYiaksoXzSgWrO/5ouFdHOp441LEiQyDE3UdiU6fnQXyVHKmiOoW16JyW50LxbqDthJXakDqnuRqONzqFktOxTHLHgduBkrIVjUsvqCNHkJwpOhDIIAi7IMhoacFFw4i4joYEBcTkiYuMESBtCllCcpgxF7Q94x2rYwNaK89E6PmdIpPb0xKG5ukCSkvJqxPs008Nc6a/c17YDz13a7p72VefWIVzon9SzfK/K9O16Jwij1Tg6SaIsPo6njmkpqu6JvbX6pS5BARp3GP4jFhbpSjPttGk5jadW+Zib+ZTM3qPc9yU7zrW9KA3Z/jHAf17YrrVNzjndL+QTbqPNaxgTLNcxf7OEIVNfK+wSOe8Gwx69a6s+4/U61cplnFt2U9fAAo6Jq+</latexit>
After optimization, our approach has the following essential merits.
• First, we prove that the optimized hypermodel can approximate the posterior distribution under the Bayesian linear regression case (see Theorem 1).
Thus, diverse Q -value functions can be sampled to select exploratory action sequences.
This retains the punchline of RLSVI for efﬁcient exploration.
We remark that this theoretical guarantee is missing in (Dwaracherla et al., 2020).
• Second, the feature mapping φ θ hidden ( · ) is updated during the training.
This deals with the feature learning problem of RLSVI (i.e., issue 1).
• Third, the posterior samples of θ predict are obtained without involving (1.2).
Speciﬁcally, the hypermodel could directly output an approximate posterior sample by taking z as its input.
This addresses the computational problem of RLSVI when the feature is changing (i.e., issue 2).
• Finally, compared with the ﬁnite ensembles in BootDQN, our approach measures the parameter uncertainty through “inﬁnite” ensembles.
This is because the hypermodel could generate “inﬁnite” parameter instances by mapping z from a continuous distribution.
Consequently, our approach could have a better posterior approximation with the same computation resources.
As such, our method is more computationally efﬁcient to achieve a near-optimal performance (see Appendix E.3)
To evaluate the efﬁciency of algorithms, we ﬁrstly consider the Atari 2600 suite (Bellemare et al., 2013).
We assess the efﬁciency in terms of the median performance over all 56 tasks in the suite provided 20M training frames.
The empirical result suggests HyperDQN achieves about 2 × improvements over its baseline version Double DQN (van Hasselt et al., 2016), the advanced randomized exploration method BootDQN, and the SOTA exploration bonus method OB2I (Bai et al., 2021).
Despite the wide coverage of Atari tasks, we ﬁnd the widely used trick reward clipping (i.e., map the game score to {− 1 , 0 , +1 } ) could preclude a meaningful comparison.
Surprisingly, there are 15 out of 56 tasks where the training and evaluation performance are weakly correlated or even uncorrelated (see Remark 6 in Appendix)!
To this end, we consider an alternative platform: SuperMarioBros (Kauten, 2018).
It provides 9 video games, covering both sparse and dense reward tasks 2 .
With the same parameter conﬁguration, HyperDQN outperforms baselines on 7 out of 9 games.
B ACKGROUND
Markov Decision Process.
In the standard reinforcement learning (RL) framework (Sutton & Barto, 2018), a learning agent interacts with an Markov Decision Process (MDP) to improve its performance via maximizing cumulative reward.
The sequential decision process is characterized as follows: at each timestep t , the agent receives a state s t from the environment and selects an action a t from its policy π ( a | s ) =
Pr { a = a t | s = s t } ; this decision is sent back to the environment, and the environment gives a reward signal r ( s t , a t ) and transits to the next state s t +1 based on the state transition probability p as,s (cid:48)
=
Pr { s (cid:48)
=
s t +1 | s = s t , a = a t } .
The main target of RL is to maximize the (expected) discounted return E (cid:2) (cid:80)
∞ t =
γ t r ( s t , a t ) (cid:12)(cid:12) s 0 ∼ ρ ( · )] , where ρ ( · ) is initial state distribution and γ ∈ (0 , 1) is a discount factor.
Deep Q-Networks (DQN).
In Deep Q-Networks (DQN) (Mnih et al., 2015), it employs a neural network to approximate the Q -value function, which is deﬁned as Q π ( s, a ) =
E [ (cid:80) ∞ t =0 γ t r ( s t , a t ) | s
=
s, a 0 = a ] .
In particular, a temporal difference (TD) based objective is applied:
[Equation] [Equation]
) where D is the experience replay buffer, Q θ is a prediction network parameterized by θ , and Q θ is the so-called target network, which is a delayed copy of Q θ for stable training.
R ELATED W ORK
Epistemic uncertainty qualiﬁcation.
It is justiﬁed that dithering strategies like (cid:15) -greedy (Mnih et al., 2015) are inefﬁcient (Osband et al., 2019).
This is intuitive since they do not have any epistemic uncertainty measure and hence cannot “write-off” sub-optimal actions after experimentation.
Importantly, epistemic uncertainty reﬂects the conﬁdence about the unknown environment in online decision-making (Osband, 2016a).
For tabular MDPs with the one-hot feature, “count” provides an epistemic uncertainty measure (Brafman & Tennenholtz, 2002; Jaksch et al., 2010; Azar et al., 2017;
Jin et al., 2018) and “covariance” serves as the counterpart for linear MDPs (Jin et al., 2020;
Cai et al., 2020).
Currently, we do not have a perfect epistemic uncertainty qualiﬁcation tool for deep RL, where a good feature is unknown in advance.
As argued in (Osband et al., 2018), approaches like dropout (Srivastava et al., 2014) and distributional operator (Bellemare et al., 2017) are not suitable since they are designed for risk estimation.
In this paper, we focus on the extension of hypermodel (Dwaracherla et al., 2020), which could capture the epistemic uncertainty for bandit tasks.
OFU-based exploration.
Based on the mentioned uncertainty qualiﬁcation, optimism in the face of uncertainty (OFU) based methods (Jaksch et al., 2010; Azar et al., 2017; Jin et al., 2018) construct upper conﬁdence bound (UCB) to direct exploration.
These theoretical works have inspired many empirical studies that encourage exploration by adding “reward/exploration bonus” to mimic UCB (Stadie et al., 2015;
Pathak et al., 2017; Tang et al., 2017; Burda et al., 2019a;b).
The challenges in this direction include: how to obtain task-relevant reward bonus (O’Donoghue et al., 2018)?
how to get an optimistic initialization with neural network (Rashid et al., 2020)?
how to properly back-propagate the uncertainty over periods to induce temporally extended behaviors (Bai et al., 2021)?
Though some of SOTA exploration bonus methods have achieved superior performance on some hard exploration tasks, (Ta¨ıga et al., 2020) report that these algorithms do not provide meaningful gains over the (cid:15) -greedy scheme on the whole Atari 2600 suite.
TS-based exploration.
On the other hand, Thompson sampling (TS) based methods, which originate in the bandit literature (see (Russo et al., 2018) and reference therein), design the exploration strategy in a Bayesian way (Osband et al., 2013; Osband & Van Roy, 2017).
Randomized least-square value iteration (RLSVI) provides a promising direction (Osband et al., 2016b; 2019).
As we mentioned in Section 1, this method, however, can only be applied when a good feature is known and ﬁxed during the training.
Following RLSVI, Osband et al.
(2016a) develop a practical algorithm called
Bootstrapped DQN, which uses ﬁnite ensembles to generate the randomized value functions.
As discussed earlier, the main bottleneck of Bootstrapped DQN is its computation complexity: it requires lots of ensembles to obtain an accurate approximation of the posterior samples (Lu & Van Roy, 2017).
Besides, (Azizzadenesheli et al., 2018) propose a direct extension of RLSVI, which is also computationally inefﬁcient and suffers the feature learning issue.
Our method is closely related to NoisyNet (Fortunato et al., 2018)
as both methods inject noise to the parameter.
However, we cannot view NoisyNet as an extension of RLSVI.
The main reason is that
NoisyNet is not ensured to approximate the posterior distribution as stated in (Fortunato et al., 2018).
We discuss other algorithmic differences and provide the numerical comparison in Appendix F.4.
M ETHODOLOGY 4. A RCHITECTURE D ESIGN
In this part, we explain the architecture design of HyperDQN shown in Figure 1.
First, the hypermodel f ν ( · ) maps a N z -dimension
random vector
z ∼ p ( z ) (e.g., a Gaussian distribution) to a speciﬁc d -dimension parameter θ := f ν ( z ) .
For example, if z follows an isotropic Gaussian distribution
N (0 , I ) and the hypermodel f ν ( z ) =
ν (cid:62) w z
+ ν b is a linear model, then θ :
= f ν ( z ) ∼ N ( ν b , ν (cid:62) w ν w ) , where ν w ∈ R N z
× d and ν b ∈ R
d are the weight and bias of the linear hypermodel with ν = ( ν w , ν b ) .
In general, the hypermodel could be a non-linear mapping so that f ν ( z ) could follow an arbitrary distribution.
In this paper, unless stated we only consider the linear hypermodel.
This is because the hypermodel would be applied to model the posterior distribution of a linear function like RLSVI does and a linear hypermodel has such a representation power to approximate the posterior distribution.
Second, the base model (i.e., a deep neural network) operates in a standard way: it takes a state s as its input and outputs Q θ ( s, a ) ∈ R for some action a .
Concretely, the base model employs a feature extractor φ θ hidden :
S → R d (i.e., the hidden layers in Figure 1) to learn a good representation.
Then, the base model outputs Q ( s, a ) by Q ( s, · ) =
θ (cid:62) predict φ θ hidden ( s ) , where θ predict ∈ R d × N a is the parameter of the prediction layer (i.e., the last layer of the base model in Figure 1).
By our formulation, θ predict is also the output of the hypermodel.
That is, θ predict = f ν ( z ) +
f ν prior ( z ) , where f ν prior ( z ) is a ﬁxed prior hypermodel; see the discussion below.
Numerically, we ﬁx v prior = v 0 , where v 0 is a random initialization.
In the sequel, we will introduce the difference between the hypermodel in Figure 1 and that in Dwaracherla et al.
4. D IFFERENCES WITH D WARACHERLA ET AL . (2020): H OW THE D IRECT E XTENSION F AILS
Readers may notice that our architecture design is different from the original one in (Dwaracherla et al., 2020).
Concretely, we only apply the hypermodel for the last layer of the base model but in (Dwaracherla et al., 2020) the hypermodel is applied for all layers of the base model.
This modiﬁcation is motivated by the trainability issue: the training will fail if otherwise (see Appendix F. for the numerical support).
To understand why, recall that parameter initialization matters for successfully training a deep neural network (Glorot & Bengio, 2010; Goodfellow et al., 2016).
For instance, the famous LeCun initialization rule (LeCun et al., 1998) suggests that we should initialize the parameter of i -th layer by sampling from the Gaussian distribution N (0 , σ 2 ) with σ = 1 / (cid:112) d i − 1 , where d i − 1 is the width of ( i − 1) -th layer.
As a result, the input signal and the output signal have the same order (cid:96) 2 -norm after processing.
Now, if we use the architecture in (Dwaracherla et al., 2020) with this technique, the output of the hypermodel is expected to lie between − 1 and 1 (up to constants).
This implies that the parameter is in [ − 1 , 1] for each layer of the base model, which disobeys the principle 1 / (cid:112) d i − 1 .
Consequently, the input signal ampliﬁes over layers and the gradient explodes (Sun, 2019).
In fact, Dwaracherla et al.
(2020) only use a two-layer base model with the width of 10 for bandit tasks so that the trainability issue is not severe in their applications.
But for deep RL, this training issue is severe and we bypass this challenge by the proposed architecture shown in Figure 1, in which the hidden layers of the base model are initialized with common techniques and the last layer could be properly initialized by normalizing the output of the hypermodel.
This deals with the parameter initialization issue.
Fortunately, this simple architecture still retains the main ingredient of RLSVI (i.e., capturing the posterior distribution over a linear prediction function).
We refer readers to Appendix F.2 for more discussion.
4. T RAINING O BJECTIVE
In this part, we introduce the objective function.
For the sake of better presentation, let us ﬁrst consider a regression task.
Let x ∈ R d be the feature and y ∈ R be the label, the objective function developed in (Dwaracherla et al., 2020) for training the hypermodel is
[Equation]
ξ
[Equation]
∈D
[Equation]
(4.1) where ξ is a random vector independently sampled from the unit hypersphere, paired with each ( x, y ) in the dataset D
= { ( x i , y i , ξ i ) :
i = 1 , . .
. , |D|} ; σ ω > 0 is the noise scale and σ p > 0 is the regularization scale.
Next, we brieﬂy explain three key parts in (4.1).
First, ( a ) is an artiﬁcial noise term exerted on the label y , which is introduced for pure technical purpose.
We defer more discussion of ( a ) to Remark 1.
Second, ( b ) contains two base models denoted by g .
Concretely, g f ν prior ( z ) ( · ) is a prior model parameterized by the output of a ﬁxed hypermodel f ν prior .
This term stores the prior information.
At the same time, g f ν ( z ) ( · ) is a differential model linked by the output of a trainable hypermodel f ν .
We remark that term ( b ) is called “additive prior models” in (Osband et al., 2018; Dwaracherla et al., 2020); see (Dwaracherla et al., 2020, Section 2.5) for a detailed discussion.
Finally, ( c ) is a regularization term, which is an essential design in Bayesian learning.
Now we explain why (4.1) is a good objective to generate approximate posterior samples.
As a “sanity check”, we provide an analysis under the linear case.
Assumption 1.
Suppose the underlying data generation follows y
= x (cid:62) θ (cid:63) + ω (cid:63) , ω (cid:63) ∼ N (0 , σ 2 ω ) and the prior distribution over θ (cid:63) is N ( θ p , σ 2 p I ) .
Furthermore, assume the base model is linear, i.e., g f ν ( z ) ( x ) =
x (cid:62) f ν ( z ) and g f ν prior ( z ) ( x ) = x (cid:62)
f
ν prior ( z ) .
Moreover, assume the hypermodel is also linear, i.e., f ν ( z )
=
ν (cid:62) w z
+ ν b and f ν prior ( z ) =
ν prior (cid:62) w z
+ ν prior b ; ν = ( ν w , ν b ) and ν prior =
( ν prior w , ν prior b ) .
Based on Assumption 1, the posterior distribution over θ (cid:63) is N ( E [ θ (cid:63) | D ] , Cov[ θ (cid:63) | D ]) with
E [ θ
[Equation]
| D ] =
[Equation]
[Equation] [Equation]
(cid:19)
[Equation]
, where X ∈ R |D|× d and Y ∈ R |D| are concatenation of x i and y i , respectively.
Also, we have that
g f ν prior ( z ) ( x ) + g f ν ( z ) ( x ) = g f ν prior ( z )+ f ν ( z ) ( x ) =
g θ ( x ) =
x (cid:62) θ , where θ := f
ν prior ( z ) + f ν ( z ) .
Next, we show that θ ∼ N ( ν prior b + ν b , ( ν prior w + ν w ) (cid:62) ( ν prior w + ν w )) is an approximate posterior sample.
Theorem 1 (Informal) .
Under Assumption 1, set ν prior w = σ p I and ν prior b = θ p .
Let ν (cid:63) =
( ν (cid:63) w , ν (cid:63)b ) be the optimal solution of (4.1) conditioned on speciﬁc realizations of ξ , then θ :=
f ν prior ( z ) + f ν (cid:63) ( z ) ∼
N ( ν prior b +
[Equation]
prior
[Equation]
satisﬁes
[Equation]
Furthermore, the error term satisﬁes E ξ [ err ( ξ )] = 0 .
[Table]
"8rA1zy3wraoZ54fWw8n59vFsH9U=">AAAC13icjVHLSsNAFD2Nr1pftS7dBIvgqiQi6LLoxmUF+xBbyySdtqFpEiYTsZTiTtz6A271j8Q/0L/wzpiCWkQnJDlz7j1n5t7rRL4XS8t6zRhz8wuLS9nl3Mrq2vpGfrNQi8NEuLzqhn4oGg6Lue8FvCo96fNGJDgbOj6vO4MTFa9fcxF7YXAuRxFvDVkv8LqeyyRR7XyhKftcsvbYmVyNm7FkYtLOF62SpZc5C+wUFJGuSph/QRMdhHCRYAiOAJKwD4aYnkvYsBAR18KYOEHI03GOCXKkTSiLUwYjdkDfHu0uUzagvfKMtdqlU3x6BSlN7JImpDxBWJ1m6niinRX7m/dYe6q7jejvpF5DYiX6xP6lm2b+V6dqkejiSNfgUU2RZlR1buqS6K6om5tfqpLkEBGncIfigrCrldM+m1oT69pVb5mOv+lMxaq9m+YmeFe3pAHbP8c5C2r7Jdsq2WcHxfJxOuostrGDPZrnIco4RQVV8r7BI57wbFwYt8adcf+ZamRSzRa+LePhA6qxl3c=</latexit>
Theorem 1 states that the optimized hypermodel can approximate the true posterior distribution.
The formal statement and proof are given in Appendix C.3.
To put our result in the literature, we remark that Dwaracherla et al.
(2020) prove that a linear hypermodel has sufﬁcient representation power to approximate any distribution (over functions)
so it is unnecessary to use a non-linear one.
However, this guarantee is unrelated to the objective (4.1): Dwaracherla et al.
(2020) only show there exists a linear hypermodel can approximate the posterior distribution but Dwaracherla et al. (2020) do not tell us whether (4.1) can lead to such a hypermodel.
Instead, this question is afﬁrmatively answered by Theorem 1.
In addition, Theoremalso conveys an important message about the noise used in (4.1), which we explain in Remark 1.
Remark 1.
Readers may ask whether an independent Gaussian noise ω (rather than the z -dependent noise z (cid:62) ξ ) is applicable in (4.1). Intuitively, ω can also form a randomized-least square problem to optimize the hypermodel.
However, by inspecting the proof of Theorem 1, we claim that this scheme does not work.
Technically speaking, ω will introduce exogenous random sources into the objective function, which jeopardizes the learning goal of mapping an index z to a posterior sample θ ; see Appendix C.4 for a formal argument.
To get a grasp of the fundamental issue involved here, consider a toy 2 -dimensional Bayesian linear regression problem (Assumption 1) with θ (cid:63)
= ( θ (cid:63)w , θ (cid:63)b ) ; see Appendix D.4 for experiment details.
After optimizing (4.1), we visualize the posterior samples generated by the linear hypermodel in Figure 2.
In particular, hypermodel( z (cid:62) ξ ) (in green) and hypermodel( ω ) (in red) correspond to learned samples via solving (4.1) with a z -dependent noise z (cid:62) ξ and an independent Gaussian noise ω , respectively.
The variant posterior sample (in purple) is obtain by the true posterior distribution
N ( E [ θ (cid:63) | D ] , Cov[ θ (cid:63) | D ]) .
We observe that the z -dependent noise is indispensable for the hypermodel to approximate the posterior distribution.
To address the limitations of RLSVI for deep RL tasks, we ﬁrst employ a feature mapping (parameterized by θ hidden ) and approximate the Q -value function by a linear function (parameterized by θ predict ) over the learned feature.
Furthermore, the parameter of this linear function is modeled by the hypermodel (i.e., θ predict = f
ν prior ( z ) + f ν ( z ) ) as discussed in Section 4.1.
To this end, we extend (4.1) and implement the following optimization problem:
[Equation]
where
[Equation]
Similar to the formulation in the regression problem, the Q -value function is the sum of the prior
Q θ prior ,f ν prior ( z ) and the differential Q θ hidden ,f ν ( z ) .
Following the target network in DQN, ( θ hidden , ν ) is the delayed copy of ( θ hidden , ν ) .
Compared with the TD loss in (2.1), there is an additional noise term σ ω z (cid:62) ξ for Bayesian learning and the prediction layer of the Q -value function is modeled as a probabilistic layer by the hypermodel.
In experiments, we use the average of ﬁnite samples of z to approximate (4.2); see the empirical loss function in (B.6) in Appendix.
We choose the stochastic gradient descent algorithm to optimize such constructed empirical loss function.
4. T HE A LGORITHM
We outline the proposed method in Algorithm 2 in Appendix.
At the beginning of each episode, a random vector z is sampled to obtain the Q -value function, which is further used for interactions with environments.
Subsequently, the randomized temporal difference objective in (4.2) is optimized to train the hypermodel and the hidden layers based on the collected data.
Next, we interpret HyperDQN from an algorithmic level introduced in (Osband et al., 2018).
In particular, HyperDQN achieves the desiderata:
• Task-relevant feature.
A task-speciﬁc feature can be obtained by the end-to-end training in (4.2).
Such a feature is good in the sense that it is optimized to approximate multiple value functions as
different invocations of z yield several individual TD loss functions.
As discussed previously, this feature learning procedure is beyond the scope of RLSVI.
• Multi-step uncertainty.
After the initialization, all state-action pairs tend to be uniformly uncertain by random features.
During the interaction, the agent updates its belief and the epistemic uncertainty could temporally propagate by the TD-style objective in (4.2) to induce “far-sighted” behaviors.
This is crucial for the deep exploration behaviors (see Figure 15b in Appendix E.3 for the visualization of the multi-step uncertainty).
• Commitment.
The agent executes action sequences that span multiple periods by obeying its intent.
In each episode, HyperDQN samples a speciﬁc value function and takes greedy actions according to this value function until the episode terminates.
Unlike BootDQN (Osband et al., 2016a) 4 and OFU-based algorithms (Ta¨ıga et al., 2020; Rashid et al., 2020; Bai et al., 2021) 5 , HyperDQN does not combine (cid:15) -greedy as the perturbation in (cid:15) -greedy ruins “far-sighted” behaviors generated by the original algorithm.
HyperDQN is closely related to BootDQN, which adopts ﬁnite ensembles to generate randomized value functions.
When the computation resources are limited, the quality of generated posterior samples by BootDQN is poor (Lu & Van Roy, 2017).
However, our approach has the advantage of measuring the uncertainty through “inﬁnite” ensembles.
As such, our method tends to be more efﬁcient; see the evidence in Table 4 in Appendix E.
As a side note in (Dwaracherla et al., 2020), if the random index z is repeatedly sampled from a ﬁnite set, the hypermodel degenerates to ﬁnite ensembles.
As such, BootDQN can be viewed as a special case of HyperDQN.
Following (Osband et al., 2018), we summarize the characteristics of BootDQN and HyperDQN in Table 1.
[Table]
E XPERIMENTS
In this section
[Table]
we present numerical experiments to validate the efﬁciency of the proposed method.
Experiment details can be found in Appendix D
[Table]
5. A TARI 2600 S UITE
Our ﬁrst experiment is on the Arcade Learning Environment (Bellemare et al., 2013), which provides a platform to assess the general competence.
As suggested by (Ta ¨ ıga et al., 2020), we evaluate algorithms across all 56 in the Atari 2600 suite.
The training and evaluation procedures (e
[Table]
g., observation preprocessing and reward clipping) follows (Mnih et al., 2015) and (van Hasselt et al., 2016).
Unless stated, reported results are based on the evaluation performance from each training epoch.
Experiments are run with 3 random seeds.
We measure the sample efﬁciency in terms of the human-normalized score during the interaction.
In particular, the raw score of each game is normalized so that 0% corresponds to a random agent and 100% to a human expert.
The ﬁnal performance is aggregated over 56 games by the robust statistics median , which is used in (Hessel et al., 2018; O’Donoghue et al., 2018; Ta¨ıga et al., 2020).
By this metric, Ta¨ıga et al.
(2020) point out that advanced exploration bonus methods do not provide meaningful gains over the simple (cid:15) -greedy scheme and conclude that the advances in some hard exploration tasks are due to the speciﬁc architecture design.
We consider the advanced exploration method BootDQN and the “SOTA” 6 OFU-based algorithm
OB2I (Bai et al., 2021).
We do not involve the Bayesian DQN (Azizzadenesheli et al., 2018) since its performance is signiﬁcantly inferior to OB2I as shown in (Bai et al., 2021) and it is not evaluated in the whole Atari suite in the original paper.
The baseline is Double DQN (van Hasselt et al., 2016) since BootDQN and our method are built on it.
[Figure]
The learning curve with the 20M frames training budget is displayed in Figure 3.
Note that this conﬁguration is commonly used in (Lee et al., 2019; Rashid et al., 2020; Bai et al., 2021).
This is because algorithms can achieve satisfactory performance on many tasks with 20M frames, while 200M frames training suffers heavy ﬁnancial burden and time cost 7 .
We see that BootDQN, OB2I, and HyperDQN could quickly explore at the initial stage and outperform the baseline Double DQN 8 .
However, only HyperDQN achieves substantial (i.e., about 2 × ) improvement over Double DQN after 20 M frames.
We notice the performance of OB2I is poor at the ﬁnal stage 9 , which is partially because the exploration bonus may guide the agent in a wrong direction.
We also consider the metric of the best performance in hindsight, which is used in (van Hasselt et al., 2016; Hessel et al., 2018; Bai et al., 2021).
The exact results are reported in Table 5 in Appendix.
Under this metric, HyperDQN is the best on 31 out of 56 games.
According to the taxonomy in (Bellemare et al., 2016), the relative improvements of HyperDQN on “easy exploration” and “hard exploration” environments are also visualized in Figure 11 and Figure 12 in Appendix E .
We see that HyperDQN could perform well on both easy exploration and hard exploration tasks.
However,
HyperDQN does not work for Montezuma’s Revenge, in which the reward is very sparse so that there is limited feedback for feature selection.
For the same reason, randomized exploration methods in (Osband et al., 2016a; 2018;
Fortunato et al., 2018) do not work well on this task, either.
Since the score range over tasks in the Atari suite varies a lot, people typically use the clipped score (i.e., clipping the one-step score to {− 1 , 0 , +1 } ) for training and the raw score for evaluation.
We observe this trick makes the training and evaluation performance weakly correlated or even uncorrelated for 15 out of 56 tasks; see the evidence and Remark 6 in Appendix E. Brieﬂy, an intelligent agent may obtain lots of +1 training reward.
But the evaluation performance could be bad because each training reward corresponds to a low raw game score.
To this end, we consider an alternative test platform: SuperMarioBros (Kauten, 2018), which is introduced as follows.
5. S UPER M ARIO B ROS
In this part, we evaluate algorithms on the SuperMarioBros suite (Kauten, 2018).
Since the raw game scores of SuperMarioBros tasks are well controlled, we use 0 .
01 × raw score as the training reward.
In this way, the training and evaluation performance are quite correlated.
Other environment preprocessing and algorithm parameters are almost the same as Atari 2600 suite.
We note that
SuperMarioBros-1-3 and SuperMarioBros-2-2 are two hard exploration tasks due to the long planning horizon and sparse reward.
Experiments are run with 3 random seeds.
We visualize the raw episode score in Figure 4.
We observe that HyperDQN outperforms all baselines for 7 out of 9 tasks; see exact numbers in Table 3 in Appendix.
Interestingly, Double DQN performs well on SuperMarioBros-2-1-v1.
By watching generated action sequences, we know that Double
DQN unintentionally ﬁnds the internal “bugs” of this game, i.e., playing the tortoise forever (see the video recorder at https://sites.google.com/view/hyperdqn ).
Thus, Double DQN obtains an immediate positive reward at each time step and has a good performance.
5. D EEP S EA
In this part, we conduct experiments to highlight the properties discussed in Section 4.4.
We consider a new testbed: deep sea (Osband et al., 2020) because this task is “simple” so that we can have a better understanding of algorithmic properties.
In particular, there are two actions in this environment:
[Figure]
move left and move right; see Figure 5a.
The reward is sparse and is only released when the agent always takes the “right” action to obtain the treasure at the corner.
We provide evidence that the dithering strategy could violate the commitment property required by deep exploration.
By the design of deep sea, once the agent takes a wrong action, it cannot obtain a positive reward during this episode.
Hence, the perturbation by (cid:15) -greedy could ruin far-sighted behaviors generated by the algorithm.
Published as a conference paper at ICLR
[Figure]
it will be rewarded with an additional reward of +1.
This presents a particularly challenging exploration problem for two reasons.
First, following the ‘gradient’ of small intermediate rewards leads the agent away from the optimal policy.
Second, a policy that explores with actions uniformly at random has probability 2 − N of reaching the rewarding state in any episode.
For the bsuite experiment we run the agent on sizes N = 10 , 12 ,.., 50 and look at the average regret compared to optimal after 10k episodes.
The summary ‘score’ computes the percentage of runs for which the average regret drops below 0.9 faster than the 2 N episodes expected by dithering.
Deep Sea is a construction, that performs well on this has mastered some properties of deep exploration.
Our summary score provides a ‘quick and dirty’ way to compareagentperformanceatahighlevel
Oursweepoverdiﬀerentsizes N canhelptopro- vide empirical evidence of the scaling properties of an algorithm beyond a simple pass/fail.
Figure 3 presents example output comparing A2C, DQN and Bootstrapped DQN on this a) Illustration for deep sea.
[Figure]
The numerical result (with 5 random seeds) is displayed in Figure 5b.
For completeness, we also show the performance of Double DQN (with (cid:15) -greedy) as a reference.
We observe the performance of
HyperDQN becomes bad if we combine (cid:15) -greedy with the randomized exploration strategy.
Similar results are also observed on Atari and SuperMarioBros (see Figure 10 and Figure 13 in Appendix).
C ONCLUSION
In this work, we present a practical exploration method to address the limitations of RLSVI and
BootDQN.
To reinforce the central idea, we leverage the hypermodel (Dwaracherla et al., 2020) and extend it from the bandit tasks to RL problems.
Several algorithmic designs are developed to release the power of randomized exploration.
Further directions include extending the idea for continuous control (Lillicrap et al., 2016;
Haarnoja et al., 2018)
(i.e., building a randomized actorcritic with a similar architecture with HyperDQN; see Appendix F.5 for preliminary results), utilizing the developed epistemic uncertainty qualiﬁcation tool for ofﬂine RL (Fujimoto et al., 2019; Levine et al., 2020), and acquiring an informative prior with human demonstrations to accelerate exploration (Hester et al., 2018;
Sun et al., 2018)
(see Appendix F.6 for the discussion).
E THICS S TATEMENT
This work focuses on designing an efﬁcient exploration method to improve the sample efﬁciency of reinforcement learning.
This work may help reinforcement learning to be better used in the real world.
There could be some unexpected consequences if the reinforcement learning is abused.
For example, a robot is trained maliciously to hurt people.
R EPRODUCIBILITY S TATEMENT
First, a formal statement and proof of Theorem 1 are given in Appendix C.3.
Second, the implementation details of the proposed algorithm HyperDQN can be found in Appendix B.2.
Third, the experiment details of numerical results in Section 5 are provided in Appendix D
A A PPENDIX O RGANIZATION
The appendix is organized as follows.
• Appendix B provides the preliminary about RLSVI (Osband et al., 2016b; 2019) and the algorithmic details of the proposed method HyperDQN.
• Appendix C reviews the Bayesian linear regression knowledge and gives the proof of Theorem 1, i.e., how the hypermodel is connected with the posterior distribution in Bayesian linear regression.
• Appendix D presents all kinds of experimental details such as environment preprocessing and the hyperparameters of algorithms.
• Appendix E offers additional empirical results to support the claims regarding the performance, algorithm design, and computation complexity covered in the main content.
• Appendix F discusses all kinds of related topics such as why (cid:15) -greedy is still used in existing efﬁcient algorithms, why a direct extension of RLSVI could fail, why a direct extension of the architecture in (Dwaracherla et al., 2020) could fail, and possible extensions of HyperDQN.
B A LGORITHM D ETAILS B. R ANDOMIZED L EAST - SQUARE V ALUE I TERATION (RLSVI)
In this part, we offer a brief introduction to randomized least-square value iteration (RLSVI)
(Osband et al., 2016b; 2019).
We hope this helps readers understand how RLSVI works and the underlying limitations of RLSVI.
In its original form, RLSVI is designed for the episodic Markov Decision Process for easy analysis.
Here the word of “episodic” means the planning horizon (i.e., the episode length) is a ﬁnite number
T > 0 .
Given a feature map φ : S × A
(cid:55)→ R d in advance, RLSVI assumes the optimal Q -value function is a linear model with respect to this feature map.
That is, for any stage t = 0 , 1 , · · · , T − 1 ,
RLSVI assumes
[Equation]
where θ (cid:63)t ∈ R
d is the unknown optimal parameter and Q (cid:63)t is the optimal value function at stage t .
Note that φ t differs from stage to stage, but they are all ﬁxed along the training.
With the above assumption, RLSVI takes three steps to balance the trade-off between exploration and exploitation.
• (Updating) Conditioned on the collected data, RLSVI estimates the optimal parameter θ (cid:63)t in a Bayesian way.
In particular, RLSVI obtains the posterior distribution over θ (cid:63)t via solving a Bayesian linear regression problem 10 .
More speciﬁcally, the Bayesian linear regression problem consists of the ﬁxed feature φ t and the label y t generated by dynamic programming; see (B.3).
Furthermore, the posterior distribution can be obtained by the closed-form solution in (B.4).
• (Sampling)
With the posterior distribution over θ (cid:63)t , RLSVI samples a speciﬁc parameter (cid:101) θ t , which forms the Q -value function (cid:101) Q t ( s, a ) =
φ t ( s, a ) (cid:62) (cid:101) θ t .
• (Interaction) RLSVI takes the greedy action argmax a (cid:101) Q ( s t , a ) at stage t to collect new data.
Let k indicates the episode count and t indicates the stage/period count.
Let [ T ] indicates the set { 0 , 1 , 2 , · · · , T − 1 } .
For instance, θ † k,t ∈ R d means the posterior mean at episode k and stage t .
With these notations, the procedure of RLSVI is outlined in Algorithm 1.
et al., 2016b
(cid:46)
Sampling
For Line 10 in Algorithm 1, RLSVI leverages the tool of Bayesian linear regression.
Abstractly, given the feature matrix X ∈ R N × d ( N is the number of samples)
[Equation]
where
[Equation]
With such deﬁned X t and Y t , RLSVI can compute the posterior distribution
N ( E [ θ (cid:63)t | D ] , Cov[ θ
[Equation]
|
D ]) over θ (cid:63)t :
[Equation]
Remark 2.
As the feature is ﬁxed in RLSVI, there exists an efﬁcient implementation of RLSVI based on the incremental update.
Speciﬁcally, let Φ be the feature covariance matrix, i.e.,
[Equation]
For easy presentation, let φ k,t denote φ ( s k,t , a k,t ) .
Then, we have the incremental update formula:
[Equation]
with [Equation] Furthermore, we have a more efﬁcient update rule for the posterior covariance in (B.4) by the
Sherman–Morrison formula (see Lemma 1).
More concretely, we have that
[Equation]
As a side note, the term X (cid:62) t Y t in (B.4) can also be implemented in an incremental update way.
In short, the posterior distribution is easy to compute when the feature is ﬁxed.
Lemma 1 (Sherman–Morrison formula) .
Suppose A ∈ R n × n is an invertible square matrix and u, v ∈ R n are column vectors.
Then A + uv (cid:62) is invertible if and only if 1 + v (cid:62)
A − 1 u (cid:54) = 0 .
In this case,
[Equation]
Here, uv (cid:62) is the outer product of two vectors u and v .
B. I MPLEMENTATION OF H YPER DQN
In this part, we provide more implementation details of HyperDQN.
Recall that we work with two value functions: the prior one and the differential one.
The prior value function and the differential value function are independently constructed, meaning there are no shared parameters.
The actual Q -value is the weighted sum of two terms:
[Equation]
where β prior > 0 and β differential > 0 are two positive scalars.
The initial value of β prior and β differential depend on the width of the last hidden layer; see Remark 5 in Appendix D. When β prior = β differential
=
1 , we recover the formulation in (4.2).
The feature extractor (i.e., the hidden layers) are task-speciﬁc, which will be discussed in Appendix D.
For the hypermodel, the differential part is initialized based on the default initializer of PyTorch while the initialization of the prior hypermodel is a little tricky.
Following (Dwaracherla et al., 2020), each row of ν w is sampled from the unit hypersphere and ν b is based on the mean of the desired prior distribution.
The purpose of this initialization is to guarantee the θ prior =
f ν prior ( z ) follows a desired Gaussian distribution.
For example, if we want to obtain a prior distribution N ( 1 , I ) for
1: agent
(cid:96)
2: for episode k = 0 , 1 , 2 , · · · do 3: generate a random vector z ∼ N (0 , I ) .
(cid:46)
Sampling 4: instantiate the Q -value function Q θ ( s, a, z ) .
5: for stage t = 0 , 1 , 2 , · · · , T − 1 do (cid:46) Interaction 6: observe state s t .
7: take the greedy action a ← argmax a Q θ ( s t , a, z ) .
8: receive the next state s t +1 and reward r ( s t , a t ) .
9: sample ξ uniformly from unit hypersphere.
10: store ( s t , a t , r, ξ, s t +1 ) into the replay buffer D .
11: agent step n ← n + 1 .
12: if mod (agent step n , train frequency M ) =
=
then (cid:46) Update 13: sample a mini-batch (cid:101) D of ( s, a, r, ξ, s (cid:48) ) from the replay buffer D .
14: sample N random vectors z (cid:48)
i : (cid:101)
Z = { z (cid:48)
i } Ni =1 .
15: optimize ν and θ hidden using the empirical loss function (B.6) with (cid:101) D and (cid:101) Z .
16: train step (cid:96) ← (cid:96) + 1 .
17: end if 18: if mod (train step (cid:96) , target update frequency G ) =
=
then 19: update the target network.
20: end if 21: end for 22: end for θ prior :
= f ν prior ( z ) , it sufﬁces to set ν b = 1 and to uniformly sample ν w from the unit hypersphere, where 1 is a vector with elements of 1 .
According to the population objective function in (4.2), the empirical objective function
L ( ν, θ hidden ; (cid:101) Z , (cid:101) D ) given the set (cid:101) Z and mini-batch (cid:101)
D is
[Equation]
where
[Equation]
In practice, (B.6) can be efﬁciently solved by Adam (Kingma & Ba, 2015).
In particular, we sample multiple z and the mini-batch from the experience replay buffer to construct the empirical objective function.
As such, Adam can be applied to optimize the hidden layers and the hypermodel as illustrated in Figure 1.
Remark 3 (Feature Representation for Prior Value Functions) .
Different from supervised learning, the prior value function is crucial for online decision making (Osband et al., 2018).
For the prior value functions, we not only need to consider the prior hypermodel but also the prior feature representation.
Importantly, the feature extractor for the prior value function should be ﬁxed rather than being shared with the differential part.
In other words, we use two independent feature networks (i.e., the hidden layers) for the prior value function and the differential value function, respectively.
Instead, the output of the prior value function would change if the prior value function and the differential value function share the same hidden layers, which violates the intuition that the prior value function should not be affected by the learning process.
B. S AMPLING F ROM U NIT H YPERSPHERE
For completeness, we describe the method for uniform sampling over unit hypersphere.
There are two steps:
(1) Generate x = ( x 1 , x 2 , . . .
, x d ) , using a zero mean and unit variance Gaussian distribution.
Thus, the probability density of x
[Equation]
is spherically symmetric.
(2) Normalize the vector x = ( x 1 , x 2 , . . .
, x d ) to a unit vector, namely x / (cid:107) x (cid:107) , which gives a sample uniformly over the unit hypersphere.
Note that once the vector is normalized, its coordinates are no longer statistically independent.
C C ONNECTION B ETWEEN B AYESIAN L INEAR R EGRESSION AND H YPERMODEL C. B AYESIAN L INEAR R EGRESSION
In this part, we review the Bayesian linear regression.
Let x ∈ R d be the feature and y = (cid:104) θ (cid:63) , x (cid:105) + ω (cid:63) be the label, where ω (cid:63) ∈ N (0 , σ 2 ω ) is the observation noise.
Here, we treat θ (cid:63) as a random variable and pose a prior distribution p 0 : N ( θ p , σ 2 p I ) on θ (cid:63) .
Given the dataset D = { ( x i , y i ) } Ni =1 , we can update the posterior probability density over θ (cid:63) by the Bayes rule:
[Equation]
Thanks to the conjugate prior, the posterior distribution over θ (cid:63) is also a Gaussian distribution
N ( E [ θ (cid:63) | D ] , Cov[ θ (cid:63) | D ]) with
[Equation]
where
X
[Equation]
In summary, given the collected dataset (i.e., the features and labels), we can obtain posterior samples through sampling from N ( E [ θ (cid:63) | D ] , Cov[ θ (cid:63) | D ]) .
Recently, Osband et al.
(2019) provide another way of obtaining posterior samples through optimizing a randomized least square problem.
Lemma 2 ((Osband et al., 2019)) .
Let x be the feature vector and the target y be generated by y = (cid:104) θ (cid:63) , x (cid:105) + ω (cid:63) , where ω (cid:63) ∼ N (0 , σ 2 ω ) is the noise and the prior on θ (cid:63) is N ( θ p , σ 2 p I ) .
Let ω be the algorithmic noise that follows the same distribution with ω (cid:63) (i.e., ω ∼ N (0 , σ 2 ω ) ) and (cid:98) θ ∼ N ( θ p , σ 2 p I ) be a prior sample.
Then,
[Equation]
or
[Equation]
could yield a sample from the posterior distribution N ( E [ θ (cid:63) | D ] , Cov[ θ (cid:63) | D ]) deﬁned by (C.1) and (C.2).
First, it is obvious that (C.3) and (C.4) are equivalent by the variable change trick.
Let us focus on (C.4) to explain the main idea of Lemma 2.
Notice that the posterior sample includes two parts: the prior (cid:98) θ and the differential θ ( θ is a speciﬁc optimal solution to (C.4)).
In particular, θ is a random variable, which depends on the randomness in ω .
Based on this observation, we can leverage the optimality condition to obtain the optimal solution of (C.4).
Subsequently, we can verify that the mean of (cid:98) θ + θ and its covariance are identical with the posterior mean E [ θ (cid:63) | D ] and posterior covariance Cov[ θ (cid:63) | D ] , respectively.
Remark 4.
Importantly, Lemma 2 recasts a sampling problem to an optimization problem, which provides a purely computational cue to obtain a posterior sample.
Consequently, it provides another way of implementing RLSVI by solving the least square problem in each iteration.
However, there is no clear advantage for this implementation under the case where RLSVI can work (e.g., tabular or linear MDPs).
C. H YPERMODEL
In this part, we provide more explanation about the hypermodel.
To proceed, note that the main issue of the procedure in Lemma 2 is its computational efﬁciency.
The reason is that we have to solve multiple randomized least square problems if we want to obtain many posterior samples.
One of the motivations of the hypermodel is to address this computational issue.
Indeed, the original optimization problem in (Dwaracherla et al., 2020) is
[Equation]
where ν prior =
ν 0 is the ﬁxed prior hypermodel parameter.
Consider the linear case where g f ν ( z ) ( x ) =
f ν ( z ) (cid:62) x , then (C.5) becomes
[Equation]
which is very similar to (C.3) and the difference is discussed in Remark 1.
Intuitively, the hypermodel wants to solve “inﬁnite” randomized least-square problems simultaneously.
Through this, the hypermodel can obtain multiple posterior samples by sampling z .
Note that (C.5) has a different form with the loss function in (4.1).
However, as discussed in (Dwaracherla et al., 2020, Section 2.5), these two loss functions are equivalent.
To derive (4.1) from (C.5), we take three steps: 1) replace f ν ( z ) in (C.5) with f ν prior ( z ) + f ν (cid:48)
( z ) ; 2) let g f ν
( z ) ( x ) =
g f ν prior ( z )+ f ν (cid:48)
( z ) ( x ) =
g f ν prior ( z ) ( x ) + g f ν (cid:48)
( z ) ( x ) ; 3) change the optimization variable from ν to ν (cid:48) .
C. P ROOF OF T HEOREM
Here, we consider the linear hypermodel, i.e., f ν ( z ) =
ν (cid:62) w z
+ ν b , where ν w ∈ R N z
× d and ν b ∈ R d .
By the property of Gaussian distribution, we know that f ν ( z ) ∼ N ( ν b , ν (cid:62) w ν w ) as z ∼ N (0 , I ) .
Therefore, ν b and ν (cid:62) w ν w qualify the mean and covariance, respectively.
By the formulation, there are two models in a complete hypermodel: the prior one and the differential one.
Speciﬁcally,
• The prior hypermodel is
[Equation]
where (cid:98) ν
= ( (cid:98) ν w , (cid:98) ν b ) is ﬁxed and a non-trainable variable.
Here, we set the (cid:98) ν (cid:62) w = σ p
I and (cid:98) ν
b = θ p as the correct prior parameters, then f (cid:98) ν
( z ) is a sample from the prior distribution
N ( θ p , σ 2 p I ) .
The
differential hypermodel is
[Equation]
where ν = ( ν w , ν b ) is a trainable variable.
Let the sample ( x i , y i ) generated under Assumption 1.
We augment each data sample ( x i , y i )
∈ D with ξ i independently sampled from unit hypersphere (refer to Appendix B.3).
Then, the dataset becomes
[Equation]
, with the augmented noise realizations Ξ = { ξ i : i = 1 , . .
. , |D|} .
Theorem 1 (Formal statement) .
Under Assumption 1, set ν prior w = σ p I and ν prior b = θ p .
Let ν (cid:63) =
( ν (cid:63)w , ν (cid:63)b ) be the optimal solution of (4.1) conditioned on speciﬁc realizations of ξ , then θ := f
ν prior ( z ) + f ν (cid:63) ( z ) ∼ N ( ν prior b + ν (cid:63)b , ( ν prior w + ν (cid:63)w ) (cid:62) ( ν prior w + ν (cid:63)w )) with ν prior b
+ ν (cid:63)b = E [ θ (cid:63) | D ] , ( ν prior w + ν (cid:63)w ) (cid:62) ( ν prior w + ν (cid:63)w ) =
Cov[ θ (cid:63) | D ] + err (Ξ) , where
[Equation]
Furthermore, the error term satisﬁes E Ξ [ err (Ξ)] = 0 .
Proof of Theorem 1.
For simplicity, we omit the subscript (cid:63) for the optimal solution to (4.1) in the proof.
That is, we use the shorthand notation ν
= ( ν ω , ν b ) for the optimal solution to (4.1)
[Equation]
The objective function (4.1) for learning hypermodel given the data D becomes:
[Equation]
According to the ﬁrst-order optimality condition:
[Equation]
It is straightforward to obtain that
[Equation]
Then
[Equation]
we can infer that
[Equation] [Equation]
where X ∈ R |D|× d , Y ∈ R |D|
[Equation]
and E [ θ | D ] is deﬁned in (C.1).
This implies that the hypermodel can recover the posterior mean.
For the variable ν w
[Equation]
we calculate its partial derivative ∂ L /∂ν (cid:62) w as
[Equation]
To proceed
[Equation]
we utilize the following helpful lemma.
Lemma 3.
Let z ∼ N (0 , I d ) .
For any ﬁxed vector a ∈ R d
[Equation]
we have
[Equation]
Proof.
The proof is easy if we look at each entry of the matrix,
[Equation]
Then, with the above Lemma 3 and (C.7)
[Equation]
we further calculate
[Equation]
By the ﬁrst order optimality condition
[Equation]
we have
[Equation]
With the posterior covariance in (C.2) Σ
[Equation]
, and deﬁne the set Ξ = ( ξ 1 , . .
. , ξ |D| ) corresponding to the ξ i in ( x i , y i , ξ i )
∈ D
[Equation]
we obtain that
[Equation]
where the second equality is due to the fact that (cid:107) ξ (cid:107) =
and we set (cid:98) ν
w = σ p
I at the beginning; and err (Ξ) =
Σ − 1 (cid:16) 1 σ 2 ω (cid:80)
[Equation]
x,ξ )
[Equation]
=
[Equation]
x
[Equation]
ξ
[Equation] [Equation]
ξ (cid:48) x
(cid:48)(cid:62)
+
[Equation]
Σ − 1 .
Taking the expectation over Ξ = ( ξ 1 , . .
. , ξ |D| ) , we have
[Equation]
Finally, we have
[Equation]
This implies the hypermodel can also recover the posterior covariance in expectation
. C. W HY I NDEPENDENT G AUSSIAN N OISE C ANNOT W ORK F OR H YPERMODEL ?
Following similar steps in Appendix C.3, we want to argue that the posterior approximation result cannot be achieved if we replace z (cid:62) ξ with an independent Gaussian noise ω in (4.1).
Similarly, we augment each sample ( x i , y i )
∈ D with ω i ∼ N (0 , σ 2 ω ) and denote ω =
( ω 1 , ω 2 , . . .
, ω |D| ) (cid:62) as the noise vector.
The dataset becomes
[Equation]
Now
[Equation]
the objective function for learning hypermodel given the data D becomes:
[Equation]
Similar to (C.6)
[Equation]
we have the ﬁrst order optimality condition
[Equation]
Then, we have
[Equation]
For the variable ν w , similar to (C.7)
[Equation]
we calculate the partial derivative ∂ L /∂ν (cid:62) w as
[Equation]
By the ﬁrst order optimality condition, we have
[Equation]
Therefore, by the fact that (cid:98) ν (cid:62) w (cid:98) ν w
[Equation]
σ 2 p
I
[Equation]
This implies that an independent Gaussian noise ω cannot work and the z -dependent noise is indispensable for posterior approximation.
D E XPERIMENT D ETAILS
D.
A TARI 2600 S UITE
Environment Preprocessing.
We follow the same preprocessing as in (Mnih et al., 2015) and (van Hasselt et al., 2016).
In particular, there are random no-operation steps (up to 30 ) before the interaction.
Each agent step corresponds to 4 environment steps by repeating the same action while each environment step corresponds to 4 frames of the simulator.
The raw score is clipped to {− 1 , 0 , +1 } for training but the evaluation performance is based on the raw score.
Episodes are early stopped after 108K frames as in (van Hasselt et al., 2016).
The observation for the agent is based on 4 stacked frames, which is reshaped to (4 , 84 , 84) .
Hyperparameters.
All agents use the same network structure as in (Mnih et al., 2015): state → conv (32 , 8 , 4) → relu → conv (64 , 4 , 2) → relu → conv (64 , 3 , 1) → mlp (512) → relu → mlp ( number of actions ) , where conv (32 , 8 , 4) means a convolution layer with 64 ﬁlters of size 8 and stride of 4 , and mlp (512) means a fully-connected layer with output size of 512 , and relu stands for Rectiﬁed Linear Units.
The algorithmic parameters follow (Mnih et al., 2015, Table 1).
For example, the replay buffer size is 1 M; the batch size is 32 ; the discount factor is 0 . 99 ; the target network update frequency is 10 K agent steps; the train frequency is 4 agent steps; and the replay starts after 50 K agent steps.
For algorithms with (cid:15) -greedy (e.g., Double DQN, BootDQN and OB2I), the exploration (cid:15) is annealed from 1 . 0 to
.linearly (from 50 K agent steps to 1 M agent steps, respectively); the test (cid:15) is 0 .
The training result of Double DQN (van Hasselt et al., 2016) is based on DQN Zoo (Quan & Ostrovski, 2020), which is a collection of DQN based algorithms.
DQN Zoo is a best-effort replication of respective baselines and publishes the learning data 11 .
The training data of OB2I is shared by authors in (Bai et al., 2021)
(private communication).
We implement Bootstrapped DQN (BootDQN) (Osband et al., 2016a) and HyperDQN with the framework of tianshou 12 .
The number of ensembles in
BootDQN is 10 and the prior scale is 1 . 0 .
BootDQN uses the same (cid:15) -greedy strategy with Double
DQN as suggested by (Osband et al., 2016a).
Note that we do not implement the “vote” mode since this evaluation strategy does not matter in practice.
The hyperparameters of HyperDQN are shown in Table 2.
Note that since each z corresponds to 32 mini-batch samples, the effective batch size of
HyperDQN is 32 × 10 = 320 .
We use three random seeds: 2021, 2022 and 2023 for HyperDQN and BootDQN.
Based on tianshou, n-step return (Mnih et al., 2016) of 3 and double Q learning (van Hasselt et al., 2016) are implemented but dueling network (Wang et al., 2016) and prioritized replay (Schaul et al., 2016) are omitted.
The optimizer for both methods is Adam (Kingma & Ba, 2015) with learning rate of 0 . 0001 .
[Table] [Table]
number
[Table]
z
[Table]
training
[Table]
noise scale σ
[Table]
in
[Table] [Table]
prior regularization scale σ p in (B.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
ARIO
[Table]
Environment Preprocessing.
The observation and action preprocessing are the same with Atari 2600 suite.
Two things are different: 1) there is no random no operation; 2) the training reward is based on 0 .
01 × raw score rather than clipping.
The maximum episode length is 1500 × 4 =frames.
11 https://github.com/deepmind/dqn_zoo/blob/master/results.tar.gzhttps
Hyperparameters.
The neural network structure and algorithmic parameters are almost identical with Atari 2600 suite.
There are two common parameters are suggested by tianshou to accelerate training without sacriﬁcing the ﬁnal performance: target network update frequency is 500 agent steps and the train frequency is 10 agent steps 13 .
Unfortunately, OB2I is sensitive to this change so that we use the original parameters that the training frequency is 4 and the target network update frequency is 10000 .
All algorithms use three random seeds: 2021, 2022, and 2023.
D.
[Equation]
Deep sea task is modiﬁed from bsuite
(Osband et al., 2020).
We use the version where the reward is deterministic and the state is in one-hot feature.
The state space size is 50 × 50 / 2 =
and the action space size is 2 .
The maximum episode length is 50 .
The neural network architecture is :
[Equation]
Different from Atari, the train frequency is 10 ; the target network update frequency is 10 ; the experience replay buffer size is 20 K. For (cid:15) -greedy, we diminish (cid:15) from 1 . 0 to 0 .
linearly (from 10 K interactions to 100 K interactions, respectively); in other words, the learning starts after 10 K interactions for Double DQN.
The test (cid:15) is 0 .
since any dithering may lead to a failure.
Other hyperparameters are given in Table 2.
Remark 5 (Parameter choice of the prior scale and differential scale) .
Since the hypermodel is used to model the parameter of the last control layer, it is helpful to obey the principle of neural network initialization (Glorot & Bengio, 2010).
Speciﬁcally, it is beneﬁcial to set β prior and β differential proportional to 1 /d , where d the width of the last hidden layer (i.e., the dimension of the feature).
Hence, for deep sea we use β prior = β differential = 1 . 0 while for Atari 2600 and SuperMarioBros we use β prior = β differential = 0 . 1 .
D. B AYESIAN L INEAR R EGRESSION
The 2 -dimensional Bayesian linear regression problem used in Section 4.3 is based on y = θ w · x + θ b + (cid:15) , where x, y, θ w , θ b , (cid:15) ∈ R .
Speciﬁcally, (cid:15) is sampled from the Gaussian distribution.
To generate the dataset, x is sampled from [ − 4 , 4] uniformly.
The total number of training samples is 50 .
The prior distribution for ( θ w , θ b ) is N (0 , I 2 ) while the actual value ( θ w , θ b ) is (6 . 06 , 0 . 47) for the dataset generation.
We obtain the exact posterior distribution by the Bayes update rule.
For hypermodel, the dimension of z is 2 .
We use gradient descent with a learning rate of 0 .
005 and momentum of 0 .
The iteration number of the gradient descent is 10000 .
After the optimization, we visualize the 5000 samples from different distributions (e.g., the posterior distribution and the one transformed by the hypermodel) in
Figure 2.
Indeed, the KL-divergence between the exact posterior distribution and the counterparts by hypermodel( z (cid:62) ξ ) and hypermodel( ω ) are 0 .
1 and 326 . 4 , respectively.
D. P ARAMETER C HOICE
Ablation study about the noise scale σ ω .
In this part, we provide the ablation study about the noise scale σ ω .
The numerical result is displayed in Figure 6a.
We observe the performance of HyperDQN is not very sensitive to σ ω .
Ablation study about the prior scale σ p .
In this part, we provide the ablation study about the prior scale σ p .
The numerical result is shown in Figure 6b.
We ﬁnd that a large prior scale results in a poor performance.
The reason is that the posterior update is slow under this case.
Ablation study about the non-linear hypermodel.
In this paper, we mainly focus on the linear hypermodel.
Here, we investigate the variant with a non-linear hypermodel (i.e., an MLP hypermodel).
In particular, we consider the hypermodel is a two-layer neural network of width
[Figure]
with ReLU activation.
We call this variant as HyperDQN(MLP) and the original HyperDQN as
HyperDQN(Linear)
[Figure]
It is intuitive that HyperDQN(MLP) has a more powerful posterior approximation ability but it may be hard to train HyperDQN(MLP) since the architecture is more complex.
The empirical results on Atari and SuperMarioBros are shown in Figure 7.
We see that HyperDQN does not obtain signiﬁcant gains by an MLP hypermodel.
We conjecture the reason is the trainability issue of the MLP hypermodel
[Figure]
Ablation study about the number of ensembles in BootDQN.
In this paper, we implement BootDQN with 10 ensembles, which follows the conﬁguration in (Osband et al., 2016a, Section 6.1).
However, in (Osband et al., 2018)
[Figure]
BootDQN is implemented with 20 ensembles.
We remark that the choice of 10 ensembles is commonly used in the previous literature (Rashid et al., 2020; Bai et al., 2021) since it is more computationally cheap.
For completeness
[Figure]
we provide the ablation study about this parameter choice; see Figure 8 for the result
[Figure]
Ablation study about the training frequency.
Here we provide the evidence that the performance does not change too much with a training frequency of 10 instead of 4 .
The results are shown in
Figure 9.
[Figure]
E A DDITIONAL R ESULTS E. A TARI 2600 S UITE
The correlation between training and test performance.
We provide the training curve and the test curve in Figure 16 and Figure 17, respectively.
Note that the former is based on the clipped score while the latter is based on the raw score.
Remark 6.
We highlight the mismatch between the training performance and the test performance due to reward clipping (i.e., clipping the raw score to {− 1 , 0 , +1 } for training).
By comparing
HyperDQN and BootDQN, we classify 56 tasks into the following 4 categories:
• The training performance of HyperDQN and BootDQN are comparative but the test performance of HyperDQN is better: amidar, choppercommand, doubledunk, mspacman, privateeye, zaxxon ( 6 ).
• The training performance of HyperDQN and BootDQN are comparative but the test performance of HyperDQN is worse: assault, asteroids, breakout, demonattack, phoenix, solaris, yarsrevenge ( 7 ).
• The training performance of HyperDQN is better but its test performance of is worse: atlantis, skiing ( 2 ).
• The training and test performance of HyperDQN and BootDQN are much correlated: alien, asterix, bankheist, battlezone, beamrider, berzerk, bowling, boxing, centipede, crazyclimber, defender, enduro, fishingderby, freeway, frostbite, gopher, gravitar, hero, icehockey, jamesbond, kangaroo, krull, kungfumaster, montezumarevenge, namethisgame, pitfall, pong, qbert, riverraid, roadrunner, robotank, seaquest, spaceinvaders, stargunner, tennis, timepilot, tutankham, upndown, venture, videopinball, wizardofwor ( 41 ).
On the commitment of randomized exploration.
Here we provide the evidence that using the (cid:15) -greedy strategy could destroy the commitment and leads to poor performance on several wellunderstood tasks; see the results in Figure 10.
We observe that combing the randomized exploration strategy with (cid:15) -greedy results in poor performance.
[Figure]
Best performance in hindsight.
In Section 5, we have evaluated the performance of algorithms in terms of the median score over 56 tasks in the Atari suite.
Here, we provide the performance on each individual task; see Table 5.
In particular, we save the best policy during the training and re-evaluate it over 200 episodes.
Following (Bai et al., 2021), the performance is based on the maximal score over 200 evaluation episodes and is aggregated over 3 random seeds.
Note that this metric is used to make a fair comparison with the results reported in (Bai et al., 2021).
We observe that HyperDQN is the best on 31 out of 56 tasks.
Relative improvement on each environment.
To better understand the improvement of HyperDQN, we visualize the relative score compared with baselines.
Speciﬁcally, the relative score is calculated as
[Equation]
− baseline max( human , baseline ) − human (Wang et al., 2016).
According to the taxonomy in (Bellemare et al., 2016),
B r e a k o u t
S t a r G unn e r P
h
o e n i x S
o l a r
i s A ss a u l t
V i
d e o P i n b a ll
Sp a c e I n v a d e r s K r u ll
T u t a n k h a m E n d u r o B o w li n g
N a m e T h i s G a m e
Y a r s r e v e n g e B e a m R i d e r G o p h e r A s t
e r i x B e
r
Z
e r k C e n t i p e d e B o x i n g P
r
i v a t e E
y e
[Table]
M
[Table]
m
[Table]
s
[Table]
d
[Table]
n A li
[Table]
n
[Table]
t
[Table]
s
J a m
[Table]
f
[Table]
n d
[Table]
a
[Table]
M
[Table] [Table]
a
[Table]
m
[Table]
n
[Table]
r C o
[Table]
o n
I c e H o c k e y H .
E .
R .
O B a n k H
e i s t T e nn i s
F i s h
i n g D e r b y
T i m e P il o t
C r a z y C li m b e r
F r o s t b i t e
R o a d R unn e r V e n t u r e
D o u b
l
e
D un k
W i z a r d o f W o r
F r ee w a y R o b
o t a n k Q
* B e r t P o n g
K un g - F u M a s t e
[Figure]
R e l a t i v e S c
o r e -80 - -
[Figure]
(Sparse
[Figure] [Figure]
[Figure] [Figure]
B r e a k o u t K r u ll A t l a n t
i s P h
o e n i
x
A ss a u l t E n d u r o
T u t a n k h a m G o p h e r B o x i n g
N a m e T h i s G a m
e A s t
e r i x
M o n t e z u m a ' s
R e v e n g e P
i t
f a ll
B e a m R i
d
e r
Sp a c e I n v a d e r s
F r ee w a y P
r
i v a t e E
y e S e a q u e s t
A s t e r o i d s P o n g B a n k H
e i s t B e r Z e r
k
I c
e H o c
k e y
Y a r s r e v e n g e D e f e n d e r
R o a d R unn e r
S t a r G unn e r B o w li n g
C h
o
pp e r C o mm a n d A
m i d a r T e nn i s A li e n G r
a v i t a r
D e m o n A tt a c k
R i v e r R a i d
F i s h
i n g D e r b y S k ii n g
M s .
P a c m a n S o l a r i s
K un g - F u M a s t e r
R o b o t a n k H .
E .
R .
O F r o s
t b
i t e J a m e s b o n d C e n t i p e
d e Z a xx o n V e n t u r e Q * B e r t
U p a n d D o w n
B a tt l e Z o n e V i
d
e o P i n b a ll
C r a z y C li m b e r
W i z a r d o f W o r
K a n g a r oo T i m e P il o t
D o u b
l e D un
[Figure] [Figure]
[Figure] [Figure]
[Figure]
we cluster environments by 4 groups: “hard exploration (dense reward)”, “hard exploration (sparse reward)”, “easy exploration” and “unknown”.
See the results in Figure 11 and Figure 12.
Remark 7.
We observe that HyperDQN has improvements on both “easy exploration” environments (e.g., Battle Zone, Jamesbond, and Pong) and “hard exploration” environments (e.g., Frostbite,
Gravitar, and Zaxxon ).
However, we notice that HyperDQN does not work well on very sparse reward tasks like Montezuma’s Revenge.
We explain the failure reason as follows.
As we have discussed in the introduction, without a good feature, randomized exploration methods are rarely competent.
On Montezuma’s Revenge, the extremely sparse reward provides limited feedback for feature selection.
As a result, it is expected that randomized exploration methods (including
BootDQN and NoisyNet) do not work well for this task.
In contrast, prediction error based methods (Pathak et al., 2017; Burda et al., 2019b; Rashid et al., 2020) could leverage speciﬁc architecture designs to provide auxiliary reward feedback to help feature selection and exploration.
As a result, these methods perform well on Montezuma’s Revenge.
We kindly remind that these methods do not perform well on other tasks because speciﬁcally designed architectures in these methods do not generalize well as pointed out in (Ta¨ıga et al., 2020).
E. S UPER M ARIO B ROS
On the commitment of randomized exploration.
Here we provide the evidence that using the (cid:15) -greedy strategy could destroy the commitment and leads to poor performance on SuperMarioBros; see the result in Figure 13.
[Figure]
In addition
[Figure]
we provide the evidence that the randomized exploration approach BootDQN (Osband et al., 2016a) could obtain some gains if (cid:15) -greedy is not used; see the result in Figure 14.
The result suggests that the original implementation of BootDQN does not satisfy the commitment property.
Note that the variant without (cid:15) -greedy is still inferior compared with HyperDQN.
[Figure]
Best performance in hindsight.
Here, we provide the comparison with the metric that the best policy in hindsight after 20 M frames.
In particular, we save the best policy after the training procedure and re-evaluate this best policy with 200 episodes.
The mean score over 200 episodes and 3 random seeds is given in Table 3.
Different from the metric in the Atari suite, we do not use the maximal score metric, which is very sensitive and provides limited insights.
We observe that HyperDQN outperforms baselines on 7 out of 9 games.
[Table]
SuperMarioBros
[Table]
SuperMarioBros
[Table]
SuperMarioBros-
[Table]
SuperMarioBros-3-2-v
[Table]
SuperMarioBros-3- v 3, 2, 3, 5,
E. D EEP S EA
On the multi-step uncertainty.
To better understand exploration behaviors by HyperDQN, we also hope to visualize the learned epistemic uncertainty.
It is intuitive to expect the epistemic uncertainty is uniformly high for all states before the interaction.
Through learning from the collected data, we hope the epistemic uncertainty becomes low for the “leaf” states (i.e., the bottom states in Figure 15a).
In contrast, the epistemic uncertainty could be relatively high for the “root” states (i.e., the top states in Figure 15a) since the transitions from these states contain more possibilities than the “leaf” states.
Once the agent could explore to ﬁnd the treasure and obtain the positive reward, the Q -value becomes high for these valuable “leaf” states.
At the same time, the epistemic uncertainty starts to diminish for these states.
Correspondingly, the temporal difference learning objective will push the “root” states to increase their Q -values and to reduce their epistemic uncertainty
Considering states on the diagonal path in Figure 15a
we visualize the standard deviation (as a proxy for the epistemic uncertainty) and the mean of Q -values in Figure 15b.
In particular, the x -axis corresponds to states along the optimal path and the y -axis corresponds to the number of interactions.
From Figure 15b, we observe that the evolution of the standard deviation and the mean values could verify our intuition about the multi-step uncertainty.
In addition, the evolution is also consistent with the learning curve in Figure 5b.
Published as a conference paper at ICLR
[Figure]
The problem is
(a
Illustration sea
Digit
[Figure]
Computation Complexity Comparision .
Here we provide the evidence that HyperDQN could be computationally efﬁcient than Bootstrapped DQN (BootDQN) (Osband et al., 2016a).
Here, we work on the hard version of the deep sea, which is slightly different from the version we used in Section 5.
In particular, actions are randomly masked so that the index 0 may correspond to “left” or “right” at different states (Osband et al., 2020).
Following (Dwaracherla et al., 2020), we measure the performance by the computation complexity 15 :
[Equation]
where n sgd is the number of SGD steps per iteration, n z is the number of ensemble (index) samples, and K is the number of episode that the episode return is 0 .
For BootDQN, it uses 10 ensembles (i.e., n sgd = n z = 10 ) while HyperDQN samples z from a two dimensional distribution (i.e., n z =and n sgd = 20 ).
For a fair comparison, the BootDQN considered here is identical with HyperDQN except that z is sampled from the set of 10 -dimensional one-hot features.
The (averaged) empirical result with 10 random seeds (from 2021 to 2030 ) is shown in Table 4.
We see that HyperDQN is clearly more computationally efﬁcient than BootDQN.
deep-seadeep-seadeep-seadeep-seadeep-seaBootDQN 130K 250K 490K 870K 1,640K
HyperDQN 48K 104K 196K 304K 1,120K
[Table]
Centipede 2,090.
Chopper Command 811.
Crazy Climber 10,780.
Defender 2,874.
Demon Attack 152.
Double Dunk -18.
Enduro 0.
Fishing Derby -91.
Freeway 0.
Frostbite 65.
Gopher 257.
Gravitar 173.
H.E.R.O 1,027.
Ice Hockey -11.
Jamesbond 29.
Kangaroo 52.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Space Invaders 148.
Star Gunner 664.
Tennis -23.
Time Pilot 3,568.
Tutankham 11.
Up and Down 533.
Venture 0.
Video Pinball 16,256.
Wizard of Wor 563.
Yarsrevenge 3,092.
Zaxxon 32.
5M 10M 15M 20M
E p
i s o d e S c
o r e
AlienNoFrameskip-v
BootDQNHyperDQN 5M 10M 15M 20M 1010 2040
AmidarNoFrameskip-v
BootDQNHyperDQN 5M 10M 15M 20M 55
AssaultNoFrameskip-v
BootDQNHyperDQN 5M 10M 15M 20M
AsterixNoFrameskip-v
BootDQNHyperDQN 5M 10M 15M 20M
E p
i s o d e S c
o r e
AsteroidsNoFrameskip-v
BootDQNHyperDQN 5M 10M 15M 20M
AtlantisNoFrameskip-v
BootDQNHyperDQN 5M 10M 15M 20M 55
BankHeistNoFrameskip-v
BootDQNHyperDQN 5M 10M 15M 20M 11
BattleZoneNoFrameskip-v4 BootDQNHyperDQN 5M 10M 15M 20M
E p
i s o d e S c
o r e
BeamRiderNoFrameskip-v4 BootDQNHyperDQN 5M 10M 15M 20M
BerzerkNoFrameskip-v4 BootDQNHyperDQN 5M 10M 15M 20M 04
BowlingNoFrameskip-v
BootDQNHyperDQN 5M 10M 15M 20M
[Figure]
[Table]
[Figure]
o
[Figure]
S c
o r e
[Figure]
[Table]
[Figure] [Figure]
o d e S c o r e
[Figure]
p i s o d e S c o r
[Figure]
F D ISCUSSION F. W HY B AYESIAN DQN AND D IRECT E XTENSION OF RLSVI C OULD F AIL ?
Based on the algorithm description of RLSVI in Appendix B.1, one may ask whether we can directly extend RLSVI for deep RL.
In fact, Bayesian DQN (Azizzadenesheli et al., 2018) is a “deep” version of RLSVI (Osband et al., 2016b; 2019).
In particular, Bayesian DQN leverage the raw feature outputted by a randomly initialized neural network to implement RLSVI.
We hope the empirical results presented in this part could help explain why RLSVI could not be directly applied in the deep
RL scenario and why the direct extension in (Azizzadenesheli et al., 2018) is not satisfying.
Like RLSVI, Bayesian DQN repeatedly takes two steps for exploration.
The ﬁrst step is to solve a Bayesian linear regression problem by ﬁxing the feature representation φ θ hidden (i.e., the hidden layers of the neural network).
The other step is to update the hidden layers by a temporal difference learning objective; see (F.1).
Speciﬁcally, it randomly samples a θ predict from the posterior distribution to substantiate the prediction layer.
And, the target network uses the posterior mean θ predict to substantiate the prediction layer.
Finally, these two steps are performed alternatively to train the feature representation and the posterior samples.
[Equation]
where
[Equation]
where x [ i ] means the i -th element of the vector x and φ θ hidden is a delayed copy of φ θ
[Equation]
Note that Bayesian DQN is not evaluated in all Atari tasks.
Despite the empirical results on several tasks shown in (Azizzadenesheli et al., 2018), we ﬁnd two bugs in the ofﬁcial code 16 , which makes the conclusions in (Azizzadenesheli et al., 2018) less reliable.
Firstly, the authors wrongly implement the posterior sampling procedure; see Listing 1.
With the authors’ implementation, the code re-samples a posterior sample (i.e., a Q -value function) for each stage/period.
Clearly, this is against the algorithm description in (Azizzadenesheli et al., 2018) and violates the principle of commitment discussed in
Section 4.4.
The second issue is the implementation of the Bayesian linear regression update.
The authors seem to additionally multiply the prior noise scale “sigma” in line 9 of Listing 2, which is different from the closed-form solution in Appendix C.1.
Listing
1: Bug 1 of Bayesian DQN Code
feature covariance
Listing
: Bug 2 of Bayesian DQN Code
In addition to the above two bugs, Bayesian DQN updates the posterior distribution with the batch size of 100 K, which is very computationally inefﬁcient.
More concretely, the procedure requires at least 20 days to run even though the above two bugs are ﬁxed.
This computation burden is unbearable like RLSVI as discussed in Section 1.
[Figure]
To investigate whether the extension Bayesian DQN is applicable
[Figure]
we ﬁx the above two bugs and use a batch size of 2560 to perform the Bayesian linear regression update.
The performance of
Bayesian DQN on Atari is displayed in Figure 18.
It turns out Bayesian DQN does not work except for FreewayNoFrameskip-v4.
Furthermore, the performance of FreewayNoFrameskip-v4 is very poor (refer to the performance of baselines in Table 5)
[Figure]
To analyze why Bayesian DQN fails
[Figure]
we investigate the training logs on the deep sea task.
We choose this toy task because we can fully understand the algorithmic behaviors on the deep sea task.
The experimental results are shown in Figure
[Figure]
where we use our approach HyperDQN as a reference
[Figure]
The analysis is given in the following.
[Figure]
• First, from Figure 19a, we observe that Bayesian DQN could not ﬁnd the optimal Q -value, indicating its performance is poor though this task is not very hard (i
[Figure]
e., the height and width of the deep sea is 20 , refer to Figure 5a).
• Second, Figure 19b suggests after the initialization, the standard deviation of the Q -value functions generated by Bayesian DQN is large (the standard deviation is a proxy for the epistemic uncertainty).
However, it suddenly shrinks to a very small value.
The time slot of this event is about 2560 , which is exactly the ﬁrst update time of Bayesian DQN (recall that the batch size of Bayesian DQN is 2560 ).
In contrast, HyperDQN could update its
conﬁdence during the interaction; that is, the standard deviation diminishes as more data is collected.
The analysis for the sudden change in Bayesian DQN is analyzed below.
Third, we ﬁnd that the feature norm ( ≈ 0 .
1603 ) does not change too much over iterations.
This implies the feature representation remains as a raw feature outputted by a randomly initialized neural network even though more data is gathered.
On the other hand, HyperDQN could learn a task-relevant feature from the interaction.
Finally, we can explain the failure reason of Bayesian DQN by the empirical loss shown in Figure 19d.
After the initialization, the feature norm of the Bayesian DQN is small as displayed in Figure 19c.
As a result, the posterior covariance is small (which could be veriﬁed by the result in Figure 19b).
Accordingly, the TD loss is small and the feature representation does not change too much.
This implies the Bayesian DQN goes to a “dead lock”, in which its epistemic uncertainty measure (by the posterior distribution) is poor and the feature cannot be updated.
F. T RAINABILITY I SSUES OF H YPER DQN
In this part, we continue the discussion of the architecture design in Section 4.1.
In particular, we illustrate why the original architecture in (Dwaracherla et al., 2020) is not suitable for RL tasks.
As we have argued in Section 4.1, the main challenge of applying the architecture in (Dwaracherla et al., 2020) is parameter initialization.
To illustrate this issue, recall that we use a two-layer MLP neural network with the width of 64 for the deep sea task.
Suppose we use the default PyTorch initialization 17 (i.e., sampling from the uniform distribution U [ − 1 / (cid:
) d i − 1 , 1 / (cid:112) d
i − 1 ] ), which is also known as “ 1 / √ f in ” with f in being the input dimension of the i -th layer.
We list the parameter information in Table 6 (recall that the state dimension is 2500 and the action dimension is 2 for the deep sea task).
We see that the expected magnitude of the parameter differs over layers.
In particular, the desired magnitude is quite small compared with the magnitude of the output of the hypermodel ( ≈ 1 ).
As a result, if we directly apply the hypermodel for all layers of the base model, the input signal over layers explodes and the gradient is quite large.
Thus, training fails when we use the architecture in (Dwaracherla et al., 2020).
Furthermore, this parameter initialization issue becomes more severe when we use deep convolution neural networks for Atari and SuperMarioBros tasks.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Readers may argue that we can normalize the output of the hypermodel by dividing proper normalization constants for each layer.
Unfortunately, we ﬁnd this variant does not work, either.
We provide the numerical evidence in Figure 20.
We observe that Q -values are regular at the initial stage by this trick but they still explode after some iterations (i.e., the magnitude could be 10 7 ).
That is, even though this trick alleviates this issue but the landscape of the normalized objective function is still ill-condition.
Combined with the changing target (due to temporal difference learning) in RL rather than the ﬁxed target (in bandit), the instability occurs as shown in Figure 20.
In summary, the architecture in (Dwaracherla et al., 2020) is hard to be applied for deep RL due to the parameter initialization and training instability issues.
F. W HY (cid:15) - GREEDY I S C OMMONLY U SED IN M ANY E FFICIENT A LGORITHMS ?
We realize many efﬁcient algorithms (Osband et al., 2016a; Rashid et al., 2020; Bai et al., 2021) still use (cid:15) -greedy in practice.
However, we argue that (cid:15) -greedy is not efﬁcient since it cannot write off sub-optimal actions after experimentation.
In addition, combining with (cid:15) -greedy violates the principle of commitment as discussed in Section 4.4.
We conjecture existing practical algorithms combine their exploration strategies with (cid:15) -greedy mainly due to imperfect imitation of theoretical algorithms, which is analyzed below.
On the one hand, we believe BootDQN (Osband et al., 2016a) uses (cid:15) -greedy is to improve the diversity since ﬁnite ensembles could offer limited diverse action sequences.
For instance, if the number of actions is larger than the number of ensembles, BootDQN cannot produce all possible action sequences to explore at the initial stage.
In this case, (cid:15) -greedy is somehow helpful.
Another reason is based on historical development.
That is, BootDQN just follows the default setting of DQN (Mnih et al., 2015).
In Figure 14, we see that the variant that drops the (cid:15) -greedy could be better than the original implementation of BootDQN in some tasks.
On the other hand, there are many practical issues to implement OFU-based algorithms in real applications, as discussed in Section 3.
In fact, directly adding an exploration bonus could yield unexpected outcomes.
Let us (re-)illustrate this using the example of the deep sea (shown in Figure 5a).
Imagine the agent visits a certain path to collect the data; for instance, the red path in Figure 21.
With the experience replay buffer, the agent can only update the Q -value function with state-action pairs from the visited path.
By exploration bonus, such state-action sequences would have larger
Q -values.
During the next episode, the agent would repeat the same action sequences by following the greedy policy.
It turns out that theoretical algorithms like (Jin et al., 2018) do not have such an issue.
This is because theoretically, the algorithm can use an optimistic initialization.
With the optimistic initialization, the algorithm instead would select other actions rather than visited actions.
In practice, however, it is not easy to obtain an optimistic initialization with deep neural networks (Rashid et al., 2020).
Again, (cid:15) -greedy could somehow avoid this issue.
Published as a conference paper at ICLR
[Figure]
[Table]
At each timestep there is a small cost r = ≠ 0 . 01 /N of movingright, and r =0formovingleft.
However, shouldtheagenttransitionrightatevery timestepoftheepisodeitwillberewardedwithanadditionalrewardof+1.
Thispresentsa particularly challenging exploration problem for two reasons.
First, following the ‘gradient’ of small intermediate rewards leads the agent away from the optimal policy.
Second, a policy that explores with actions uniformly at random has probability 2 ≠ N of reaching the rewarding state in any episode.
For the bsuite experiment we run the agent on sizes N = 10 , 12 ,.., 50 and look at the average regret compared to optimal after 10k episodes.
The summary ‘score’ computes the percentage of runs for which the average regret drops below 0.9 faster than the 2 N episodes expected by dithering.
Deep Sea is a good bsuite experiment because it is targeted, simple, challenging, scalable and fast.
By construction, an agent that performs well on this task has mastered some key properties of deep exploration.
Our summary score provides a ‘quick and dirty’ way to compareagentperformanceatahighlevel.
Oursweepoverdierentsizes N canhelptopro- vide empirical evidence of the of
/fail.
Published as a conference paper at ICLR
[Figure]
consequences of its actions towards cumulative rewards, an agent seeking to ‘explore’ must consider how its actions can position it to learn more eectively in future timesteps.
The literature on ecient exploration broadly states that only agents that perform deep exploration can expect polynomial sample complexity in learning (Kearns & Singh, 2002).
This literature has focused, for the most part, on uncovering possible strategies for deep explo- ration through studying the tabular setting analytically (Jaksch et al., 2010; Azar et al., 2017).
Our approach in bsuite is to complement this understanding through a series of behavioural experiments that highlight the need for ecient exploration.
The deep sea problem is implemented as an N ◊ N grid with a one-hot encoding for state.
The agent begins each episode in the top left corner of the grid and descends one row per timestep.
Each episode terminates after N steps, when the agent reaches the bottom row.
In each state there is a random but ﬁxed mapping between actions
A = { 0 , 1 } and the transitions ‘left’ and ‘right’.
At each timestep there is a small cost r = ≠ 0 . 01 /N of movingright, and r =0formovingleft.
However, shouldtheagenttransitionrightatevery timestepoftheepisodeitwillberewardedwithanadditionalrewardof+1.
Thispresentsa particularly challenging exploration problem for two reasons.
First, following the ‘gradient’ of small intermediate rewards leads the agent away from the optimal policy.
Second, a policy that explores with actions uniformly at random has probability 2 ≠ N of reaching the rewarding state in any episode.
For the bsuite experiment we run the agent on sizes N = 10 , 12 ,.., 50 and look at the average regret compared to optimal after 10k episodes.
The summary ‘score’ computes the percentage of runs for which the average regret drops below 0.9 faster than the 2 N episodes expected by dithering.
[Table]
[Table] [Table]
/OmVmPsr2bs/Vt/cUqDWvmYa7N8Gp2SRfsfb/On2B3tey5Za/mlSoVvI9hzGEei3Sfa6hgG1XUKfsSd7jHg3PhXDnXzs271Cnknll8Gc7tG1fup94=</latexit>
In addition to the initialization issue, the uncertainty should propagate over stages/periods with exploration bonus in a backward manner.
In (Jin et al., 2018), this operation can be achieved by dynamic programming.
In contrast, practical algorithms randomly draw mini-batch from the experience buffer to update.
This indeed may not properly propagate the uncertainty.
As a result, the Q -values are not always optimistic, and induced action sequences may not be diverse.
This phenomenon is pointed out in (Bai et al., 2021).
Again, (cid:15) -greedy may be helpful to address this issue.
F. C OMPARISON WITH N OISY N ET
In this part, we explain the differences between HyperDQN and NoisyNet in detail.
The following discussion aims to provide insights about what HyperDQN can achieve while NoisyNet cannot .
Note that we are by no means criticizing NoisyNet.
Instead, NoisyNet is simple and has strong empirical performance.
We hope the above discussion could provide intuitions (or explanations) why NoisyNet successes or fails under different cases.
• First, as remarked in (Fortunato et al., 2018), NoisyNet is not ensured to approximate the posterior distribution of parameters.
Therefore, NoisyNet is not a typical Thompson sampling based algorithm.
The implication is that we cannot use the well-known theory (Osband et al., 2013; 2019; Zanette et al., 2020) to analyze NoisyNet.
• Second, NoisyNet re-samples a new policy every time step (Line 5 of Algorithm 1 in (Fortunato et al., 2018)).
Consequently, it does not implement deep exploration like RLSVI (see (Osband, 2016b, Section 4.1)).
This explains the empirical result that NoisyNet can not solve the deep sea task when the problem size is large than 20 (see Figure
(Osband et al., 2018, Figure 9)).
Instead, BootDQN and HyperDQN can solve the deep sea task even if the problem size is large than 20 as shown in Appendix E.3.
• Third, NoisyNet does not have the mechanism of “prior” (Osband et al., 2018) even though it is randomized.
As a consequence, NoisyNet cannot leverage an informative prior to accelerate exploration when such information is available.
In contrast, HyperDQN can achieve this goal as discussed in Appendix F.6.
Next, we conduct experiments to empirically compare NoisyNet and HyperDQN.
Since there is no public implementation of NoisyNet, we reproduce NoisyNet according to the description in (Fortunato et al., 2018, Algorithm 1) and the hyper-parameters in (Fortunato et al., 2018, Section 3.2).
The neural network architecture and learning parameters are the same as the ones described in Appendix D.
The empirical results on Atari and SuperMarioBros are shown in Figure 22 and
Figure 23, respectively.
For tasks in the Atari suite (see Figure 22), the reproduced results of Noisy basically match the results reported in (Fortunato et al., 2018, Figure 6).
In particular, both methods do not work for Montezuma’s Revenge.
On the one hand, NoisyNet is better than HyperDQN on Beam Rider and Seaquest.
On the other hand, HyperDQN outperforms NoisyNet on the remaining 6 tasks including Alien, Amidar, Battle Zone, Pong, Venture, and Zaxxon.
For tasks in the SuperMarioBros suite (see Figure 23), NoisyNet is only better than HyperDQN on SuperMarioBros-1-1.
They are competitive on SuperMarioBros-2-3 and SuperMarioBros-3-3.
For the remaining 6 tasks, NoisyNet is inferior to HyperDQN. In particular, on the hard exploration task SuperMarioBros-1-3 and SuperMarioBros-2-2, HyperDQN performs better than NoisyNet since these tasks require deep exploration while NoisyNet is hard to generate “far-sighted” behaviors.
In summary, HyperDQN is better than NoisyNet on 12 out of 18 tasks considered here.
We are still running more experiments and we will provide full results in the later revision.
F. E XTENSION TO C ONTINUOUS C ONTROL
In this part, we brieﬂy discuss how to extend the idea in this paper for continuous control tasks.
We also present some preliminary results to support this direction.
Following the idea presented in Section 4.3, we should leverage the hypermodel to capture the posterior distribution of the Q -value function.
Considering the standard actor-critic methods, we can replace the vanilla critic with the one that is built by the hypermodel.
In particular, we replace the last layer of the critic with a hypermodel.
In this way, each z corresponds to a speciﬁc critic from the posterior distribution.
To perform policy optimization, we notice that each actor (i.e., a policy network) should be greedy to a speciﬁc critic with the same index z .
To this end, the last layer of the actor network is also built by a hypermodel.
The architecture design is illustrated in Figure 24.
Let ( θ critichidden , ν critic ) be the parameter for the critic network and ( θ actorhidden , ν actor ) be the parameter for the actor network.
Following the optimization framework in (Lillicrap et al., 2016; Haarnoja et al., 2018),
[Figure]
[Figure]
i s o d
[Figure]
c o r
[Figure]
(cid:68)(cid:70)(cid:87)(cid:82)(cid:85)(cid:3)(cid:81)(cid:72)(cid:87)(cid:90)(cid:82)(cid:85)(cid:78) (cid:70)(cid:85)(cid:76)(cid:87)(cid:76)(cid:70)(cid:3)(cid:81)(cid:72)(cid:87)(cid:90)(cid:82)(cid:85)(cid:78)
Algorithm for episode k = 0 , 1 , 2 , · · · do generate a random vector z ∼ N (0 , I ) .
(cid:46)
Sampling instantiate an actor π θ ( · ; z ) .
for stage t = 0 , 1 , 2 , · · · , T − 1 do (cid:46) Interaction observe state s t .
sample the action a t ∼ π θ ( s t ; z ) .
receive the next state s t +1 and reward r ( s t , a t ) .
sample ξ uniformly from unit hypersphere.
store ( s t , a t , r, ξ, s t +1 ) into the replay buffer D .
agent step n ← n + 1 .
if mod (agent step n , train frequency M ) =
=
then (cid:46) Update sample a mini-batch (cid:101) D of ( s, a, r, ξ, s (cid:48) ) from the replay buffer D .
sample N random vectors z (cid:48)
i : (cid:101)
Z = { z (cid:48)
i } Ni =1 .
optimize the critic network using the loss function (F.3) with (cid:101) D and (cid:101) Z .
optimize the actor network using the loss function (F.2) with (cid:101) D and (cid:101) Z .
update the critic target network with exponential moving average.
end if end for end for the training objective for the actor network is
[Equation]
a
[Equation] with a z ∼ π θ actor hidden ,ν actor ( s ; z [Equation] , (F.
[Equation]
where Q θ critic hidden ,ν critic is the critic network and π θ actor θ hidden ,ν actor is the actor network.
In particular, objective (F.2) states that we should sample actions from the actor network to maximize the Q -value function.
The difference with the traditional actor-critic methods is that we sample many actor networks (indexed by z ) to optimize.
Similarly, the training objective for the critic network is:
[Equation]
(F.3) with a (cid:48)
z ∼ π θ actorhidden ,ν actor ( s (cid:48) , z ) .
In (F.3), Q θ critichidden ,ν critic is the target network.
The goal of (F.3) is to perform the temporal difference learning to optimize the critic.
Note that the prior regularization term is not appeared in (F.3) for easy presentation.
We call the above method as HyperActorCritic(HAC).
Its implementation is outlined in Algorithm 3, which shares many features with Algorithm 2.
Now, we consider the Cart Pole task from dm control 18 (Tunyasuvunakool et al., 2020) as a testbed; see Figure 25a .
Detailed environment information is provided as follows: the dimension of the state space is 5 , the dimension of the action space is 1 , the planning horizon is 1000 and the reward is between 0 and 1 .
This environment is a standard platform to test the exploration efﬁciency for continuous control algorithms because the agent can only obtain +1 reward when it succeeds and obtain 0 reward otherwise.
We compare our extension (HAC) with the strong baseline SAC (Haarnoja et al., 2018).
Even though the policy entropy reward is used to guide exploration, SAC does not have the epistemic uncertainty qualiﬁcation like HAC.
As a result, we expect SAC does not perform well for Cart Pole.
The numerical result is displayed in Figure 25b.
We see that HAC outperforms SAC on this hard exploration task.
from dm_control.suite.wrappers import pixels env = suite.load("cartpole", "swingup") # Replace existing features by pixel observations: env_only_pixels =
pixels.
Wrapper(env)
# Pixel observations in addition to existing features.
env_plus_pixels =
pixels.
Wrapper(env, pixels_only=False) Reward visualisation: Models in the Control Suite use a common set of colours and textures for visual uniformity.
As illustrated in the video, this also allows us to modify colours in proportion to the reward, providing a convenient visual cue.
env = suite.load("fish", "swim", task_kwargs, visualize_reward=True) 6.
Domains and Tasks A domain refers to a physical model, while a task refers to an instance of that model with a particular MDP structure.
For example the diﬀerence between the swingup and balance tasks of the cartpole domain is whether the pole is ini- tialised pointing downwards or upwards, respectively.
In some cases, e.g. when the model is procedurally generated, diﬀerent tasks might have diﬀerent physical proper- ties.
Tasks in the Control Suite are collated into tuples according to predeﬁned tags.
Tasks used for benchmarking are in the BENCHMARKING tuple (Figure 1), while those not used for benchmarking (because they are particularly diﬃcult, or because they do not conform to the standard structure) are in the EXTRA tuple.
All suite tasks are accessible via the ALL_TASKS tuple.
In the domain descriptions below, names are followed by three integers specifying the dimensions of the state , control and observation spaces: Name (cid:16) dim( S ) , dim( A ) , dim( O ) (cid:17) .
Pendulum (2, 1, 3):
The classic inverted pendulum.
The torque- limited actuator is 1 / 6 th as strong as required to lift the mass from motionless horizontal, necessitating several swings to swing up and balance.
The swingup task has a simple sparse reward: 1 when the pole is within 30 ◦ of the vertical position and 0 otherwise.
Acrobot (4, 1, 6): The underactuated double pendulum, torque applied to the second joint.
The goal is to swing up and balance.
Despite being low-dimensional, this is not an easy control problem.
The physical model conforms to (Coulom, 2002) rather than the earlier Spong 1995.
The swingup and swingup_sparse tasks have smooth and sparse rewards, respectively.
Cart-pole (4, 1, 5): Swing up and balance an unactuated pole by applying forces to a cart at its base.
The physical model conforms to Barto et al. 1983.
Four benchmarking tasks: in swingup and swingup_sparse the pole starts pointing down while in balance and balance_sparse the pole starts near the upright position.
Cart-k-pole (2k + 2 , 1 , 3k + 2): The cart-pole domain allows to procedurally adding more poles, connected serially.
Two non-benchmarking tasks, two_poles and three_poles are available.
[Figure]
In this part, we brieﬂy discuss the role of prior value functions in our formulation. In particular, we argue that if an informative prior value function is available, the exploration efﬁciency can be signiﬁcantly improved.
[Figure]
Here we show the learning curve with an informative prior in Figure 26.
In particular, this informative prior value function is obtained from the pre-trained model after 20 M frames.
We see that this informative prior value function improves efﬁciency a lot.
Note that the main goal of this experiment is to illustrate that an informative prior could accelerate exploration.
We will consider how to automatically acquire an informative prior with human demonstrations (Hester et al., 2018; Sun et al., 2018) in the future.