Lifting the Veil on Hyper-parameters for Value-based Deep Reinforcement Learning
Abstract
Successful applications of deep reinforcement learning (deep RL) combine algorithmic design and careful hyper-parameter selection.
The former often comes from iterative improvements over existing algorithms, while the latter is either inherited from prior methods or tuned for the speciﬁc method being introduced.
Although critical to a method’s performance, the effect of the various hyper-parameter choices are often overlooked in favour of algorithmic advances.
In this paper, we perform an initial empirical investigation into a number of often-overlooked hyperparameters for value-based deep RL agents, demonstrating their varying levels of importance.
We conduct this study on a varied set of classic control environments which helps highlight the effect each environment has on an algorithm’s hyper-parameter sensitivity.
Introduction
Deep reinforcement learning (deep RL) is a burgeoning research area with an astounding number of theoretical and empirical advances.
Algorithmic progress is usually measured by comparing to pre-existing baselines on a set of established benchmarks, where the proposed method is typically evaluated using a single (or small set of) hyper-parameter choices.
While this approach helps demonstrate the potential of new algorithms, they are less effective at demonstrating their robustness to varying hyper-parameters, especially those that have been "inherited" from prior methods.
We will refer to these types of publications as academic deep RL .
There have been a number of recent success stories in applying deep RL to large-scale real-world tasks [Silver et al., 2016, Bellemare et al., 2020, Mirhoseini et al., 2021] but in all these cases very speciﬁc design decisions (including hyper-parameter choices) were necessary for them to work effectively; often, differing from the design choices made in the algorithms leveraged by these works.
We will refer to these types of publications as applied deep RL .
Although tackling the same underlying problem, there exists a gap between academic and applied deep RL.
While progress in the applied side leverages advances made in the academic side, it typically requires a group of highly-specialized researchers to successfully adapt the academic insights into a workable agent; this produces a de-facto barrier for non-academics to successfully apply deep RL to their problem.
On the other hand, many of the components ultimately used for large-scale real-world problems tend to be speciﬁc, resulting in less-than-ideal academic uptake.
I n i t i a li z a t i o n A c t
i v a t i o n F n N o r m a li z a t i o n W i d t
h D e p t h
L e a r n i n g r a t e O p
t
i m i z e r
R e w a r d c li pp i n g
D i s c o un t
f a c t o r
M i n r e p l a y h
i s t o r y
U pd a t e h
o r i z
o n
O n li n e u pd a t e p e r i o d
T a r g e t u pd a t e p e r i o d
N u m b e r o f a t
o m s
F i n a l r e t u r n
Acrobot
I n i t i a li z a t i o n A c t
i v a t i o n F n N o r m a li z a t i o n W i d t
h D e p t h
L e a r n i n g r a t e O p
t
i m i z e r
R e w a r d c li pp i n g
D i s c o un t
f a c t o r
M i n r e p l a y h
i s t o r y
U pd a t e h
o r i z
o n
O n li n e u pd a t e p e r i o d
T a r g e t u pd a t e p e r i o d
N u m b e r o f a t
o m s
[Table] [Table]
[Table]
CartPole
I n i t i a li z a t i o n A c t
i v a t i o n F n N o r m a li z a t i o n W i d t
h D e p t h
L e a r n i n g r a t e O p
t
i m i z e r
R e w a r d c li pp i n g
D i s c o un t
f a c t o r
M i n r e p l a y h
i s t o r y
U pd a t e h
o r i z
o n
O n li n e u pd a t e p e r i o d
T a r g e t u pd a t e p e r i o d
N u m b e r o f a t o m s
LunarLander
DQN
Rainbow
I n i t i a li z a t i o n A c t
i v a t i o n F n N o r m a li z a t i o n W i d t
h D e p t h
L e a r n i n g r a t e O p
t
i m i z e r
R e w a r d c li pp i n g
D i s c o un t
f a c t o r
M i n r e p l a y h
i s t o r y
U pd a t e h
o r i z
o n
O n li n e u pd a t e p e r i o d
T a r g e t u pd a t e p e r i o d
N u m b e r o f a t
o m s
MountainCar
I n i t i a li z a t i o n A c t
i v a t i o n F n N o r m a li z a t i o n W i d t
h D e p t h
L e a r n i n g r a t e O p
t
i m i z e r
R e w a r d c li pp i n g
D i s c o u n t
f
a c t o r
M i n r e p l a y h
i s t o r y
U pd a t e h
o r i z
o n
O n li n e u pd a t e p e r i o d
T a r g e t u pd a t e p e r i o d
N u m b e r o f a t
o m s
F i n a l r e t u r n
Asterix
I n i t i a li z a t i o n A c t
i v a t i o n F n N o r m a li z a t i o n W i d t
h D e p t h
L e a r n i n g r a t e O p
t
i m i z e r
R e w a r d c li pp i n g
D i s c o u n t
f
a c t o r
M i n r e p l a y h
i s t o r y
U pd a t e h
o r i z
o n
O n li n e u pd a t e p e r i o d
T a r g e t u pd a t e p e r i o d
N u m b e r o f a t
o m s
Breakout
I n i t i a li z a t i o n A c t
i v a t i o n F n N o r m a li z a t i o n W i d t
h D e p t h
L e a r n i n g r a t e O p
t
i m i z e r
R e w a r d c li pp i n g
D i s c o u n t
f
a c t o r
M i n r e p l a y h
i s t o r y
U pd a t e h
o r i z
o n
O n li n e u pd a t e p e r i o d
T a r g e t u pd a t e p e r i o d
N u m b e r o f a t o m s
Freeway
I n i t i a li z a t i o n A c t
i v a t i o n F n N o r m a li z a t i o n W i d t
h D e p t h
L e a r n i n g r a t e O p
t
i m i z e r
R e w a r d c li pp i n g
D i s c o u n t
f
a c t o r
M i n r e p l a y h
i s t o r y
U pd a t e h
o r i z
o n
O n li n e u pd a t e p e r i o d
T a r g e t u pd a t e p e r i o d
N u m b e r o f a t o m s
SpaceInvaders
I n i t i a li z a t i o n A c t
i v a t i o n F n N o r m a li z a t i o n W i d t
h D e p t h
L e a r n i n g r a t e O p
t
i m i z e r
R e w a r d c li pp i n g
D i s c o u n t
f
a c t o r
M i n r e p l a y h
i s t o r y
U pd a t e h
o r i z
o n
O n li n e u pd a t e p e r i o d
T a r g e t u pd a t e p e r i o d
N u m b e r o f a t o m s
Seaquest
We argue that a key contributor to the gap between academic and applied deep RL is a poor empirical understanding of the impact of the aforementioned hyper-parameters on the performance of the overall algorithm.
In this paper, we begin to "lift the veil" on some commonly overlooked hyper-parameters for value-based deep RL in the hope that they help bridge this gap.
We aim to (at least partially) answer the following questions:
How sensitive are deep reinforcement learning algorithms to variations in hyper-parameter selection?
Does the choice of environment vary the sensitivity to hyper-parameter selection?
Are existing methods under-performing due to a poor hyper-parameter choice?
Background
Reinforcement learning problems are typically formulated as a Markov decision process (MDP), which consists of a 5-tuple (cid:104)X , A , R, P , γ (cid:105) , where X denotes the (possibly inﬁnite) state space, A denotes a ﬁnite set of actions, R :
X ×
A → R is the reward function, P :
X ×
A → Dist ( X ) encodes the transition dynamics (often written as P ( x (cid:48)
| x, a ) ), and γ ∈ [0 , 1) is a discount factor.
The principal objective of reinforcement learning is to learn a behaviour policy π : X → Dist ( A ) that maximizes the discounted sum of expected rewards.
While there are a number of valid approaches (see, e.g. Sutton and Barto [1998]), in this paper we focus on value-based methods.
These aim to learn an approximation to the so-called Q ∗ -values, deﬁned via the Bellman recurrence:
Q ∗ ( x, a ) :=
R ( x, a ) + γ E x (cid:48) ∼P
( x,a ) [max a (cid:48) ∈A Q ∗ ( x (cid:48) , a (cid:48) )] .
The optimal policy π ∗ can then be obtained from Q ∗ as π ∗ ( x ) := max a ∈A Q ∗ ( x, a ) .
One of the most common approaches for learning Q ∗ is via the method of temporal differences: given an estimate Q and a transition tuple ( x, a, r, x (cid:48) ) , we can obtain a new estimate via: Q ( x, a ) ←
Q ( x, a ) + α [ r + γ max a (cid:48) ∈A Q ( x (cid:48) , a (cid:48) )
− Q ( x, a )] , where α is a learning rate.
We often refer to the term [ r + γ max a (cid:48) ∈A Q ( x (cid:48) , a (cid:48) )] as the Bellman target .
2.1 DQN
Mnih et al.
[2015] deﬁned DQN by combining temporal-difference learning with deep networks, and demonstrated its capabilities in achieving super-human performance on the Arcade Learning
Environment (ALE)
[Bellemare et al., 2012].
Speciﬁcally, a deep network, parameterized by a vector θ , was trained to approximate Q ∗ : Q θ ≈ Q ∗ .
Mnih et al.
[2015] introduced two key components that helped stabilize the learning process.
The ﬁrst was the use of a large replay buffer to store experienced transitions and, after collecting a sufﬁcient number of transitions (referred to as the min replay history ), use samples from that buffer to update the network.
By having many transitions in the buffer and sampling from it one can reduce the dependency between the elements of a training batch, which helps with neural network training, in addition to leveraging speedups from hardware like GPUs.
However, this results in off-policy learning, which means that the agent is learning from experience obtained from a different (e.g. older) policy than the policy it is currently using to act.
Exploring the difﬁculties in off-policy learning is an active area of research; nonetheless, the use of replay buffers is ubiquitous in deep RL.
The second was the use of a target network (parameterized by ¯ θ ), in addition to the main online network, for stabler bootstrapping targets.
The Q -update then becomes
[Equation]
The target network parameters are not updated by gradient descent, but rather they are synchronized with the online parameters at a lower frequency (e.g. ¯ θ ← θ ).
We refer to the frequency of online network updates (via gradient descent) as the online update period , and the frequency of the target network updates as the target update period .
2.2 Rainbow
Although DQN benchmarked on the 57 ALE games with the same set of hyper-parameters, Anschel et al.
[2017] and Cini et al.
[2020] demonstrated that in some environments it can prove to be rather unstable resulting in degraded performance.
There were a number of papers that improved on it to improve its stability and performance.
Hessel et al.
[2018] combined many of these into a single agent they called "Rainbow".
Speciﬁcally, they combined DQN with double Q-learning [van Hasselt et al., 2016b], prioritized experience replay [Schaul et al., 2016], dueling networks [Wang et al., 2016], multi-step learning [Sutton, 1988], noisy nets [Fortunato et al., 2018], and distributional reinforcement learning [Bellemare et al., 2017].
2.3 Experimental details
As suggested by Obando-Ceron and Castro [2021], we evaluate on four classic control environments (CartPole, Acrobot, LunarLander, and MountainCar) 2 , which can be useful for conducting thorough investigations with numerous independent runs.
Our implementation is based on the Dopamine framework [Castro et al., 2018], but the default hyper-parameter settings used for the online experiments are those speciﬁed by Obando-Ceron and Castro [2021].
All experiments were run on a CPU, and we report the mean and 95% conﬁdence intervals, averaged over 30 independent seeds.
We also evaluated on the MinAtar environment
[Young and Tian, 2019], as they can help shed some light into the effect of convolutional layers and more complex environments.
For the MinAtar experiments we use the default settings suggested by Obando-Ceron and Castro [2021], except where noted.
Given the increased computational complexity of this suite, we ran each setting withindependent seeds (each on a separate GPU), and we report the mean 75% conﬁdence intervals.
Lifting the veil
We organize our experiments into three groups of hyper-parameters: network components (subsection 3.1), optimization hyper-parameters (subsection 3.2), and algorithmic parameters (subsection 3.3).
Figure 1 presents an overall summary of these experiments, and we will present
[Figure]
l r
[Figure]
l r
[Figure]
n
[Figure]
n W i d t
h D e p t
h
SpaceInvaders
I n i t i a li z a t i o n A c t
i v a t i o n F n N o r m a li z a t i o n W i d t
h D e p t h
Seaquest and discuss each individual group below.
Due to space constraints we include all the ﬁgures in Appendix A and include only aggregate plots in the main paper.
All our code is available at https://github.com/JohanSamir/rainbow_extend.
3.1 Network components
The aggregate results for these experiments are summarized in Figure 2.
Initializations Initialization is the term used to deﬁne the initial values for the parameters of a neural network.
While there exists a broad literature studying initialization strategies for neural networks [Glorot and Bengio, 2010, He et al., 2015, Saxe et al., 2014].
With a few exceptions, these strategies have not been studied in detail in deep reinforcement learning [Andrychowicz et al., 2020,
Hussenot et al., 2021, Paine et al., 2020].
In Figure 5 and Figure 6 we can observe that, for the most part, there is little difference when varying the initialization scheme.
This is somewhat expected due to RL’s "self-correcting nature"; that is, it is not optimizing towards a ﬁxed target, but rather a shifting one (as a consequence of bootstrapping).
There are two notable exceptions worth mentioning:
(1) All-zeros and all-ones initialization almost always fails to learn with DQN.
This is most likely due to zero gradients occurring as a consequence of the combination of constant predictions with little reward variability.
In Rainbow this issue is not present, perhaps due to the presence of noisy networks (which break the constancy of predictions).
(2) DQN on MountainCar seems to be rather sensitive to the choice of initialization.
This is likely due to the properties of this particular environment, which we discuss below.
In Figure 7 we can see that we have the same level of stability in the MinAtar environments.
Non-linear activation functions are a fundamental part of Deep Neural Networks, as their removal effectively turns the network into a linear function approximator.
Many different activation functions have been proposed for different settings ([Devlin et al., 2019, Elfwing et al., 2018, Dauphin et al., 2017]), yet it’s rare to see comparisons between the many possible options [Shamir et al., 2020] and, to the best of our knowledge, there are no previous examples of doing such comparison in the Reinforcement Learning setting.
In Figure 8 (and in Figure 10 for MinAtar environments) we can see that there is little difference across the top activation functions, with DQN on MountainCar once again being a notable exception.
However, in Figure 9 we can see that there is a fair bit of variability across the 16 possible activation
[Figure]
functions. Of note is how sensitive Rainbow seems to be to these on CartPole, which is arguably the simplest of all the environments.
A more detailed investigation of the effect (and interaction with other components) of activation functions is an interesting avenue for future work.
Normalization Normalization plays an important role in various deep learning applications [Figure] and it is well established in supervised learning [Tan and Le, 2020, Xie et al., 2017].
Surprisingly, the use of normalization is fairly rare in deep reinforcement learning, with a few exceptions [Bhatt et al., 2019, Arpit et al., 2019, Lillicrap et al., 2019, Silver et al., 2017].
We explore two types of normalization: batch normalization [Ioffe and Szegedy, 2015] and layer normalization [Ba et al., 2016].
In Figure 11 we can see that although there are some minor differences, there appears to be little gain to using normalization.
This may very well be a consequence of the simplicity of the environments (indeed, the biggest differences are observed in MountainCar) and the shallowness of the networks we’re using.
Although batch normalization does not seem to provide clear beneﬁts, it also does not seem to hurt (with the exception of MountainCar), a ﬁnding that is somewhat at odds with that of Salimans and Kingma [2016], who afﬁrm that batch normalization is not well suited for
Deep Reinforcement Learning.
We investigated this further on the MinAtar suite in Figure 12.
It is worth noting that since the default setting for MinAtar is to use a single convolutional layer with no dense layers, we added one dense layer (based on the depth results from Figure 14) and applied the normalization on this extra dense layer.
The results provide even more evidence that there is little gain to using normalization, and this can in fact hurt performance.
It is worth investigating whether applying normalization to the convolutional layer can help address this.
Network capacity Improvements in deep learning architectures have played a vital role in moving forward the state of supervised and unsupervised learning in computer vision, but neural network architecture design for deep reinforcement learning is relatively unexplored, with a few exceptions [Sinha et al., 2020, Andrychowicz et al., 2020].
We vary network capacity via the depth (e.g. the number of hidden layers) and the width (e.g. the number of neurons) of each hidden layer.
In Figure 13 we observe an interesting phenomenon where fewer layers seems to help DQN (e.g. MoutainCar) whereas in Rainbow performance is correlated with the number of layers.
We hypothesize that this form of network capacity has a non-trivial interaction with the type of loss (e.g. expectational versus distributional), and may help further understand the difference between the two forms of TD-learning [Lyle et al., 2019].
Huh et al.
[2021] and Kumar et al.
[2020] demonstrated that deeper networks tend to be biased towards lower rank
R e w a r d c li pp i n g
D i s c o un t
f a c t o r
M i n r e p l a y h
i s t o r y
U pd a t e h
o r i z
o n
O n li n e u pd a t e p e r i o d
T a r g e t u pd a t e p e r i o d
N u m b e r o f a t o m
[Figure]
F i n a l r e t u r n
[Figure]
R e w a r d c li pp i n g
D i s c o un t
f a c t o r
M i n r e p l a y h
i s t o r y
[Figure]
h
[Figure]
r i
[Figure]
o
[Figure]
O
[Figure]
p e r
[Figure]
e p e r
[Figure]
N u m
[Figure]
r o f a
[Figure]
R e
[Figure]
a r d c
[Figure]
i n g
D i s c o un t
f a c t o r
M i n r e p l a y h
i s t o r y
U pd a t e h
o r i z
o n
O n li n e u pd a t e p e r i o d
T a r g e t u pd a t e p e r i o d
N u m b e r o f a t
o m s
[Figure]
LunarLander
[Figure]
R e w a r d c li pp i n g
D i s c o un t
f a c t o r
M i n r e p l a y h
i s t o r y
U pd a t e h
o r i z
o n
O n li n e u pd a t e p e r i o d
T a r g e t u pd a t e p e r
i
o
[Figure]
N u m
[Figure]
r o f a t
[Figure]
MountainCar
R e w a r d c li pp i n g
D i s c o un t
f a c t o r
M i n r e p l a y h
i s t o r y
U p
d a t e h
o r i z
o n
O n li n e u p d a t e p e r i o d
T a r g e t u p d a t e p e r i o d
N u m b e r o f a t
o m s
F i n a l r e t u r n
Asterix
R e w a r d c li pp i n g
D i s c o un t
f a c t o r
M i n r e p l a y h
i s t o r y
U p
d a t e h
o r i z
o n
O n li n e u p d a t e p e r i o d
T a r g e t u p d a t e p e r
[Table]
d
N u m
[Table]
f
[Table]
m s
[Table]
i n g
D
[Table]
t f a c t
o r
M i n r e p l a
[Table]
h
[Table]
U p
d a
[Table]
n
O n li n e u p d a t e p e r i o d
T a r g e t u p d a t e p e r i o d
N u m b e r o f a t
o m s
[Table] [Table]
R e w a r d c li pp i n g
D i s c o un t
f a c t o r
M i n r e p l a y h
i s t o r y
U p
d a t e h
o r i z
o n
O n li n e u p d a t e p e r i o d
T a r g e t u p d a t e p e r i o d
N u m b e r o f a t o m s
SpaceInvaders
R e w a r d c li pp i n g
D i s c o un t
f a c t o r
M i n r e p l a y h
i s t o r y
U p
d a t e h
o r i z
o n
O n li n e u p d a t e p e r i o d
T a r g e t u p d a t e p e r i o d
N u m b e r o f a t o m s
[Figure]
(e.g. simpler) solutions.
This may help explain why in DQN, shallower networks work best with
Mountaincar (the most complex off the four classic control environments).
In Figure 14 we investigate the effect of depth on the MinAtar suite.
The default setting for MinAtar is to use a single convolutional layer (depth = 0), so we investigated the effect of adding dense layers after the convolutional layer.
In DQN there is a clear beneﬁt to adding more layers, but the best results are obtained when adding only a single layer.
It is interesting to note that in Freeway, the simplest of the ﬁve games, the best performance is obtained without any extra layers.
This strongly suggests that there is a close relationship between network depth and environment complexity, where the best performance is obtained in a “goldilocks region”, and this region varies across environments.
In Rainbow, we can further observe how much this “goldilocks region” varies from game to game: from a wide region (Asterix) to a very narrow one (Seaquest).
In Figure 15 we can see that, in general, increasing width improves performance.
It is somewhat surprising the signiﬁcant differences observed with Rainbow in CartPole (arguably the easiest of the environments), yet there is no noticeable difference in DQN.
Given the results of the depth experiments, we explored varying width in the MinAtar environments by adding one dense layer after the convolutional layer.
In Figure 16 we see surprising variation across environments where performance is sometimes proportional to width (SpaceInvaders), and sometimes inversely proportional (Asterix and Freeway).
3.2 Optimizer hyper-parameters
Although the choice of optimizer is a design choice in itself, Obando-Ceron and Castro [2021] demonstrated the superior performance of the Adam optimizer [Kingma and Ba, 2015] relative to candidates such as RMSProp.
We explore two hyper-parameters of this optimizer: learning rate and epsilon values.
The aggregate results for these hyper-parameters are summarized in Figure 3.
In Figure 17 we can see that both algorithms are relatively robust to varying learning rates between 10 − 4 and 0 .
01 ; however in Figure 18 we explore a wider range and observe a deterioration in performance in both algorithms.
In Figure 19 we can see a fair bit of variation across the MinAtar environments for the learning rates considered, with the ordering based on performance varying between environments (e.g. Asterix versus Breakout on DQN).
We also observe that DQN seems to exhibit greater stability with larger (cid:15) values than Rainbow (see
Figure 20 and Figure 21).
To further investigate this observation, we ran the (cid:15) experiment on the
MinAtar suite (Figure 22).
We observe that the qualitative differences between DQN and Rainbow are mostly gone, suggesting that the observations in the classic control environment are perhaps due more to Rainbow’s stability on simpler environments.
3.3 Algorithmic parameters
There are a number of algorithmic design choices for RL agents that are often overlooked, but can play a signiﬁcant role in performance.
Hessel et al.
[2019] explored this to some extent, focusing on variants of the A2C algorithm
[Mnih et al., 2016].
The aggregate results for this group of hyper-parameters is presented in Figure 4.
Reward clipping Reward clipping was an important design decision for the original DQN algorithm, as it enabled normalizing scores across games [Mnih et al., 2015].
Clipping rewards changes the objective, which can result in qualitatively different learned behaviours.
This design decision has often been adopted by derivative algorithms for other domains without properly evaluating its efﬁcacy.
Indeed, Figure 23 suggests that the agents can actually perform better without reward clipping.
This is somewhat at odds with the ﬁndings of van Hasselt et al.
[2016a], where they found reward-clipping to be useful for learning.
In the MinAtar environments we do not see much difference between the two (Figure 24).
Discount factor The importance of γ has been observed in a number of recent works [Amit et al., 2020, Hessel et al., 2019, Gelada and Bellemare, 2019, van Seijen et al., 2019, François-Lavet et al., 2016], and in particular the discrepancy between the γ value used for training and the one used for evaluation .
As Figures 25 and 26 demonstrate, algorithmic performance is rather sensitive to the choice of γ ; a result consistent with the ﬁndings of Hessel et al.
In Figure 27 we evaluate three values of γ on the MinAtar environments and see a fair bit of variability across environments (compare DQN on SpaceInvaders versus Seaquest, for instance).
Minimum replay history As mentioned above, the agent stores its experience in a replay memory, from which it then samples mini-batches for learning.
It is common practice to only begin sampling from the replay buffer when a minimum number of transitions have been recorded: the minimum replay history.
The purpose of this parameter is to avoid overﬁtting to a sample set of samples at the early stages of training.
Perhaps, surprisingly, Figure 28 and Figure 29 suggest that the choice of this parameter has very little effect on the performance of either algorithm.
Update horizon Multi-step learning [Sutton, 1988] computes the temporal difference error using multi-step transition, instead of a single step.
DQN uses a single-step update by default, whereas
Rainbow chose a 3-step update.
In Figure 30 we compare various update horizons.
It is interesting to note that DQN is mostly unaffected by the update horizon, whereas Rainbow seems to have degraded performance with a single-step update horizon.
It thus seems that one of the Rainbow components beneﬁts from the multi-step update, an interesting question left for future work.
The update horizon has been argued to trade-off between the bias and the variance of the return estimate [Kearns and Singh, 2000].
This effect has been observed in the linear function approximation case, but it has not been very well studied with deep networks.
Hernandez-Garcia and Sutton [2019] perform a statistical analysis on the effect the update horizon has on the performance of six reinforcement algorithms in MountainCar and ﬁnd that the performance of each algorithm at the beginning of training was better with larger update horizons.
These ﬁndings are conﬁrmed by our results on MountainCar (Figure 30), but they are less evident on the other environments.
Indeed, DQN seems to perform worse with a larger update horizon on Acrobot.
These ﬁndings further highlight the sensitivity of the different DRL components relative to the environment on which they are run.
In Figure 31 we see this effect in high relief, where the beneﬁt of the update horizon seems to be closely tied with environmental complexity.
Compare, for instance, both algorithms on Freeway and
Seaquest: in the former, an update horizon of 1 performs best, while in the latter an update horizon of 10 yields a dramatic improvement for both algorithms.
Update periods For synchronous agents (like DQN and Rainbow), the update of the online network parameters is not done after every step taken in the environment; instead, it is performed at a frequency speciﬁed by the update period .
The default value used is 4 , which means that the online network parameters are only updated after every 4 steps taken in the environment.
The effect of varying the update period for the online network can be seen in Figure 32.
The results vary from environment to environment (compare the performance of an update period equal to 8 in
Acrobot and MountainCar with DQN); this variability is likely due to the complex learning dynamics of RL and merits further study.
Interestingly, this parameter does not seem to have much of an effect on the MinAtar environments (Figure 31).
As mentioned previously, the target network parameters are updated less frequently than the online parameters.
It has been observed that less frequent updates can result in improvements [HernandezGarcia and Sutton, 2019].
However, our ﬁndings in Figure 34 suggest that the best performance is obtained between values of 50-200, but any higher or lower results in decreased performance.
We also ﬁnd that DQN seems to be more sensitive to this choice than Rainbow in the classic control environments, but there is little variation for both DQN and Rainbow in the MinAtar environments (Figure 35).
Number of atoms One of the key components of the Rainbow algorithm is distributional reinforcement learning.
With this approach, the output layer predicts the distribution of the returns for each action a in a state s , instead of the mean Q π ( s, a ) .
Rainbow uses the distribution parameterization originally proposed by Bellemare et al.
[2017]; namely, representing the return distribution as a categorical distribution parameterized by N "atoms".
Bellemare et al.
[2017] found that empirically setting N = 51 proved best for the ALE; in Figure 36 we revisit this design choice.
Somewhat surprisingly, we observe little sensitivity to this choice in the four environments considered.
Obando-Ceron and Castro [2021] hypothesized that the role of the number of atoms may be larger when the deep networks include convolutional layers.
To investigate this, we repeated this experiment on the MinAtar suite (see Figure 37) and observe that, rather than being dependent on the use of convolutional layers, the role of this parameter is affected by the environment .
In particular, we observe very little difference between the number of atoms in Asterix, but see a clear difference in
SpaceInvaders, where more atoms results in better performance.
Discussion
There is a growing interest and concern in the effect of hyperparameter choice, and the ensuing reproducibiliity, for deep reinforcement learning.
Henderson et al.
[2019] and Islam et al.
[2017] highlight issues with reproducibility in RL, including performance differences between different code implementations, hyperparameters, and the high level of non-determinism due to random seeds.
Fu et al.
[2019] experimentally investigate potential issues of deep Q-learning algorithms.
Other large-scale studies similar to ours have been carried out by Andrychowicz et al.
[2020], Hussenot et al.
[2021], and Paine et al.
[2020], but not for online value-based methods.
Figure 1 summarizes our experiments for both agents.
We hearken back to the questions raised in the introduction and discuss them with respect to our ﬁndings.
How sensitive are DQN and Rainbow to variations in hyper-parameter selection?
While in aggregate the two agents have comparable sensitivity, there are some notable exceptions.
Network capacity seems to be strongly affected by environment complexity ; indeed, in the simpler classic control environments Rainbow has a higher sensitivity to over-parameterization, while in the more difﬁcult MinAtar suite both algorithms have noticeable sensitivity.
Rainbow seems to be more affected by the choice of (cid:15) for the Adam optimizer.
High values of (cid:15) make Adam behave more like
SGD with Momentum then as diagonal natural gradient descent [Choi et al., 2020]; this leads us to wonder about the relationship between the distributional loss and these different forms of optimization.
In the classic control environments, Rainbow seems to be unaffected by different choices of the number of atoms (in contrast with the original ﬁndings of Bellemare et al.
[2017]); this result may very well be a consequence of the relative simplicity of these environments, as further evidenced by the ﬁndings in MinAtar (Figure 37).
Does the choice of environment vary the sensitivity to hyper-parameter selection?
This is most certainly the case, as we observed stark qualitative differences between MountainCar and the other classic control environments.
Indeed, any differences between hyper-parameter values seem to be brought into high-relief when evaluated in MountainCar.
Additionally, a somewhat surprising ﬁnding is that CartPole (often considered the simplest of all four) proved quite effective at highlighting important qualitative differences between DQN and Rainbow (e.g. network width/depth, optimizer (cid:15) ).
We observed similar qualitative differences between the different MinAtar games (e.g. Figuredemonstrated the effect of number of atoms is best observed in SpaceInvaders, relative to the other games).
Some of our experiments are interesting in that their behavior is different from all other combinations of actor-environment:
• While for most environments initializations seemed to have little effect on the performance of our agents, we see a much higher effect when varying initializations on DQN on Mountaincar.
• Although in general changing activation functions did not impact performance signiﬁcantly, we can see that in both DQN on Mountaincar, and Rainbow on Breakout there are clearly activation functions that are better than the rest, most interestingly is how relu6 shows a clear improvement over all other activations in Rainbow on Breakout.
• The next striking behavior comes from the depth experiments in DQN on Mountaicar, where we see shallower networks being signiﬁcantly better than deeper networks, a result opposite to what we see in the other combinations of agents and classic control environments.
• The experimental results on MinAtar show that the optimal choice for a number of hyperparameters are quite sensitive to the environment itself.
Speciﬁcally, see the results and discussion on varying learning rates, layer width, γ , and the update horizon.
Are existing methods under-performing due to a poor hyper-parameter choice?
There are no canonical hyper-parameter values for the environments considered in this paper; the ones we have labeled as "default" are using the values provided by Obando-Ceron and Castro [2021].
However, some interesting ﬁndings that merit further investigation are:
• As mentioned previously, there seems to be a “goldilocks region” for network capacity that is conditional on the environment.
As we observed in Figure 13, a network with a single hidden layer seems to be sufﬁcient for DQN, and in fact yields improved performance on MountainCar.
On the other hand, in Figure 14 we see that deeper networks can help, but only up to a point .
• As discussed above, while reward clipping may be useful for the ALE, it is worth revisiting for any new environment as it can sometimes prove detrimental to performance.
• Our results suggest both algorithms are quite sensitive to the choice of γ , and this sensitivity varies across environments.
As such, this parameter must be selected with care for new environments.
• While 51 atoms may have proved best for the ALE experiments conducted by Bellemare et al.
[2017], our results suggest smaller values can sufﬁce for simpler environments.
However, this question can be somewhat sidestepped by using different parameterizations of the return distribution [Dabney et al., 2018a,b].
• Although we provided some insights into some aspects of the replay buffer and network updates, a more thorough investigation into the relationship between the size and frequency of updates, dubbed the replay ratio by Fedus et al.
[2020] is warranted.
The present work aims at investigating the importance of a broad set of hyper-parameters that need to be chosen when designing and implementing off-policy learning algorithms.
Although we found surprising insights on the classic control environments, we would like to run further experiments on the ALE
[Bellemare et al., 2012].
B Best Hyperparameters settings for DQN and Rainbow accros the environments
Table 1 shows the best values for each hyperparameter across all the classic control environments and
MinAtar games.
[Table]
C Hyperparameters settings for Classic control environments and MinAtar games
Table 2, 3, 4 list the choice conﬁgurations we used to run several experiments on classic control environments and MinAtar games.
[Table]
[Table] [Table]