V ARIANCE D OUBLE -D OWN : T HE S MALL B ATCH S IZE A NOMALY IN M ULTISTEP D EEP R EINFORCEMENT
L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In deep reinforcement learning, multi-step learning is almost unavoidable to achieve state-of-the-art performance.
However, the increased variance that multistep learning brings makes it difﬁcult to increase the update horizon beyond relatively small numbers.
In this paper, we report the counterintuitive ﬁnding that decreasing the batch size parameter improves the performance of many standard deep RL agents that use multi-step learning.
It is well-known that gradient variance decreases with increasing batch sizes, so obtaining improved performance by increasing variance on two fronts is a rather surprising ﬁnding.
We conduct a broad set of experiments to better understand what we call the variance doubledown phenomenon.
I NTRODUCTION
Deep reinforcement learning (DRL), which combines traditional reinforcement learning (RL) techniques with neural networks, has had a number of recent successes, including achieving superhuman performance on challenging games (Mnih et al., 2015; Schrittwieser et al., 2020;
Perolat et al., 2022), overcoming difﬁcult robotics challenges (Andrychowicz et al., 2020; Smith et al., 2022), and being successfully applied to large-scale real-world tasks (Bellemare et al., 2020; Degrave et al., 2022).
Yet successful application of DRL to new problems remains a challenge, in large part due to the difﬁculty in understanding how neural network training is affected by the vast number of hyperparameters involved.
Despite a number of recent works developing a greater understanding of the dynamics of training neural networks for reinforcement learning (Ceron & Castro, 2021; Ara´ujo et al., 2021; Nikishin et al., 2022; Ostrovski et al., 2021; Schaul et al., 2022), the relationship between particular hyper-parameter conﬁgurations and performance on a given environment remains hard to predict.
One generally held desire in training neural networks is to reduce the variance of gradient updates, so as to avoid unstable and unreliable learning.
For example, in the reinforcement learning literature there has been a growing trend to use multi-step (or n -step) learning (Hessel et al., 2018; Schwarzer et al., 2020; Kapturowski et al., 2018; Agarwal et al., 2022) for improved performance.
Despite their demonstrated advantage, researchers have been limited to small values of n to avoid performance collapse, in part due to the increased variance arising from larger n .
The supervised learning literature suggests that an effective mechanism for mitigating variance is through the choice of batch size: Shallue et al.
(2019) empirically demonstrate that larger batch sizes result in reduced variance and increased performance.
In this paper, we report the counterintuitive ﬁnding that reducing the batch size can help avoid performance collapse with larger n -step updates.
This is effectively doubling down on increased variance for improved performance.
We showcase this anomaly in a broad set of training regimens and value-based RL agents, and conduct an empirical analysis to develop a better understanding of its causes.
Additionally, we demonstrate that reduced batch sizes also results in reduced overall computation time during training.
In Appendix A we provide background on deep reinforcement learning, including a description of n -step updates and batch sizes.
[Figure]
I Q M
H u m a n N o r m a li z e d S c
o r e
[Figure]
u m a n N o
[Figure]
m a
[Figure]
e
[Figure]
c o
[Figure]
C ASE S TUDY : T HE A RCADE L EARNING E NVIRONMENT
Advances in deep reinforcement learning (DRL) often build on prior algorithms, network architectures
[Figure]
and hyper-parameter selections.
Given the large number of options, new work typically re-tunes only those components necessary for the new methods being considered.
Thus, we have accumulated a set of, mostly static, parameters upon which new ideas are tested (this may be a form of the “social dynamics of research” hypothesized by Schaul et al.
One of the static parameters for training single-agent value-based agents has been the choice of batch size.
Since the introduction of DQN by Mnih et al.
(2015), single-agent training on the Arcade Learning
Environment (ALE, Bellemare et al., 2013) has used a batch size of 32, where this value was carefully tuned by the authors for performance.
Since then, this value has rarely been changed, save for distributed agent training (Kapturowski et al., 2018; Espeholt et al., 2018).
If one takes the general advice from the supervised learning literature, we should be aiming to increase the batch size so as to reduce variance and improve performance (Shallue et al., 2019).
We focus on the effect of changing the batch size, while keeping all else equal.
2. E XPERIMENTAL SETUP
For this case study, we use JAX implementations of agents provided by the Dopamine library (Castro et al., 2018) and applied to game-playing in the ALE (Bellemare et al., 2013).
For computational reasons, we evaluate our agents on 20 games chosen by Fedus et al.
(2020) in their analysis of replay ratios; these were picked to offer a diversity of difﬁculty and dynamics.
Similarly, we run each learning trial for 100 million frames (as opposed to the standard 200 million).
In exploratory experiments, we determined that for our purposes there are unsubstantial differences at 100M and 200M frames.
The four agents we consider are: DQN (Mnih et al., 2015), Rainbow (Hessel et al.,
2018) 2 , QR-DQN (Dabney et al., 2018a), and IQN (Dabney et al., 2018b).
These all use the default hyper-parameter values given in Dopamine.
All experiments were run with 3 independent seeds on
NVIDIA Tesla P100 GPUs.
For evaluation, we follow the robust evaluation guidelines of Agarwal et al.
Speciﬁcally, we report the human-normalized median , interquantile mean (IQM) , mean , and optimality gap , aggregated over the 20 games.
For all plots we report the mean with 95% stratiﬁed bootstrap conﬁdence intervals.
Agarwal et al.
(2021) suggest IQM as the more robust of these metrics, so we place a stronger emphasis on it.
Throughout the main paper we present the main ﬁndings, but include extra ﬁgures in the appendix.
2. W HEN DO SMALLER BATCH SIZES IMPROVE PERFORMANCE ?
We ﬁrst varied the batch size for all agents (Figure 1).
There are two surprising observations from this result.
The ﬁrst is that aggregate agent performance is relatively stable with respect to changing batch sizes.
The second, and perhaps more surprising, result is that agent performance seems to improve with reduced batch size .
Indeed, we can observe that the default batch size is in fact not optimal for any of the agents and, with the exception of DQN, all agents seem to beneﬁt from a reduced batch size.
The four agents considered differ in a number of respects.
Three important considerations are that, of the 4, DQN is the only agent without distributional training (Bellemare et al., 2017), prioritized experience replay, and the only one without n -step returns.
To get a better sense for whether either of these components is responsible for the reduced batch size performance boost, we performed ablation studies similar to those conducted by Ceron & Castro (2021).
Since the version of Rainbow provided with the Dopamine library (Castro et al., 2018) is effectively DQN with three added components, we can investigate the changing dynamics as these components are added or removed from DQN and Rainbow, respectively.
Figure 2 depicts the outcome of this ablation study.
We ﬁnd a striking pattern: while the four variants that use 1-step learning see their performance increase with greater batch sizes, as might be expected, the relationship is almost completely reversed for the variants using 3-step learning.
Additionally, the other two components do not seem to present such a relationship with batch size.
See Appendix B for further results on QR-DQN and IQN.
[Figure]
The last results demonstrated there is a strong performance relationship between batch size and update horizon.
We systematically explored this by evaluating various choices of these two parameters for three of the agents.
As Figure 3 shows, the optimal batch size decreases as n increases.
This is most stark in QR-DQN, where simply reducing the batch size to 8 improves performance by close to 70% on the subset of games we consider.
3 With Rainbow a batch size of 8 is able to maintain performance for n -step values as high as 9; in contrast, performance for the default batch size ofcollapses beyond an n -step of 3.
11 13 15 NStep 0.
I Q M     11 13 15 NStep 0.
11 13 15 NStep 0.
U NDERSTANDING THE PHENOMENON
Having observed the advantages that can come when combining reduced batch sizes with increased update horizons, we would like to better understand the phenomenon.
Given the dramatic change in performance observed in QR-DQN with an n -step value of 3 when reducing the batch size to 8 (Figure 3), we focus on this agent and on the game Asterix, where the effect is prominent.
We provide results on additional games in Appendix C., where we observe the same qualitative ﬁndings.
3. T HE EFFECT ON VARIANCE
Reducing batch size and increasing the update horizon are generally thought to increase variance .
We thus start by conﬁrming this through measuring the variance throughout learning.
In Figure 4 we measure, from left to right, the training returns, the variance of the TD-loss, the variance of the last layer’s weight gradients, and the variance of the last layer’s bias gradients.
The ﬁrst observation is that, in aggregate, we see an increased variance with both reduced batch size and increased update horizon.
Analyzing the results in more detail, we can observe the following:
Loss variance: As expected, this type of variance is correlated with the update horizon, but inversely correlated with batch size.
In both cases, the variance seems to have an upward trend throughout training.
Weight gradient variance: While we see a reduction in variance with increased batch size (as expected), we seem to observe a reduction in variance with increased update horizon, which is the opposite of what was expected.
In all cases, the variance goes down as training progresses.
Bias gradient variance:
Here we have
the expected relationship: reduced batch size and increased update horizon both bring higher variance.
In contrast to the other two types of variance, this one seems to stabilize relatively early in training.
It thus appears that, in aggregate, the performance boost is correlated with increased variance on both these fronts; we are dubbing this the variance double-down phenomenon .
The curious behaviour of the reduced variance of the weight gradients when increasing n -steps may be an important component of this phenomenon: perhaps it is generally advantageous to have a certain degree of variance in the weight gradients, and thus the reduced batch size helps counteract the effect of increased update horizon.
Nevertheless, it is unlikely to be the only cause, so we investigate a number of additional possible causes for the phenomenon below.
Do adaptive learning rates work better with lower batch sizes?
All our experiments, like most modern RL agents, use the Adam optimizer (Kingma & Ba, 2015), a variant of stochastic gradient descent (SGD) that adapts its learning rate based on the ﬁrst- and second-order moments of the gradients, as estimated from mini-batches used for training.
It is thus possible that smaller batch sizes have a second-order effect on the learning-rate adaptation that beneﬁts agent performance.
To investigate this we evaluated, for each training step, performing multiple gradient updates on subsets of the original sampled batch; the parameter MiniBatchSplit deﬁnes the number of gradient steps and dividing factor (where a value of 1 is the default setting).
Thus, for a MiniBatchSplit of 4, we would perform 4 gradient updates with subsets of size 8 instead of a single gradient update with the full mini-batch of size 32.
With an optimizer like SGD this has no effect (as they are mathematically
R e t u r n s
Return BatchSize 832    80
V a r i a n c e
Loss    80 100 0.
1.2 1e 5 WeightGradient    800.
1e 5 BiasGradient
[Figure] [Figure]
NStep
[Figure] [Figure]
[Figure] [Figure]
6 1e
[Figure]
1.4 1.61e
[Figure]
equivalent),
but we may see differing performance due to Adam’s adaptive learning rates.
Indeed, the left panel in Figure 5 suggests that while there are differences, these are not signiﬁcant enough to explain the performance boost observed in Figure 3 when reducing the batch size to 8.
We provide further analyses on all games in Appendix C.2.
[Figure]
Is it a side effect of policy churn?
Schaul et al.
(2022) highlighted the phenomenon of policy churn , whereby the greedy action of a value network is rapidly and consistently changing throughout training.
This effect was hypothesized to provide a training beneﬁt in the form of implicit exploration .
In the middle panel of Figure 5 we measured the average policy churn throughout training.
We can see that policy churn increases with batch size.
This is somewhat expected, as increased batch sizes effectively increases the replay ratio, which was shown to be correlated with policy churn.
However, given that Schaul et al.
(2022) found increased policy churn to be beneﬁcial, it is likely that the reduced policy churn is correlated with, but not the cause of, the improved performance observed with reduced batch size.
Is variance all one needs?
The main thesis of this work is the performance beneﬁt obtained from the variance double-down phenomenon.
Although we have focused on increasing variance in a “structured”, yet indirect, way by varying update horizon and batch size, one can also increase variance in more direct ways.
We explore this by adding noise to the target values used by the
Bellman update during learning.
Speciﬁcally, we sample a zero mean isotropic Gaussian matching the shape of the target values, scale it by a multiplier, and add it to the target values.
In the right panel of Figure 5 we compare the performance when using different scaling values.
Rather surprisingly, scaling the target noise by 5.0 gives a dramatic performance boost on the game Asterix.
Although this striking result does not hold across all games (see Appendix C.3), it does hold for many.
This suggests that indeed, prediction variance may play an incompletely understood, yet beneﬁcial role in deep reinforcement learning.
E FFECT UNDER D IFFERENT L EARNING R EGIMES
In the preceding sections we established that learning from minibatches that are much smaller than standard results in improved performance across the gamut of Atari 2600 games.
Speciﬁcally, our analysis takes place in the 100-million frames regime.
At this point, one may wonder whether the root cause of the double-down phenomenon is due to peculiarities of this regime, for example the interplay between online exploration and neural network predictions (what Ostrovski et al.
(2021) call the tandem effect ).
To investigate this further, we now study how the batch size parameter affects performance in other learning regimes, and how this relates to the degree to which multi-step learning is used by high-performing algorithms for this regime.
4. T HE LOW DATA REGIME
We ﬁrst consider algorithms designed for the low data regime , speciﬁcally the Atari 100k benchmark introduced by Kaiser et al.
5 This is of particular interest to us as all algorithms that achieve competitive performance on this benchmark do so by increasing the n parameter beyond what is used for longer training periods.
Here we consider three methods, all of which use n = 10 : Data-efﬁcient
Rainbow (DER), a version of the Rainbow algorithm with hyper-parameters tuned for faster early learning (van Hasselt et al., 2019); SPR, which incorporates self-supervised learning to improve sample efﬁciency (Schwarzer et al., 2020); and DrQ( (cid:15) ), which in addition uses data augmentation (Agarwal et al., 2021).
Our results in this section evaluate performance on 26 games (the standard for this setting), aggregated over 6 independent trials.
Figure 6 (top) depicts the performance of all three agents after 100,000 agent steps, measured in terms of interquartile mean (IQM) of human-normalized scores (Agarwal et al., 2021).
We observe that DER exhibits the same trend as the preceding experiments: reducing the batch size from its default value of 32 transitions improves performance.
This is expected given that DER is a tuned version of Rainbow.
The trend is less clear for SPR and DrQ( (cid:15) ), although in the the former case similar performance is achieved for a smaller batch size.
These results concern published agents, whose hyper-parameters (including n and, to some extent, the batch size) have been tuned to maximize performance in the 100k regime.
To understand whether the double-down relationship between n and the batch size parameter also holds in this regime, we evaluated these agents on a wider range of parameter values (Figure 6, middle).
We ﬁnd that indeed,
DER exhibits this double-down relationship, with the optimal batch size varying as a function of n ; the result is also present, to a lesser extent, with SPR and appears to be absent from DrQ( (cid:15) ).
This suggests that additional algorithmic components present in the latter may obviate the need for reducing the batch size in this particular regime.
Given that performance in the 100k regime depends on a number of considerations and is difﬁcult to measure precisely, in a further experiment we trained DER and DrQ for the longer duration of 30M frames.
This provides an interesting in-between the low data regime and the classic 100M regime.
On this longer time frame, we indeed ﬁnd that both algorithms exhibit the double-down phenomenon (Figure 6, bottom): for DER, a batch size of 8 performs substantially better, while for DrQ a batch size of 16 slightly outperforms the default value.
Combined with the other results presented here, this suggests that the double-down phenomenon may only emerge with longer training regimes.
4. T HE OFFLINE REGIME
We next turn our attention to the ofﬂine reinforcement learning regime (Gulcehre et al., 2020; Levine et al., 2020), 6 where we are given a dataset of sample transitions from which we would like to obtain a policy that performs well.
Compared to the online regime, learning ofﬂine is more challenging as there is more room for overﬁtting to the ﬁxed dataset, and there is no possibility for the agent to
[Figure]
correct its estimation mistakes by interacting with the environment (as argued by Ostrovski et al.
We study the effect of varying n and the batch size parameter for three algorithms: DQN, CQL (Kumar et al., 2020)
[Figure]
and CQL+DR3 (Kumar et al., 2021).
Except for DQN, these algorithms are speciﬁcally tailored to the ofﬂine regime, incorporating among other things a penalty to mitigate value overestimation.
We follow the training scheme of Kumar et al.
(2021): each agent is trained on 17 games from the ALE for 200 iterations (where each iteration consists of 62.5K gradient steps), and after each iteration the agent is evaluated for 125K steps on the environment.
The ofﬂine dataset consists of the transitions experienced during the full training of a DQN agent (Agarwal et al., 2020).
Figure 7 illustrates the impact of jointly varying our two parameters of interest on performance.
In the case of CQL+DR3 (the highest-performing method), it is clear that when n is increased from 1 to 3, it is beneﬁcial to also reduce the batch size (from 32 to 4 or 8), in line with our previous ﬁndings.
For CQL alone, the relative performance gap between batch sizes is reduced.
We ﬁnd a similar trend for DQN.
One might argue that reducing the batch size without additional training effectively mitigates overestimation, simply because each transition is trained on fewer times.
A closer look at the learning curves (Fig 29 in the appendix) suggests reduced overﬁtting is not the main factor explaining our results, at least regarding CQL and CQL+DR3.
R ELATED W ORK
There has been a growing interest in developing a better understanding of reinforcement learning dynamics with neural networks, and our work falls in this category.
Ceron & Castro (2021) demonstrated the surprising ﬁnding that a simple switch of optimizer and loss can dramatically improve the performance of DRL agents.
Andrychowicz et al.
(2021) performed a broad examination of the
[Figure]
impact varying hyper-parameters can one agent performance, for policy-gradient methods; Ara´ujo et al.
(2021) performed a similar analysis, but for value-based agents.
Lyle et al.
identiﬁed a mechanism by which non-stationary prediction targets can prevent learning progress in deep RL agents.
They term this phenomenon as capacity loss .
Wang et al.
(2022) extensively and systematically investigated the properties of representations learned by deep reinforcement learning systems.
Fujimoto et al.
(2022) studied the relationship between the Bellman error and the accuracy of value functions through theoretical analysis and empirical study.
They found that the Bellman error is a poor proxy for the accuracy of the value function.
The use of overparameterized deep models in value based RL still exhibits mysteries in stability and performance.
To better understand the utility of deep models in RL, Xiao et al.
(2022) presented an anaylsis of recursive value estimation using overparameterized linear representations.
Nikishin et al.
identiﬁed the primacy bias in deep RL, a damaging tendency of artiﬁcial agents to overﬁt early experiences.
They proposed a resetting mechanism allowing the agent to forget a part of its knowledge.
Fedus et al.
(2020) conducted a depth study of how replay affects performance in value-based deep RL agents.
Lahire et al.
; Stooke & Abbeel (2018) studied the key role of batch size in deep RL agents over a signiﬁcant range of classical RL benchmarks.
D ISCUSSION
The long-term goal of reinforcement learning research is to develop generally capable agents that can adapt to uncertain environments.
Although theoretical results spanning multiple decades have given us a crisp insight into the mathematical properties of these algorithms, these theories unfortunately do not hold for non-linear function approximators such as neural networks.
Given that neural networks have played a key role in the impact RL has had since 2015, it behooves the community to develop a better understanding of the the interplay of the various components and how changes can affect learning dynamics.
Our work has revealed the striking ﬁnding that doubling down on variance by increasing n and reducing batch size seems to, overwhelmingly so, produce improved performance.
This ﬂies in the face of traditional beliefs from the supervised learning community that reduced variance is best.
Indeed, the remarkable performance gains obtained from simply adding noise to the target values (right panel of Figure 5) suggest that our relationship with learning variance needs to be better understood.
One important aspect of our ﬁndings is that it seems to be limited to the use of deep neural networks; indeed, in Appendix F we ﬁnd that the phenomenon is not present when using linear function approximators.
[Figure]
I Q M
H u m a n N o r m a li z e d
S c
o r e
[Figure]
One may wonder whether the same effect can be observed if one were to increase the stochasticity of action selection by, for instance, increasing the value of (cid:15) of the (cid:15) -greedy strategy used by all these agents.
The results in Appendix H suggest otherwise: decreasing (cid:15) seems to be more beneﬁcial.
It is interesting to observe that the variance double-down phenomenon remains present despite varying values of (cid:15) .
One natural area for further exploration is to explore this phenomenon in actor-critic methods (Haarnoja et al., 2018; Fujimoto et al., 2018) or those agents which their return estimator is based on multi-step (Tang et al., 2022), such as Q( λ ) (Harutyunyan et al., 2016), Retrace (Munos et al., 2016) or emphatic algorithms (Jiang et al., 2021).
We present some preliminary ﬁndings in Appendix G that suggests the variance double-down phenomenon is present in these scenarios.
Additionally, because this paper focuses mainly on an empirical investigation of the doubling down phenomenon, an exciting future work it is to develop a theoretical understanding about the nature and dynamics of this phenomenon.
6. C OMPUTATIONAL C ONSEQUENCES
Empirical advances in deep reinforcement learning are generally measured with respect to sample efﬁciency; that is, the number of environment interactions required before achieving a certain level of performance (as we have done throughout this paper).
While a valid metric, it fails to capture computational differences between algorithms.
If two algorithms have the same performance with respect to environment interactions, but one takes twice as long to perform each training step, one would clearly opt for the faster of the two.
This important distinction, however, is largely overlooked in the standard evaluation methodologies used by the DRL community.
Many of our results have demonstrated the performance beneﬁts obtained when reducing the batch size, but an additional important consequence is the reduction in computation wall-time.
Figuredemonstrates that not only can we obtain better performance with reduced batch size, but we can do so at a fraction of the runtime.
We invite the reader to revisit the results presented above under this lens.
For example, when evaluated with respect to environment interactions, the top row of Figure 6 would suggest that there is no real advantage to reducing the batch size from 32 to 16 for SPR; however, if evaluated with respect to computation time, the advantages of using a reduced batch would become apparent.
As argued by Ceron & Castro (2021), the ALE as a benchmark proves quite onerous for communities with limited access to compute; thus, computational gains like the one presented here can help reduce this barrier to entry.
We encourage others to consider not just sample efﬁciency, but also computational efﬁciency, when evaluating new methods.
A B ACKGROUND A. DQN
Mnih et al.
introduced DQN, which combined Q-learning with deep neural networks.
Some of the most important design contributions are: ( 1 ) The Q function is represented using a feed forward neural network consisting of three convolutional layers followed by two fully connected layers.
Two copies of the Q -network are maintained: an online network (parameterized by θ ) and a target network (parameterized by ¯ θ ).
The online network is updated via the learning process described below, while the target network remains ﬁxed and is synced with the online weights at less frequent (but regular) intervals.
( 2 ) A large replay buffer D is maintained to store experienced transitions ( s, a, r, s (cid:48) )
(Lin, 1992).
The temporal difference update is implemented using the following loss function to update the online network:
L ( θ ) =
E ( s,a,r,s (cid:48) ) ∼ U ( D ) [( Y DQN − Q θ ( s, a )) 2 ] using Y DQN =
( r + γ max a (cid:48) ∈A Q ¯ θ ( s (cid:48) , a (cid:48) )) and mini-batches of size 32 sampled from the replay buffer D .
A. R AINBOW
In this section we brieﬂy present the enhancements to DQN that were combined by Hessel et al.
(2018) for the Rainbow agent.
Hessel et al.
(2018) proposed Rainbow, a combination of several incremental improvements on top of DQN (Mnih et al., 2015) that increased sample efﬁciency, stability and ﬁnal performance of the DQN algorithm.
Rainbow rests upon years of model-free RL research and has been adapted to work well in data-limited regimes.
Rainbow agent is composed of six components: double Qlearning (van Hasselt et al., 2016), prioritized experience replay (PER) (Schaul et al., 2016), dueling networks (Wang et al., 2016), multi-step learning (Sutton, 1988), noisy nets (Fortunato et al., 2018), and distributional reinforcement learning (Bellemare et al., 2017).
The two main components are (PER) and multi-step learning, removing either component caused a large drop in median performance (Hessel et al., 2018; Ceron & Castro, 2021).
PER proposed to sample a trajectory t =
( s, a, r, s (cid:48) ) with probability p t proportional to the temporal difference error instead of sampling uniformly from the replay buffer.
In multi-step learning, instead of computing the temporal difference error using a single-step transition, one can use multi-step targets instead Sutton (1988), where for a trajectory ( s 0 , a 0 , r 0 , s 1 , a 1 , · · · ) and update horizon n : R ( n ) t := (cid:80) n − 1 k =0 γ k r t + k +1 , yielding the multi-step temporal difference: R ( n ) t
[Equation]
[Equation]
A. D ISTRIBUTIONAL RL AGENTS
The experiments in the previous sections were conducted using the JAX Dopamine DQN and Rainbow implementations.
However, we did not evaluate if low batch size values behave similar in
Distributional Q-learning variants.
In this section, we investigate the interaction of Quantile Regression for Distributional RL QR-DQN (Dabney et al., 2018a) and Implicit Quantile Networks
IQN (Dabney et al., 2018b) with low batch sizes values on 20 Atari games (same setup of previous experiments).
QR-DQN (Dabney et al., 2018a) computes the return quantile values for N ﬁxed, uniform probabilities.
This has no restrictions or bound for value, as the distribution of the random return is approximated by a uniform mixture of N Diracs: Z θ ( x, a ) :
= 1 N (cid:80) Ni =
δ θ i ( x,a ) , with each θ i assigned a quantile value trained with quantile regression.
IQN uses implicit quantile networks (IQN) as the parameterization of the return distribution Dabney et al.
(2018b).
IQN learns to transform a base distribution (typically a uniform distribution in [0 , 1] ) to the quantile values of the return distribution.
This can result in greater representation power in comparison to QR-DQN, as well as the ability to incorporate distortion risk measures .
B T HE E FFECT OF R EDUCED B ATCH S IZE
In this appendix we provide additional results investigating the effect of reduced batch size, complementing the results presented in section
[Figure] [Figure]
[Figure] [Figure]
[Figure]
C U NDERSTANDING OF THE V ARIANCE D OUBLE -D OWN P HENOMENON
In this section we provide additional results to complement those presented in section
. C. T HE EFFECT ON VARIANCE
[Figure] [Figure]
V a r i a n c e
Loss    80 100 0.
WeightGradient    80 1001357 1e 5 BiasGradient    80 100 Frames (x1M)
R e t u r n s
NStep37    80 100 Frames (x1M) 0.
V
[Figure] i [Figure] n
[Figure] [Figure] 1.75 1e    80 100 Frames ( [Figure]
1e
[Figure] [Figure]
5 BiasGradient    80 100 Frames (x1M) {+
+}
R e t u r n s
NStep37    80 100 Frames (x1M)
V a r i a n c e    80 100 Frames (x1M) 0.
1.41e    80 100 Frames (x1M) 0.5 1.
1e
C. D O ADAPTIVE LEARNING RATES WORK BETTER WITH LOWER BATCH SIZES ?
[Figure]
d
[Figure]
c
[Figure]
e
MiniBatchSplits
[Figure]
C. I S VARIANCE ALL ONE NEEDS ?
[Figure]
d
[Figure] c
[Figure] e
TargetNoiseScale
None [Figure] [Figure]
[Figure]
D E FFECT UNDER DIFFERENT LEARNING REGIMES
In this appendix we provide additional results to complement those presented in section
[Figure]
D. L OW DATA REGIME
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
E C OMPUTATIONAL CONSEQUENCES
[Figure] [Figure]
F E VALUATION WITH LINEAR FUNCTION APPROXIMATORS
[Figure]
G I S THE PHENOMENON PRESENT IN OTHER ENVIRONMENT TYPES ?
Although not as comprehensive as the results in our main paper
[Figure]
we found that the variance doubledown phenomenon does appear to be present in continuous control tasks (Figure 33), as well as in procedurally generated environments (Figure 35)
[Figure] [Figure]
[Figure]
H E XPERIMENTS WITH VARYING (cid:15) EXPLORATION VALUES
[Figure]
li z
[Figure]
d
[Figure]
( n s t
[Figure]
I Q M H u m a n N o r m a li z e d S c
o r e ( n s t e p = 3 )
= 0.
[Figure]
I E XPERIMENTS WITH VARYING LEARNING RATES
[Figure]
a li z e d c o r e ( n s t e p = 3 )
[Figure]
I Q M H u m a n N o r m a li z e d S c
o r e ( n s t e p = 3 )
LearningRate
= 0.
Batch Size
J O N - POLICY EVALUATION
Most of our experiments are in the online, off-policy setting, so it is informative to evaluate whether it is present in an on-policy situation.
To do so, we populate the replay buffer with a ﬁxed policy π coming from a pre-trained agent, and do policy estimation with the same learning rule (based on the
Bellman update).
Figure 38 plots the Q -estimation error for a set of ﬁve games.
The results suggest that the phenomenon is somewhat present, but only mildly so.
[Figure]
t i m a t i o n
[Figure]
rr
[Figure]
[Table]
[Figure]
i
[Figure]
n
[Figure]
(n=3)
[Figure]
n=
[Figure]
[Figure]
Seaquest (n=3) 0.