I S H IGH V ARIANCE U NAVOIDABLE IN RL? A C ASE S TUDY IN C ONTINUOUS C ONTROL
A BSTRACT
Reinforcement learning (RL) experiments have notoriously high variance, and minor details can have disproportionately large effects on measured outcomes.
This is problematic for creating reproducible research and also serves as an obstacle when applying RL to sensitive real-world applications.
In this paper, we investigate causes for this perceived instability.
To allow for an in-depth analysis, we focus on a speciﬁcally popular setup with high variance – continuous control from pixels with an actor-critic agent.
In this setting, we demonstrate that poor outlier runs which completely fail to learn are an important source of variance, but that weight initialization and initial exploration are not at fault.
We show that one cause for these outliers is unstable network parametrization which leads to saturating nonlinearities.
We investigate several ﬁxes to this issue and ﬁnd that simply normalizing penultimate features is surprisingly effective.
For sparse tasks, we also ﬁnd that partially disabling clipped double Q-learning decreases variance.
By combining ﬁxes we signiﬁcantly decrease variances, lowering the average standard deviation across 21 tasks by a factor > 3 for a state-of-the-art agent.
This demonstrates that the perceived variance is not necessarily inherent to RL.
Instead, it may be addressed via simple modiﬁcations and we argue that developing low-variance agents is an important goal for the RL community.
I NTRODUCTION
With the advent of deep learning, reinforcement learning (RL) with function approximators has become a promising strategy for learning complex behavior (Silver et al., 2017; Mnih et al., 2015).
While RL can work remarkably well, deep RL experiments have been observed to exhibit high variance (Chan et al., 2020; Clary et al., 2018; Lynnerup et al., 2020).
Speciﬁcally, the rewards obtained by RL agents commonly vary signiﬁcantly depending on the random seeds used for training, even if no other modiﬁcations are made.
Furthermore, implementation level details (Engstrom et al., 2019) and even hardware scheduling (Henderson et al., 2018) often have an outsized effect on performance.
High variance is problematic for creating reproducible RL research, and achieving statistically signiﬁcant results over large RL benchmarks is often prohibitively time-consuming (Colas et al., 2018;
Agarwal et al., 2021).
Furthermore, high variance provides an obstacle for real-world applications of RL, where safety and predictability can be critical (Garcıa & Fern´andez, 2015; Srinivasan et al., 2020).
Especially relevant RL applications include medical testing (Bastani et al., 2021), autonomous vehicles (Bellemare et al., 2020) and communications infrastructure (Liu et al., 2020).
In this paper, we investigate causes for high variance in RL.
To limit the paper’s scope and computational footprint we focus on continuous control from pixels with a state-of-the-art (SOTA) actorcritic algorithm
(Lillicrap et al., 2021; Yarats et al., 2021b).
This is a popular setup that exhibits high variance, and where actor-critic algorithms often achieve SOTA results (Laskin et al., 2020b;a;
Kostrikov et al., 2020).
In this setting, we ﬁrst show that an important cause of variance is outlier runs that fail early and never recover.
It is natural to hypothesize that the typical sources of randomness, network initialization, and initial exploration, are to blame.
However, we demonstrate that these do in fact have little impact on the ﬁnal performance.
We similarly provide experiments that suggest that neither poor feature representations nor excessive sensitivity of the environment is at fault.
We instead demonstrate that unstable network parametrization can drive the variance and lead to excessively large activations that saturate action distributions.
We propose several methods to avoid such issues and ﬁnd that they decrease variance.
The most effective method is simply setting penultimate features to unit norm – a strategy we call penultimate normalization – which avoids exploding activations by design.
We also ﬁnd that partially disabling double clipped Q-learning (Fujimoto et al., 2018), a strategy we call asymmetrically double clipped Q-learning , reduces variance for many sparse tasks.
By combining multiple approaches we increase average rewards while decreasing variance as measured of 21 tasks.
The average standard deviation decreases by a factor > 3 and for some tasks by more than an order of magnitude.
Our experiments suggest that high variance is not necessarily inherent to RL.
Instead, one can directly optimize for lower variance without hurting the average reward.
We argue that developing such low-variance agents is an important goal for the RL community – both as RL becomes deployed in sensitive real-world applications and as a way of enabling reproducible research.
B ACKGROUND
We consider continuous control modeled as a Markov decision process (MDP) with a continuous action space A and the state space S .
In practice, the action space A is often bounded to account for physical limitations, e.g., how much force robotic joints can exert.
A popular choice is simply
A =
[ − 1 , 1] dim ( A ) .
Within this setting, Deep Deterministic Policy Gradients (DDPG) (Lillicrap et al., 2021) is a performant algorithm that forms the backbone of many state-of-the-art agents (Laskin et al.
, 2020b;a; Kostrikov et al., 2020; Yarats et al., 2021b).
DDPG uses a neural network (the critic ) to predict the Q-value Q φ ( a t , s t ) for each state-action pair.
The parameters φ of the critic are trained by minimizing the soft Bellman residual:
[Equation]
Here, r t is the reward and E [ ˆ Q ( a t +1 , s t +1 )] is the Q-value estimated by the target critic – a network whose weights can e.g. be the exponentially averaged weights of the critic.
One can use multi-step returns for the critic target (Sutton, 1988).
Another common trick is double clipped q-learning, i.e. using two Q-networks Q 1 , Q 2 and taking Q = min( Q 1 , Q 2 ) to decrease Q-value overestimation (Fujimoto et al., 2018).
The policy is obtained via an actor -network with parameters θ , which maps each state s t to a distribution π θ ( s t ) over actions.
The action is obtained by calculating a preθ ( s ) , the vector output of the actor network.
Thereafter a tanh non-linearity is applied elementwise to ensure a ∈ [ − 1 , 1] n for n = dim ( A ) .
The policy is then obtained as:
[Equation]
(cid:15) is some additive noise which encourages exploration, e.g. a Gaussian variable with time-dependent variance σ 2 t .
Various noise types have been employed (Achiam, 2018; Lillicrap et al., 2021; Yarats et al., 2021b).
Since the Q-values correspond to the expected discounted rewards, the actions a θ should maximize Q φ ( a θ , s ) .
Thus, to obtain a gradient update for the actor one uses the gradient ∇ θ Q φ ( a θ , s ) , effectively calculating gradients through the critic network.
In control from pixels, the state s is typically an image (possibly data augmented) processed by a convolutional network.
B ENCHMARKING V ARIANCE
Experimental Setup.
There is a large number of available RL agents and environments to consider.
To limit the paper’s scope and computational footprint, we consider continuous control from pixels with an actor-critic algorithm.
We consider the standard continuous control benchmark deepmind control (dm-control)
(Tassa et al., 2020).
Within dm-control, a long line of state-of-the-art models relies on the actor-critic setup (Laskin et al., 2020b;a; Kostrikov et al., 2020).
We consider the recent data-augmentation based DDPG agent DRQv2 of Yarats et al.
(2021b) which provides stateof-the-art performance and runs fast.
We use the default hyperparameters that Yarats et al.
(2021b) uses on the medium benchmark (listed in Appendix A) throughout the paper.
We will refer to this agent as the baseline agent.
We are interested in using a large number of random seeds to obtain accurate statistics and thus need to limit the number of environments we consider.
To select environments we simply rank the dm-control tasks based upon their relative variance as measured by Yarats et al.
(2021b).
We select the ﬁve tasks with the highest relative variance – ﬁnger turn hard, walker walk, hopper hop, acrobot swingup, and reacher hard (see Table 4 in Appendix A for details).
In Section 4.2 we consider a larger set of tasks.
We will refer to these as environments or tasks interchangeably.
For each run, we train the agent for one million frames, or equivalently 1, episodes, and evaluate over ten episodes.
We use 10 or 40 seeds for each method we consider, these are not shared between experiments, but we reuse the runs for the baseline agent across ﬁgures.
3. V ERIFYING V ARIANCE
We ﬁrst aim to verify that the environments indeed exhibit high variance.
To this end, we run the baseline agent with 40 random seeds on each task.
Figure 1 shows the result of this experiment for three tasks, see Appendix A for the two other tasks.
We show both individual runs as well as mean and standard deviations.
For some tasks, the variance is dominated by outlier runs that fail to learn early in the training and never recover.
E.g. for the walker task, we see several runs which have not improved during the ﬁrst 500 episodes, and most of these do not improve in the followingepisodes either.
Except for these outliers, the variance of the walker task is relatively small.
We also note that these outliers separate themselves relatively early on, suggesting that there are some issues in early training that cause variance.
If it was possible to eliminate poor outlier runs, we would expect the variance to go down.
Motivated by these observations, and the walker task in particular, we will investigate various potential causes of outlier runs and high variance.
3. D OES INITIALIZATION MATTER ?
There are multiple sources of randomness speciﬁc to the early training iterations.
The agent networks are randomly initialized and the agent is given some initial experience, collected by random actions, to start training on.
Could these two initial factors be the cause of the high variance?
To study this hypothesis we measure the correlation between two runs that share the same network initialization and initial experience, but otherwise have independent randomness (from e.g. the environment, action sampling, and mini-batch sampling).
Across all tasks, we repeat this process for ten combinations of initializations and initial experience, from each such combination we start two runs A, B which do not share other sources of randomness.
For each time-step we measure the Pearson correlation between score ( A ) and score ( B ) .
This correlation, averaged across all timesteps, is shown in Figure 2 (left).
We see that the correlation is very small, suggesting that the initialization and initial experience does not have a large effect on the outcome.
Speciﬁcally, the hypothesis that
[Figure]
For some tasks, the variance is dominated by a few outlier runs that completely fail to learn.
This is especially clear in the walker task.
Removing such outlier runs would decrease the variance.
[Figure]
there are good and bad initializations that leads to good or bad performance is inconsistent with the results of Figure 2 (left).
Correlation across time is noisy but centers around zero, see Appendix A.
3. A RE THE E NVIRONMENTS TOO S ENSITIVE ?
Another cause of variance is the environments.
Perhaps the environments require very speciﬁc policies to yield rewards, and unless the agent stumbles upon a good policy early on, learning fails.
To study the sensitivity of the environments we consider gradually exchanging a good learned policy with a random policy.
Speciﬁcally, at every timestep t , with probability p , we exchange the policy’s action with a random action.
We use the ﬁnal policy of a performant run with the baseline agent as the learned policy.
The effect of this, for ten ﬁnal policies learned for the walker environment, is shown in Figure 2 (right).
Performance degrades gracefully as we gradually move to a random policy.
This suggests that the environment is relatively stable with regard to actions.
Results for other environments are similar and are shown in Appendix A. Furthermore, in Appendix E we verify that the variance arising from evaluating the agent on a ﬁnite number of episodes is small compared to the inter-run variance.
Thus, environment sensitivity and evaluation methodology do not appear to explain the excessive variance.
3. I S FEATURE LEARNING TO BLAME ?
In the early stages of training, the image encoder will extract poor features.
Without good representations, it is hard to learn a good policy, and without a policy generating interesting behavior it is hard to learn good representations.
If such a vicious cycle was causing runs to fail, jump-starting the encoder to extract good features should help to decrease the variance.
To test this hypothesis, we consider initializing the weights of the convolutional encoder to some good pretrained values before training as normal.
For each task, we pick a run with above-average ﬁnal performance and use its convolutional weights at episode 500.
We refer to this strategy as pretrain.
We also consider the same strategy but decrease the learning rate of the pre-trained weights by a factor of 10x.
We refer to this strategy as pretrain + small lr.
In addition to this ﬁx, we also consider improving the features with self-supervised learning during early training (Jing & Tian, 2020).
We use the loss function proposed in Grill et al.
(2020) which is useful for smaller batch sizes (we use 256) and has already successfully been applied in RL (Schwarzer et al., 2021).
We use standard methods in self-supervised learning: a target network with exponentially moving weights and a head network.
See Appendix A for details.
We consider adding the self-supervised loss to the ﬁrst 10,000 and 20,000 steps and refer to these modiﬁcations as self-supervised 10k and self-supervised 20k.
The results for this experiment are shown in Table 1.
For these ﬁxes, the large variance persists even
, Table 2 shows that performance can be improved significantly without better representation learning.
Metrics are calculated over
[Table]
though the performance improves marginally with self-supervised learning.
We thus hypothesize that representation learning is not the main bottleneck.
Indeed, experiments in later sections of the paper show that one can signiﬁcantly decrease variance without improving representation learning.
3. S TABILITY OF N ETWORK P ARAMETRIZATION
We have considered the effects of initialization, feature learning, and environment sensitivity and found that none of these factors satisfactorily explain the observed variance.
Another factor to consider is the stability of the network parametrization.
Unstable parametrization could make the learning process chaotic and thus cause high variance.
To study this hypothesis we investigate the failure modes of individual runs.
In Figure 3 we consider some runs that fail as well as some successful runs for the walker task.
Plots for all runs are available in
Appendix A. We plot three [Table] [Equation] [Table]
[Table]
[Equation]
[Table] [Equation] [Table]
[Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Figure]
latexit>
[Table] , and [Table] absolute value of
[Table]
| a | .
All agents initially have poor rewards, but once the rewards start to improve, they improve steadily. During this initial period of no or little learning, the gradients of the actor-network (row two in Figure 3) are small. Recall that the actor is optimized via gradients: ∇ θ Q φ ( s , a θ ( s )) = (cid:80) i ∂Q φ ∂ a i ∂ a i ∂θ .
If this gradient is close to zero, we might suspect that ∂ a
i ∂θ ≈ 0 .
This could easily happen if the tanh non-linearity in equation 2 is saturated.
In row three of Figure 3 we show the average absolute value of the actions, which are all bound to lie in [ − 1 , 1] .
We see that for the runs that fail the average absolute action stays around 1 throughout training, which implies that the gradients for the actor are small since | tanh( x ) | ≈ 1 implies ∂∂x tanh( x ) ≈ 0 .
The cause of saturating tanh must be excessively large activations, i.e. a preθ ( s ) in equation 2 becoming too large, which saturates the tanh non-linearity.
In fact, all runs seem to initially suffer from saturating tanh, but most runs escape from this poor policy.
Somewhat similarly, Wang et al.
(2020) has observed that saturating nonlinearities can impede exploration when removing the entropy bonus from SAC agents.
However, for the DDPG agent saturation does not harm exploration since the exploration noise is added outside the non-linearity as per eq.
Instead, the issue here appears to be vanishing gradients which impedes learning.
The Q-values could of course also suffer from exploding activations, but since there are no saturating nonlinearities for the Q-values, such issues are harder to pinpoint.
These experiments suggest that avoiding excessively large activations could reduce the number of poor outlier runs, and thus decrease variance.
I MPROVING S TABILITY
Motivated by our qualitative observations we now consider a few ideas for improving the stability of learning, hoping that these will decrease the variance.
We consider several methods:
Normalizing activations.
As per eq.
(2), the actor selects actions using a preθ ( s ) , where a preθ ( s ) is a deep neural network.
Figure 3 shows that learning can fail due to excessively large activations.
To avoid this we propose to parametrize a preθ ( s ) as a linear function of features with a ﬁxed magnitude.
Speciﬁcally, let Λ θ ( s ) be the penultimate features of the actor-network.
We propose to simply normalize Λ nθ ( s ) =
Λ θ ( s ) / (cid:107) Λ θ ( s ) (cid:107) and let a pre be a linear function of Λ nθ , i.e. a preθ ( s ) =
L Λ nθ ( s ) for a linear function L .
Note that
no mean subtraction is performed.
We refer to this straightforward modiﬁcation as penultimate normalization or pnorm for short.
When applied to the actor, we simply call the strategy actor pnorm .
Our motivation has been to stabilize the actor output, but the critic network could also beneﬁt from the stability offered by normalizing penultimate features.
We refer to the strategy of normalizing both the actor and the critic as both pnorm .
We also consider layer norm (Ba et al., 2016) applied similarly to both the actor and critic.
We also consider the normalization scheme of Wang et al.
(2020), which directly constrains the output of the actor.
We refer to this method as output norm .
At last, following Gogianu et al.
(2021), we consider adding spectral normalization (Miyato et al., 2018) to the penultimate layer.
baseline actor pnorm both pnorm
[Equation]
[Table]
[Figure]
Penalizing saturating actions.
Another strategy for avoiding actions with large magnitude is to add a term to the loss function which directly penalizes large actions.
One can imagine many such loss functions, we here consider the simple and natural choice of squared loss (cid:96) (
[Equation]
We can simply add this term to the loss function times some hyperparameter λ .
By tuning λ , one should be able to remove the saturating actions illustrated in Figure 3.
We heuristically ﬁnd that λ
=
.works well in practice (see Appendix A) and use this value.
We refer to this modiﬁcation as penalty .
Since the Adam optimizer is invariant under re-scaling the loss, the speciﬁc value of λ has a small effect as long as ∇ θ Q φ ( s , a θ ( s ))
≈ 0 , which happen as per Figure 3.
Learning rate warmup.
One common strategy to improve stability is to start the learning rate at zero and increase it linearly to its top value during early training (Goyal et al., 2017).
This can improve numerical stability (Ma & Yarats, 2021).
We consider using this strategy and linearly increasing the learning rate from zero over the ﬁrst 10,000 steps.
We refer to this strategy as lr warmup .
[Table]
Small weights ﬁnal layer.
Following Andrychowicz et al.
(2020), we consider using smaller weights for the ﬁnal layers of the actor and the critic.
Speciﬁcally, we downscale these weights by a factor of 100 at initialization time.
We refer to this strategy as scale down .
Gradient clipping.
Another stabilizing strategy is gradient clipping (Zhang et al., 2020).
To avoid occasional exploding gradients, one simply clips the norm of gradients that are above some threshold.
This strategy is popular in NLP (Gehring et al., 2017; Peters et al., 2018) and is sometimes used in RL (Rafﬁn et al., 2019).
We consider clipping the gradient when the norm, calculated independently for the actor, critic, and convolutional encoder, is larger than 1 or 10.
We refer to these strategies as grad clip 1 and grad clip 10 .
Results.
In Table 2 we evaluate the effects of these ideas.
Penultimate normalization has the best performance and lowest variability on average, whereas layer norm can make performance worse.
Since normalizing the critic improves performance, the critic might also suffer from instabilities.
This instability can be observed by simply plotting the average Q-values during training for various agents, see Figure 4.
We study this question in detail in Appendix C.
The penalty method improves the performance while many other methods decrease average reward, and lowering variance by using a poor policy is not hard to achieve and not particularly desirable.
The output normalization method of Wang et al.
(2020) performs worse than actor normalization, and importantly fails completely on the acrobot task.
We hypothesize that directly constraining the output space is harmful on the acrobot task.
These experiments show that it is possible to signiﬁcantly decrease the variance of the baseline agent by applying stabilization methods.
Additionally, decreasing the variance often increases the average reward – likely as bad outlier runs are eliminated.
4. C OMBINING F IXES
We have found that many of the proposed methods can decrease variance and improve average reward.
For optimal performance, we now consider combining three orthogonal methods which improve performance individually – actor and critic penultimate normalization, pre-tanh penalty, [Table] and self-supervised learning (we apply it to the ﬁrst 10,000 steps).
In Table 3 we compare the performance between the baseline and an agent using these three modiﬁcations (which we refer to as combined ) over 40 seeds.
We see that the variances go down while the average score goes up.
In fact, for the walker task, the variance decreases by orders of magnitude.
Eliminating poor outlier runs seems to beneﬁt the average performance in addition to decreasing variance.
While our methods have mostly been motivated by outlier runs that fail catastrophically, they likely help more normal runs too.
Learning curves for the combined agent are given in Appendix A. In Appendix D, we verify that our ﬁxes enable the agent to learn successfully with a larger learning rate.
4. A DDITIONAL T ASKS
We now apply the combined agent of Table 3 to additional tasks.
Speciﬁcally, we consider alltasks identiﬁed as easy or medium by Yarats et al.
(2021b).
Note that the combined agent has been constructed using only 5 of these tasks, so this experiment measures generalization.
For computational reasons, the additional 16 tasks only use 10 seeds each.
We use the baseline hyperparameters across all tasks without tuning (Yarats et al.
(2021b)
tunes parameters on a per-task basis).
Results are shown in Figure 5.
Our methods consistently improve the average reward while the average standard deviation decreases by a factor > 2 .
Furthermore, a few additional tasks (ﬁnger spin, hopper stand, walker run, walker stand) have their variance decreased by an order of magnitude or more.
The tasks cheetah run and quadruped walk perform worse than the baseline, although this could potentially be alleviated by tuning hyperparameters.
Just improving learning rate on these tasks improve performance, see Figure 13 in Appendix A.
These results demonstrate that our modiﬁcations signiﬁcantly decrease variance and that directly optimizing variance can be feasible and fruitful.
Performance proﬁles (Agarwal et al., 2021) are available in Figure 16 of Appendix A.
4. S PARSE T ASKS
Consider the ﬁve tasks in Figure 5 where the combined agent has the highest standard deviation.
Four out of these have sparse rewards – ﬁnger turn hard, reacher hard, pendulum swingup, and cartpole swingup sparse.
Once stability has been addressed, sparsity is a major driver of residual variance.
Motivated by this, we consider three ﬁxes that could help for sparse tasks.
Firstly, not training until at least one non-zero reward has been observed.
Secondly, extending the self-supervised training throughout the whole training process.
Thirdly, exchanging the clipped double Q-learning
Q = min( Q 1 , Q 2 ) (Fujimoto et al., 2018) with Q = avg ( Q 1 , Q 2 ) , to decrease pessimistic bias.
Through ablations (see Appendix B), we ﬁnd that doing this for the critic loss (eq.
but retaining the clipping Q = min( Q 1 , Q 2 ) for the actor loss performs the best.
We dub this strategy asymmetrically clipped double Q-learning.
In Appendix B we present an ablation study over these three modiﬁcations, ﬁnding that using more self-supervised learning and asymmetrically clipped double
Q-learning improves performance.
We add these two modiﬁcations to our combined agent, which we now call combined++.
In Figure 5 we show the performance of this agent (seeds are independent from experiments in Appendix B).
Performance improves, especially for sparse tasks, and compared to the baseline agent the average standard deviation decreases by a factor > 3 .
E PILOGUE
Related Work.
There are multiple papers documenting the high variance of RL (Henderson et al., 2018), focusing on implementation details Engstrom et al.
(2019), continuous control Mania et al.
(2018), Atari games Clary et al.
and real-world robots Lynnerup et al.
Given this high variance, there is ample work on how to accurately measure performance in RL (Colas et al., 2019; Chan et al., 2020; Jordan et al., 2020; Agarwal et al., 2021; Islam et al., 2017).
Our strategy of simply reducing the variance of RL algorithms is complementary to developing robust metrics.
Within Q-learning, there is a large amount of work aiming to improve and stabilize Q-estimates (Fujimoto et al., 2018;
Nikishin et al., 2018; Anschel et al., 2017; Jia et al., 2020; Chen et al., 2018).
These apply to the discrete DQN setting, distinct from continuous control.
There is also work on stabilizing RL for speciﬁc architectures Parisotto et al.
and
task types Kumar et al.
Mao et al.
(2019) Normalization in RL have been investigated by Salimans & Kingma (2016);
Bjorck et al.
; Gogianu et al.
(2021), and Wang et al.
Wang et al.
(2020) directly normalizes the output of the actor, which e.g. for a one-dimensional task shrinks the action space to [tanh( − 1) , tanh(1)]
(modulo noise).
As per Table 2, this method underperforms our methods.
Limitations.
We have only considered a single SOTA agent (DRQv2 of (Yarats et al., 2021b)) on one popular suite of tasks.
Furthermore, we focus on an agent with tanh nonlinearities for the policy.
While this is an important agent class, other important agent types (Hafner et al., 2021) and environments (Bellemare et al., 2013) remain.
We show that it is possible to decrease variance with simple modiﬁcations, although which methods are effective likely depends upon the base algorithm.
While many of our ﬁxes can potentially be added to other agents, it remains to be experimentally tested if doing so improves performance.
We leave this question for future work.
While our ﬁxes typically are only a few lines of code, they nonetheless increase the agent complexity.
Conclusion.
We have investigated variance in a popular continuous control setup and proposed several methods to decrease it.
We demonstrate across 21 tasks that these dramatically decrease variance while improving average rewards for a state-of-the-art agent.
Our experiments show that directly optimizing for lower variance can be both feasible and fruitful.
[Table]
=</latexit>
Ethics Statement.
There are many positive RL applications – examples include safe autonomous cars, accelerated scientiﬁc discovery, and medical testing.
Studying causes of high variance, and proposing ﬁxes to this issue, can beneﬁt real-world adoption of RL techniques.
Additionally, reducing the variance of RL experiments would aid the research community as reproducibility can be problematic in the ﬁeld.
However, there are also many potentially harmful military applications.
Our study is academic in nature and only considers simulated toy environments, but insights from our work could nonetheless be useful in harmful applications.
We do not believe that our work differs from other RL work signiﬁcantly concerning ethics
Reproducability Statement.
In Appendix A we detail the infrastructure and software used.
The perhaps most important component of generating reproducible ﬁndings in RL is to use many seeds.
To this end, we have focused on ﬁve environments but evaluated these thoroughly.
The main experiments of the paper (Table 3) use 40 seeds whereas all others use 10 seeds.
A A PPENDIX
Tuning λ .
For tuning the hyperparameters λ we search over the set { 1e − 1 , 1e − 2 , 1e − 3 , 1e − 4 , 1e − 5 , 1e − 6 } .
For each value, we run the hopper task for a short amount of time with ﬁve different seeds.
We found that too large λ penalized actions too much, and that even very small values like 1e − 6 sufﬁced to give improvement.
Details on self-supervised Learning.
We extract features from the convolutional network and the ﬁrst feedforward layer of the critic, which projects the feature map to 50-dimensions.
Feature vectors are normalized to unit norm before being fed into the self-supervised loss (Bachman et al., 2019).
We use exponentially moving weights for the target network (He et al., 2020), using the same decay parameters as for the target critic.
We use a head for the online network which is simply a two-layer
MLP with hidden dimension 512 and ﬁnal dimension 50 (Chen et al., 2020).
The two augmentations we consider are simply two adjacent frames with augmentations that the DDPG agent of Yarats et al.
(2021b) considers naturally.
We use the square loss as proposed in Grill et al.
(2020), which has been shown to be useful in smaller batch sizes.
The self-supervised loss is then simply added to the normal loss function for the critic.
Infrastructure.
We run our experiments on Nvidia Tesla V100 GPUs and Intel Xeon CPUs.
The
GPUs use CUDA 11.1 and CUDNN 8.0.0.5.
We use PyTorch 1.9.0 and python 3.8.10.
Our experiments are based upon the open-source DRQv2 implementation of Yarats et al.
(2021b).
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
acrobot
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
spin 0.
[Table] [Table]
walk 0.
cartpole
[Table]
sparse 0.
[Table]
cartpole
[Table]
sparse 0.
cheetah
[Table]
[Table]
pendulum swingup 0.
hopper
[Table]
cartpole
[Table]
[Table]
cartpole balance 0.
[Figure]
[Table]
[Figure] [Figure]
[Table]
/RJlGhbCslM/T2R09iYcRzazpji0Cx6U/E/r5thdBXkQqUZcsXmi6JMEkzINADSF5ozlGNLKNPC3krYkGrK0MZUsSF4iy8vk/ZZ3XPr3t1FrXFdxFGGIziGU/DgEhpwC01oAYMUnuEV3pzMeXHenY95a8kpZg7hD5zPHxdAkbM
[Figure] [Figure]
[Figure] [Figure]
[Table] [Table]
[Table]
=
=</latexit>
[Table]
<
[Table]
+
[Table]
+V/dfkKtZAqoIMZ0wiCFKCMaOBVsWupaw1JCx2TIOo4qIpmJsvnhU3zqlD4eJNqVAjxXf09kRBozkbHrlARGZtmbif95HQuDqyjjKrXAFF0sGliBIcGzFHCfa0ZBTBwhVHN3K6YjogkFl1XJhRAuv7xKmufVMKiGdxeV2nUeRxEdoxN0hkJ0iWroFtVRA1Fk0TN6RW/ek/fivXsfi9aCl88coT/wPn8ApdeTDw==</latexit> | a | <
[Table] [Table]
=
[Table]
+
[Table] [Table]
[Table] [Table]
a | <
[Table]
[Figure]
[Table]
[Figure] [Figure]
Our combined agents improve performance, especially for the lowest quantiles.
The effect is most dramatic for the 5 original tasks, this is natural since we initially selected these as they suffered from high variance.
Also note that the baseline is the recent SOTA DRQv2 agent of Yarats et al.
(2021b) which is very competitive – so any improvements to it can be very signiﬁcant
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
B I MPROVEMENTS FOR S PARSE T ASKS
In Figure 5, consider the ﬁve tasks where the combined agent suffers from the highest standard deviation.
Four of these tasks have sparse rewards.
This suggests that sparsity is a major driver of variance.
To further decrease the variance, we now investigate ﬁxes for four sparse tasks – cartpole swingup sparse, pendulum swingup, reacher hard, and ﬁnger turn hard.
We will consider three additional modiﬁcations which aim to speciﬁcally combat reward sparsity:
Avoiding training without rewards.
For the sparse tasks, an agent might not see any rewards during e.g. the ﬁrst 100 episodes.
Perhaps training in the absence of any rewards harms the network weights.
The ﬁx we consider is simple, do not use any gradient updates before at least one non-zero reward has been observed.
Increasing self-supervised learning.
Since the agent might potentially not observe any rewards for 100 episodes, we consider extending the self-supervised learning to ensure some supervisory signal.
We simply extend the self-supervised time from 10k steps to the entire training process.
Removing clipped double q-learning.
The agent of Yarats et al.
(2021a)
employs clipped double q-learning (Fujimoto et al., 2018) – i.e. the agent uses two critic networks Q 1 , Q 2 and takes
Q = min( Q 1 , Q 2 ) to avoid overestimation.
We hypothesise that the min-operator causes excessive pessimism in sparse environments where reward often are zero.
For example, if Q 1 = 0 and Q 2 >we have Q = min( Q 1 , Q 2 ) = 0 .
Thus, if one Q-network outputs only zeros and the rewards are almost only zeros, the critic target of eq.
(1) will be mostly zeros and the other Q-network will be encouraged to output mostly zeros too.
Thus, the Q-network could potentially get stuck with mostly outputting zeros, an issue shown in Figure 18.
We thus propose to use Q a =
avg ( Q 1 , Q 2 ) .
Through ablation experiments in Table 6, we ﬁnd that using Q a = avg ( Q 1 , Q 2 ) only for the critic loss (i.e. eq.
but retaining Q = min( Q 1 , Q 2 ) for the actor update, works the best.
We call this strategy asymmetrically clipped double Q-learning.
Results for these experiments are shown in Table 7.
We see that decreasing pessimism and increasing self-supervised learning help the most.
Increasing self-supervised learning helps the most for cartpole swingup sparse and pendulum swingup.
Note that these two tasks were not used in the ablation experiments of Table 2.
Additionally, whereas feature learning was not the bottleneck for the baseline agent, it might be the bottleneck for the combined agent.
Avoiding training without rewards does not seem to improve performance signiﬁcantly.
We conclude that increased self-supervised learning and asymmetrically clipped double Q-learning improve performance for tasks with sparse rewards.
We add these two modiﬁcations to the combined agent and call the resulting agent combined++.
Its performance across all 21 tasks is shown in Figure 5.
It achieves even better results, and its average standard deviation is smaller than the baseline by a factor > 3 .
[Table]
Increasing the self-supervised learning helps the most for cartpole swingup sparse and pendulum swingup, two tasks which were not part of the ﬁve tasks we consider ﬁrst.
[Table]
[Table]
asXL3K4yiQQ3JETohHzkmV3JAaqRNONHkmr+
[Table]
+
[Table]
=
=</
[Table]
>
[Table]
10 <latexit sha1_
[Table]
="rMZt6lLsG4QW0FzbkSd9wwyTKHE=
[Table]
[Figure] [Figure]
C N ORMALIZATION FOR T HE C RITIC
We now study why penultimate normalization is helpful for the critic network.
We consider three agents from Table 2: baseline, actor pnorm, and both pnorm.
The both pnorm agent uses penultimate normalization for the critic, whereas the others do not.
We plot two additional quantities during early training for the reacher hard task, using 10 new seeds.
In Figure 19 we show the average Q-values, measured over training batches, achieved by actions generated by the actor network.
We see that the curves are much smoother when normalizing the critic, whereas they oscillate aggressively otherwise.
Quickly oscillating Q-values makes it hard for the actor to learn a good policy.
We hypothesize that stabilizing the Q-values over time is the primary mechanism by which critic normalization helps.
In Figure 20 we show the average absolute difference between Q-values before and after a gradient update, as measured over training batches.
We see that normalizing the critic often leads to Q-value changes which are smaller and more stable over time.
This in turn likely stabilizes learning as per Figure 19.
Whereas the Q-value changes can trivially be made smaller by decreasing the learning rate, changes in Q-values depend non-linearly on the learning rate.
Furthermore, too low learning rates often hurt generalization (Li et al., 2019).
For increased legibility, the curves are smoothed by averaging with a window of width ten.
[Table]
/+sOYpRFKwwTVuue5ifEzqgxnAmelfqoxoWxCR9izVNIItZ8tDp2RC6sMSRgrW9KQhfp7IqOR1tMosJ0RNWO96s3F/7xeasIbP+MySQ1KtlwUpoKYmMy/JkOukBkxtYQyxe2thI2poszYbEo2BG/15XXSvqp6btVrXlfqt3kcRTiDc7gED2pQh3toQAsYIDzDK7w5j86L8+58LFsLTj5zCn/gfP4Aqx2M1Q==</latexit>
[Table]
[Figure]
D L EARNING R ATE S TABILITY
To verify that our ﬁxes increase the stability we now consider changing the learning rate.
A high learning rate can cause divergence (Li et al., 2019)
[Table]
and learning rate stability can thus serve as a proxy for learning stability.
We consider learning rates in the set { 1e − 4 , 3e − 4 , 1e − 3 , 3e − 3 } , where 1e − 4 is the default learning rate used by Yarats et al.
(2021b).
Results for the walker task are shown in Table 8.
We see that the baseline algorithm fails for large learning rates, the average rewards drop quickly while variance decreases when no effective policy is learned.
We show the same experiment for additional tasks, but only one learning rate, in Table 9.
Again, we see that the agent can tolerate much higher learning rates when using our modiﬁcations.
[Table] [Table]
[Table] [Table]
E E VALUATION V ARIANCE
We now consider the variance arising from evaluating RL agents on a ﬁnite set of episodes.
Similar to the bias-variance decomposition, we will see that the variance here can be divided into one part that depends upon the randomness of the algorithm, and one term that depends on the randomness from the ﬁnite set of episodes we evaluate on.
We let R be a set of training runs and S be a set of evaluation episodes sampled.
The performance is measured by
[Equation]
Let us calculate the variance of this quantity.
We will assume that rewards from different runs are independent, and that rewards from the same training run are independent conditioned on that run.
We now have
[Equation]
The variance of a sum of independent variables is simply the sum of their variances.
Thus we have
[Equation]
Let µ = E [ r ij ] .
We then have
[Equation]
Here we explicitly calculate the expectation over random training runs ( alg ) and the random episodes sampled ( samples ).
We can condition the expectation upon the random training run and then integrate over this expectation.
Furthermore, let µ i =
E [ r ij | i ] , i.e. the expected value an episode sampled from training run i .
We then have
[Equation]
The middle part vanishes since µ i = E
[ r ij | i ] .
For the ﬁrst term, given a ﬁxed training run the terms r ij are independent.
This gives us
[Table]
We then have
[Equation]
[Table]
[Equation]
Here µ alg = µ i .
The ﬁrst term is what we are interested in measuring.
The second term can be large if the environment is noisy and we sample too few episodes.
The second term is straightforward to estimate: run the algorithm N times and measure the performance over M samples.
For each run, calculate the variance over the samples.
Then average this variance over all runs.
In Table 10 we estimate these quantities over ten training runs and ten episodes for the baseline algorithm.
As we can see, the environments have very different characteristics.
Some have high sample variance and some have very low sample variance.
It is common to evaluate with ten training seeds and ten episodes.
Can we improve the variance by evaluating on more episodes, say 100?
Even for the task with the highest sample noise (reacher), the relative improvement in the standard deviation of the sample variance, i.e.,
[Equation]
(4) is no larger than 20 %.
Thus, we conclude that only minor gains can come from evaluating on more episodes.
Other environments outside the ﬁve we consider, however, might beneﬁt more from using more samples.
F E XTENDED R ELATED W ORK
There are plenty of important applications of RL.
Examples include RL for medical testing (Bastani et al., 2021), autonomous vehicles (Bellemare et al., 2020), chemistry (Zhou et al., 2019;
Liu et al., 2019), robotics (Kober et al., 2013) and communications infrastructure (Liu et al., 2020).
Many more applications remain.
Below we discuss work most closely related to ours in detail.
Engstrom et al.
(2019) studies how implementation level details affects performance from policy gradient methods.
They speciﬁcally ﬁnd that code-level optimizations can have a signiﬁcant impact on performance, often rivaling that of algorithmic innovations.
This insight agrees with our experimental observations that small modiﬁcations can signiﬁcantly inﬂuence performance.
Henderson et al.
(2018) presents among the ﬁrst studies of variability and reproducibility in deep reinforcement learning.
They demonstrate that many minor details can have an outsized effect on the outcomes – seeds, environment randomness, and even hardware non-determinism.
In our work, we have demonstrated how improved implementations can signiﬁcantly decrease variance – and our improved agent suffers from much smaller variability between seeds.
Islam et al.
(2017) studies variance and reproducibility for continuous control with policy gradient methods.
They discuss what metrics are suitable for comparing algorithms and highlight how hyperparameter tuning signiﬁcantly affects outcomes and thus fairness of evaluations.
Our results show that variance in continuous control can be signiﬁcantly decreased with minor ﬁxes, but we do not study variance from hyperparameter tuning which is an important direction.
Mania et al.
(2018) demonstrates that random search is in fact a competitive method for ﬁnding policies for continuous control tasks.
This strategy is computationally cheap, and the authors present a large study of over 100 seeds, demonstrating that the variance is high.
RL still suffers from high variance, but it is not clear if random search still is competitive compared to RL agents.
Clary et al.
(2018) studies variability for the Atari benchmark and ﬁnds large variability.
In light of this, it is proposed that post-training performance should be reported as a distribution rather than as a point estimate.
Our paper does not study the Atari suite, but this is an important benchmark that can model discrete tasks rather than continuous robot control problems.
Lynnerup et al.
(2020) presents a study over variance for evaluating RL agents on real-world robots.
It is concluded that documenting hyperparameters and performing rigorous statistical analysis of the results is imperative.
We only consider simulated tasks which are easy to reproduce as long as source code is available, reproducibility becomes much harder in the real world.
Colas et al.
(2019) discusses hypotheses testing in the context of RL experiments.
They make recommendations regarding signiﬁcance testing and discuss violations of statistical assumptions.
Statistical testing has been criticized in later research (Agarwal et al., 2021), and we refrain from using it.
Chan et al.
(2020) proposes multiple metrics to measure notations of reliability in reinforcement learning, and presents an open-sources library to help with this.
They discuss the distinction between variability during or after training and the use of signiﬁcance testing.
We have refrained from using hypothesis testing as it suffers from some issues (Agarwal et al., 2021).
Jordan et al.
(2020) shows that hyperparameters signiﬁcantly affect the outcome of RL experiments, and proposes an evaluation strategy that takes this into account.
To limit the computational footprint, we have used ﬁxed hyperparameters.
While we do not study it, hyperparameter stability is very important for practical applications, and developing RL agents with such stability is an outstanding challenge.
Agarwal et al.
(2021) studies variance and statistical signiﬁcance for RL, focusing especially on
Atari tasks where they provide experiments over 100 seeds which shows outlier runs.
Among other things, they propose using inter-quartile-means and performance proﬁles for measuring performance.
We adopt the use of performance proﬁles, but since we are especially interested in outliers inter-quartile-mean is not suitable for our purposes.
Fujimoto et al.
(2018) discusses how variance can be harmful to Q-learning, and propose several ﬁxes to it, culminating in the TD3 agent.
Some of these ﬁxes have become very popular, especially double clipped Q-values.
In Appendix B we revisit double clipped Q-values and ﬁnd that it can hurt performance for sparse tasks.
Anschel et al.
(2017) shows that averaging Q-values can improve the stability and improve performance for a DQN agent.
This strategy increases the computational footprint, and such ensembling is known to often work well for supervised learning.
The paper mainly focuses on Atari tasks, in contrast, we consider continuous control tasks which arguably are closer to real-world applications.
Nikishin et al.
(2018), in a similar spirit, demonstrates that averaging weights can improve the stability of learning.
The authors demonstrate that this can improve performance on Atari tasks, although only a handful of seeds appears to be used.
To the best of our knowledge, this method is not widely adopted in RL, and we do not experimentally study it.
Jia et al.
(2020) considers using classical variance reduction methods such as SVRG (Johnson &
Zhang, 2013) to improve deep Q-learning.
The authors introduce a recursive framework for computing the stochastic gradient estimate and demonstrate that it can improve the performance of DQN.
Again, this method is not widely adopted in RL to the best of our knowledge, and we do not empirically study it.
Chen et al.
(2018) considers RL as applied to a dynamic environment, motivated by applications to online recommendation.
They propose stratiﬁed sampling replay, which relies on a prior customer distribution to decrease variance and the use of approximate regret.
These ideas improve performance for DQN agents for a tip recommendation system, while this is an important setting, it is far from continuous control.
Greensmith et al.
(2004) studies the use of variance reduction methods for policy gradient methods.
Speciﬁcally, they study the use of baseline methods, which can signiﬁcantly decrease the variance and often is adopted in deep reinforcement learning.
The use of baselines for policy gradient methods is distinct from the setup we consider.
Parisotto et al.
(2020) considers the use of transformers for reinforcement learning and ﬁnds that naively using them leads to poor performance.
To address this, architectural modiﬁcations are proposed that lead to more stable training and improve performance.
Studying transformer architectures is important for RL, but in this study, we only consider classical CNN and MLP networks.
Kumar et al.
(2019) considers stability for off-policy reinforcement learning, focusing on a setting where the off-policy data is ﬁxed and there is no further interaction with the environment.
In this setting, it is demonstrated that bootstrapping error is a key driver of instability, a method to address this issue is proposed.
Off-policy reinforcement learning is very important for practical applications, but not the focus of our work.
Mao et al.
(2019) considers RL where a stochastic exogenous process inﬂuences the environment.
In such a setting, a gradient policy method with a standard state-dependent baseline suffers from high variance, and the authors consider ﬁxes to this.
The setting considered in this work is distinct from the continuous control tasks we use.
Gogianu et al.
(2021) studies the use of spectral normalization (Miyato et al., 2018) for improving the performance of DQN for Atari tasks.
They ﬁnd that this method can stabilize learning when applied to the penultimate layer.
Due to the similarity with the normalization methods we consider, we consider this strategy in Table 2.
Wang et al.
(2020) studies the behavior of a SAC agent when the entropy bonus is removed.
They ﬁnd that the tanh activations saturate which effectively eliminates exploration since SAC adds exploration noise inside the tanh non-linearity.
This argument does not apply to DDPG which adds the exploration noise outside the non-linearity.
We study the proposed method Table 2 and ﬁnd that it underperforms, especially for the acrobot task.
Salimans & Kingma (2016) proposes weight normalization, which reparametrizes the weights by decoupling magnitude and direction.
They demonstrate that this idea improves performance in DQN and other settings.
Despite the paper being well-known, this strategy has not become widely adopted in RL to the best of our knowledge.