I S H IGH V ARIANCE U NAVOIDABLE IN RL? A C ASE S TUDY IN C ONTINUOUS C ONTROL
Anonymous authors
Paper under double-blind review
A BSTRACT
Reinforcement learning (RL) experiments have notoriously high variance, and minor details can have disproportionately large effects on measured outcomes.
This is problematic for creating reproducible research and also serves as an obstacle for real-world applications, where safety and predictability are paramount.
In this paper, we investigate causes for this perceived instability.
To allow for an in-depth analysis, we focus on a speciﬁcally popular setup with high variance – continuous control from pixels with an actor-critic agent.
In this setting, we demonstrate that poor outlier runs which completely fail to learn are an important source of variance, but that weight initialization and initial exploration are not at fault.
We show that one cause for outliers is numerical instability which leads to saturating nonlinearities.
We investigate several ﬁxes to this issue and ﬁnd that one particular method is surprisingly effective and simple – normalizing penultimate features.
Addressing the instability signiﬁcantly decreases the variance of outcomes, lowering the average standard deviation across 21 tasks by a factor > 3 for a state-of-the-art agent.
This demonstrates that the perceived variance in RL is not necessarily inherent to the problem deﬁnition and may be addressed via simple modiﬁcations.
I NTRODUCTION
With the advent of deep learning, reinforcement learning (RL) with function approximators has become a promising strategy for learning complex behavior (Silver et al., 2017; Mnih et al., 2015).
While RL can work remarkably well, deep RL experiments have been observed to exhibit high variance (Chan et al., 2019; Clary et al., 2019; Lynnerup et al., 2020).
Speciﬁcally, the rewards obtained by RL agents commonly vary signiﬁcantly depending on the random seeds used for training, even if no other modiﬁcations are made.
Furthermore, implementation level details often have an outsized effect on performance (Engstrom et al., 2020) and even hardware scheduling can signiﬁcantly affect outcomes (Henderson et al., 2017).
In comparison to RL, experiments in computer vision and natural language processing (NLP) often have lower variance (Bouthillier et al., 2021).
High variance is problematic for creating reproducible RL research, and achieving statistically signiﬁcant results over large RL benchmarks is often prohibitively time-consuming (Colas et al., 2018;
Agarwal et al., 2021).
Furthermore, this variance provides an obstacle for real-world applications of RL, where safety and predictability can be critical (Garcıa & Fern´andez, 2015; Srinivasan et al., 2020).
RL is already being evaluated for medical testing (Bastani et al., 2021), autonomous vehicles (Bellemare et al., 2020) and communications infrastructure (Liu et al., 2020).
RL agents are typically evaluated on the average performance, but for these sensitive real-world applications, the drastic performance drop for the worst-performing training runs could yield disastrous outcomes.
In this paper, we aim to investigate causes for high variance in RL.
To limit the paper’s scope and computational footprint we focus on continuous control from pixels with a state-of-the-art (SOTA) actor-critic algorithm (Lillicrap et al., 2015; Yarats et al., 2021b).
This is a popular setup that exhibits high variance, and where actor-critic algorithms often give SOTA results (Laskin et al., 2020b;a;
Kostrikov et al., 2020).
Additionally, this class of tasks is relatively close to real-world robotics (Zhan et al., 2020).
In this setting, we ﬁrst show that an important cause of variance is outlier runs that fail early and never recover.
It is natural to hypothesize that the typical sources of randomness, network initializa- tion and initial exploration, are to blame.
However, we demonstrate that these do in fact have little impact on the ﬁnal performance.
We similarly provide experiments that suggest that neither poor feature representations nor excessive sensitivity of the environment is at fault.
We instead demonstrate that numerical instability can drive the variance and show that excessively large activations that saturate action distributions can often yield poor performance.
We propose several methods to avoid such issues and ﬁnd that they decrease variance.
The most effective method is simply normalizing the penultimate features which avoids exploding activations by design.
By combining multiple approaches we increase average rewards while decreasing variance as measured of 21 tasks.
The average standard deviation decreases by a factor > 3 and for some tasks by more than an order of magnitude.
Our experiments suggest that high variance is not necessarily inherent to RL and that addressing numerical issues is a promising avenue for stabilizing RL agents.
We summarize our contributions:
• We provide extensive experiments on variance in a popular continuous control setup.
We show that outlier runs that completely fail to learn is an important driver of variance, but that poor initialization, sensitive environments, and feature learning are likely not at fault.
• We demonstrate that excessively large activations that saturate action distributions can cause poor outlier runs and propose ﬁxes to stabilize learning and avoid this issue.
• Over 21 continuous control tasks and with a SOTA agent, we show that our modiﬁcations decrease the average standard deviation by a factor > 3 while improving average reward.
This shows that directly optimizing for lower variance can be both feasible and fruitful.
B ACKGROUND
Reinforcement Learning.
Reinforcement learning tasks can be formulated as Markov decision processes (MDPs) deﬁned by a state-space S , an action space A , transition probabilities P , and a reward distribution r (Sutton & Barto, 2018).
For continuous control tasks, the action space
A and the state space S are typically continuous.
Each dimension of the action space
A might for example correspond to a joint of a robotic arm, and the space is often bounded to account for physical limitations, e.g., how much force the robotic arm can exert.
A common setting is simply
A =
[ − 1 , 1] dim ( A ) .
At every time step t , the agent is in some state s t ∈S and takes an action a t ∈A to arrive at a new state s t +1 ∈ S .
The transition between states given an action is random with transition probability P : S × S ×
A → [0 , ∞ ) .
Given some action a t at timestep t , the agent also receives a reward r t from the reward distribution r .
When benchmarking RL agents, it is common to measure performance by the cumulative rewards (cid:80) t r t .
DDPG.
Deep Deterministic Policy Gradients (DDPG) (Lillicrap et al., 2015) is an algorithm for continuous control with roots in Q-learning (Watkins, 1989).
Together with its offshoot soft actorcritic (SAC) (Haarnoja et al., 2018), DDPG forms the backbone of many state-of-the-art algorithms for continuous control (Laskin et al., 2020b;a; Kostrikov et al., 2020; Yarats et al., 2021b).
DDPG uses a neural network (the critic ) to predict the Q-value Q φ ( a t , s t ) for each state-action pair.
The parameters φ of the critic are trained by minimizing the soft Bellman residual:
[Equation]
Here, r t is the reward and E [ ˆ Q ( a t +1 , s t +1 )] is the Q-value estimated by the target critic – a network whose weights can e.g. be the exponentially averaged weights of the critic.
One can use multi-step returns for deﬁning the critic target (Sutton, 1988) and double clipped q-learning (i.e. use two Qnetworks Q 1 , Q 2 and take Q = min( Q 1 , Q 2 ) ) to decreased Q-value overestimation (Fujimoto et al., 2018).
The policy π ( s ) =
arg max a Q φ ( s , a ) is intractable since it requires ﬁnding arg max over a continuous action space.
Instead the policy is given by an actor -network with parameters θ , which maps each state s t to a distribution π θ ( s t ) over actions.
The action is obtained by calculating a preθ ( s ) , the vector output of the actor network.
Thereafter a tanh non-linearity is applied elementwise to ensure a ∈ [ − 1 , 1] n for n = dim ( A ) .
The policy is then obtained as:
[Equation]
(cid:15) is some additive noise which encourages exploration, e.g. a Gaussian variable with time-dependent variance σ 2 t .
Various noise types have been employed (Achiam, 2018; Lillicrap et al., 2015), Yarats et al.
(2021b) for example use Gaussian noise which is truncated outside of acting.
Since the Qvalues correspond to the expected discounted rewards, the actions a θ should maximize Q φ ( a θ , s ) .
Thus, to obtain a gradient update for the actor one uses the gradient ∇ θ Q φ ( a θ , s ) , effectively calculating gradients through the critic network.
In control from pixels, the state s is typically an image (possibly data augmented) processed by a convolutional network (Kostrikov et al., 2020; Laskin et al., 2020a).
B ENCHMARKING V ARIANCE 3. E XPERIMENTAL S ETUP
There is a large number of available RL agents and environments to consider.
To limit the paper’s scope and computational footprint, we consider continuous control from pixels with an actor-critic algorithm.
Continuous control from pixels is an environment type close to real-world applications such as autonomous driving (Kiran et al., 2021) and robotics (Zhu et al., 2020).
Investigating causes of high variance in this domain could therefore have a large potential impact on downstream applications.
We consider the standard continuous control benchmark deepmind control (dm-control)
(Tassa et al., 2020).
Within dm-control, a long line of state-of-the-art models relies on the actor-critic setup (Laskin et al., 2020b;a; Kostrikov et al., 2020; Yarats et al., 2021b).
We consider DDPG (Lillicrap et al., 2015), using the recent data-augmentation based DDPG agent of Yarats et al.
(2021b) which provides state-of-the-art performance and runs fast.
We use the default hyperparameters that
Yarats et al.
(2021b) uses on the medium benchmark (listed in Appendix A) throughout the paper.
We will refer to this agent as the baseline agent.
We are interested in using a large number of random seeds to obtain accurate statistics and thus need to limit the number of environments we consider.
To select environments we simply rank the dm-control tasks based upon their relative variance as measured by Yarats et al.
(2021a).
We select the ﬁve tasks with the highest relative variance – ﬁnger turn hard, walker walk, hopper hop, acrobot swingup, and reacher hard (see Table 5 in Appendix A for details).
In Section 4.2 we consider a larger set of tasks.
We will refer to these as environments or tasks interchangeably.
For each run, we train the agent for one million frames, or equivalently 1, episodes, and evaluate over ten episodes.
We use 10 or 40 seeds for each method we consider, these are not shared between experiments, but we reuse the runs for the baseline agent across ﬁgures.
[Figure] [Figure]
3. V ERIFYING V ARIANCE
We ﬁrst aim to verify that the environments indeed do exhibit high variance.
To this end, we run the baseline algorithm with 40 random seeds on each task.
Figure 1 shows the result of this experiment for three tasks, see Appendix A for the two other tasks.
We show both individual runs as well as mean and standard deviations.
For some tasks, the variance is dominated by outlier runs that fail to learn early in the training and never recover.
E.g. for the walker task, we see several runs which have not improved during the ﬁrst 500 episodes, and most of these do not improve in the following 500 episodes either.
Except for these outliers, the variance of the walker task is relatively small.
We also note that these outliers separate themselves relatively early on, suggesting that there are some issues in early training that cause variance.
If it was possible to eliminate poor outlier runs, we would expect the variance to go down.
Motivated by these observations, and the walker task in particular, we will investigate various potential causes of outlier runs.
3. D OES INITIALIZATION MATTER ?
There are multiple sources of randomness speciﬁc to the early training iterations.
The agent networks are randomly initialized and the agent is given some initial experience, collected by random actions, to start training on.
Could these two initial factors be the cause of the high variance?
To study this hypothesis we measure the correlation between two runs that share the same network initialization and initial experience, but otherwise have independent randomness (from e.g. the environment, action sampling, and mini-batch sampling).
Across all tasks, we repeat this process for ten combinations of initializations and initial experience, from each such combination we start two runs
A, B which do not share other sources of randomness.
For each time-step t we measure the Pearson correlation between score ( A ) and score ( B ) .
This correlation, averaged across all timesteps, is shown in Figure 2 (left).
We see that the correlation is very small, suggesting that the initialization and initial experience does not have a large effect on the outcome.
Speciﬁcally, the hypothesis that there are good and bad initializations that leads to good or bad performance is inconsistent with the results of Figure 2 (left).
Correlation across time is noisy but centers around zero, see Appendix A.
3. A RE THE E NVIRONMENTS TOO S ENSITIVE ?
Another cause of variance is the environments.
Perhaps the environments require very speciﬁc policies to yield rewards, and unless the agent stumbles upon a good policy early on, learning fails.
To study the sensitivity of the environments we consider gradually exchanging a good learned policy
[Table]
with a random policy.
Speciﬁcally, at every timestep t , with probability p
[Table]
we exchange the policy’s action with a random action.
We use the ﬁnal policy of a performant run with the baseline agent as the learned policy.
The effect of this, for ten ﬁnal policies learned for the walker environment, is shown in Figure 2 (right).
Performance degrades gracefully as we gradually move to a random policy.
This suggests that the environment is relatively stable with regard to actions.
Results for other environments are similar and are shown in Appendix A. Furthermore, in Appendix B we investigate how much variance comes from evaluating the agent on a ﬁnite number of episodes.
When using ten seeds and ten episodes (the lowest numbers we use), the evaluation variance is dominated by the variance of the random training process.
Thus, environment sensitivity and evaluation methodology do not appear to explain the excessive variance.
3. I S FEATURE LEARNING TO BLAME ?
In the early stages of training, the image encoder will extract poor features.
Without good representations, it is hard to learn a good policy, and without a policy generating interesting behavior it is hard to learn good representations.
If such a vicious cycle was causing runs to fail, jump-starting the encoder to extract good features should help to decrease the variance.
To test this hypothesis, we consider initializing the weights of the convolutional encoder to some good pretrained values before training as normal.
For each task, we pick a run with an above-average ﬁnal performance and extract the convolutional weights at episode 500.
We refer to this strategy as pretrain.
We also consider the same strategy but decrease the learning rate of the pre-trained weights by a factor of 10x.
We refer to this strategy as pretrain + small lr.
In addition to this ﬁx, we also consider improving the features with self-supervised learning during early training (Jing & Tian, 2020).
We use the loss function proposed in Grill et al.
(2020) which is useful for smaller batch sizes (we use 256) and has already successfully been applied in RL (Schwarzer et al., 2020).
We use standard methods in self-supervised learning: a target network with exponentially moving weights and a head network.
See Appendix A for details.
We consider adding the self-supervised loss to the ﬁrst 10,000 and 20,000 steps and refer to these modiﬁcations as self-supervised 10k and self-supervised 20k.
The results for this experiment are shown in Table 1.
For these ﬁxes, the large variance persists even though the performance improves marginally with self-supervised learning.
We thus hypothesize that representation learning is not the main bottleneck.
Indeed, experiments in later sections of the paper show that one can signiﬁcantly decrease variance without improving representation learning.
3. S TABILITY OF N ETWORK P ARAMETRIZATION
We have considered the effects of initialization, feature learning, and environment sensitivity and found that none of these factors satisfactorily explain the observed variance.
Another factor to consider is the stability of the learning algorithms.
Such instability could make the learning process “chaotic”, which would cause high variance.
To study this hypothesis we investigate the failure modes of individual runs.
In Figure 3 we consider some runs that fail as well as some successful runs for the walker task.
Plots for all runs are available in
Appendix A. We plot four quantities during training: rewards, actor gradient norm (cid:107)∇ (cid:96) (cid:107) , actor weight norm (cid:107) w (cid:107) , and average absolute value of actions | a | .
All agents initially have poor rewards, but once the rewards start to improve, they improve steadily.
During this initial period of no or little learning, the gradients of the actornetwork (row two in Figure 3) are small.
Similarly, the weight norm of the actor seems to be constant (row three in Figure 3).
This suggests that the actor isn’t changing its weight much (technically the weights could be “rotating”, but this seems unlikely).
Why would the actor weights not change?
Recall that the actor is optimized via gradients:
[Equation]
If this gradient is close to zero, we might suspect that ∂ a
i ∂θ ≈ 0 .
This could easily happen if the tanh non-linearity in equation 2 is saturated.
In row four of Figure 3 we show the average absolute value of the actions, which are all bound to lie in [ − 1 , 1] .
We see that for the runs that fail the average absolute action stays around 1 throughout training, which implies that the gradients for the actor are small since | tanh( x ) | ≈ 1 implies ∂∂x tanh( x ) ≈ 0 .
The cause of saturating tanh must be excessively large activations, i.e. a pre θ ( s ) in equation 2 becoming too large, which saturates the tanh non-linearity.
In fact, all runs seem to initially suffer from saturating tanh, but most runs escape from this poor policy.
The Q-values could of course also suffer from exploding activations, but since there are no saturating nonlinearities for the Q-values, such issues are harder to pinpoint.
These experiments suggest that improving numerical stability and avoiding excessively large activations could reduce the number of poor outlier runs, and thus decrease variance
[Table]
r e w a r d
[Table] [Table]
/RJlGhbCslM/T2R09iYcRzazpji0Cx6U/E/r5thdBXkQqUZcsXmi6JMEkzINADSF5ozlGNLKNPC3krYkGrK0MZUsSF4iy8vk/ZZ3XPr3t1FrXFdxFGGIziGU/DgEhpwC01oAYMUnuEV3pzMeXHenY95a8kpZg7hD5zPHxdAkbM=</latexit>
| a | .
Each column corresponds to an independent run of the walker task: run 1 is successful, runs 2 and 3 eventually learn, while run 4 fails to learn.
Before learning starts, all runs suffer from small gradients and weights which barely change.
During this period, the average absolute value of actions is close to 1.
This causes small gradients since the gradient of tanh( x ) is small when | tanh( x ) |
≈ 1 .
This suggests that numerical issues may hamper learning.
I MPROVING S TABILITY
Motivated by our qualitative observations we now consider a few ideas for improving the stability of learning, hoping that these will decrease the variance.
We consider several methods:
Normalizing activations.
Recall that the actor selects actions using a preθ ( s ) , where a preθ ( s ) is a deep neural network.
Figure 3 shows that learning can fail due to excessively large activations.
To avoid this we propose to parametrize a preθ ( s ) as a linear function of features with a ﬁxed magnitude.
Speciﬁcally, let Λ θ ( s ) be the penultimate features of the actor-network.
We propose to simply normalize Λ nθ ( s )
=
Λ
[Equation]
) / (cid:107) Λ θ ( s ) (cid:107) and let a pre be a linear function of Λ nθ , i.e. a preθ ( s ) =
L Λ nθ
[Equation]
s ) for a linear function L .
We refer to this straightforward modiﬁcation as actor norm .
Our motivation has been to stabilize the actor output, but rapidly changing critic output might also be harmful and the critic network could similarly beneﬁt from normalizing penultimate features.
We refer to the strategy of normalizing both the actor and the critic as both norm .
We also consider layer norm (Ba et al., 2016) applied similarly to both the actor and critic.
At last, following Gogianu et al.
(2021), we consider applying spectral normalization (Miyato et al., 2018) to the second to last layer.
Penalizing saturating actions.
Another strategy for avoiding actions with large magnitude is to add a term to the loss function which directly penalizes large actions.
One can imagine many such loss functions, we here consider the simple and natural choice of squared loss:
[Equation]
We can simply add this term to the loss function times some hyperparameter λ .
By tuning λ , one should be able to remove the saturating actions illustrated in Figure 3.
We heuristically ﬁnd that λ = 0 .
000001 works well in practice (see Appendix A) and use this value.
We refer to this modiﬁcation as penalty .
Since the Adam optimizer is invariant under re-scaling the loss, the speciﬁc value of λ has a small effect as long as ∇ θ Q φ ( s , a θ ( s ))
≈
(which happen as per Figure 3).
Learning rate warmup.
One common strategy to improve stability is to start the learning rate at zero and increase it linearly to its top value during early training (Goyal et al., 2017).
This can improve numerical stability (Ma & Yarats, 2019).
We consider using this strategy and linearly increasing the learning rate from zero over the ﬁrst 10,000 steps.
We refer to this strategy as lr warmup .
[Table]
Small weights ﬁnal layer.
Following Andrychowicz et al.
, we consider using small weights for the ﬁnal layers of the actor and the critic.
Speciﬁcally, we downscale these weights by a factor of 100 at initialization time.
We refer to this strategy as scale down .
Gradient clipping.
Another stabilizing strategy is gradient clipping (Zhang et al., 2019).
To avoid occasional exploding gradients, one simply clips the norm of gradients that are above some threshold.
This strategy is popular in NLP (Gehring et al., 2017; Peters et al., 2018) and is sometimes used in RL (Rafﬁn et al., 2019).
We consider clipping the gradient when the norm, calculated independently for the actor, critic, and convolutional encoder, is larger than 1 or 10.
We refer to these strategies as grad clip 1 and grad clip 10 .
Results.
In Table 2 we evaluate the effects of these ideas.
Normalizing activations has the best performance and lowest variability on average, whereas layer norm can make performance worse.
Since normalizing the critic improves performance, the critic might also suffer from instabilities.
We study this point in Appendix D. The penalty method improves the performance while other methods decrease average reward, and lowering variance by using a poor policy is not hard to achieve and not particularly desirable.
These experiments show that it is possible to signiﬁcantly decrease the variance of the baseline agent by applying stabilization methods.
Additionally, decreasing the variance often increases the average reward – likely as bad outlier runs are eliminated.
4. C OMBINING F IXES
We have found that many of the proposed methods can decrease variance and improve average reward.
For optimal performance, we now consider combining three orthogonal methods which improve performance individually – actor and critic penultimate normalization, pre-tanh penalty, and self-supervised learning (we apply it to the ﬁrst 10,000 steps).
In Table 3 we compare the performance between the baseline and an agent using these three modiﬁcations (which we refer to as combined ) over 40 seeds.
We see that the variances go down while the average score goes up.
In fact, for the walker task, the variance decreases by orders of magnitude.
Eliminating poor outlier runs seems to beneﬁt the average performance in addition to decreasing variance.
While our methods have mostly been motivated by outlier runs that fail catastrophically, they likely help more normal runs too.
Learning curves for the combined agent are given in Appendix A.
[Table]
A DDITIONAL T ASKS
We now apply the combined agent of Table 3 to additional tasks.
Speciﬁcally, we consider alltasks identiﬁed as easy or medium by Yarats et al.
(2021b).
Note that the combined agent has been constructed using only 5 of these tasks, so this experiment measures generalization.
For computational reasons, the additional 16 tasks only use 10 seeds each.
We use the baseline hyperparameters across all tasks without tuning (Yarats et al.
(2021b)
tunes parameters on a per-task basis).
Results are shown in Figure 4.
Our methods consistently improve the average reward while the average standard deviation decreases by a factor > 2 .
Furthermore, a few additional tasks (ﬁnger spin, hopper stand, walker run, walker stand) have their variance decreased by an order of magnitude or more.
The tasks cheetah run and quadruped walk perform worse than the baseline, although this could potentially be alleviated by tuning hyperparameters.
Just improving learning rate on these tasks improve performance, see Figure 12 in Appendix A.
These results demonstrate that our modiﬁcations signiﬁcantly decrease variance and that directly optimizing variance can be feasible and fruitful.
Consider the ﬁve tasks in Figure 4 where combined has the highest standard deviation.
Four out of these have sparse rewards – ﬁnger turn hard, reacher hard, pendulum swingup, and cartpole swingup sparse.
Once stability has been addressed, sparsity seems to be a major driver of residual variance.
In Appendix C we consider further modiﬁcation to improve performance speciﬁcally for sparse tasks.
We ﬁnd that removing the double clipped Q-learning for the critic loss and extending the selfsupervision helps for these tasks.
With these changes, the agent achieves even better performance and an average standard deviation which is > 3 times lower than the baseline.
Performance proﬁles (Agarwal et al., 2021) are available in Figure 15 of Appendix A.
4. L EARNING RATE STABILITY
To verify that our ﬁxes indeed increase the stability of learning we now consider changing the learning rate.
A high learning rate can cause divergence (Li et al., 2019), and can thus serve as a proxy for learning stability.
We consider learning rates in the set { 1e − 4 , 3e − 4 , 1e − 3 , 3e − 3 } , where 1e − 4 is the default learning rate used by Yarats et al.
(2021b).
Results for the walker task are shown in Table 4.
We see that the baseline algorithm fails for large learning rates, the average rewards drop quickly while variance decreases when no effective policy is learned.
With our modiﬁcations, the agent can tolerate much higher learning rates while retaining good average rewards and small standard deviation.
Finally, to verify that the actions do not saturate when using our ﬁxes, we repeat the experiments of Figure 3 with the combined agent.
The actions are shown in Figure 5, where we see that the action magnitude indeed goes up quickly early in training but never saturates.
This supports our hypothesis that normalizing activations can stabilize training.
[Table]
[Table] [Table]
=
[Table]
latexit>
[Table]
=</latexit>
R ELATED WORK
Variance in RL.
Many researchers have studied variance in RL: Engstrom et al.
(2020) demonstrates variability from small implementation details;
Henderson et al.
(2017) shows how even hardware non-determinism can cause variability; Islam et al.
(2017) studies reproducibility of benchmarking in RL; Mania et al.
(2018) demonstrates variability and visualizes outliers in continuous control over 100 seeds; Clary et al.
(2019) studies variability in the Atari suite; Lynnerup et al.
(2020) studies variance in real-world robots.
Given this high variance, there is ample work on how to accurately measure performance in RL (Colas et al., 2019).
Chan et al.
(2019) proposes several metrics that can measure different notations of reliability in reinforcement learning.
Jordan et al.
(2020) proposes an evaluation methodology that takes into account the effects of hyperparameter tuning.
Recently Agarwal et al.
(2021) shows outlier runs for atari tasks and proposes new robust estimates of performance.
Given this plethora of proposed metrics, we simply report average mean and standard deviation which is commonly reported in practice.
Our strategy of simply reducing the variance of RL algorithms is complementary to developing robust metrics.
Stabilizing RL.
Within Q-learning, it is known that high variance Q-estimates can impede learning (Fujimoto et al., 2018).
Hence, there is a large amount of work on improving the Q-estimates for
DQN: averaging weights (Nikishin et al., 2018), averaging Q-estimators (Anschel et al., 2017), using classical variance reduction methods (Jia et al., 2020) and regretted reward (Chen et al., 2018).
These papers focus on Q-value variance instead of performance variance and apply to the discrete DQN setting, distinct from continuous control.
In policy learning there are several methods for decreasing variance of gradient estimates (Greensmith et al., 2004; Wang et al., 2017), however, these do not directly apply to DDPG.
On the architectural side, Parisotto et al.
(2020) stabilizes transformers for RL.
On the task-speciﬁc side, Kumar et al.
(2019) stabilizes learning for ofﬂine tasks while Mao et al.
(2018) considers reducing gradient variance for externally driven environments.
Normalization strategies in RL have been investigated by Salimans & Kingma (2016) and Gogianu et al.
D ISCUSSION
Limitations.
We have considered a single SOTA algorithm on one popular suite of tasks.
Furthermore, we focus on an agent with tanh nonlinearities for the policy.
While this is an important agent class, other important agent types (Hafner et al., 2020) and environments (Bellemare et al., 2013) remain.
We show that it is possible to decrease variance with simple modiﬁcations, although which methods are effective likely depends upon the base algorithm.
We leave evaluating how well our modiﬁcations transfer to other settings for future work.
While our ﬁxes typically are only a few lines of code, they nonetheless increase the agent complexity.
Conclusion.
We have investigated causes of variance in a popular continuous control setup.
Whereas feature learning and network initialization do not appear to be at fault, we observe that learning instability causes signiﬁcant variance.
We propose several ﬁxes to this issue and demonstrate across 21 tasks that these dramatically decrease variance while improving average rewards for a state-of-the-art agent.
Our experiments demonstrate that the perception that RL has intrinsically high variance might need to be revisited – in some cases, numerical issues might instead be at fault.
Ethics Statement.
There are many positive RL applications – examples include safe autonomous cars, accelerated scientiﬁc discovery, and medical testing.
Studying causes of high variance, and proposing ﬁxes to this issue, can beneﬁt real-world adoption of RL techniques.
Additionally, reducing the variance of RL experiments would aid the research community as reproducibility can be problematic in the ﬁeld.
However, there are also many potentially harmful military applications.
Our study is academic in nature and only considers simulated toy environments, but insights from our work could nonetheless be useful in harmful applications.
We do not believe that our work differs from other RL work signiﬁcantly concerning ethics.
Reproducability Statement.
To aid the reproducibility of our work, anonymized code for the experiments is available 1 .
Furthermore, in Appendix A we detail the infrastructure and software used.
The perhaps most important component of generating reproducible ﬁndings in RL is to use many seeds.
To this end, we only consider ﬁve environments but evaluated these thoroughly.
The main experiments of the paper (Table 3) use 40 seeds whereas all others use 10 seeds.
A A PPENDIX Tuning λ .
For tuning the hyperparameters λ deﬁned in equation 4 we search over the set { 1e − 1 , 1e − 2 , 1e − 3 , 1e − 4 , 1e − 5 , 1e − 6 } .
For each value, we run the hopper task for a short amount of time with ﬁve different seeds.
We found that too large λ penalized actions too much, and that even very small values like 1e − 6 sufﬁced to give improvement.
Details on self-supervised Learning.
We extract features from the convolutional network and the ﬁrst feedforward layer of the critic, which projects the feature map to 50-dimensions.
Feature vectors are normalized to unit norm before being fed into the self-supervised loss (Bachman et al., 2019).
We use exponentially moving weights for the target network (He et al., 2019), using the same decay parameters as for the target critic.
We use a head for the online network which is simply a two-layer
MLP with hidden dimension 512 and ﬁnal dimension 50 (Chen et al., 2020).
The two augmentations we consider are simply two adjacent frames with augmentations that the DDPG agent of Yarats et al.
(2021b) considers naturally.
We use the square loss as proposed in Grill et al.
(2020), which has been shown to be useful in smaller batch sizes.
The self-supervised loss is then simply added to the normal loss function for the critic.
Infrastructure.
We run our experiments on Nvidia Tesla V100 GPUs and Intel Xeon CPUs.
The
GPUs use CUDA 11.1 and CUDNN 8.0.0.5.
We use PyTorch 1.9.0 and python 3.8.10.
Our experiments are all based upon the open-source implementation of Yarats et al.
(2021b).
task σ/µ µ σ ﬁnger
[Table]
hard
[Table]
walker walk 0.
hopper hop 0.
ﬁnger turn easy 0.
acrobot swingup 0.
reacher hard 0.
walker run 0.
cup catch 0.
ﬁnger spin 0.
quadruped run 0. quadruped walk 0.
cartpole balance sparse 0.
reach duplo 0.
cartpole swingup sparse 0.
cheetah run 0.
reacher easy 0.
pendulum swingup 0.
hopper stand 0.
cartpole swingup 0.
walker stand 0.
cartpole balance 0.
[Figure]
[Table]
[Figure] [Figure]
[Table]
/RJlGhbCslM/T2R09iYcRzazpji0Cx6U/E/r5thdBXkQqUZcsXmi6JMEkzINADSF5ozlGNLKNPC3krYkGrK0MZUsSF4iy8vk/ZZ3XPr3t1FrXFdxFGGIziGU/DgEhpwC01oAYMUnuEV3pzMeXHenY95a8kpZg7hD5zPHxdAkbM=</latexit>
[Figure]
[Figure] [Figure]
[Table]
=
=</latexit>
[Table] [Table]
=
=</latexit
[Table]
=
[Table]
a | <latexit sha1_
[Table]
=
[Table]
="
[Table]
[Table] [Table]
[Figure]
L1oKTzxzDHzifP4Z
[Figure]
latexit>
[Table]
[Figure] [Figure] Our combined agents improve performance, especially for the lowest quantiles. The effect is most dramatic for the 5 original tasks, this is natural since we initially selected these as they suffered from high variance.
Also note that the baseline is the recent SOTA agent of Yarats et al
), which is very competitive – so any improvements to it can be very signiﬁcant
[Table]
[Figure]
B E VALUATION V ARIANCE
We will now consider the variance arising from evaluating RL agents on a ﬁnite set of episodes.
Similar to the bias-variance decomposition, we will see that the variance here can be divided into one part that depends upon the randomness of the algorithm, and one term that depends on the randomness from the ﬁnite set of episodes we evaluate on.
We let R be a set of training runs and S be a set of evaluation episodes sampled.
The performance is measured by
[Equation]
Let us calculate the variance of this quantity.
We will assume that rewards from different runs are independent, and that rewards from the same training run are independent conditioned on that run.
We now have
[Equation]
The variance of a sum of independent variables is simply the sum of their variances.
Thus we have
[Equation]
Let µ = E [ r ij ] .
We then have
[Equation]
Here we explicitly calculate the expectation over random training runs ( alg ) and the random episodes sampled ( samples ).
We can condition the expectation upon the random training run and then integrate over this expectation.
Furthermore, let µ i =
E [ r ij | i ] , i.e. the expected value an episode sampled from training run i .
We then have
[Equation]
The middle part vanishes since µ i = E
[ r ij | i ] .
For the ﬁrst term, given a ﬁxed training run the terms r ij are independent.
This gives us
[Table]
We then have
Var
[Equation]
[Table]
[Equation]
Here µ alg = µ i .
The ﬁrst term is what we are interested in measuring.
The second term can be large if the environment is noisy and we sample too few episodes.
The second term is straightforward to estimate: run the algorithm N times and measure the performance over M samples.
For each run, calculate the variance over the samples.
Then average this variance over all runs.
In Table 7 we estimate these quantities over ten training runs and ten episodes for the baseline algorithm.
As we can see, the environments have very different characteristics.
Some have high sample variance and some have very low sample variance.
It is common to evaluate with ten training seeds and ten episodes.
Can we improve the variance by evaluating on more episodes, say 100?
Even for the task with the highest sample noise (reacher), the relative improvement in the standard deviation of the sample variance, i.e.,
[Equation]
(6) is no larger than 20 %.
Thus, we conclude that only minor gains can come from evaluating on more episodes.
Other environments outside the ﬁve we consider, however, might beneﬁt more from using more samples.
C I MPROVEMENTS FOR S PARSE T ASKS
In Figure 4, consider the ﬁve tasks where the combined agent suffers from the highest standard deviation.
Four out of these have sparse rewards.
This suggests that sparsity is a major driver of variance.
To further decrease the variance, we now investigate ﬁxes for four sparse tasks – cartpole balance sparse, pendulum swingup, reacher hard, and ﬁnger turn hard.
We will consider three additional modiﬁcations which aim to speciﬁcally combat reward sparsity:
Avoiding training without rewards.
For the sparse tasks, an agent might not see any rewards during the ﬁrst 100 episodes.
Perhaps training in the absence of any rewards can ruin the network weights.
The ﬁx we consider is simple, do not apply any gradient updates before at least one non-zero reward has been observed.
Increasing self-supervised learning.
Since the agent might potentially not observe any rewards for a hundred episodes, we consider extending the self-supervised learning.
This could still support a supervisory signal to the network.
We simply extend the self-supervised time from 10k steps to the entire training process.
Removing clipped double q-learning.
The agent of Yarats et al.
(2021a)
employs clipped double q-learning (Fujimoto et al., 2018) – i.e. the agent uses two critic networks Q 1 , Q 2 and takes
Q = min( Q 1 , Q 2 ) to avoid overestimation.
We hypothesise that the min-operator causes excessive pessimism in sparse environments where reward often are zero.
For example, if Q 1 = 0 and Q 2 >we have Q = min( Q 1 , Q 2 ) = 0 .
Thus, if one Q-network outputs only zeros and the rewards are almost only zeros, the critic target of eq.
(1) will be mostly zeros and the other Q-network will be encouraged to output mostly zeros too.
Thus, the Q-network could potentially get stuck with mostly outputting zeros, an issue shown in Figure 18.
We thus propose to use Q a =
avg ( Q 1 , Q 2 ) for deﬁning the targets in Equation (1).
We still use double clipping for the actor, i.e. the actor gradients are ∇ θ min( Q 1 φ ( a θ , s ) , Q 2 φ ( a θ , s , ) as before.
Results for these experiments are shown in Table 8.
We see that decreasing pessimism and increasing self-supervised learning help the most.
Increasing self-supervised learning helps the most for cartpole swingup sparse and pendulum swingup.
Note that these two tasks were not used in the ablation experiments of Table 2.
Additionally, whereas feature learning was not the bottleneck for the baseline agent, it might be the bottleneck for the combined agent.
Avoiding training without rewards does not seem to improve performance signiﬁcantly.
We conclude that increased self-supervised learning and decreasing pessimism improve performance for tasks with sparse rewards.
We add these two modiﬁcations to the combined agent and call the resulting agent combined++.
Its performance across all 21 tasks is shown in Figure 17.
It achieves even better results, and its average standard deviation is smaller than the baseline by a factor > 3 .
[Table]
+
[Table]
/asXL3K4yiQQ3JETohHzkmV3JAaqRNONHkmr+
[Table]
+
[Table]
=
=</latexit>
[Table]
10 <latexit sha1_base64="rMZt6lLsG4QW0FzbkSd9wwyTKHE=">AAAB+HicbVBNSwMxEM3Wr1o/
[Table]
+
[Table]
/Fm//
[Table]
[Figure]
[Table]
D N ORMALIZATION FOR T HE C RITIC
We here study why normalization is helpful for the critic network.
We plot two additional quantities during early training for the reacher hard task over 10 new seeds.
We consider the three agents from
Table 2: baseline, actor norm, and both norm.
The both norm agent uses normalization for the critic, whereas the others do not.
In Figure 19 we show the average Q-values achieved by the policy as measured over training batches.
We see that the curves are much smoother when normalizing the critic, whereas they oscillate aggressively otherwise.
With quickly oscillating Q-values, it becomes hard for the actor to learn a good policy.
Normalization appears to stabilize the output of both critic and actor networks – whereas unstable actor output leads to saturating nonlinearities, an unstable critic seems to result in oscillating Q-values.
In Figure 20 we show the average absolute difference between Q-values before and after a gradient update, as measured over training batches.
We see that normalizing the critic often leads to Q-value changes which are smaller and more stable over time. This in turn appears to stabilize learning as per Figure 19. Whereas the Q-value changes can trivially be made smaller by decreasing the learning rate, changes in Q-values depend non-linearly on the learning rate. Furthermore, too low learning rates often hurt generalization (Li et al., 2019).
For increased legibility, the curves are smoothed by averaging with a window of width ten.
[Table]
/+sOYpRFKwwTVuue5ifEzqgxnAmelfqoxoWxCR9izVNIItZ8tDp2RC6sMSRgrW9KQhfp7IqOR1tMosJ0RNWO96s3F/7xeasIbP+MySQ1KtlwUpoKYmMy/JkOukBkxtYQyxe2thI2poszYbEo2BG/15XXSvqp6btVrXlfqt3kcRTiDc7gED2pQh3toQAsYIDzDK7w5j86L8+58LFsLTj5zCn/gfP4Aqx2M1Q==</latexit>
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Figure]