R EGULARIZATION M ATTERS IN P OLICY O PTIMIZATION - A N E MPIRICAL S TUDY ON C ONTINUOUS C ONTROL
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep Reinforcement Learning (Deep RL) has been receiving increasingly more attention thanks to its encouraging performance on a variety of control tasks.
Yet, conventional regularization techniques in training neural networks (e.g., Lregularization, dropout) have been largely ignored in RL methods, possibly because agents are typically trained and evaluated in the same environment, and because the deep RL community focuses more on high-level algorithm designs.
In this work, we present the ﬁrst comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks.
Interestingly, we ﬁnd conventional regularization techniques on the policy networks can often bring large improvement, especially on harder tasks.
We also compare these techniques with the more widely used entropy regularization.
Our ﬁndings are shown to be robust against training hyperparameter variations.
In addition, we study regularizing different components and ﬁnd that only regularizing the policy network is typically the best.
Finally, we discuss and analyze why regularization may help generalization in RL from four perspectives - sample complexity, return distribution, weight norm, and noise robustness.
We hope our study provides guidance for future practices in regularizing policy optimization algorithms.
Our code is available at https://github.com/anonymouscode114/iclr2021_rlreg .
I NTRODUCTION
The use of regularization methods to prevent overﬁtting is a key technique in successfully training neural networks.
Perhaps the most widely recognized regularization methods in deep learning are Lregularization (also known as weight decay) and dropout (Srivastava et al., 2014).
These techniques are standard practices in supervised learning tasks across many domains.
Major tasks in computer vision, e.g., image classiﬁcation (Krizhevsky et al., 2012;
He et al., 2016), object detection (Ren et al., 2015; Redmon et al., 2016), use L 2 regularization as a default option.
In natural language processing, for example, the Transformer (Vaswani et al., 2017) uses dropout, and the popular BERT model (Devlin et al., 2018) uses L 2 regularization.
In fact, it is rare to see state-of-the-art neural models trained without regularization in a supervised setting.
However, in deep reinforcement learning (deep RL), those conventional regularization methods are largely absent or underutilized in past research, possibly because in most cases we are maximizing the return on the same task as in training.
In other words, there is no generalization gap from the training environment to the test environment (Cobbe et al., 2018).
Heretofore, researchers in deep
RL have focused on high-level algorithm design and largely overlooked issues related to network training, including regularization.
For popular policy optimization algorithms like Trust Region Policy
Optimization (TRPO) (Schulman et al., 2015), Proximal Policy Optimization (PPO) (Schulman et al., 2017), and Soft Actor Critic (SAC) (Haarnoja et al., 2018), conventional regularization methods were not considered.
In popular codebases such as the OpenAI Baseline (Dhariwal et al., 2017), Lregularization and dropout were not incorporated.
Instead, a commonly used regularization in RL is the entropy regularization which penalizes the high-certainty output from the policy network to encourage more exploration and prevent the agent from overﬁtting to certain actions.
The entropy regularization was ﬁrst introduced by (Williams & Peng, 1991) and now used by many contemporary algorithms (Mnih et al., 2016; Schulman et al., 2017; Teh et al., 2017; Farebrother et al., 2018).
In this work, we take an empirical approach to assess the conventional paradigm which omits common regularization when learning deep RL models.
We study agent performance on current task (the environment which the agent is trained on), rather than its generalization ability to different environments as in many recent works (Zhao et al., 2019;
Farebrother et al., 2018; Cobbe et al., 2018).
We speciﬁcally focus our study on policy optimization methods, which are becoming increasingly popular and have achieved top performance on various tasks.
We evaluate four popular policy optimization algorithms, namely SAC, PPO, TRPO, and the synchronous version of Advantage Actor
Critic (A2C), on multiple continuous control tasks.
Various conventional regularization techniques are considered, including L 2 / L 1 weight regularization, dropout, weight clipping (Arjovsky et al., 2017) and Batch Normalization (BN) (Ioffe & Szegedy, 2015).
We compare the performance of these regularization techniques to that without regularization, as well as the entropy regularization.
Surprisingly, even though the training and testing environments are the same, we ﬁnd that many of the conventional regularization techniques, when imposed to the policy networks, can still bring up the performance, sometimes signiﬁcantly.
Among those regularizers, L 2 regularization tends to be the most effective overall.
L 1 regularization and weight clipping can boost performance in many cases.
Dropout and Batch Normalization tend to bring improvements only on off-policy algorithms.
Additionally, all regularization methods tend to be more effective on more difﬁcult tasks.
We also verify our ﬁndings with a wide range of training hyperparameters and network sizes, and the result suggests that imposing proper regularization can sometimes save the effort of tuning other training hyperparameters.
We further study which part of the policy optimization system should be regularized, and conclude that generally only regularizing the policy network sufﬁces, as imposing regularization on value networks usually does not help.
Finally we discuss and analyze possible reasons for some experimental observations.
Our main contributions can be summarized as follows:
• To our best knowledge, we provide the ﬁrst systematic study of common regularization methods in policy optimization, which have been largely ignored in the deep RL literature.
We ﬁnd conventional regularizers can be effective on continuous control tasks (especially on harder ones) with statistical signiﬁcance, under randomly sampled training hyperparameters.
Interestingly, simple regularizers ( L 2 , L 1 , weight clipping) could perform better than entropy regularization, with L 2 generally the best.
BN and dropout can only help in off-policy algorithms.
• We study which part of the network(s) should be regularized.
The key lesson is to regularize the policy network but not the value network.
• We analyze why regularization may help generalization in RL through sample complexity, return distribution, weight norm, and training noise robustness.
R ELATED W ORKS
Regularization in Deep RL.
There have been many prior works studying the theory of regularization in policy optimization (Farahmand et al., 2009; Neu et al., 2017; Zhang et al., 2020).
In practice, conventional regularization methods have rarely been applied in deep RL.
One rare case of such use is in Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2016), where Batch Normalization is applied to all layers of the actor and some layers of the critic, and L 2 regularization is applied to the critic.
Some recent studies have developed more complicated regularization approaches to continuous control tasks.
Cheng et al.
(2019) regularizes the stochastic action distribution π ( a | s ) using a control prior and dynamically adjusts regularization weight based on the temporal difference (TD) error.
Parisi et al.
(2019) uses TD error regularization to penalize inaccurate value estimation and Generalized Advantage Estimation (GAE) (Schulman et al., 2016) regularization to penalize
GAE variance.
However, most of these regularizations are rather complicated (Cheng et al., 2019) or catered to certain algorithms (Parisi et al., 2019).
Also, these techniques consider regularizing the output of the network, while conventional methods mostly directly regularize the parameters.
In this work, we focus on studying these simpler but under-utilized regularization methods.
Generalization in Deep RL typically refers to how the model perform in a different environment from the one it is trained on.
The generalization gap can come from different modes/levels/difﬁculties of a game (Farebrother et al., 2018), simulation vs. real world (Tobin et al., 2017), parameter variations (Pattanaik et al., 2018), or different random seeds in environment generation (Zhang et al., 2018b).
There are a number of methods designed to address this issue, e.g., through training the agent over multiple domains/tasks (Tobin et al., 2017; Rajeswaran et al., 2017), adversarial training (Tobin et al., 2017), designing model architectures (Srouji et al., 2018), adaptive training (Duan et al., 2016), etc.
Meta RL (Finn et al., 2017; Gupta et al., 2018; Al-Shedivat et al., 2017) try to learn generalizable agents by training on many environments drawn from the same family/distribution.
There are also some comprehensive studies on RL generalization with interesting ﬁndings (Zhang et al., 2018a;b;
Zhao et al., 2019; Packer et al., 2018), e.g., algorithms performing better in training environment could perform worse with domain shift (Zhao et al., 2019).
Recently, several studies have investigated conventional regularization’s effect on generalization across tasks.
(Farebrother et al., 2018) shows that in Deep Q-Networks (DQN), L 2 regularization and dropout are sometime beneﬁcial when evaluated on the same Atari game with mode variations.
(Cobbe et al., 2018) shows that L 2 regularization, dropout, BN, and data augmentation can improve generalization performance, but to a less extent than entropy regularization and (cid:15) -greedy exploration.
Different from those studies, we focus on regularization’s effect in the same environment, yet on which conventional regularizations are under-explored.
E XPERIMENTS 3. S ETTINGS
Regularization Methods.
We study six regularization methods, namely, L 2 and L 1 weight regularization, weight clipping, Dropout (Srivastava et al., 2014), Batch Normalization (Ioffe & Szegedy, 2015), and entropy regularization.
See Appendix A for detailed introduction.
Note that we consider entropy as a separate regularization method because it encourages exploration and helps to prevent premature convergence (Mnih et al., 2016).
In Appendix N, we show that in the presence of certain regularizers, adding entropy on top does not lead to signiﬁcant performance difference.
Algorithms.
We evaluate regularization methods on four popular policy optimization algorithms, namely, A2C (Mnih et al., 2016), TRPO (Schulman et al., 2015), PPO (Schulman et al., 2017), and
SAC (Haarnoja et al., 2018).
The ﬁrst three algorithms are on-policy while the last one is off-policy.
For the ﬁrst three algorithms, we adopt the code from OpenAI Baseline (Dhariwal et al., 2017), and for SAC, we use the ofﬁcial implementation at (Haarnoja, 2018).
Tasks.
The algorithms with different regularizers are tested on nine continuous control tasks: Hopper,
Walker, HalfCheetah, Ant, Humanoid, and HumanoidStandup from MuJoCo (Todorov et al., 2012);
Humanoid, AtlasForwardWalk, and HumanoidFlagrun from RoboSchool (OpenAI).
Among the
MuJoCo tasks, agents for Hopper, Walker, and HalfCheetah are easier to learn, while Ant, Humanoid,
HumanoidStandup are relatively harder (larger state-action space, more training examples).
The three Roboschool tasks are even harder than the MuJoCo tasks as they require more timesteps to converge (Klimov & Schulman, 2017).
To better understand how different regularization methods work on different difﬁculties, we roughly categorize the ﬁrst three environments as “easy” tasks and the last six as “hard” tasks.
Besides continuous control, we provide results on randomly sampled
Atari environments (Bellemare et al., 2012) in Appendix S, which have discrete action space and different reward properties.
Our observations are mostly similar to those on continuous control tasks.
Training.
On MuJoCo tasks, we keep all hyperparameters unchanged as in the codebase adopted.
Since hyperparameters for the RoboSchool tasks are not included, we brieﬂy tune the hyperparameters for each algorithm before we apply regularization (details in Appendix D).
For details on regularization strength tuning, please see Appendix C.
The results shown in this section are obtained by only regularizing the policy network , and a further study on this will be presented in Section 5.
We run each experiment independently with ﬁve seeds, then use the average return over the last 100 episodes as the ﬁnal result.
Each regularization method is evaluated independently, with other regularizers turned off.
We refer to the result without any regularization as the baseline.
For BN and dropout, we use its training mode in updating the network, and test mode in sampling trajectories.
During training, negligible computation overhead is induced when a regularizer is applied.
Speciﬁcally, the increase in training time for BN is ∼ 10% , dropout ∼ 5% , while L 2 , L 1 , weight clipping, and entropy regularization are all < 1% .
We used up to 16 NVIDIA Titan Xp GPUs and 96 Intel Xeon
E5-2667 CPUs, and all experiments take roughly 57 days with resources fully utilized.
Additional Notes.
Note that entropy regularization is still applicable for SAC, despite it already incorporates the maximization of entropy in the reward.
In our experiments, we add the entropy
    H     H $&+XPDQRLG     H     H $&$QW      H     H $&$WODV)RUZDUG:DON     H    H 7532+XPDQRLG     H    H 7532$QW 
    H    H
7532$WODV)RUZDUG:DON     H     H 332+XPDQRLG     H     H 332$QW       H    H 332$WODV)RUZDUG:DON     
H    H 6$&+XPDQRLG    H
    H 6$&$QW      H   
 H 6$&$WODV)RUZDUG:DON      H     
 H $&5RERVFKRRO+XPDQRLG      H    H 75325RERVFKRRO+XPDQRLG      H     H 3325RERVFKRRO+XPDQRLG      H    H 6$&5RERVFKRRO+XPDQRLG EDVHOLQH HQWURS\
/ / ZHLJKWFOLS GURSRXW EDWFKQRUP regularization term to the policy loss function in equation (12) of (Haarnoja et al., 2018).
In our experiments, L 2 regularization loss is added to the training loss, which is then optimized using Adam (Kingma & Ba, 2015).
(Loshchilov & Hutter, 2019) observes that L 2 regularization interacts poorly with Adam and proposes AdamW to decouple weight decay from the optimization steps.
However, in policy optimization algorithms, we ﬁnd that the performance of AdamW with decoupled weight decay is slightly worse than the performance of Adam with L 2 loss directly added.
Comparisons are shown in Appendix O. 3.
Policy network dropout is not applicable to TRPO because during policy updates, different neurons in the old and new policy networks are dropped out, causing different shifts in the old and new action distributions given the same state, which violates the trust region constraint.
In this case, the algorithm fails to perform any update from network initialization.
3. R ESULTS
Training curves.
We plot the training curves from four environments (rows) in Figure 1, on four algorithms (columns).
Figures for the rest ﬁve environments are deferred to Appendix P. In the ﬁgure, different colors are used to denote different regularization methods, e.g., black is the baseline method.
Shades are used to denote ± 1 standard deviation range.
Notably, these conventional regularizers can frequently boost the performance across different tasks and algorithms, demonstrating that a study on the regularization in deep RL is highly demanding.
We observe that BN always signiﬁcantly hurts the baseline for on-policy algorithms.
The reason will be discussed later.
For the off-policy SAC algorithm, dropout and BN sometimes bring large improvement on hard tasks like
AtlasForwardWalk and RoboschoolHumanoid.
Interestingly, in some cases where the baseline (with the default hyperparameters in the codebase) does not converge to a reasonable solution, e.g., A2C
Ant, PPO Humanoid, imposing some regularization can make the training converge to a high level.
How often do regularizations help?
To quantitatively measure the effectiveness of the regularizations on each algorithm across different tasks, we deﬁne the condition when a regularization is said to “improve” upon the baseline in a certain environment.
Denote the baseline mean return over ﬁve seeds on an environment as µ env ,b , and the mean and standard deviation of the return obtained with a certain regularization method over ﬁve seeds as µ env ,r and σ env ,r .
We say the performance is “improved” by the regularization if µ env ,r − σ env ,r > max( µ env ,b , T ( env )) , where T ( env ) is the minimum return threshold of an environment.
The threshold serves to ensure the return is at least in a reasonable level.
We set the threshold to be 10 5 for HumanoidStandup and 10 3 for all other tasks.
[Table]
The results are shown in Table 1.
Perhaps the most signiﬁcant observation is that L 2 regularization is the most often to improve upon the baseline.
A2C algorithm is an exception, where entropy regularization is the most effective.
L 1 regularization behaves similar to L 2 regularization
[Table]
but is outperformed by the latter.
Weight clipping’s usefulness is highly dependent on the algorithms and environments.
Despite in total it only helps at 30.6% times, it can sometimes outperform entropy regularization by a large margin, e.g., in TRPO Humanoid and PPO Humanoid as shown in Figure 1.
BN is not useful at all in the three on-policy algorithms (A2C, TRPO, and PPO).
Dropout is not useful in A2C at all, and sometimes helps in PPO.
However, BN and dropout can be useful in SAC.
All regularization methods generally improve more often when they are used on harder tasks, perhaps because for easier ones the baseline is often sufﬁciently strong to reach a high performance.
Note that under our deﬁnition, not “improving” does not indicate “hurting”.
If we deﬁne “hurting” as µ env ,r + σ env ,r < µ env ,b (the return minimum threshold is not considered here), then total percentage of hurting is 0.0% for L 2 , 2.8% for L 1 , 5.6% for weight clipping, 44.4% for dropout, 66.7% for BN, and 0.0% for entropy.
In other words, under our parameter tuning range, L 2 and entropy regularization never hurt with appropriate strengths.
For BN and dropout, we also note that almost all hurting cases are in on-policy algorithms, except one case for BN in SAC.
In sum, all regularizations in our study very rarely hurt the performance except for BN/dropout in on-policy methods.
How much do regularizations improve?
For each algorithm and environment (for example, PPO on Ant), we calculate a z -score for each regularization method and the baseline, by treating results produced by all regularizations (including baseline) and all ﬁve seeds together as a population, and calculate each method’s average z -scores from its ﬁve ﬁnal results (positively clipped).
z -score is also known as “standard score”, the signed fractional number of standard deviations by which the value of a data point is above the mean value.
For each algorithm and environment, a regularizer’s z -score roughly measures its relative performance among others.
The z -scores are then averaged over environments of a certain difﬁculty (easy/hard), and the results are shown in Table 2.
In terms of the average improved margin, we can draw mostly similar observations as the improvement frequency (Table 1): L 2 tops the average z -score most often, and by large margin in total; entropy regularization is best used with A2C; Dropout and BN are only useful in the off-policy SAC algorithm; the improvement over baseline is larger on hard tasks.
Notably, for all algorithms, any regularization on average outperforms the baseline on hard tasks, except dropout and BN in on-policy algorithms.
On hard tasks, L 1 and weight clipping also perform higher than entropy in total, besides L 2 .
To further verify our observations
[Table]
we present z -scores for MuJoCo environments in Appendix G where we increase the number of seeds from 5 to 10.
Our observations are consistent with those in Table
[Table]
[Table]
Besides the improvement percentage (Table 1) and the z -score (Table 2), we provide more metrics of comparison (e.g., average ranking, min-max scaled return) to comprehensively compare the different regularization methods.
We also conduct statistical signiﬁcance tests on these metrics, and the improvement are mostly statistically signiﬁcant ( p < 0.05).
We believe evaluating under a variety of metrics make our conclusions more reliable.
Detailed results are in Appendix F, I, and
J. In addition, we provide detailed justiﬁcation in Appendix K that, because we test on the entire set of environments instead of on a single environment, our sample size is large enough to satisfy the condition of signiﬁcance tests and provide reliable results.
R OBUSTNESS WITH H YPERPARAMETER C HANGES
In the previous section, the experiments are conducted mostly with the default hyperparameters in the codebase we adopt, which are not necessarily optimized.
For example, PPO Humanoid baseline performs poorly using default hyperparameters, not converging to a reasonable solution.
Meanwhile, it is known that RL algorithms are very sensitive to hyperparameter changes (Henderson et al., 2018).
Thus, our ﬁndings can be vulnerable to such variations.
To further conﬁrm our ﬁndings, we evaluate the regularizations under a variety of hyperparameter settings.
For each algorithm, we sample ﬁve hyperparameter settings for the baseline and apply regularization on each of them.
Due to the heavy computation cost, we only evaluate on ﬁve environments: Hopper, Walker, Ant, Humanoid,
HumanoidStandup.
Under our sampled hyperparameters, poor baselines are mostly signiﬁcantly improved.
See Appendix E/ Q for details on sampling and curves.
The z -scores are shown in Table 3.
We note that our main ﬁndings in Section 3 still hold.
Interestingly, compared to the previous section,
L 2 , L 1 , and weight clipping all tend to be better than entropy regularization by larger margins.
For the p -scores of statistical signiﬁcance/improvement percentages, see Appendix F/H. ﬁve sampled hyperparameter settings.
[Table]
To better visualize the robustness against change of hyperparameters
[Table]
we show the result when a single hyperparameter is varied in Figure 2.
We note that the certain regularizations can consistently improve the baseline with different hyperparameters.
In these cases, proper regularizations can ease the hyperparameter tuning process, as they bring up performance of baselines with suboptimal hyperparameters to be higher than that with better ones.
[Figure]
    H
[Figure]
      
[Figure]
     H 332+XPDQRLG
H     /HDUQLQJ5DWH       H 6$&$QW H     /HDUQLQJ5DWH       H
[Figure]

[Figure]
P OLICY AND V ALUE N ETWORK R EGULARIZATION
[Table]
Our experiments so far only impose regularization on policy network.
To investigate the relationship between policy and value network regularization, we compare four options: 1) no regularization
[Table]
and regularizing 2) policy network, 3) value network, 4) policy and value networks.
For 2) and 3) we tune the regularization strengths independently and then use the appropriate ones for 4) (more details in
Appendix C).
We evaluate all four algorithms on the six MuJoCo tasks and present the improvement percentage in Table 4.
Note that entropy regularization is not applicable to the value network.
We observe that generally, only regularizing the policy network is the most often to improve almost all algorithms and regularizations.
Regularizing the value network alone does not bring improvement as often as other options.
Though regularizing both is better than regularizing value network alone, it is worse than only regularizing the policy network.
For detailed training curves, refer to Appendix R.
We also note that the policy optimization algorithms in our study have adopted multiple techniques to train the value function.
For example, SAC uses the replay buffer and the clipped double-Q learning.
A2C, TRPO, and PPO adopt multi-step roll-out, and the sum of discounted rewards is used as the value network objective.
However, analyzing the individual effects of these techniques is not the main focus of our current work.
We would like to leave the interaction between these techniques and value network regularization for future work.
A NALYSIS AND C ONCLUSION
Why does regularization beneﬁt policy optimization?
In RL, when we are training and evaluating on the same environment, there is no generalization gap across different environments.
However, there is still generalization between samples: the agents is only trained on the limited trajectories it has experienced, which cannot cover the whole state-action space of the environment.
A successful policy needs to generalize from seen samples to unseen ones, which potentially makes regularization necessary.
This might also explain why regularization could be more helpful on harder tasks, which have larger state space, and the portion of the space that have appeared in training tends to be smaller.
We study how regularization helps generalization through the following perspectives:
Sampling Complexity.
We compare the return with varying number of training samples/timesteps, since the performance of learning from fewer samples is closely related to generalization ability.
From the results in Figure 3, we ﬁnd that for regularized models to reach the same return level as baseline, they need much fewer training samples.
This suggests that certain regularizers can signiﬁcantly reduce the sampling complexity of baseline and thus lead to better generalization.
[Figure]
Return Distribution.
We evaluate agents trained with and without regularization on 100 different trajectories and plot the return distributions over trajectories in Figure 4.
These trajectories represent unseen samples during training, since the state space is continuous (so it is impossible to traverse
 
     
  H 
   33 2  + X P DQR L G %DVHOLQH  
     
  H 
   /  
     
  H 
   /  
     
  H 
   :HLJKW&OLS     
     H     7 5 3 2  $ Q W     
     H         
    
[Figure]
  
[Figure]
   identical trajectories).
For baseline, some trajectories yield relatively high returns, while others yield low returns, demonstrating the baseline cannot stably generalize to unseen examples; for regularized models, the returns are more concentrated at a high level, demonstrating they can more stably generalize to unseen samples.
This suggests that certain conventional regularizers can improve the model’s generalization ability to larger portion of unseen samples.
Weight Norm.
We observe that on many tasks, smaller policy weight norm correlates with better generalization ability.
An example is illustrated in Table 5 and Figure 5.
We observe that Lregularization accomplishes the effect of entropy regularization and, at the same time, limits the policy norm.
Even though both the entropy-regularized model and the L 2 -regularized model have similar ﬁnal policy entropy, L 2 -regularized model have much higher ﬁnal performance, which suggests that simply increasing the policy entropy is not enough.
We conjecture that L 2 -encouraged small weight norm makes the network less prone to overﬁtting and provides a better optimization landscape for the model.
[Table]
[Table]
W X U Q H
[Table]
  
[Table] R OL F\  : H L JK W 1 R U P EDVHOLQH/HQWURS\    7LPHVWHSV H  
[Table]
[Table]
Robustness to Training Noise.
Recent works (Kostrikov et al., 2020; Laskin et al., 2020) have applied data augmentation (DA) to
RL, mainly on image-based inputs, to improve data efﬁciency and generalization.
Laskin et al.
(2020) adds noise to state-based input observations by random scaling them as a form of DA.
We apply this technique to both baseline and L 2 regularization on PPO Humanoid.
At each time step, we randomly scale the input state by a factor of s , where s ∼ Unif (1 − k, 1+ k ) , k ∈ { 0 . 05 , 0 . 1 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } .
We select the k with the highest performance on the original environment and report the results in Table 6.
Interestingly, while DA cannot improve the baseline performance, it can signiﬁcantly improve the performance of L 2 -regularized model.
This suggests L 2 regularizer can make the model robust to, or even beneﬁt from, noisy/augmented input during training.
Why do BN and dropout work only with off-policy algorithms?
One ﬁnding in our experiments is BN and dropout can sometimes improve on the off-policy algorithm SAC, but mostly hurt on-policy algorithms.
We further conﬁrm this observation through experiments on Deep Deterministic Policy Gradient (DDPG, Lillicrap et al.
(2016)), another off-policy algorithm, and present the results in
Appendix M. We hypothesize two possible reasons: 1) for both BN and dropout, training mode is used to train the network, and testing mode is used to sample actions during interaction with the environment, leading to a discrepancy between the sampling policy and optimization policy (the same holds if we always use training mode).
For on-policy algorithms, if such discrepancy is large, it can cause severe “off-policy issues”, which hurts the optimization process or even crashes it since their theory necessitates that the data is “on policy”, i.e., data sampling and optimization policies are the same.
For off-policy algorithms, this discrepancy is not an issue, since they sample data from replay buffer and do not require the two policies to be the same.
2) BN can be sensitive to input distribution shifts, since the mean and std statistics depend on the input, and if the input distribution changes too quickly in training, the mapping functions of BN layers can change quickly too, which can possibly destabilize training.
One evidence for this is that in supervised learning, when transferring a ImageNet pretrained model to other vision datasets, sometimes the BN layers are ﬁxed (Yang et al., 2017) and only other layers are trained.
In off-policy algorithms, the sample distributions are relatively slow-changing since we always draw from the whole replay buffer which holds cumulative data; in on-policy algorithms, we always use the samples generated from the latest policy, and the faster-changing input distribution for on-policy algorithms could be harmful to BN.
In summary , we conducted the ﬁrst systematic study of regularization methods on multiple policy optimization algorithms.
We found that conventional regularizations ( L 2 , L 1 , weight clipping) could be effective at improving performance, sometimes more than entropy regularization.
BN and dropout could be useful but only on off-policy algorithms.
Our ﬁndings were conﬁrmed with multiple sampled hyperparameters.
Further experiments have shown that generally, the best practice is to regularize the policy network but not the value network or both.
Finally we analyze why regularization can help in
RL with experiments and discussions.
A PPENDIX T ABLE OF C ONTENTS
A Regularization Methods
B Policy Optimization Algorithms
C Regularization Implementation & Tuning Details
D Default Hyperparameter Settings
E Hyperparameter Sampling Details
F Statistical Signiﬁcance Test of z -scores
G z -score Statistics under More Random Seeds on MuJoCo
H Improvement Percentage for Hyperparameter Experiments
I
Statistical Signiﬁcance Test of z -scores (Entropy Regularization)
J Additional Metrics
J.
Ranking all regularizers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
J.
Scaled Returns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
K Justiﬁcation of Methodology and Statistical Signiﬁcance
L Regularization with a Fixed Strength
M DDPG Results
N Regularizing with L 2 and Entropy
O L 2 Regularization vs. Fixed Weight Decay (AdamW)
P Additional Training Curves (Default Hyperparameters)
Q Training Curves for Hyperparameter Experiments
R Training Curves for Policy vs. Value Experiments
S Atari Experiments
A R EGULARIZATION M ETHODS
There are in general two types of common approaches for imposing regularization.
One is to discourage complex models (e.g., weight regularization, weight clipping), and the other is to inject certain noise in network activations (e.g., dropout and Batch Normalization).
Here we brieﬂy introduce the methods we investigate in our experiments.
L 2 / L 1 Weight Regularization.
Suppose L is the original empirical loss we want to minimize.
When applying L 2 regularization, we add an additional L 2 -norm squared loss term 12 λ
||
θ
||
to
L , where θ are model parameters and λ is a hyperparameter.
Similarly, in the case of L 1 weight regularization, the additional loss term is λ ||
θ
|| 1 .
In our experiments, the total loss is optimized using Adam (Kingma & Ba, 2015).
Using L 2 / L 1 regularization can encourage the model to be simpler/sparse.
From a Bayesian view, they impose certain prior distributions on model weights.
Weight Clipping.
Weight clipping is a simple operation: after each gradient update step, each individual weight is clipped to range [ − c, c ] , where c is a hyperparameter.
This could be formally described as θ i ← max(min( θ i , c ) , − c ) .
In Wasserstein GANs (Arjovsky et al., 2017), weight clipping is used to enforce the constraint of Lipschitz continuity.
This plays an important role in stabilizing the training of GANs (Goodfellow et al., 2014)
Weight clipping can also be seen as a regularizor since it reduce the complexity of the model space, by preventing any weight’s magnitude from being larger than c .
Dropout.
Dropout (Srivastava et al., 2014) is one of the most successful regularization techniques developed speciﬁcally for neural networks.
During training, a certain percentage of neurons is deactivated; during testing, all neurons in the neural network are kept, and rescaling is applied to ensure the scale of the activations is the same as training.
One explanation for its effectiveness in reducing overﬁtting is they can prevent “co-adaptation” of neurons.
In the policy optimization algorithms we investigate, when the policy or the value network performs updates using minibatches of trajectory data or replay buffer data, we use the training mode of dropout.
When the policy network samples trajectories from the environment, we use the testing mode of dropout.
Batch Normalization.
Batch Normalization (BN) (Ioffe & Szegedy, 2015) is invented to address the problem of “internal covariate shift”, and it does the following transformation:
ˆ z
=
z in − µ B √ σ 2 B + (cid:15) ; z out = γ ˆ z + β , where µ B and σ B are the mean and standard deviation of input activations over B , and γ and β are trainable afﬁne transformation parameters.
BN turns out to greatly accelerate the convergence and bring up the accuracy.
It also acts as a regularizer (Ioffe & Szegedy, 2015): during training, the statistics µ B and σ B depend on the current batch, and BN subtracts and divides different values in each iteration.
This stochasticity can encourage subsequent layers to be robust against such input variation.
In policy optimization algorithms, we switch between training and testing modes the same way as we do in dropout.
Entropy Regularization.
In a policy optimization framework, the policy network is used to model a conditional distribution over actions, and entropy regularization is widely used to prevent the learned policy from overﬁtting to one or some of the actions.
More speciﬁcally, in each step, the output distribution of the policy network is penalized to have a high entropy.
Policy entropy is calculated at each step as H s i = − E
a i ∼ π ( a i | s i ) log π ( a i | s i ) , where ( s i , a i ) is the state-action pair.
Then the per-sample entropy is averaged within the batch of state-action pairs to obtain the regularization term L H = 1 N (cid:80) s i
H s i .
A coefﬁcient λ is also needed, and λL H is added to the policy objective
J ( θ ) .
The sum is then maximized during policy updates.
Entropy regularization also encourages exploration and prevents premature convergence due to increased randomness in actions, leading to better performance in run.
B P OLICY O PTIMIZATION A LGORITHMS
The policy optimization family of algorithms is one of the most popular methods for solving reinforcement learning problems.
It directly parameterizes and optimizes the policy to gain more cumulative rewards.
Below, we give a brief introduction to the algorithms we evaluate in our work.
A2C. Sutton et al.
(2000) developed a policy gradient to update the parametric policy in a gradient descent manner.
However, the gradient estimated in this way suffers from high variance.
Advantage
Actor Critic (A3C) (Mnih et al., 2016) is proposed to alleviate this problem by introducing a function approximator for values and replacing the Q-values with advantage values.
A3C also utilizes multiple actors to parallelize training.
The only difference between A2C and A3C is that in a single training iteration, A2C waits for parallel actors to ﬁnish sampling trajectories before updating the neural network parameters, while A3C updates in an asynchronous manner.
TRPO.
Trust Region Policy Optimization (TRPO) (Schulman et al., 2015) proposes to constrain each update within a safe region deﬁned by KL divergence to guarantee policy improvement during training.
Though TRPO is promising at obtaining reliable performance, approximating the KL constraint is quite computationally heavy.
PPO.
Proximal Policy Optimization (PPO) (Schulman et al., 2017) simpliﬁes TRPO and improves computational efﬁciency by developing a surrogate objective that involves clipping the probability ratio to a reliable region, so that the objective can be optimized using ﬁrst-order methods.
SAC.
Soft Actor Critic (SAC) (Haarnoja et al., 2018) optimizes the maximum entropy objective in reward (Ziebart et al., 2008), which is different from the objective of the on-policy methods above.
SAC combines soft policy iteration, which maximizes the maximum entropy objective, and clipped double Q learning (Fujimoto et al., 2018), which prevents overestimation bias, during actor and critic updates, respectively.
C R EGULARIZATION I MPLEMENTATION & T UNING D ETAILS
As mentioned in the paper, in Section 3 we only regularize the policy network; in Section 5, we investigate regularizing both policy and value networks.
For L 1 and L 2 regularization, we add λ ˙ ||
θ ||
and λ 2 ˙
||
θ || 22 , respectively, to the loss of policy network or value network of each algorithm (for SAC’s value regularization, we apply regularization only to the V network instead of also to the two Q networks).
L 1 and L 2 loss are applied to all the weights of the policy or value network.
For A2C, TRPO, and PPO, we tune λ in the range of [1 e − 5 , 5 e − 5 , 1 e − 4 , 5 e − 4] for L 1 and [5 e − 5 , 1 e − 4 , 5 e − 4 , 1 e − 3] for L 2 .
For SAC, we tune λ in the range of [5 e − 4 , 1 e − 3 , 5 e − 3 , 1 e − 2] for L 1 and [1 e − 3 , 5 e − 3 , 1 e − 2 , 5 e − 2] for L 2 .
For weight clipping, the OpenAI Baseline implementation of the policy network of A2C, TRPO, and PPO outputs the mean of policy action from a two-layer fully connected network (MLP).
The log standard deviation of the policy action is represented by a standalone trainable vector.
We ﬁnd that when applied only to the weights of MLP, weight clipping makes the performance much better than when applied to only the logstd vector or both.
Thus, for these three algorithms, the policy network weight clipping results shown in all the sections above come from clipping only the MLP part of the policy network.
On the other hand, in the SAC implementation, both the mean and the log standard deviation come from the same MLP, and there is no standalone log standard deviation vector.
Thus, we apply weight clipping to all the weights of the MLP.
For all algorithms, we tune the policy network clipping range in [0 . 1 , 0 . 2 , 0 . 3 , 0 . 5] .
For value network, the MLP produces a single output of estimated value given a state, so we clip all the weights of the MLP.
For A2C, TRPO, and
PPO, we tune the clipping range in [0 . 1 , 0 . 2 , 0 . 3 , 0 . 5] .
For SAC, we only clip the V network and do not clip the two Q networks for simplicity.
We tune the clipping range in [0 . 3 , 0 . 5 , 0 . 8 , 1 . 0]
due to its weights having larger magnitude.
For BatchNorm/dropout, we apply it before the activation function of each hidden layer/immediately after the activation function.
When the policy or the value network is performing update using minibatches of trajectory data or minibatches of replay buffer data, we use the train mode of regularization and update the running mean and standard deviation.
When the policy is sampling trajectory from the environment, we use the test mode of regularization and use the existing running mean and standard deviation to normalize data.
For Batch Normalization/dropout on value network, only training mode is applied since value network does not participate in sampling trajectories.
Note that adding policy network dropout on TRPO causes the KL divergence constraint E s ∼ ρ θ old [ D KL ( π θ old ( ·| s ) (cid:107) π θ ( ·| s ))] ≤ δ to be violated almost every time during policy network update.
Thus, policy network dropout causes the training to fail on TRPO, as the policy network cannot be updated.
For entropy regularization, we add − λL H to the policy loss.
λ is tuned from [5 e − 5 , 1 e − 4 , 5 e − 4 , 1 e − 3] for A2C, TRPO, PPO and [0 . 1 , 0 .
5 , 1 . 0 , 5 . 0] for SAC.
Note that for SAC, our entropy regularization is added directly on the optimization objective (equation 12 in Haarnoja et al.
(2018)), and is different from the original maximum entropy objective inside the reward term.
Note that for the three on-policy algorithms (A2C, TRPO, PPO)
we use the same tuning range, and the only exception is the off-policy SAC.
The reason why SAC’s tuning range is different is that SAC uses a hyperparameter that controls the scaling of the reward signal, while A2C, TRPO, and PPO do not.
In the original implementation of SAC, the reward signals are pre-tuned to be scaled up by a factor ranging from 5 to 100 , according to speciﬁc environments.
Also, unlike A2C, TRPO, and
PPO, SAC uses unnormalized reward because if the reward magnitude is small, then, according to the original paper, the policy becomes almost uniform.
Due to the above reasons, the reward magnitude of SAC is much higher than the magnitude of rewards used by A2C, TRPO, and PPO.
Thus, the policy network loss and the value network loss have larger magnitude than those of A2C, TRPO, and
PPO, so the appropriate regularization strengths become higher.
Considering the SAC’s much larger reward magnitude, we selected a different range of hyperparameters for SAC before we ran the whole experiments.
The optimal policy network regularization strength we selected for each algorithm and environment used in Section 3 can be seen in the legends of Appendix R.
In addition to the results with environmentspeciﬁc strengths presented in Section 3, we also present the results when the regularization strength is ﬁxed across all environments for the same algorithm.
The results are shown in Appendix L.
In Section 5, to investigate the effect of regularizing both policy and value networks, we combine the tuned optimal policy and value network regularization strengths.
The detailed training curves are presented in Appendix R.
As a side note, when training A2C, TRPO, and PPO on the HalfCheetah environment, the results have very large variance.
Thus, for each regularization method, after we obtain the best strength, we rerun it for another ﬁve seeds as the ﬁnal result in Table 1 and 2.
D D EFAULT H YPERPARAMETER S ETTINGS
Training timesteps.
For A2C, TRPO, and PPO, we run 5 e 6 timesteps for Hopper, Walker, and
HalfCheetah; 2 e 7 timesteps for Ant, Humanoid (MuJoCo), and HumanoidStandup; 5 e 7 timesteps for
Humanoid (RoboSchool); and 1 e 8 timesteps for AtlasForwardWalk and HumanoidFlagrun.
For SAC, since its simulation speed is much slower than A2C, TRPO, and PPO (as SAC updates its policy and value networks using a minibatch of replay buffer data at every timestep), and since it takes fewer timesteps to converge, we run 1 e 6 timesteps for Hopper and Walker; 3 e 6 timesteps for HalfCheetah and Ant; 5 e 6 timesteps for Humanoid and HumanoidStandup; and 1 e 7 timesteps for the RoboSchool environments.
Hyperparameters for RoboSchool.
In the original PPO paper (Schulman et al., 2017), hyperparameters for the Roboschool tasks are given, so we apply the same hyperparameters to our training, except that instead of linear annealing the log standard deviation of action distribution from − 0 .
7 to − 1 .
6 , we let it to be learnt by the algorithm, as implemented in OpenAI Baseline (Dhariwal et al., 2017).
For TRPO, due to its proximity to PPO, we copy PPO’s hyperparameters if they exist in both algorithms.
We then tune the value update step size in [3 e − 4 , 5 e − 4 , 1 e − 3] .
For A2C, we keep the original hyperparameters and tune the number of actors in [32 , 128] and the number of timesteps for each actor between consecutive policy updates in [5 , 16 , 32] .
For SAC, we tune the reward scale from [5 , 20 , 100] .
The detailed hyperparameters used in our baselines for both MuJoCo and RoboSchool are listed in
Tables 7-10.
for A2C MuJoCo and RoboSchool tasks
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure] [Table]
[Figure]
[Table]
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Table]
In Section 4, we present results based on ﬁve hyperparameter settings. To obtain such hyperparameter variations, we consider varying the learning rates and the hyperparameters that each algorithm is very sensitive to. For A2C, TRPO, and PPO, we consider a range of rollout timesteps between consecutive policy updates by varying the number of actors or the number of trajectory sampling timesteps for each actor.
For SAC, we consider a range of reward scale and a range of target smoothing coefﬁcient.
More concretely, for A2C, we sample the learning rate from [2 e − 4 , 7 e − 4 , 2 e − 3] linear decay, the number of trajectory sampling timesteps (nsteps) for each actor from [3 , 5 , 16 , 32] , and the number of actors (nenvs) from [1 , 4] .
For TRPO, we sample the learning rate of value network (vf_stepsize) from [3 e − 4 , 5 e − 4 , 1 e − 3] and the number of trajectory sampling timesteps for each actor (nsteps) in [1024 , 2048 , 4096 , 8192] .
The policy update uses conjugate gradient descent and is controlled by the max KL divergence.
For PPO, we sample the learning rate from [1 e − 4 linear , 3 e − 4 constant ] , the number of actors (nenvs) from [1 , 2 , 4 , 8] , and the probability ratio clipping range (cliprange) in [0 . 1 , 0 , 2] .
For SAC, we sample the learning rate from [1 e − 4 , 3 e − 4 , 1 e − 3] the target smoothing coefﬁcient ( τ ) from [0 .
01] , and the reward scale from small, default, and large mode.
The default reward scale of 5 is changed to (3 , 5 , 20) ; 20 to (4 , 20 , 100) ; 100 to (20 , 100 , 400) for each mode, respectively.
Sampled hyperparameters 1-5 for each algorithms are listed in Table 11-14.
[Table]
[Table]
[Table]
[Table]
F S TATISTICAL S IGNIFICANCE T EST OF z - SCORES
For each regularization method, we collect the z -scores produced by all seeds and all environments of a certain difﬁculty (e.g. for L 2 on PPO and hard environments, we have 6 envs × 5 seeds = 30 zscores), and perform Welch’s t
-test (two-sample t -test with unequal variance) with the corresponding z -scores produced by the baseline. The resulting p -values for Table 2 in Section 3 and Table 3 in
Section 4 are presented in Table 15 and Table 16, respectively.
Note that whether the signiﬁcance indicates improvement or harm depends on the relative mean z -score in Table 2 and Table 3.
For example, for BN and dropout in on-policy algorithms, the statistical signiﬁcance denotes harm, and in most other cases it denotes improvement.
From the results, we observe that the improvement is statistically signiﬁcant ( p < 0 . 05 ) for hard tasks in general, with only a few exceptions.
In total, L 2 ,
L 1 , entropy and weight clipping are all statistically signiﬁcantly better than baseline.
For Welch’s t -test between entropy regularization and other regularizers, see Appendix I.
[Table]
[Table]
G z - SCORE S TATISTICS UNDER M ORE R ANDOM S EEDS ON M U J O C O
To further verify our result
[Table]
we increase the number of seeds from 5 to 10 and present z -scores for the six MuJoCo environments (easy: Hopper, Walker, HalfCheetah; hard: Ant, Humanoid,
HumanoidStandup) in Table 17.
We also present tests of statistical signiﬁcance in Table 18.
Our observations are consistent with those in Table 2.
Due to the large computation cost required, we do not include the three hard Roboschool environments in the calculation of z -scores.
[Table]
[Table]
H I MPROVEMENT P ERCENTAGE FOR H YPERPARAMETER E XPERIMENTS
We provide the percentage of improvement result in Table 19 as a complement with Table 3, for the experiments with multiple sampled hyperparameters
[Table]
I S TATISTICAL S IGNIFICANCE T EST OF z - SCORES (E NTROPY R EGULARIZATION )
As a complement to Table 2 in Section 3 and Table 3 in Section
[Table]
we present the p -value results from Welch’s t -test comparing the z -scores of entropy regularization with other regularizers in Table 20 and Table 21.
Note that whether the signiﬁcance indicates improvement or harm over entropy regularization depends on the relative mean z -score in Table 2 under default hyperparameter setting and Table 3 under sampled hyperparameter setting.
We observe that in total, L 2 has signiﬁcant improvement over entropy in both default hyperparameter setting and sampled hyperparameter setting.
L 1 and weight clipping are signiﬁcantly better than entropy under sampled hyperparameter setting.
In general, the improvement over entropy is statistically more signiﬁcant for hard tasks.
[Table]
[Table]
J A DDITIONAL M ETRICS J. R ANKING ALL REGULARIZERS
We compute the “average ranking” metric to compare the relative effectiveness of different regularization methods.
Note that the average ranking of different methods across a set of tasks/datasets has been adopted as a metric before, as in (Ranftl et al., 2019) and (Knapitsch et al., 2017).
Here, we rank the performance of all the regularization methods, together with the baseline, for each algorithm and task, and present the average ranks in Table 22 and Table 23, with statistical signiﬁcance tests in Table 24 and 25.
The ranks of returns among different regularizers are collected for each environment (after averaging over 5 random seeds)
[Table]
and then the mean rank over all seeds is calculated.
From Tableand Table 23, we observe that, except for BN and dropout in on-policy algorithms, all regularizations on average outperform baselines.
Again, L 2 regularization is the strongest in most cases.
Other similar observations can be made as in previous tables.
For every algorithm, baseline ranks lower on harder tasks than easier ones; in total, it ranks 3.50 for easier tasks and 5.25 for harder tasks.
This indicates that regularization is more effective when the tasks are harder.
[Table]
[Table]
[Table]
[Table]
J. S CALED R ETURNS
Min-max scaling is a linear-mapping operation to map values ranging from [ min ( x ) , max ( x )] to [0 , 1] , using x (cid:48) = x − min ( x ) max ( x ) − min ( x ) .
For each environment and policy optimization algorithm (for example, PPO on Ant), we calculate a "scaled return" for each regularization method and the baseline, using the maximum mean return obtained using any regularization method (including baseline) as max ( x ) and 0 as min ( x ) , on positively clipped returns. We then average the scaled returns of mean return over environments of a certain difﬁculty (easy/hard).
We present the results under the default hyperparameter setting in Table 26-28 and the results under sampled hyperparameter settings in
Table 29-31.
To analyze whether regularization signiﬁcantly improves over the baseline and whether conventional regularizers signiﬁcantly improves over entropy, we perform Welch’s t -test on the scaled returns, using an identical approach to the one we used for z -score.
Our observation is similar to the ones we made in Section 3 and Section
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
K J USTIFICATION OF M ETHODOLOGY AND S TATISTICAL S IGNIFICANCE
In this section, we provide rigorous justiﬁcation that, (1) when the sample size is large enough ( n ≥ 30) , the normality assumption for the sampling distribution is not needed (loc); (2) since we test on the entire set of environments instead of on a single environment, our sample size is large enough to satisfy the condition of Welch’s t -test and provide reliable results.
Consider two distributions with mean and variance pairs ( µ 1 , σ 21 ) and ( µ 2 , σ 22 ) , respectively, where neither distribution needs to be normal , and the mean and variances are unknown. Let H 0 : µ 1 = µ 2 be the null hypothesis, and H 1 : µ 1 (cid:54) = µ 2 be the alternate hypothesis.
Let ( X 1 , X 2 , . . .
, X n ) and ( Y 1 , Y 2 , . .
, Y n ) be independent samples from the two distributions.
Then, under the null hypothesis, the t statistic from Welch’s t -test converges in distribution to N (0 , 1) as n → ∞ .
We formalize the above statement below.
Theorem K.1.
Consider two distributions with mean and variance pairs ( µ 1 , σ 21 ) and ( µ 2 , σ 22 ) , where the mean and variances are unknown.
Deﬁne H 0 : µ 1 = µ 2 and H 1 : µ 1 (cid:54) = µ 2 .
Let ( X 1 , X 2 , . . .
, X n ) and ( Y 1 , Y 2 , . .
, Y n ) be independent samples from the two distributions.
Then, under H 0 , the t statistic from Welch’s t -test converges in distribution to the standard normal distribution as n → ∞ .
That is, t n = √ n ( X n − Y n ) √ S 2 X,n + S 2 Y,n d −→ N (0 , 1) , where X n , Y n are the sample means for ( X 1 , X 2 , . . .
, X n ) and ( Y 1 , Y 2 , . .
, Y n ) ; S 2 X,n and S 2 Y,n are the sample variances.
Proof.
We have S 2 X,n p −→ σ 21 and S 2 Y,n p −→ σ 22 .
Then due to independence, ( S 2 X,n , S 2 Y,n ) p −→ ( σ 21 , σ 22 ) .
By the continuous mapping theorem, (cid:113) S 2 X,n + S 2 Y,n p −→ (cid:112) σ 21 + σ 22 .
The rejection / acceptance region of t n is based on the null hypothesis.
Under the null hypothesis, according to the
Central Limit Theorem, √ n ( X n − µ 1 ) d −→ N (0 , σ 21 ) , √ n ( Y n − µ 1 ) d −→ N (0 , σ 22 ) .
Then due to independence, ( √ n ( X n − µ 1 ) , √ n ( Y n − µ 1 ))
d −→ ( N (0 , σ 21 ) , N (0 , σ 22 )) .
By Slutsky’s theorem, √ n ( X n − Y n ) d −→ N (0 , σ 21 + σ 22 ) .
Again by Slutsky, √ n ( X n − Y n ) √ S 2 X,n + S 2 Y,n d −→ N (0 , 1) .
Therefore, if n ≥ 30 (i.e. the sample size is large), we do not need the normality assumption of our distribution to apply Welch’s t -test, and we can use our t -statistic to obtain the p -value the same way as from the z -test (loc) (i.e. the p -value equals 2 · Φ( −| t | ) , where Φ is the cumulative distribution function (CDF) of the standard normal distribution).
Also, the t -test can be applied when n grows much larger than 30 (lar).
We now show that our sample size is large enough to apply the above theorem.
For each algorithm and regularizer, we calculate the average z -score, the average ranking, and the average scaled return over a set of environments and all seeds.
We then test whether the performance of a regularizer is signiﬁcantly different from that of baseline.
We take the average z -score metric as an example.
Let E be the set of environments with uniform distribution over the environments, and let S be the set of seeds.
For e ∈ E and s ∈ S , let f reg ( e, s ) denote the z -score of a certain regularizer under an environment e and seed s , and let f baseline ( e, s ) denote the z -score under the baseline.
We use
Welch’s t -test to test whether µ reg (cid:54) = µ baseline , given unknown σ reg , σ baseline , on a policy optimization algorithm.
For experiments in Section 3 (e.g. Table 2), for each policy optimization algorithm, we test the distribution of { f reg ( e, s ) : e ∈ E, s ∈ S } versus { f baseline ( e, s ) : e ∈ E, s ∈ S } on 9 environments (3 easy, 6 hard).
We obtain 5 seeds * 3 envs = 15 data samples for "easy" environments, and 5 seeds * 6 envs = 30 data samples for "hard" environments, so that the "total" column has 15 + 30 =data samples.
In Appendix G, we increase the number of seeds from 5 to 10, so that we obtaindata samples for "easy" environments.
In the last three columns, we aggregate the data from each policy optimization algorithm and test whether a regularizer performs signiﬁcantly different from the baseline across algorithms, environments, and seeds.
Since there are 4 algorithms, we obtain 15 * 4 = 60 samples for "easy", 30 * 4 = 120 for "hard", and 60 + 120 = 180 for "total".
The sample size is large enough and satisfy our condition for Welch’s t -test.
For experiments in Section 4 (e.g. Table 3), for each policy optimization algorithm, we test { f reg ( h, e, s ) : h ∈ H, e ∈ E, s ∈ S } versus { f baseline ( h, e, s ) : h ∈ H, e ∈ E, s ∈ S } , where
H is the set of training hyperparameters.
In other words, we test whether a regularizer’s performance over training hyperparameters, environments, and seeds is signiﬁcantly different from that of baseline.
We conducted experiments on 2 easy environments and 3 hard environments.
We obtain 5 hyperparameters * 5 seeds * 2 envs =
50 data samples for "easy" environments, 5 * 5 * 3 = 75 for "hard", and 50 + 75 = 125 samples for "total".
In the last three columns, we aggregate the data from each policy optimization algorithm.
We obtain 50 * 4 = 200 data samples for "easy", 75 * 4 = 300 for "hard", and 200 + 300 = 500 for "total".
The sample size is large enough and satisfy our condition for Welch’s t -test.
We further plot the distribution of our z -score metric in the quantile-quantile (Q-Q) plots in Figureand Figure 7.
A Q-Q plot plots the quantiles of two distributions X and Y against each other, where in our case X is normal.
If the plot approximately follows the line y = x , then the two distributions have approximately the same cumulative distribution function (CDF).
In our case, this means that Y is approximately normal.
We observe that empirically, the distribution of our performance metric is close to normal.
As a result, the t -statistic we calculate from our samples is close to the t -distribution with parameter n , which converges to N (0 , 1) quickly as n increases. [Figure] SAC baseline TOTAL L    2 Normal Theoretical Quantiles
TOTAL baseline
We have presented the mean values for our performance metrics ( z -scores in Table 2, 3; average ranking in Table 22, 23; scaled return in Table 26, 29).
Given statistical signiﬁcance, whether a regularizer improves upon baseline depends on whether the performance metric is higher than the baseline.
For example, in the "Total" column and "hard" subcolumn of Table 2, the z -score of Lregularization is 0.58, while the z -score of baseline is -0.27.
The p -value in the corresponding entry of Table 15 is 0.00.
Therefore, L 2 regularization signiﬁcantly improves over the baseline on hard tasks.
Note that the p -value is not a standalone performance metric.
It only serves as a complement to our metrics and indicates whether the performance of a regularizer differs signiﬁcantly from our baseline.
In addition, we note that the Figure 5 in Henderson et al.
(2018) shows that, under the same hyperparameter conﬁguration, two sets of 5 different runs on the HalfCheetah environment can be
S a m p l e
Q u a n t il e s
A2C L    2 Normal Theoretical Quantiles 1.
S a m p l e
Q u a n t il e s A2C baseline PPO L     2 Normal Theoretical Quantiles PPO baseline
TOTAL L   2 Normal Theoretical Quantiles TOTAL baseline signiﬁcantly different from each other. We ﬁnd that the unique environment property of HalfCheetah contributes to such observation. For A2C, PPO, and TRPO on HalfCheetah, there is a certain probability that the policy found is suboptimal, where the half cheetah robot runs upside-down using its head.
In this case, the ﬁnal return never rises above 2200.
In other cases, the half cheetah robot runs using its legs, and the ﬁnal return is almost always above 4000.
Therefore, it is possible that in a set of 5 runs, 4 of the runs have ﬁnal returns above 4000, while for another set of 5 runs, 4 of the runs have ﬁnal returns below 2200.
This causes a signiﬁcant performance difference between the two sets of runs.
However, for all other environments, the ﬁnal return is approximately normally distributed with respect to seeds, instead of categorically distributed like HalfCheetah.
The variance on the other environments is much smaller than that of HalfCheetah.
For example, according to Table 3 of Henderson et al.
(2018), PPO Walker’s 95% conﬁdence interval for the ﬁnal return has a range of 800, while HalfCheetah has a range of 2200.
Thus, other environments do not yield as much ﬂuctuations as HalfCheetah.
In fact, in our experiments under the default hyperparameter setting in Section 3, we do not ﬁnd any regularization on any algorithm to "improve" upon HalfCheetah, according to our deﬁnition that a regularizer "improves" upon the baseline in Section 3.
Thus, our observations do not change if we take away the HalfCheetah environment.
L R EGULARIZATION WITH A F IXED S TRENGTH
In previous sections
[Table]
we tune the strength of regularization for each algorithm and environment, as described in Appendix C.
Now we restrict the regularization methods to a single strength for each algorithm, across different environments.
The results are shown in Table 32 and 33.
The selected strength are presented in Table 34. We see that the L 2 regularization is still generally the best performing one, but SAC is an exception, where BN is better. This can be explained by the fact that in SAC, the reward scaling coefﬁcient is different for each environment, which potentially causes the optimal L 2 and L 1 strength to vary a lot across different environments, while BN does not have a strength parameter
[Table]
[Table]
[Table]
algorithm to obtain results in Table 32 and Table 33.
[Table]
M DDPG R ESULTS
To study the effect of regularization on off-policy algorithms, besides the SAC results, we also present results on DDPG (Lillicrap et al., 2016) in Table 35.
We run DDPG on 5 MuJoCo environments:
Hopper, Walker, Ant, Humanoid, and HumanoidStandup.
We did not run DDPG on HalfCheetah due to its large variance.
We then analyze the performance through the calculation of z -scores, and we also perform Welch’s t -test.
Note that entropy regularization is not applicable here because
DDPG’s policy network outputs a deterministic action.
We obtain similar observations as we did in SAC.
Notably, Dropout and Batch Normalization can be useful in DDPG, as indicated by the higher average z -score than the baseline, which supports our hypothesis that they can be helpful on off-policy algorithms.
[Table]
N R EGULARIZING WITH L 2 AND E NTROPY
We also investigate the effect of combining L 2 regularization with entropy regularization, given that both cases of applying one of them alone yield performance improvement.
We take the optimal strength of L 2 regularization and entropy regularization together and compare with applying Lregularization or entropy regularization alone.
From Figure 8, we ﬁnd that the performance increases for PPO HumanoidStandup, approximately stays the same for TRPO Ant, and decreases for A2C
HumanoidStandup.
Thus, the regularization beneﬁts are not always addable.
This phenomenon is possibly caused by the fact that the algorithms already achieve good performance using only Lregularization or entropy regularization, and further performance improvement is restrained by the intrinsic capabilities of algorithms.
    H
[Figure]
   H
[Figure]
H
[Figure]
O L 2 R EGULARIZATION VS . F IXED W EIGHT D ECAY (A DAM W)
For the Adam optimizer (Kingma & Ba, 2015), “ﬁxed weight decay” (AdamW in Loshchilov & Hutter (2019)) differs from L 2 regularization in that the gradient of 12 λ || θ || 2 is not computed with the gradient of the original loss, but the weight is “decayed” ﬁnally with the gradient update.
For Adam these two procedures are very different (see Loshchilov & Hutter (2019) for more details).
In this section, we compare the effect of adding L 2 regularization with that of using AdamW, with PPO on
Humanoid and HumanoidStandup.
The result is shown in Figure 9.
Similar to L 2 , we brieﬂy tune the strength of weight decay in AdamW and the optimal one is used.
We ﬁnd that while both Lregularization and AdamW can signiﬁcantly improve the performance over baseline, the performance of AdamW tends to be slightly lower than the performance of L 2 regularization.
[Figure]
P
A DDITIONAL T RAINING C URVES (D EFAULT H YPERPARAMETERS )
     H     
H $&+RSSHU     H     H $&+XPDQRLG6WDQGXS      H     
  H 7532+RSSHU     H       H 7532+XPDQRLG6WDQGXS      H     
  H 332+RSSHU     H       H 332+XPDQRLG6WDQGXS      H    H 6$&+RSSHU      H     
 H 6$&+XPDQRLG6WDQGXS      H     
  H $&:DONHU      H         H $&+DOI&KHHWDK     
H        H $&+XPDQRLG)ODJUXQ      H      H 7532:DONHU      H     H 7532+DOI&KHHWDK      H      
H 7532+XPDQRLG)ODJUXQ      H  
    H 332:DONHU      H      H 332+DOI&KHHWDK     
H      
H 332+XPDQRLG)ODJUXQ      H     H
6$&:DONHU    H        H 6$&+DOI&KHHWDK      H      
H 6$&+XPDQRLG)ODJUXQ EDVHOLQH HQWURS\
/ / ZHLJKWFOLS GURSRXW EDWFKQRUP
As a complement with Figure 1 in Section 3, we plot the training curves of the other ﬁve environments in Figure 10.
Q T RAINING C URVES FOR H YPERPARAMETER E XPERIMENTS
In this section, we plot the full training curves of the experiments in Section 4 with ﬁve sampled hyperparameter settings for each algorithm in Figure 11 to 14.
The strength of each regularization is tuned according to the range in Appendix C.
R T RAINING C URVES FOR P OLICY VS . V ALUE E XPERIMENTS
We plot the training curves with our study in Section 5 on policy and value network regularizations in
Figure 15-18.
S A TARI E XPERIMENTS
We present results on 5 randomly sampled Atari environments (Asteroids, Pacman, Qbert, Roadrunner,
Riverraid) in Table 36.
Note that SAC not applicable here because it requires the environment to have continuous action space, while Atari environments have discrete action space.
We ﬁnd that
L 2 regularization can still signiﬁcantly improve over the baseline, while L 1 and weight clipping are slightly less effective.
Interestingly, while BN still signiﬁcantly harms performance on on-policy environments (A2C, TRPO, PPO), dropout can signiﬁcantly outperform the baseline.
We also observe that, different from continuous control tasks, entropy regularization can improve a lot on baseline, perhaps due to the action space being discrete.
[Table]