Paper:
Deep Reinforcement Learning (Deep RL) has been receiving increasingly more attention thanks to its encouraging performance on a variety of control tasks. Yet, conventional regularization techniques in training neural networks (e.g., L 2 regularization, dropout) have been largely ignored in RL methods, possibly because agents are typically trained and evaluated in the same environment. In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks. Interestingly, we find conventional regularization techniques on the policy networks can often bring large improvements on the task performance, and the improvement is typically more significant when the task is more difficult. We also compare with the widely used entropy regularization and find L 2 regularization is generally better. Our findings are further confirmed to be robust against the choice of training hyperparameters. We also study the effects of regularizing different components and find that only regularizing the policy network is typically the best option. We hope our study provides guidance for future practices in regularizing policy optimization algorithms.

Review:
This paper investigates the use of conventional regularizers for neural networks in the reinforcement learning setting. Contrary to the standard practice of foregoing regularizers in deep RL, the paper finds that their addition can improve the performance of policy gradient algorithms on a standard suite of continuous control tasks. Various regularizers are tried, including l2/l1 regularization, entropy regularization and dropout in a combination with a few standard deep RL algorithms such as TRPO, PPO and SAC. Other experiments also verify the impact of these regularizers on the sensitivity of other hyperparameters and whether regularization should be applied to the value or policy networks. 
Overall, I find this paper to be a solid empirical study of regularization in deep reinforcement learning. The experiments are thorough, with various aspects being examined in more detail. I find several of the findings interesting, such as the importance of regularizing solely the policy network and that batch norm/dropout are effective for off-policy methods but not on-policy ones. There were certain points which warranted some further clarification. 
I would be willing to increase my score based on the authors' response to the following points:
1) In section 6, the last two sentences ("For A2C, TRPO, and PPO ... so further regularization is unnecessary.") are unclear to me.
	- "rewards are already normalized using running mean filters." I thought that rewards are also normalized for SAC, so I'm not sure how this could explain the difference between the on-policy algorithms and the off-policy ones.
	- "mitigates the overestimation bias...further regularization is unncessary." Could you clarify the connection between regularization and overestimation bias? Related to this point, in section 2 of the paper, it is written that "L2 regularization is applied to the critic Q network because it tends to have overestimation bias (Fujimoto et al., 2018)" but I was not able to find such an explanation in the cited paper though I may have missed it.
2) In section 7, in the paragraph on BN/Dropout, could you clarify the point starting from "1) For both BN and dropout,..."? In particular, which discrepancy between the sampling policy and the optimization policy is being referred to here? 
3) Did you consider trying weight decay ("Fixing Weight Decay Regularization in Adam", Loschilov et al. 2018) as a regularizer? Given the success of L2 regularization, it could be possible that weight decay is even more effective.
4) For the hyperparameter sensitivity plots, where one hyperparameter is varied at a time, why are the step sizes for the policy and value networks not included in these experiments? They are usually a critical hyperparameter.
Minor comments and typos:
- On p.5, when defining "hurting", perhaps it could be better to choose a looser definition such as "\mu_r < \mu_b" or "\mu_r - \sigma_r < \mu_b - \sigma_b". This way, there could be a larger distinction between the most effective methods. Currently, both l2 and entropy regularization achieve 0.0% and the next best two regularizers are also under 10%. 
- In abstract: "regularizing the policy network is typically enough." Rephrase perhaps? The experiments seem to show that applying a regularizer to only the policy network is better than on both.
- In abstract: "large improvement" -> "large improvements"
- p.2, par. 2: "those regularizations" -> "those regularizers"
- p.3, Weight Clipping: "This greatly stablizes" -> "This greatly stabilizes". This sentence could be rephrased since "This" seems to refer to only weight clipping, but is not the only change in WGANs. 
- p.3, Dropout: "regularization technique" -> "regularization techniques"
- p.4, par. 1: "due to more stochasticity" -> "due to increased stochasticity"
- p.4, 2nd to last par.: "during policy update" -> "during policy updates"
- p.5, 2nd to last par.: "sometimes help" -> "sometimes helps", "easier ones baseline is" -> "easier ones the baseline is"
- p.8, 2nd to last par.: "it naturally accepts" -> "they naturally accept", "been shown effective" -> "been shown to be effective"
- p.8, last sentence: "policy network without the value network." -> "policy network but not the value network."