Stabilizing Off-Policy Reinforcement Learning with Conservative Policy Gradients
Authors: Chen Tessler, Nadav Merlis, Shie Mannor
Keywords: Deep Reinforcement Learning, Variance Reduction, Policy Gradient
TL;DR: We propose a conservative update rule for off-policy policy-gradient methods (e.g., DDPG) in order to reduce the variance of the training regime.
Abstract: In recent years, advances in deep learning have enabled the application of reinforcement learning algorithms in complex domains. However, they lack the theoretical guarantees which are present in the tabular setting and suffer from many stability and reproducibility problems \citep{henderson2018deep}. In this work, we suggest a simple approach for improving stability and providing probabilistic performance guarantees in off-policy actor-critic deep reinforcement learning regimes. Experiments on continuous action spaces, in the MuJoCo control suite, show that our proposed method reduces the variance of the process and improves the overall performance.